From rcolebur at SCIRES.COM Tue Feb 1 01:48:21 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Message-ID: Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. Do you have a plan to fix? Anything I can do to help? --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 01:53:21 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 00:53:21 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: References: Message-ID: Oops, I was just wondering to self if I had fixed that. I can get to it fairly soon. We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling convention (which imho is a good reason to make it a separate m3cg call...) (The input parameter is in eax, and the output is in esp.) (Tony, you really don't want a separate m3cg call?) - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. Do you have a plan to fix? Anything I can do to help? --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 01:55:07 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 00:55:07 +0000 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: , , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, , Message-ID: ps: I'm happy to report that my available time here has become significantly reduced. Partly with that in mind, but even w/o it, I am willing to go back to the old way. m3front is full of mystery and I'm not sure I'm keen on understanding it. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 15:26:20 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I think we need to have a way to get the scope of the containing procedure. TryStmt.Parse is not the right place. If I get some time soon I can take a look at this. On Jan 31, 2011, at 2:19 PM, Jay K wrote: Right..I also tried Scope.Insert, didn't seem to help. But I think I might be doing it too late. I'll try doing it in Parse instead of Check. Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat more closely resembles the old pattern: IF Foo() THEN RETURN ELSE TRY FINALLY in the old way would use the stack, so we could too. But the allocation is slower/larger now (alloca possibly aligning up more than needed). Anyway, this point isn't the problem. The problem is my inability to work with m3front. I'll look at it a bit more -- i.e. as I said, moving the code to Parse. The other improvement I'd like to make: currently, if I had it working, what I'm aiming for is: struct EF1{ ... jmpbuf* jb } ef1; jmpbuf* jb = 0; jb = jb ? jb : alloca(sizeof(*jb)); ef1.jb = jb; setjmp(ef1.jb); but it'd be better to have: struct EF1{ ... jmpbuf* jb } ef1; ef1.jb = 0; ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); setjmp(ef1.jb); ie. we don't need the extra pointer variable, the pointer in the record should suffice. [not sure where to put this in email] I tried figuring out how to determine the current vs. topmost-in-current-function scope. I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, so I kinda hoped it'd just use the correct scope. Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. Even if it doesn't get used, that might be ok, since there is tracking of use, and declaring it might not cause any waste if it isn't used. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 12:14:09 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? Hi Jay, I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Jan 30, 2011, at 5:45 AM, Jay K wrote: I'm trying to fix the inadvertant change I made where FOR i := 1 TO 10 DO TRY FINALLY END will call alloca 10 times. I'm not familiar with the frontend, and it is hard for me to understand. It seems like it is probably very well factored, written by someone who really knew what they were doing, but it is dense, and not verbosely commented. Something like this?? But this doesn't work. What am I missing? And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? ? 1.txt Index: src/misc/Marker.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v retrieving revision 1.1.1.1 diff -u -r1.1.1.1 Marker.i3 --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 @@ -64,9 +64,12 @@ PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); (* generate the call to acquire or release a mutex *) -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T); (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) PROCEDURE Reset (); +PROCEDURE NewJumpbuf (): Variable.T; + END Marker. Index: src/misc/Marker.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v retrieving revision 1.9 diff -u -r1.9 Marker.m3 --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 @@ -10,7 +10,7 @@ MODULE Marker; IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; TYPE Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, @@ -231,8 +231,10 @@ END; END CallFinallyHandler; -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T) = VAR new: BOOLEAN; + label := CG.Next_label (); BEGIN (* int setjmp(void* ); *) IF (setjmp = NIL) THEN @@ -262,13 +264,33 @@ Target.Word.size, Target.Word.align, Target.Word.cg_type, 0); END; - - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) + + (* IF jumpuf # NIL THEN + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); + * END + *) + + CG.Load_nil (); + Variable.Load (jumpbuf); + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); + CG.Start_call_direct (alloca, 0, Target.Address.cg_type); CG.Load_int (Target.Word.cg_type, Jumpbuf_size); CG.Pop_param (Target.Word.cg_type); CG.Call_direct (alloca, Target.Address.cg_type); + + (* FUTURE: We should actually have a Variable + * for the entire EF1, including initializing the jumpbuf + * in it. That would save us this extra load/store, + * and save a local variable. + *) + Variable.LoadLValue (jumpbuf); + CG.Swap (); + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); + Variable.Load (jumpbuf); + CG.Store_addr (frame, M3RT.EF1_jmpbuf); + CG.Set_label (label); (* setmp(frame.jmpbuf) *) CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); @@ -806,5 +828,15 @@ tos := 0; END Reset; +PROCEDURE NewJumpbuf (): Variable.T = + VAR jumpbuf: Variable.T; + BEGIN + jumpbuf := Variable.New (M3ID.NoID, TRUE); + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, + readonly := FALSE, open_array_ok := FALSE, + needs_init := TRUE); + RETURN jumpbuf; + END NewJumpbuf; + BEGIN END Marker. Index: src/stmts/TryFinStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v retrieving revision 1.6 diff -u -r1.6 TryFinStmt.m3 --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -10,6 +10,7 @@ IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; +IMPORT Variable; FROM Stmt IMPORT Outcome; TYPE @@ -20,6 +21,7 @@ viaProc : BOOLEAN; scope : Scope.T; handler : HandlerProc; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -63,6 +65,11 @@ RETURN p; END Parse; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + RETURN NOT (Target.Has_stack_walker OR p.viaProc); + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; BEGIN @@ -95,6 +102,9 @@ END; END; TryStmt.PopHandler (); + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE HandlerName (uid: INTEGER): TEXT = @@ -106,11 +116,16 @@ END HandlerName; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN - IF Target.Has_stack_walker THEN RETURN Compile1 (p); - ELSIF p.viaProc THEN RETURN Compile2 (p); - ELSE RETURN Compile3 (p); + IF Target.Has_stack_walker THEN result := Compile1 (p); + ELSIF p.viaProc THEN result := Compile2 (p); + ELSE usesSetjmp := TRUE; + result := Compile3 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -302,7 +317,9 @@ l := CG.Next_label (3); CG.Set_label (l, barrier := TRUE); Marker.PushFrame (frame, M3RT.HandlerClass.Finally); - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) Marker.PushFinally (l, l+1, frame); Index: src/stmts/TryStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v retrieving revision 1.3 diff -u -r1.3 TryStmt.m3 --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -22,6 +22,7 @@ hasElse : BOOLEAN; elseBody : Stmt.T; handled : ESet.T; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -153,6 +154,14 @@ p.handles := h3; END ReverseHandlers; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + IF (p.handles = NIL) AND (NOT p.hasElse) THEN + RETURN FALSE; + END; + RETURN NOT Target.Has_stack_walker; + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR h: Handler; handled: ESet.T; BEGIN @@ -182,6 +191,10 @@ WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; Stmt.TypeCheck (p.elseBody, cs); PopHandler (); + + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = @@ -245,14 +258,19 @@ END CheckHandler; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN IF (p.handles = NIL) AND (NOT p.hasElse) THEN - RETURN Stmt.Compile (p.body); - END; - IF Target.Has_stack_walker - THEN RETURN Compile1 (p); - ELSE RETURN Compile2 (p); + result := Stmt.Compile (p.body); + ELSIF Target.Has_stack_walker THEN + result := Compile1 (p); + ELSE + usesSetjmp := TRUE; + result := Compile2 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -423,7 +441,9 @@ END; (* capture the machine state *) - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) oc := Stmt.Compile (p.body); Thanks, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 04:39:36 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 31 Jan 2011 22:39:36 -0500 Subject: [M3devel] link failure on NT386 In-Reply-To: References: Message-ID: Why can't we just have a run-time hook that resolves differently on different targets? Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. On Jan 31, 2011, at 7:53 PM, Jay K wrote: > Oops, I was just wondering to self if I had fixed that. > I can get to it fairly soon. > We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling > convention (which imho is a good reason to > make it a separate m3cg call...) > (The input parameter is in eax, and the output is in esp.) > (Tony, you really don't want a separate m3cg call?) > > > - Jay > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 31 Jan 2011 19:48:21 -0500 > Subject: [M3devel] link failure on NT386 > > Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. > Do you have a plan to fix? > Anything I can do to help? > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 04:45:53 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 31 Jan 2011 22:45:53 -0500 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: , , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, , Message-ID: <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu> I'd like to do what you envisage. It is a significant improvement, and should not require too much effort: 1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3) 2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. On Jan 31, 2011, at 7:55 PM, Jay K wrote: > ps: I'm happy to report that my available time here > has become significantly reduced. Partly with that in mind, > but even w/o it, I am willing to go back to the old way. > m3front is full of mystery and I'm not sure I'm keen on understanding it. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 15:26:20 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I think we need to have a way to get the scope of the containing procedure. > TryStmt.Parse is not the right place. > If I get some time soon I can take a look at this. > > On Jan 31, 2011, at 2:19 PM, Jay K wrote: > > Right..I also tried Scope.Insert, didn't seem to help. > > But I think I might be doing it too late. I'll try doing it in Parse instead of Check. > Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. > I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat > more closely resembles the old pattern: > > IF Foo() THEN > RETURN > ELSE TRY FINALLY > > in the old way would use the stack, so we could too. > But the allocation is slower/larger now (alloca possibly aligning up more than needed). > Anyway, this point isn't the problem. The problem is my inability to work with m3front. > I'll look at it a bit more -- i.e. as I said, moving the code to Parse. > > The other improvement I'd like to make: > > currently, if I had it working, what I'm aiming for is: > struct EF1{ ... jmpbuf* jb } ef1; > jmpbuf* jb = 0; > > jb = jb ? jb : alloca(sizeof(*jb)); > ef1.jb = jb; > setjmp(ef1.jb); > > but it'd be better to have: > > struct EF1{ ... jmpbuf* jb } ef1; > ef1.jb = 0; > > ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); > setjmp(ef1.jb); > > > ie. we don't need the extra pointer variable, the pointer in the record should suffice. > > > [not sure where to put this in email] > I tried figuring out how to determine the current vs. topmost-in-current-function scope. > I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps > some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, > so I kinda hoped it'd just use the correct scope. > > > Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. > Even if it doesn't get used, that might be ok, since there is tracking of use, and > declaring it might not cause any waste if it isn't used. > > > Thanks, > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 12:14:09 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > Hi Jay, > > I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On Jan 30, 2011, at 5:45 AM, Jay K wrote: > > I'm trying to fix the inadvertant change I made where > FOR i := 1 TO 10 DO TRY FINALLY END > > will call alloca 10 times. > > I'm not familiar with the frontend, and it is hard for me to understand. > It seems like it is probably very well factored, written by someone who really knew what they were doing, > but it is dense, and not verbosely commented. > > Something like this?? But this doesn't work. What am I missing? > And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? > > > ? 1.txt > Index: src/misc/Marker.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v > retrieving revision 1.1.1.1 > diff -u -r1.1.1.1 Marker.i3 > --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 > +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 > @@ -64,9 +64,12 @@ > PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); > (* generate the call to acquire or release a mutex *) > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T); > (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) > > PROCEDURE Reset (); > > +PROCEDURE NewJumpbuf (): Variable.T; > + > END Marker. > Index: src/misc/Marker.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v > retrieving revision 1.9 > diff -u -r1.9 Marker.m3 > --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 > +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,7 +10,7 @@ > MODULE Marker; > > IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; > -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; > +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; > > TYPE > Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, > @@ -231,8 +231,10 @@ > END; > END CallFinallyHandler; > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T) = > VAR new: BOOLEAN; > + label := CG.Next_label (); > BEGIN > (* int setjmp(void* ); *) > IF (setjmp = NIL) THEN > @@ -262,13 +264,33 @@ > Target.Word.size, Target.Word.align, > Target.Word.cg_type, 0); > END; > - > - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) > + > + (* IF jumpuf # NIL THEN > + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); > + * END > + *) > + > + CG.Load_nil (); > + Variable.Load (jumpbuf); > + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); > + > CG.Start_call_direct (alloca, 0, Target.Address.cg_type); > CG.Load_int (Target.Word.cg_type, Jumpbuf_size); > CG.Pop_param (Target.Word.cg_type); > CG.Call_direct (alloca, Target.Address.cg_type); > + > + (* FUTURE: We should actually have a Variable > + * for the entire EF1, including initializing the jumpbuf > + * in it. That would save us this extra load/store, > + * and save a local variable. > + *) > + Variable.LoadLValue (jumpbuf); > + CG.Swap (); > + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); > + Variable.Load (jumpbuf); > + > CG.Store_addr (frame, M3RT.EF1_jmpbuf); > + CG.Set_label (label); > > (* setmp(frame.jmpbuf) *) > CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); > @@ -806,5 +828,15 @@ > tos := 0; > END Reset; > > +PROCEDURE NewJumpbuf (): Variable.T = > + VAR jumpbuf: Variable.T; > + BEGIN > + jumpbuf := Variable.New (M3ID.NoID, TRUE); > + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, > + readonly := FALSE, open_array_ok := FALSE, > + needs_init := TRUE); > + RETURN jumpbuf; > + END NewJumpbuf; > + > BEGIN > END Marker. > Index: src/stmts/TryFinStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v > retrieving revision 1.6 > diff -u -r1.6 TryFinStmt.m3 > --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 > +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,6 +10,7 @@ > > IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; > IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; > +IMPORT Variable; > FROM Stmt IMPORT Outcome; > > TYPE > @@ -20,6 +21,7 @@ > viaProc : BOOLEAN; > scope : Scope.T; > handler : HandlerProc; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -63,6 +65,11 @@ > RETURN p; > END Parse; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + RETURN NOT (Target.Has_stack_walker OR p.viaProc); > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; > BEGIN > @@ -95,6 +102,9 @@ > END; > END; > TryStmt.PopHandler (); > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE HandlerName (uid: INTEGER): TEXT = > @@ -106,11 +116,16 @@ > END HandlerName; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > - IF Target.Has_stack_walker THEN RETURN Compile1 (p); > - ELSIF p.viaProc THEN RETURN Compile2 (p); > - ELSE RETURN Compile3 (p); > + IF Target.Has_stack_walker THEN result := Compile1 (p); > + ELSIF p.viaProc THEN result := Compile2 (p); > + ELSE usesSetjmp := TRUE; > + result := Compile3 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -302,7 +317,9 @@ > l := CG.Next_label (3); > CG.Set_label (l, barrier := TRUE); > Marker.PushFrame (frame, M3RT.HandlerClass.Finally); > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > Marker.PushFinally (l, l+1, frame); > Index: src/stmts/TryStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v > retrieving revision 1.3 > diff -u -r1.3 TryStmt.m3 > --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 > +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -22,6 +22,7 @@ > hasElse : BOOLEAN; > elseBody : Stmt.T; > handled : ESet.T; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -153,6 +154,14 @@ > p.handles := h3; > END ReverseHandlers; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + IF (p.handles = NIL) AND (NOT p.hasElse) THEN > + RETURN FALSE; > + END; > + RETURN NOT Target.Has_stack_walker; > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR h: Handler; handled: ESet.T; > BEGIN > @@ -182,6 +191,10 @@ > WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; > Stmt.TypeCheck (p.elseBody, cs); > PopHandler (); > + > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = > @@ -245,14 +258,19 @@ > END CheckHandler; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > IF (p.handles = NIL) AND (NOT p.hasElse) THEN > - RETURN Stmt.Compile (p.body); > - END; > - IF Target.Has_stack_walker > - THEN RETURN Compile1 (p); > - ELSE RETURN Compile2 (p); > + result := Stmt.Compile (p.body); > + ELSIF Target.Has_stack_walker THEN > + result := Compile1 (p); > + ELSE > + usesSetjmp := TRUE; > + result := Compile2 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -423,7 +441,9 @@ > END; > > (* capture the machine state *) > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > oc := Stmt.Compile (p.body); > > Thanks, > - Jay > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 08:12:13 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 07:12:13 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: References: , , Message-ID: Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. Why are the atomics in m3cg? I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. alloca is simpler just by virtue of it only taking one parameter. Why aren't and/or/xor/add runtime hooks, or special functions? Or the set operations? I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. And that there may be a desire to in general avoid growing m3cg. esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. I'm also wishy washy. I can see it is easy to implement easily any way. I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. At some point, I realize, you can model just about everything as function calls, except, calling functions. :) I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. Since anything implemented in Modula-3 and exposed must be in some interface. However runtime hooks can be implemented in C as well. It depends somewhat if they are only called by generated code or also hand written Modula-3. You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, there is just relying on m3front and m3core agreeing. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:39:36 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] link failure on NT386 Why can't we just have a run-time hook that resolves differently on different targets?Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. On Jan 31, 2011, at 7:53 PM, Jay K wrote:Oops, I was just wondering to self if I had fixed that. I can get to it fairly soon. We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling convention (which imho is a good reason to make it a separate m3cg call...) (The input parameter is in eax, and the output is in esp.) (Tony, you really don't want a separate m3cg call?) - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier.Do you have a plan to fix?Anything I can do to help?--Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 08:16:40 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 07:16:40 +0000 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu> References: , , , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, , , , , , <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu> Message-ID: These look like good hints. I wouldn't know to make such slightly large changes, that the existing infrastructure doesn't suffice. I'll have a go at it..as my now-very-limited time permits. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:45:53 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I'd like to do what you envisage. It is a significant improvement, and should not require too much effort:1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3)2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. On Jan 31, 2011, at 7:55 PM, Jay K wrote:ps: I'm happy to report that my available time here has become significantly reduced. Partly with that in mind, but even w/o it, I am willing to go back to the old way. m3front is full of mystery and I'm not sure I'm keen on understanding it. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 15:26:20 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I think we need to have a way to get the scope of the containing procedure.TryStmt.Parse is not the right place.If I get some time soon I can take a look at this. On Jan 31, 2011, at 2:19 PM, Jay K wrote:Right..I also tried Scope.Insert, didn't seem to help. But I think I might be doing it too late. I'll try doing it in Parse instead of Check. Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat more closely resembles the old pattern: IF Foo() THEN RETURN ELSE TRY FINALLY in the old way would use the stack, so we could too. But the allocation is slower/larger now (alloca possibly aligning up more than needed). Anyway, this point isn't the problem. The problem is my inability to work with m3front. I'll look at it a bit more -- i.e. as I said, moving the code to Parse. The other improvement I'd like to make: currently, if I had it working, what I'm aiming for is: struct EF1{ ... jmpbuf* jb } ef1; jmpbuf* jb = 0; jb = jb ? jb : alloca(sizeof(*jb)); ef1.jb = jb; setjmp(ef1.jb); but it'd be better to have: struct EF1{ ... jmpbuf* jb } ef1; ef1.jb = 0; ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); setjmp(ef1.jb); ie. we don't need the extra pointer variable, the pointer in the record should suffice. [not sure where to put this in email] I tried figuring out how to determine the current vs. topmost-in-current-function scope. I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, so I kinda hoped it'd just use the correct scope. Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. Even if it doesn't get used, that might be ok, since there is tracking of use, and declaring it might not cause any waste if it isn't used. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 12:14:09 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? Hi Jay, I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. Antony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Jan 30, 2011, at 5:45 AM, Jay K wrote:I'm trying to fix the inadvertant change I made where FOR i := 1 TO 10 DO TRY FINALLY END will call alloca 10 times. I'm not familiar with the frontend, and it is hard for me to understand. It seems like it is probably very well factored, written by someone who really knew what they were doing, but it is dense, and not verbosely commented. Something like this?? But this doesn't work. What am I missing? And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? ? 1.txt Index: src/misc/Marker.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v retrieving revision 1.1.1.1 diff -u -r1.1.1.1 Marker.i3 --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 @@ -64,9 +64,12 @@ PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); (* generate the call to acquire or release a mutex *) -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T); (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) PROCEDURE Reset (); +PROCEDURE NewJumpbuf (): Variable.T; + END Marker. Index: src/misc/Marker.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v retrieving revision 1.9 diff -u -r1.9 Marker.m3 --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 @@ -10,7 +10,7 @@ MODULE Marker; IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; TYPE Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, @@ -231,8 +231,10 @@ END; END CallFinallyHandler; -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T) = VAR new: BOOLEAN; + label := CG.Next_label (); BEGIN (* int setjmp(void* ); *) IF (setjmp = NIL) THEN @@ -262,13 +264,33 @@ Target.Word.size, Target.Word.align, Target.Word.cg_type, 0); END; - - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) + + (* IF jumpuf # NIL THEN + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); + * END + *) + + CG.Load_nil (); + Variable.Load (jumpbuf); + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); + CG.Start_call_direct (alloca, 0, Target.Address.cg_type); CG.Load_int (Target.Word.cg_type, Jumpbuf_size); CG.Pop_param (Target.Word.cg_type); CG.Call_direct (alloca, Target.Address.cg_type); + + (* FUTURE: We should actually have a Variable + * for the entire EF1, including initializing the jumpbuf + * in it. That would save us this extra load/store, + * and save a local variable. + *) + Variable.LoadLValue (jumpbuf); + CG.Swap (); + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); + Variable.Load (jumpbuf); + CG.Store_addr (frame, M3RT.EF1_jmpbuf); + CG.Set_label (label); (* setmp(frame.jmpbuf) *) CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); @@ -806,5 +828,15 @@ tos := 0; END Reset; +PROCEDURE NewJumpbuf (): Variable.T = + VAR jumpbuf: Variable.T; + BEGIN + jumpbuf := Variable.New (M3ID.NoID, TRUE); + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, + readonly := FALSE, open_array_ok := FALSE, + needs_init := TRUE); + RETURN jumpbuf; + END NewJumpbuf; + BEGIN END Marker. Index: src/stmts/TryFinStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v retrieving revision 1.6 diff -u -r1.6 TryFinStmt.m3 --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -10,6 +10,7 @@ IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; +IMPORT Variable; FROM Stmt IMPORT Outcome; TYPE @@ -20,6 +21,7 @@ viaProc : BOOLEAN; scope : Scope.T; handler : HandlerProc; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -63,6 +65,11 @@ RETURN p; END Parse; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + RETURN NOT (Target.Has_stack_walker OR p.viaProc); + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; BEGIN @@ -95,6 +102,9 @@ END; END; TryStmt.PopHandler (); + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE HandlerName (uid: INTEGER): TEXT = @@ -106,11 +116,16 @@ END HandlerName; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN - IF Target.Has_stack_walker THEN RETURN Compile1 (p); - ELSIF p.viaProc THEN RETURN Compile2 (p); - ELSE RETURN Compile3 (p); + IF Target.Has_stack_walker THEN result := Compile1 (p); + ELSIF p.viaProc THEN result := Compile2 (p); + ELSE usesSetjmp := TRUE; + result := Compile3 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -302,7 +317,9 @@ l := CG.Next_label (3); CG.Set_label (l, barrier := TRUE); Marker.PushFrame (frame, M3RT.HandlerClass.Finally); - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) Marker.PushFinally (l, l+1, frame); Index: src/stmts/TryStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v retrieving revision 1.3 diff -u -r1.3 TryStmt.m3 --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -22,6 +22,7 @@ hasElse : BOOLEAN; elseBody : Stmt.T; handled : ESet.T; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -153,6 +154,14 @@ p.handles := h3; END ReverseHandlers; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + IF (p.handles = NIL) AND (NOT p.hasElse) THEN + RETURN FALSE; + END; + RETURN NOT Target.Has_stack_walker; + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR h: Handler; handled: ESet.T; BEGIN @@ -182,6 +191,10 @@ WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; Stmt.TypeCheck (p.elseBody, cs); PopHandler (); + + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = @@ -245,14 +258,19 @@ END CheckHandler; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN IF (p.handles = NIL) AND (NOT p.hasElse) THEN - RETURN Stmt.Compile (p.body); - END; - IF Target.Has_stack_walker - THEN RETURN Compile1 (p); - ELSE RETURN Compile2 (p); + result := Stmt.Compile (p.body); + ELSIF Target.Has_stack_walker THEN + result := Compile1 (p); + ELSE + usesSetjmp := TRUE; + result := Compile2 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -423,7 +441,9 @@ END; (* capture the machine state *) - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) oc := Stmt.Compile (p.body); Thanks, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 15:46:22 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 1 Feb 2011 09:46:22 -0500 Subject: [M3devel] link failure on NT386 In-Reply-To: References: Message-ID: <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu> The atomics are not necessarily functions. They approximate real instructions generated inline. Shall I just go ahead and add it? Sent from my iPad On 01/02/2011, at 2:12 AM, Jay K wrote: > Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? > In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. > Why are the atomics in m3cg? > I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. > alloca is simpler just by virtue of it only taking one parameter. > > Why aren't and/or/xor/add runtime hooks, or special functions? > Or the set operations? > > I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. > > And that there may be a desire to in general avoid growing m3cg. > esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. > > I'm also wishy washy. I can see it is easy to implement easily any way. > I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. > > > At some point, I realize, you can model just about everything as function calls, except, calling functions. :) > > I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. > Since anything implemented in Modula-3 and exposed must be in some interface. > However runtime hooks can be implemented in C as well. > It depends somewhat if they are only called by generated code or also hand written Modula-3. > You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, > there is just relying on m3front and m3core agreeing. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:39:36 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] link failure on NT386 > > Why can't we just have a run-time hook that resolves differently on different targets? > Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. > > On Jan 31, 2011, at 7:53 PM, Jay K wrote: > > Oops, I was just wondering to self if I had fixed that. > I can get to it fairly soon. > We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling > convention (which imho is a good reason to > make it a separate m3cg call...) > (The input parameter is in eax, and the output is in esp.) > (Tony, you really don't want a separate m3cg call?) > > > - Jay > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 31 Jan 2011 19:48:21 -0500 > Subject: [M3devel] link failure on NT386 > > Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. > Do you have a plan to fix? > Anything I can do to help? > --Randy Coleburn > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 15:48:02 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 1 Feb 2011 09:48:02 -0500 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu> <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu> Message-ID: Let me know if you need some help. Sent from my iPad On 01/02/2011, at 2:16 AM, Jay K wrote: > These look like good hints. I wouldn't know to make such slightly large changes, > that the existing infrastructure doesn't suffice. > I'll have a go at it..as my now-very-limited time permits. > > Thanks, > - Jay > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:45:53 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I'd like to do what you envisage. It is a significant improvement, and should not require too much effort: > 1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3) > 2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. > > On Jan 31, 2011, at 7:55 PM, Jay K wrote: > > ps: I'm happy to report that my available time here > has become significantly reduced. Partly with that in mind, > but even w/o it, I am willing to go back to the old way. > m3front is full of mystery and I'm not sure I'm keen on understanding it. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 15:26:20 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I think we need to have a way to get the scope of the containing procedure. > TryStmt.Parse is not the right place. > If I get some time soon I can take a look at this. > > On Jan 31, 2011, at 2:19 PM, Jay K wrote: > > Right..I also tried Scope.Insert, didn't seem to help. > > But I think I might be doing it too late. I'll try doing it in Parse instead of Check. > Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. > I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat > more closely resembles the old pattern: > > IF Foo() THEN > RETURN > ELSE TRY FINALLY > > in the old way would use the stack, so we could too. > But the allocation is slower/larger now (alloca possibly aligning up more than needed). > Anyway, this point isn't the problem. The problem is my inability to work with m3front. > I'll look at it a bit more -- i.e. as I said, moving the code to Parse. > > The other improvement I'd like to make: > > currently, if I had it working, what I'm aiming for is: > struct EF1{ ... jmpbuf* jb } ef1; > jmpbuf* jb = 0; > > jb = jb ? jb : alloca(sizeof(*jb)); > ef1.jb = jb; > setjmp(ef1.jb); > > but it'd be better to have: > > struct EF1{ ... jmpbuf* jb } ef1; > ef1.jb = 0; > > ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); > setjmp(ef1.jb); > > > ie. we don't need the extra pointer variable, the pointer in the record should suffice. > > > [not sure where to put this in email] > I tried figuring out how to determine the current vs. topmost-in-current-function scope. > I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps > some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, > so I kinda hoped it'd just use the correct scope. > > > Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. > Even if it doesn't get used, that might be ok, since there is tracking of use, and > declaring it might not cause any waste if it isn't used. > > > Thanks, > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 12:14:09 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > Hi Jay, > > I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On Jan 30, 2011, at 5:45 AM, Jay K wrote: > > I'm trying to fix the inadvertant change I made where > FOR i := 1 TO 10 DO TRY FINALLY END > > will call alloca 10 times. > > I'm not familiar with the frontend, and it is hard for me to understand. > It seems like it is probably very well factored, written by someone who really knew what they were doing, > but it is dense, and not verbosely commented. > > Something like this?? But this doesn't work. What am I missing? > And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? > > > ? 1.txt > Index: src/misc/Marker.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v > retrieving revision 1.1.1.1 > diff -u -r1.1.1.1 Marker.i3 > --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 > +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 > @@ -64,9 +64,12 @@ > PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); > (* generate the call to acquire or release a mutex *) > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T); > (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) > > PROCEDURE Reset (); > > +PROCEDURE NewJumpbuf (): Variable.T; > + > END Marker. > Index: src/misc/Marker.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v > retrieving revision 1.9 > diff -u -r1.9 Marker.m3 > --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 > +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,7 +10,7 @@ > MODULE Marker; > > IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; > -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; > +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; > > TYPE > Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, > @@ -231,8 +231,10 @@ > END; > END CallFinallyHandler; > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T) = > VAR new: BOOLEAN; > + label := CG.Next_label (); > BEGIN > (* int setjmp(void* ); *) > IF (setjmp = NIL) THEN > @@ -262,13 +264,33 @@ > Target.Word.size, Target.Word.align, > Target.Word.cg_type, 0); > END; > - > - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) > + > + (* IF jumpuf # NIL THEN > + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); > + * END > + *) > + > + CG.Load_nil (); > + Variable.Load (jumpbuf); > + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); > + > CG.Start_call_direct (alloca, 0, Target.Address.cg_type); > CG.Load_int (Target.Word.cg_type, Jumpbuf_size); > CG.Pop_param (Target.Word.cg_type); > CG.Call_direct (alloca, Target.Address.cg_type); > + > + (* FUTURE: We should actually have a Variable > + * for the entire EF1, including initializing the jumpbuf > + * in it. That would save us this extra load/store, > + * and save a local variable. > + *) > + Variable.LoadLValue (jumpbuf); > + CG.Swap (); > + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); > + Variable.Load (jumpbuf); > + > CG.Store_addr (frame, M3RT.EF1_jmpbuf); > + CG.Set_label (label); > > (* setmp(frame.jmpbuf) *) > CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); > @@ -806,5 +828,15 @@ > tos := 0; > END Reset; > > +PROCEDURE NewJumpbuf (): Variable.T = > + VAR jumpbuf: Variable.T; > + BEGIN > + jumpbuf := Variable.New (M3ID.NoID, TRUE); > + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, > + readonly := FALSE, open_array_ok := FALSE, > + needs_init := TRUE); > + RETURN jumpbuf; > + END NewJumpbuf; > + > BEGIN > END Marker. > Index: src/stmts/TryFinStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v > retrieving revision 1.6 > diff -u -r1.6 TryFinStmt.m3 > --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 > +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,6 +10,7 @@ > > IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; > IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; > +IMPORT Variable; > FROM Stmt IMPORT Outcome; > > TYPE > @@ -20,6 +21,7 @@ > viaProc : BOOLEAN; > scope : Scope.T; > handler : HandlerProc; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -63,6 +65,11 @@ > RETURN p; > END Parse; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + RETURN NOT (Target.Has_stack_walker OR p.viaProc); > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; > BEGIN > @@ -95,6 +102,9 @@ > END; > END; > TryStmt.PopHandler (); > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE HandlerName (uid: INTEGER): TEXT = > @@ -106,11 +116,16 @@ > END HandlerName; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > - IF Target.Has_stack_walker THEN RETURN Compile1 (p); > - ELSIF p.viaProc THEN RETURN Compile2 (p); > - ELSE RETURN Compile3 (p); > + IF Target.Has_stack_walker THEN result := Compile1 (p); > + ELSIF p.viaProc THEN result := Compile2 (p); > + ELSE usesSetjmp := TRUE; > + result := Compile3 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -302,7 +317,9 @@ > l := CG.Next_label (3); > CG.Set_label (l, barrier := TRUE); > Marker.PushFrame (frame, M3RT.HandlerClass.Finally); > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > Marker.PushFinally (l, l+1, frame); > Index: src/stmts/TryStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v > retrieving revision 1.3 > diff -u -r1.3 TryStmt.m3 > --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 > +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -22,6 +22,7 @@ > hasElse : BOOLEAN; > elseBody : Stmt.T; > handled : ESet.T; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -153,6 +154,14 @@ > p.handles := h3; > END ReverseHandlers; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + IF (p.handles = NIL) AND (NOT p.hasElse) THEN > + RETURN FALSE; > + END; > + RETURN NOT Target.Has_stack_walker; > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR h: Handler; handled: ESet.T; > BEGIN > @@ -182,6 +191,10 @@ > WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; > Stmt.TypeCheck (p.elseBody, cs); > PopHandler (); > + > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = > @@ -245,14 +258,19 @@ > END CheckHandler; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > IF (p.handles = NIL) AND (NOT p.hasElse) THEN > - RETURN Stmt.Compile (p.body); > - END; > - IF Target.Has_stack_walker > - THEN RETURN Compile1 (p); > - ELSE RETURN Compile2 (p); > + result := Stmt.Compile (p.body); > + ELSIF Target.Has_stack_walker THEN > + result := Compile1 (p); > + ELSE > + usesSetjmp := TRUE; > + result := Compile2 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -423,7 +441,9 @@ > END; > > (* capture the machine state *) > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > oc := Stmt.Compile (p.body); > > Thanks, > - Jay > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 20:20:50 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 19:20:50 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu> References: , <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu> Message-ID: Modeling something as a function in m3cg does not necessarily correlate to generating a function call. E.g. the existing m3_alloca. And vice versa. e.g some of the set operations. Almost anything can be modeled as a function call in m3cg, but not end up generating one. Many things can be not modeled as a function call, but end up generating one. gcc has a huge number of "builtin functions". So does Visual C++. In Visual C++, basically any "new instruction" (mmx, sse2/3/4, avx) ends up modeled as a function, but generates roughly one instruction per function calls. It's a slightly high level assembly -- i.e. you get to chose the important algorithmic instructions, the compiler does register allocation, stack/local allocation, etc. I can add it -- surely my diff from a few weeks ago wasn't terrible? I'm ok either way. I'm more curious to understand than to "do". I admit. - Jay CC: m3devel at elegosoft.com From: hosking at cs.purdue.edu Subject: Re: [M3devel] link failure on NT386 Date: Tue, 1 Feb 2011 09:46:22 -0500 To: jay.krell at cornell.edu The atomics are not necessarily functions. They approximate real instructions generated inline.Shall I just go ahead and add it? Sent from my iPad On 01/02/2011, at 2:12 AM, Jay K wrote: Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. Why are the atomics in m3cg? I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. alloca is simpler just by virtue of it only taking one parameter. Why aren't and/or/xor/add runtime hooks, or special functions? Or the set operations? I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. And that there may be a desire to in general avoid growing m3cg. esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. I'm also wishy washy. I can see it is easy to implement easily any way. I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. At some point, I realize, you can model just about everything as function calls, except, calling functions. :) I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. Since anything implemented in Modula-3 and exposed must be in some interface. However runtime hooks can be implemented in C as well. It depends somewhat if they are only called by generated code or also hand written Modula-3. You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, there is just relying on m3front and m3core agreeing. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:39:36 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] link failure on NT386 Why can't we just have a run-time hook that resolves differently on different targets?Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. On Jan 31, 2011, at 7:53 PM, Jay K wrote:Oops, I was just wondering to self if I had fixed that. I can get to it fairly soon. We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling convention (which imho is a good reason to make it a separate m3cg call...) (The input parameter is in eax, and the output is in esp.) (Tony, you really don't want a separate m3cg call?) - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier.Do you have a plan to fix?Anything I can do to help?--Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 21:01:44 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 1 Feb 2011 15:01:44 -0500 Subject: [M3devel] link failure on NT386 In-Reply-To: References: <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu> Message-ID: <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu> RTHooks makes most sense. Sent from my iPhone On Feb 1, 2011, at 2:20 PM, Jay K wrote: > Modeling something as a function in m3cg does not necessarily correlate to generating a function call. > E.g. the existing m3_alloca. > And vice versa. e.g some of the set operations. > Almost anything can be modeled as a function call in m3cg, but not end up generating one. > Many things can be not modeled as a function call, but end up generating one. > gcc has a huge number of "builtin functions". So does Visual C++. > In Visual C++, basically any "new instruction" (mmx, sse2/3/4, avx) ends up modeled > as a function, but generates roughly one instruction per function calls. > It's a slightly high level assembly -- i.e. you get to chose the important algorithmic instructions, > the compiler does register allocation, stack/local allocation, etc. > > I can add it -- surely my diff from a few weeks ago wasn't terrible? > > I'm ok either way. > I'm more curious to understand than to "do". I admit. > > - Jay > > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > Subject: Re: [M3devel] link failure on NT386 > Date: Tue, 1 Feb 2011 09:46:22 -0500 > To: jay.krell at cornell.edu > > The atomics are not necessarily functions. They approximate real instructions generated inline. > Shall I just go ahead and add it? > > Sent from my iPad > > On 01/02/2011, at 2:12 AM, Jay K wrote: > > Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? > In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. > Why are the atomics in m3cg? > I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. > alloca is simpler just by virtue of it only taking one parameter. > > Why aren't and/or/xor/add runtime hooks, or special functions? > Or the set operations? > > I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. > > And that there may be a desire to in general avoid growing m3cg. > esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. > > I'm also wishy washy. I can see it is easy to implement easily any way. > I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. > > > At some point, I realize, you can model just about everything as function calls, except, calling functions. :) > > I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. > Since anything implemented in Modula-3 and exposed must be in some interface. > However runtime hooks can be implemented in C as well. > It depends somewhat if they are only called by generated code or also hand written Modula-3. > You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, > there is just relying on m3front and m3core agreeing. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:39:36 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] link failure on NT386 > > Why can't we just have a run-time hook that resolves differently on different targets? > Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. > > On Jan 31, 2011, at 7:53 PM, Jay K wrote: > > Oops, I was just wondering to self if I had fixed that. > I can get to it fairly soon. > We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling > convention (which imho is a good reason to > make it a separate m3cg call...) > (The input parameter is in eax, and the output is in esp.) > (Tony, you really don't want a separate m3cg call?) > > > - Jay > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 31 Jan 2011 19:48:21 -0500 > Subject: [M3devel] link failure on NT386 > > Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. > Do you have a plan to fix? > Anything I can do to help? > --Randy Coleburn > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 22:12:05 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 21:12:05 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu> References: , , , , <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu>, , <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu> Message-ID: Why? You do realize, though it doesn't really matter, that the function cannot be implemented in Modula-3, or C, and isn't implemented on most platforms. It is only implemented on NT, in assembly. Maybe RTHooks is the holder for most function calls from generated code? The interface between any library and m3front? No matter how and if they are implemented? (it could perhaps be implemented in C, void* m3_alloca(size_t n) { return alloca(n); } but I have a "problem" with that, not sure it is a real problem or not, that it strongly resembles returning the address of a local, and if tools had enough visibility into our code, they would flag it as a bug and/or misoptimize it.) As well, this still requires the backend to recognize and special-case the function. As well, it seems like atomics could fit the same model. Why are they in m3cg? I'm trying not to insist on anything in particular, but I'd like to understand why the different options. m3cg offers the most direct, easily implemented, easily optimized, special-case free approach, though it makes m3cg "wider" and m3cg is kind of scary and large. But making special-case functions is just a way to hide the interface widening..in a way that is easier to remove or omit per-target if applicable perhaps.. - Jay From: hosking at cs.purdue.edu Date: Tue, 1 Feb 2011 15:01:44 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] link failure on NT386 RTHooks makes most sense. Sent from my iPhone On Feb 1, 2011, at 2:20 PM, Jay K wrote: Modeling something as a function in m3cg does not necessarily correlate to generating a function call. E.g. the existing m3_alloca. And vice versa. e.g some of the set operations. Almost anything can be modeled as a function call in m3cg, but not end up generating one. Many things can be not modeled as a function call, but end up generating one. gcc has a huge number of "builtin functions". So does Visual C++. In Visual C++, basically any "new instruction" (mmx, sse2/3/4, avx) ends up modeled as a function, but generates roughly one instruction per function calls. It's a slightly high level assembly -- i.e. you get to chose the important algorithmic instructions, the compiler does register allocation, stack/local allocation, etc. I can add it -- surely my diff from a few weeks ago wasn't terrible? I'm ok either way. I'm more curious to understand than to "do". I admit. - Jay CC: m3devel at elegosoft.com From: hosking at cs.purdue.edu Subject: Re: [M3devel] link failure on NT386 Date: Tue, 1 Feb 2011 09:46:22 -0500 To: jay.krell at cornell.edu The atomics are not necessarily functions. They approximate real instructions generated inline. Shall I just go ahead and add it? Sent from my iPad On 01/02/2011, at 2:12 AM, Jay K wrote: Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. Why are the atomics in m3cg? I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. alloca is simpler just by virtue of it only taking one parameter. Why aren't and/or/xor/add runtime hooks, or special functions? Or the set operations? I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. And that there may be a desire to in general avoid growing m3cg. esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. I'm also wishy washy. I can see it is easy to implement easily any way. I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. At some point, I realize, you can model just about everything as function calls, except, calling functions. :) I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. Since anything implemented in Modula-3 and exposed must be in some interface. However runtime hooks can be implemented in C as well. It depends somewhat if they are only called by generated code or also hand written Modula-3. You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, there is just relying on m3front and m3core agreeing. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:39:36 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] link failure on NT386 Why can't we just have a run-time hook that resolves differently on different targets? Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. On Jan 31, 2011, at 7:53 PM, Jay K wrote: Oops, I was just wondering to self if I had fixed that. I can get to it fairly soon. We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling convention (which imho is a good reason to make it a separate m3cg call...) (The input parameter is in eax, and the output is in esp.) (Tony, you really don't want a separate m3cg call?) - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. Do you have a plan to fix? Anything I can do to help? --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Feb 2 00:57:37 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 1 Feb 2011 18:57:37 -0500 Subject: [M3devel] link failure on NT386 In-Reply-To: References: , , , , <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu>, , <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu> Message-ID: I still don't understand what you are proposing? M3CG.alloca? On Feb 1, 2011, at 4:12 PM, Jay K wrote: > Why? You do realize, though it doesn't really matter, that the function cannot be implemented in Modula-3, or C, and isn't implemented on most platforms. It is only implemented on NT, in assembly. > Maybe RTHooks is the holder for most function calls from generated code? > The interface between any library and m3front? > No matter how and if they are implemented? > > (it could perhaps be implemented in C, void* m3_alloca(size_t n) { return alloca(n); } but I have a "problem" with that, not sure it is a real problem or not, that it strongly resembles returning the address of a local, and if tools had enough visibility into our code, they would flag it as a bug and/or misoptimize it.) > > As well, this still requires the backend to recognize and special-case the function. > As well, it seems like atomics could fit the same model. Why are they in m3cg? > > I'm trying not to insist on anything in particular, but I'd like to understand why the different options. > m3cg offers the most direct, easily implemented, easily optimized, special-case free approach, though it makes m3cg "wider" and m3cg is kind of scary and large. But making special-case functions is just a way to hide the interface widening..in a way that is easier to remove or omit per-target if applicable perhaps.. > > > - Jay > > From: hosking at cs.purdue.edu > Date: Tue, 1 Feb 2011 15:01:44 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] link failure on NT386 > > RTHooks makes most sense. > > Sent from my iPhone > > On Feb 1, 2011, at 2:20 PM, Jay K wrote: > > Modeling something as a function in m3cg does not necessarily correlate to generating a function call. > E.g. the existing m3_alloca. > And vice versa. e.g some of the set operations. > Almost anything can be modeled as a function call in m3cg, but not end up generating one. > Many things can be not modeled as a function call, but end up generating one. > gcc has a huge number of "builtin functions". So does Visual C++. > In Visual C++, basically any "new instruction" (mmx, sse2/3/4, avx) ends up modeled > as a function, but generates roughly one instruction per function calls. > It's a slightly high level assembly -- i.e. you get to chose the important algorithmic instructions, > the compiler does register allocation, stack/local allocation, etc. > > I can add it -- surely my diff from a few weeks ago wasn't terrible? > > I'm ok either way. > I'm more curious to understand than to "do". I admit. > > - Jay > > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > Subject: Re: [M3devel] link failure on NT386 > Date: Tue, 1 Feb 2011 09:46:22 -0500 > To: jay.krell at cornell.edu > > The atomics are not necessarily functions. They approximate real instructions generated inline. > Shall I just go ahead and add it? > > Sent from my iPad > > On 01/02/2011, at 2:12 AM, Jay K wrote: > > Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? > In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. > Why are the atomics in m3cg? > I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. > alloca is simpler just by virtue of it only taking one parameter. > > Why aren't and/or/xor/add runtime hooks, or special functions? > Or the set operations? > > I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. > > And that there may be a desire to in general avoid growing m3cg. > esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. > > I'm also wishy washy. I can see it is easy to implement easily any way. > I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. > > > At some point, I realize, you can model just about everything as function calls, except, calling functions. :) > > I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. > Since anything implemented in Modula-3 and exposed must be in some interface. > However runtime hooks can be implemented in C as well. > It depends somewhat if they are only called by generated code or also hand written Modula-3. > You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, > there is just relying on m3front and m3core agreeing. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:39:36 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] link failure on NT386 > > Why can't we just have a run-time hook that resolves differently on different targets? > Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. > > On Jan 31, 2011, at 7:53 PM, Jay K wrote: > > Oops, I was just wondering to self if I had fixed that. > I can get to it fairly soon. > We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling > convention (which imho is a good reason to > make it a separate m3cg call...) > (The input parameter is in eax, and the output is in esp.) > (Tony, you really don't want a separate m3cg call?) > > > - Jay > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 31 Jan 2011 19:48:21 -0500 > Subject: [M3devel] link failure on NT386 > > Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. > Do you have a plan to fix? > Anything I can do to help? > --Randy Coleburn > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Feb 2 01:10:46 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 2 Feb 2011 00:10:46 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: References: , , , , , , , , <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu>, , , , <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu>, , Message-ID: I still don't understand what you are proposing? M3CG.alloca? Yes. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Thu Feb 3 03:02:27 2011 From: hendrik at topoi.pooq.com (Hendrik Boom) Date: Wed, 2 Feb 2011 21:02:27 -0500 Subject: [M3devel] About Juno-2 and automating debugging software In-Reply-To: <22454.41538.qm@web29715.mail.ird.yahoo.com> References: <459675.89026.qm@web29709.mail.ird.yahoo.com> <22454.41538.qm@web29715.mail.ird.yahoo.com> Message-ID: <20110203020227.GB1094@topoi.pooq.com> On Mon, Jan 31, 2011 at 04:20:58AM +0000, Daniel Alejandro Benavides D. wrote: > Hi all: > yes, I believe is doable especially if you think this issue comes again in another form, the AI guys thinking what went wrong, well nothing went wrong as I realize now, there is a lot of research certainly, but in the end is just the same thing like before, and you can see this like the reuse of concepts of prior knowledge as is mentioned between other by Dr Andrew Tanenbaum, it just happens when something like this: see, the GPGPU GPU stuff alike is a nother come of the concept of explicit parallelism and its implication in computing fields, surely more developed in terms of the hardware this was I believe, but at the end is just another way of calling Transputers in earlier times, is just like it, then it was not an abandoned thing like before, just a reuse of the concept, and in practical terms this what it is the Occam Occam2 and alike languages; there is one in particular of my interest here the one built for drawing using CSP of Hoare, the language > is called Armadillo see a slide show here, specially check for the Computer magazine backcover in p. 6 in: > https://lsl.unal.edu.co/fl18/conferencias/09.20_ArmadilloTortugav02.pdf The TUNES website is a good place to look for plans and musings in the direction of formally verified programming: http://tunes.org/ It's a project that aims to use formal verification for security, and has produces a lot of interesting theorizing about the issues, but not much practical code. At least, that was the case last time I looked at the web site seriously a few years ago. -- hendrik From dabenavidesd at yahoo.es Sun Feb 6 01:07:56 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 6 Feb 2011 00:07:56 +0000 (GMT) Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: <20110205221326.967B12474007@birch.elegosoft.com> Message-ID: <692616.32152.qm@web29703.mail.ird.yahoo.com> Hi all: it seems that by denying itself from being well behaved program is checking for the unmatched traces or good behaviour programs, sort of analyzing many times more (incorrect traces with correct one) than checking the traces are well behaved and just checking wheh its not, better to check when its not, and not asumming the traces are ill anyway, then most of the time is spend in good standing than in bad or ill behaved program nalysis. The question how do you get those traces in good state, for instance checking by itself several times you would gain a pattern is repeatable, opposite of and ill corrupt and unmatched execution traces. In that case it makes a trace pattern you are in good stands but of course if it doens't then you have sort of a problem there. The next step is where to get the break point before it goes into an unrepeatable trace, that is when you must have an static analyzer which run a simulation and verifies program is well stated at each assert of it. Then it seems like you were getting the opposite direction going to see where it is not working yet at each part it makes asserts it, sort of looking for not well behaved errors but rather irrepeatable ones, which I think is harder. A error one you get by denying it's safe guarded makes illegal steps in the program you state or forbidden ones, is an unsafe error, a safe error is if you get it to pass it but you can't get it works by its results (sort of visbile) so getting in the right results but not in its good states are plausible to find, but ill ones, sort of what you are looking for i.e. looking for forbidden states that are ill-behaved and get wrong results are almost the harder path to get it such a bug Just to let you know if you see this point. --- El s?b, 5/2/11, Mika Nystrom escribi?: > De: Mika Nystrom > Asunto: [M3commit] CVS Update: cm3 > Para: m3commit at elegosoft.com > Fecha: s?bado, 5 de febrero, 2011 18:13 > CVSROOT: /usr/cvs > Changes by: mika at birch. > 11/02/05 23:13:26 > > Modified files: > cm3/m3-libs/m3core/tests/thread/src/: > Main.m3 > > Log message: > removed a lot of unnecessary heap memory > allocation, which could interfere with the threading tests. > > Most output to Stdio.stdout, not sure if > this is right (compared with Stdio.stderr). > > From hendrik at topoi.pooq.com Sun Feb 6 01:49:35 2011 From: hendrik at topoi.pooq.com (Hendrik Boom) Date: Sat, 5 Feb 2011 19:49:35 -0500 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: <692616.32152.qm@web29703.mail.ird.yahoo.com> References: <20110205221326.967B12474007@birch.elegosoft.com> <692616.32152.qm@web29703.mail.ird.yahoo.com> Message-ID: <20110206004935.GA31175@topoi.pooq.com> On Sun, Feb 06, 2011 at 12:07:56AM +0000, Daniel Alejandro Benavides D. wrote: > Hi all: ... ... > Just to let you know if you see this point. It sounds as if you're saying it's possible to prove a program is faulty by running it, but it's not possible to say it's correct. And concurrency makes things nonrepeatable, especially when there's bugs. -- hendrik From dabenavidesd at yahoo.es Sun Feb 6 02:01:16 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 6 Feb 2011 01:01:16 +0000 (GMT) Subject: [M3devel] About Juno-2 and automating debugging software In-Reply-To: <20110203020227.GB1094@topoi.pooq.com> Message-ID: <749746.64343.qm@web29720.mail.ird.yahoo.com> Hi all: thanks for your asnwer I came to see this today: http://www.espresso-soft.org/main.php?button=supervisors expand the last (show/hide) abstract selector. I think definitively this guys looks better things than many of us !! Or not? In a sense is good I guess, but what if not? Just hope to see what it brings to it. :) --- El mi?, 2/2/11, Hendrik Boom escribi?: > De: Hendrik Boom > Asunto: Re: [M3devel] About Juno-2 and automating debugging software > Para: m3devel at elegosoft.com > Fecha: mi?rcoles, 2 de febrero, 2011 21:02 > On Mon, Jan 31, 2011 at 04:20:58AM > +0000, Daniel Alejandro Benavides D. wrote: > > Hi all: > > yes, I believe is doable especially if you think this > issue comes again in another form, the AI guys thinking what > went wrong, well nothing went wrong as I realize now, there > is a lot of research certainly, but in the end is just the > same thing like before, and you can see this like the reuse > of concepts of prior knowledge as is mentioned between other > by Dr Andrew Tanenbaum, it just happens when something like > this: see, the GPGPU GPU stuff alike is a nother come of the > concept of explicit parallelism and its implication in > computing fields, surely more developed in terms of the > hardware this was I believe, but at the end is just another > way of calling Transputers in earlier times, is just like > it, then it was not an abandoned thing like before, just a > reuse of the concept, and in practical terms this what it is > the Occam Occam2 and alike languages; there is one in > particular of my interest here the one built for drawing > using CSP of Hoare, the language > > is called Armadillo see a slide show here, > specially check for the Computer magazine backcover in p. 6 > in: > > https://lsl.unal.edu.co/fl18/conferencias/09.20_ArmadilloTortugav02.pdf > > The TUNES website is a good place to look for plans and > musings in the > direction of formally verified programming: > > http://tunes.org/ > > It's a project that aims to use formal verification for > security, and > has produces a lot of interesting theorizing about the > issues, but not > much practical code. At least, that was the case last > time I looked > at the web site seriously a few years ago. > > -- hendrik > From dabenavidesd at yahoo.es Sun Feb 6 02:14:01 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 6 Feb 2011 01:14:01 +0000 (GMT) Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: <20110206004935.GA31175@topoi.pooq.com> Message-ID: <879742.59049.qm@web29710.mail.ird.yahoo.com> Hi all: yes in a sense, as we don't have the static checker or any static analyzer for its check yet, that is what it is for now, or Am I wrong? Surely, they showed that is possible to find; Damian Dominguez with Dr Nelson to see a thread deadlock in object oriented code but with a special tool very primitive, but in general we don't have that, I mean we just can get where a thread implementation makes this things illegal, not when the program is illegal by itself, that's the aim for me now. Better to think as you say even better. I don't dispute the last point though you can get a tool to analyze a sample of outputs and make a distribution of it thats what I think a professor told me a day when looking for one of threading without locking, just yielding its control, here surely the case is very different. Thanks in advance --- El s?b, 5/2/11, Hendrik Boom escribi?: > De: Hendrik Boom > Asunto: Re: [M3devel] [M3commit] CVS Update: cm3 > Para: m3devel at elegosoft.com > Fecha: s?bado, 5 de febrero, 2011 19:49 > On Sun, Feb 06, 2011 at 12:07:56AM > +0000, Daniel Alejandro Benavides D. wrote: > > Hi all: > ... > ... > > Just to let you know if you see this point. > > It sounds as if you're saying it's possible to prove a > program is faulty > by running it, but it's not possible to say it's correct. > > And concurrency makes things nonrepeatable, especially when > there's > bugs. > > -- hendrik > From jay.krell at cornell.edu Tue Feb 8 15:06:34 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 8 Feb 2011 14:06:34 +0000 Subject: [M3devel] assert in Modula-3 C code? Message-ID: It was recently suspect that asserts aren't enabled in the C code in m3core.h I don't think so. I think there might just be a missing check of a return value in m3core/thread/*.m3 However, if we want to dispell this fear now and forever, we could do e.g.: m3core.h: M3_DLL_LOCAL void __stdcall M3C__AssertFailed (const char* file, unsigned line, const char* expression); #define M3_ASSERT(expr) ((expr) ? ((void)0) ? M3C__AssertFailed ( \ __FILE__, __LINE__, #expression)) hand.c: M3_DLL_LOCAL void __stdcall M3C__AssertFailed (const char* file, unsigned line, const char* expression) { fprintf (stderr, "%s(%u): assertion failed: %s\n", file, line, expression); abort (); } This might slightly reduce quality on some platforms, e.g.: Assertion failed: (0), function main, file 1.c, line 2. Abort trap jbook2:libm3 jay$ cat 1.c #include int main() { assert(0); return 0; } that is -- see how it printed the file name? Anyway, we could get that back too, like #if __GNUC__ > whatever #define M3_FUNCTION something #elif _MSC_VER > something #define M3_FUNCTION __FUNCTION__ #else #define M3_FUNCTION 0 #endif ? But I think probably using standard assert is adequate or preferable. We could #undef NDEBUG in m3core.h, in case someone added to the C compiler on the config files. or #ifdef NDEBUG, #error. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 8 15:27:57 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 8 Feb 2011 09:27:57 -0500 Subject: [M3devel] assert in Modula-3 C code? In-Reply-To: References: Message-ID: I think I was mistaken. It does appear to be turned on. On Feb 8, 2011, at 9:06 AM, Jay K wrote: > It was recently suspect that asserts aren't enabled in the C code in m3core.h > I don't think so. I think there might just be a missing check of a return value in m3core/thread/*.m3 > > However, if we want to dispell this fear now and forever, we could do e.g.: > > m3core.h: > > M3_DLL_LOCAL > void > __stdcall > M3C__AssertFailed (const char* file, unsigned line, const char* expression); > > #define M3_ASSERT(expr) ((expr) ? ((void)0) ? M3C__AssertFailed ( \ > __FILE__, __LINE__, #expression)) > > > > hand.c: > > M3_DLL_LOCAL > void > __stdcall > M3C__AssertFailed (const char* file, unsigned line, const char* expression) > { > fprintf (stderr, "%s(%u): assertion failed: %s\n", file, line, expression); > abort (); > } > > This might slightly reduce quality on some platforms, e.g.: > > Assertion failed: (0), function main, file 1.c, line 2. > Abort trap > jbook2:libm3 jay$ cat 1.c > #include > int main() { assert(0); return 0; } > > > that is -- see how it printed the file name? Anyway, we could get that back too, like > #if __GNUC__ > whatever > #define M3_FUNCTION something > #elif _MSC_VER > something > #define M3_FUNCTION __FUNCTION__ > #else > #define M3_FUNCTION 0 > #endif > > > ? > > But I think probably using standard assert is adequate or preferable. > We could #undef NDEBUG in m3core.h, in case someone added to the C compiler on the config files. > or #ifdef NDEBUG, #error. > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 8 23:41:44 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 8 Feb 2011 22:41:44 +0000 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: ,,, , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, ,,, , , , , , , <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu>, Message-ID: "Are you sure?" I mean, are you sure the right thing wouldn't be to declare Variables in Try*.Parse or Try*.Check? Possibly they'll even be at a high enough scope as to be correct? ie: they don't really have to be at the function scope, as long as they are just outside the try and the except/finally...depending on how except/finally are generated..ok, this is the less important point. I can see putting them at function-scope being easy enough and more clearly correct. But I do still wonder about the approach using Variable.T and such, instead of something "custom" involving Cg.declare_local and separate hooks in ProcBody esp. for initialization. (Though I can see a "custom" approach having other benefits: e.g. doing a multiplication jmpbuf_size times number of trys and making one larger call to alloca. Furthermore, the approach that uses just one jmpbuf for the entire function, plus a local integer, would more reasonly need a special access to the function scope.) I have a bit more time to spend here now maybe..but I'm tempted to put in a temporary hack of using a 1K jmpbuf for all targets..at least that keeps the per-target code out... and an assert m3core that it is large enough (as was there already). We could also pick some liberal values per-processor, like 64 bytes for x86, 128 for amd64, 512 for sparc, 1K for powerpc/alpha/mips/other, along with the assertion. (powerpc does tend to need quite large jmpbuf, alpha/mips/other are rare and can suffer until a better solution is in place, 64 for x86 is still bloated but not so bad temporarily..)) Again, this is just a stopgap. The fact that very long running loops with a try allocate more and more stack is quite bad. I didn't realize it at the time, sorry. I also think the frontend should declare the EF1 type correctly and itself, so as to initialize the pointer in it, instead of having a separate pointer local variable that gets copied to it. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Tue, 1 Feb 2011 07:16:40 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? These look like good hints. I wouldn't know to make such slightly large changes, that the existing infrastructure doesn't suffice. I'll have a go at it..as my now-very-limited time permits. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:45:53 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I'd like to do what you envisage. It is a significant improvement, and should not require too much effort: 1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3) 2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. On Jan 31, 2011, at 7:55 PM, Jay K wrote: ps: I'm happy to report that my available time here has become significantly reduced. Partly with that in mind, but even w/o it, I am willing to go back to the old way. m3front is full of mystery and I'm not sure I'm keen on understanding it. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 15:26:20 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I think we need to have a way to get the scope of the containing procedure. TryStmt.Parse is not the right place. If I get some time soon I can take a look at this. On Jan 31, 2011, at 2:19 PM, Jay K wrote: Right..I also tried Scope.Insert, didn't seem to help. But I think I might be doing it too late. I'll try doing it in Parse instead of Check. Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat more closely resembles the old pattern: IF Foo() THEN RETURN ELSE TRY FINALLY in the old way would use the stack, so we could too. But the allocation is slower/larger now (alloca possibly aligning up more than needed). Anyway, this point isn't the problem. The problem is my inability to work with m3front. I'll look at it a bit more -- i.e. as I said, moving the code to Parse. The other improvement I'd like to make: currently, if I had it working, what I'm aiming for is: struct EF1{ ... jmpbuf* jb } ef1; jmpbuf* jb = 0; jb = jb ? jb : alloca(sizeof(*jb)); ef1.jb = jb; setjmp(ef1.jb); but it'd be better to have: struct EF1{ ... jmpbuf* jb } ef1; ef1.jb = 0; ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); setjmp(ef1.jb); ie. we don't need the extra pointer variable, the pointer in the record should suffice. [not sure where to put this in email] I tried figuring out how to determine the current vs. topmost-in-current-function scope. I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, so I kinda hoped it'd just use the correct scope. Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. Even if it doesn't get used, that might be ok, since there is tracking of use, and declaring it might not cause any waste if it isn't used. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 12:14:09 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? Hi Jay, I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Jan 30, 2011, at 5:45 AM, Jay K wrote: I'm trying to fix the inadvertant change I made where FOR i := 1 TO 10 DO TRY FINALLY END will call alloca 10 times. I'm not familiar with the frontend, and it is hard for me to understand. It seems like it is probably very well factored, written by someone who really knew what they were doing, but it is dense, and not verbosely commented. Something like this?? But this doesn't work. What am I missing? And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? ? 1.txt Index: src/misc/Marker.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v retrieving revision 1.1.1.1 diff -u -r1.1.1.1 Marker.i3 --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 @@ -64,9 +64,12 @@ PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); (* generate the call to acquire or release a mutex *) -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T); (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) PROCEDURE Reset (); +PROCEDURE NewJumpbuf (): Variable.T; + END Marker. Index: src/misc/Marker.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v retrieving revision 1.9 diff -u -r1.9 Marker.m3 --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 @@ -10,7 +10,7 @@ MODULE Marker; IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; TYPE Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, @@ -231,8 +231,10 @@ END; END CallFinallyHandler; -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T) = VAR new: BOOLEAN; + label := CG.Next_label (); BEGIN (* int setjmp(void* ); *) IF (setjmp = NIL) THEN @@ -262,13 +264,33 @@ Target.Word.size, Target.Word.align, Target.Word.cg_type, 0); END; - - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) + + (* IF jumpuf # NIL THEN + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); + * END + *) + + CG.Load_nil (); + Variable.Load (jumpbuf); + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); + CG.Start_call_direct (alloca, 0, Target.Address.cg_type); CG.Load_int (Target.Word.cg_type, Jumpbuf_size); CG.Pop_param (Target.Word.cg_type); CG.Call_direct (alloca, Target.Address.cg_type); + + (* FUTURE: We should actually have a Variable + * for the entire EF1, including initializing the jumpbuf + * in it. That would save us this extra load/store, + * and save a local variable. + *) + Variable.LoadLValue (jumpbuf); + CG.Swap (); + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); + Variable.Load (jumpbuf); + CG.Store_addr (frame, M3RT.EF1_jmpbuf); + CG.Set_label (label); (* setmp(frame.jmpbuf) *) CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); @@ -806,5 +828,15 @@ tos := 0; END Reset; +PROCEDURE NewJumpbuf (): Variable.T = + VAR jumpbuf: Variable.T; + BEGIN + jumpbuf := Variable.New (M3ID.NoID, TRUE); + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, + readonly := FALSE, open_array_ok := FALSE, + needs_init := TRUE); + RETURN jumpbuf; + END NewJumpbuf; + BEGIN END Marker. Index: src/stmts/TryFinStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v retrieving revision 1.6 diff -u -r1.6 TryFinStmt.m3 --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -10,6 +10,7 @@ IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; +IMPORT Variable; FROM Stmt IMPORT Outcome; TYPE @@ -20,6 +21,7 @@ viaProc : BOOLEAN; scope : Scope.T; handler : HandlerProc; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -63,6 +65,11 @@ RETURN p; END Parse; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + RETURN NOT (Target.Has_stack_walker OR p.viaProc); + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; BEGIN @@ -95,6 +102,9 @@ END; END; TryStmt.PopHandler (); + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE HandlerName (uid: INTEGER): TEXT = @@ -106,11 +116,16 @@ END HandlerName; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN - IF Target.Has_stack_walker THEN RETURN Compile1 (p); - ELSIF p.viaProc THEN RETURN Compile2 (p); - ELSE RETURN Compile3 (p); + IF Target.Has_stack_walker THEN result := Compile1 (p); + ELSIF p.viaProc THEN result := Compile2 (p); + ELSE usesSetjmp := TRUE; + result := Compile3 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -302,7 +317,9 @@ l := CG.Next_label (3); CG.Set_label (l, barrier := TRUE); Marker.PushFrame (frame, M3RT.HandlerClass.Finally); - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) Marker.PushFinally (l, l+1, frame); Index: src/stmts/TryStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v retrieving revision 1.3 diff -u -r1.3 TryStmt.m3 --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -22,6 +22,7 @@ hasElse : BOOLEAN; elseBody : Stmt.T; handled : ESet.T; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -153,6 +154,14 @@ p.handles := h3; END ReverseHandlers; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + IF (p.handles = NIL) AND (NOT p.hasElse) THEN + RETURN FALSE; + END; + RETURN NOT Target.Has_stack_walker; + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR h: Handler; handled: ESet.T; BEGIN @@ -182,6 +191,10 @@ WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; Stmt.TypeCheck (p.elseBody, cs); PopHandler (); + + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = @@ -245,14 +258,19 @@ END CheckHandler; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN IF (p.handles = NIL) AND (NOT p.hasElse) THEN - RETURN Stmt.Compile (p.body); - END; - IF Target.Has_stack_walker - THEN RETURN Compile1 (p); - ELSE RETURN Compile2 (p); + result := Stmt.Compile (p.body); + ELSIF Target.Has_stack_walker THEN + result := Compile1 (p); + ELSE + usesSetjmp := TRUE; + result := Compile2 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -423,7 +441,9 @@ END; (* capture the machine state *) - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) oc := Stmt.Compile (p.body); Thanks, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Feb 9 15:18:59 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 9 Feb 2011 09:18:59 -0500 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: , , , , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, , , , , , , , , , <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu>, Message-ID: <65642FC4-4F9C-4F8B-817E-EE32B93D4E11@cs.purdue.edu> That is overkill and not particularly modular. I'd prefer to keep the changes as localised as possible. On Feb 8, 2011, at 5:41 PM, Jay K wrote: > "Are you sure?" > I mean, are you sure the right thing wouldn't be to declare Variables in Try*.Parse or Try*.Check? > Possibly they'll even be at a high enough scope as to be correct? > > > ie: they don't really have to be at the function scope, as long as they are just outside the try and the except/finally...depending on how except/finally are generated..ok, this is the less important point. I can see putting them at function-scope being easy enough and more clearly correct. But I do still wonder about the approach using Variable.T and such, instead of something "custom" involving Cg.declare_local and separate hooks in ProcBody esp. for initialization. > > > (Though I can see a "custom" approach having other benefits: e.g. doing a multiplication jmpbuf_size times number of trys and making one larger call to alloca. Furthermore, the approach that uses just one jmpbuf for the entire function, plus a local integer, would more reasonly need a special access to the function scope.) > > > I have a bit more time to spend here now maybe..but I'm tempted to put in a temporary hack of using a 1K jmpbuf for all targets..at least that keeps the per-target code out... and an assert m3core that it is large enough (as was there already). > > > > We could also pick some liberal values per-processor, like 64 bytes for x86, 128 for amd64, 512 for sparc, 1K for powerpc/alpha/mips/other, along with the assertion. (powerpc does tend to need quite large jmpbuf, alpha/mips/other are rare and can suffer until a better solution is in place, 64 for x86 is still bloated but not so bad temporarily..)) Again, this is just a stopgap. The fact that very long running loops with a try allocate more and more stack is quite bad. I didn't realize it at the time, sorry. > > > I also think the frontend should declare the EF1 type correctly and itself, so as to initialize the pointer in it, instead of having a separate pointer local variable that gets copied to it. > > > - Jay > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Tue, 1 Feb 2011 07:16:40 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > These look like good hints. I wouldn't know to make such slightly large changes, > that the existing infrastructure doesn't suffice. > I'll have a go at it..as my now-very-limited time permits. > > Thanks, > - Jay > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:45:53 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I'd like to do what you envisage. It is a significant improvement, and should not require too much effort: > 1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3) > 2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. > > On Jan 31, 2011, at 7:55 PM, Jay K wrote: > > ps: I'm happy to report that my available time here > has become significantly reduced. Partly with that in mind, > but even w/o it, I am willing to go back to the old way. > m3front is full of mystery and I'm not sure I'm keen on understanding it. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 15:26:20 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I think we need to have a way to get the scope of the containing procedure. > TryStmt.Parse is not the right place. > If I get some time soon I can take a look at this. > > On Jan 31, 2011, at 2:19 PM, Jay K wrote: > > Right..I also tried Scope.Insert, didn't seem to help. > > But I think I might be doing it too late. I'll try doing it in Parse instead of Check. > Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. > I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat > more closely resembles the old pattern: > > IF Foo() THEN > RETURN > ELSE TRY FINALLY > > in the old way would use the stack, so we could too. > But the allocation is slower/larger now (alloca possibly aligning up more than needed). > Anyway, this point isn't the problem. The problem is my inability to work with m3front. > I'll look at it a bit more -- i.e. as I said, moving the code to Parse. > > The other improvement I'd like to make: > > currently, if I had it working, what I'm aiming for is: > struct EF1{ ... jmpbuf* jb } ef1; > jmpbuf* jb = 0; > > jb = jb ? jb : alloca(sizeof(*jb)); > ef1.jb = jb; > setjmp(ef1.jb); > > but it'd be better to have: > > struct EF1{ ... jmpbuf* jb } ef1; > ef1.jb = 0; > > ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); > setjmp(ef1.jb); > > > ie. we don't need the extra pointer variable, the pointer in the record should suffice. > > > [not sure where to put this in email] > I tried figuring out how to determine the current vs. topmost-in-current-function scope. > I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps > some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, > so I kinda hoped it'd just use the correct scope. > > > Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. > Even if it doesn't get used, that might be ok, since there is tracking of use, and > declaring it might not cause any waste if it isn't used. > > > Thanks, > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 12:14:09 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > Hi Jay, > > I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On Jan 30, 2011, at 5:45 AM, Jay K wrote: > > I'm trying to fix the inadvertant change I made where > FOR i := 1 TO 10 DO TRY FINALLY END > > will call alloca 10 times. > > I'm not familiar with the frontend, and it is hard for me to understand. > It seems like it is probably very well factored, written by someone who really knew what they were doing, > but it is dense, and not verbosely commented. > > Something like this?? But this doesn't work. What am I missing? > And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? > > > ? 1.txt > Index: src/misc/Marker.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v > retrieving revision 1.1.1.1 > diff -u -r1.1.1.1 Marker.i3 > --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 > +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 > @@ -64,9 +64,12 @@ > PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); > (* generate the call to acquire or release a mutex *) > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T); > (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) > > PROCEDURE Reset (); > > +PROCEDURE NewJumpbuf (): Variable.T; > + > END Marker. > Index: src/misc/Marker.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v > retrieving revision 1.9 > diff -u -r1.9 Marker.m3 > --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 > +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,7 +10,7 @@ > MODULE Marker; > > IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; > -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; > +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; > > TYPE > Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, > @@ -231,8 +231,10 @@ > END; > END CallFinallyHandler; > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T) = > VAR new: BOOLEAN; > + label := CG.Next_label (); > BEGIN > (* int setjmp(void* ); *) > IF (setjmp = NIL) THEN > @@ -262,13 +264,33 @@ > Target.Word.size, Target.Word.align, > Target.Word.cg_type, 0); > END; > - > - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) > + > + (* IF jumpuf # NIL THEN > + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); > + * END > + *) > + > + CG.Load_nil (); > + Variable.Load (jumpbuf); > + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); > + > CG.Start_call_direct (alloca, 0, Target.Address.cg_type); > CG.Load_int (Target.Word.cg_type, Jumpbuf_size); > CG.Pop_param (Target.Word.cg_type); > CG.Call_direct (alloca, Target.Address.cg_type); > + > + (* FUTURE: We should actually have a Variable > + * for the entire EF1, including initializing the jumpbuf > + * in it. That would save us this extra load/store, > + * and save a local variable. > + *) > + Variable.LoadLValue (jumpbuf); > + CG.Swap (); > + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); > + Variable.Load (jumpbuf); > + > CG.Store_addr (frame, M3RT.EF1_jmpbuf); > + CG.Set_label (label); > > (* setmp(frame.jmpbuf) *) > CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); > @@ -806,5 +828,15 @@ > tos := 0; > END Reset; > > +PROCEDURE NewJumpbuf (): Variable.T = > + VAR jumpbuf: Variable.T; > + BEGIN > + jumpbuf := Variable.New (M3ID.NoID, TRUE); > + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, > + readonly := FALSE, open_array_ok := FALSE, > + needs_init := TRUE); > + RETURN jumpbuf; > + END NewJumpbuf; > + > BEGIN > END Marker. > Index: src/stmts/TryFinStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v > retrieving revision 1.6 > diff -u -r1.6 TryFinStmt.m3 > --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 > +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,6 +10,7 @@ > > IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; > IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; > +IMPORT Variable; > FROM Stmt IMPORT Outcome; > > TYPE > @@ -20,6 +21,7 @@ > viaProc : BOOLEAN; > scope : Scope.T; > handler : HandlerProc; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -63,6 +65,11 @@ > RETURN p; > END Parse; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + RETURN NOT (Target.Has_stack_walker OR p.viaProc); > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; > BEGIN > @@ -95,6 +102,9 @@ > END; > END; > TryStmt.PopHandler (); > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE HandlerName (uid: INTEGER): TEXT = > @@ -106,11 +116,16 @@ > END HandlerName; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > - IF Target.Has_stack_walker THEN RETURN Compile1 (p); > - ELSIF p.viaProc THEN RETURN Compile2 (p); > - ELSE RETURN Compile3 (p); > + IF Target.Has_stack_walker THEN result := Compile1 (p); > + ELSIF p.viaProc THEN result := Compile2 (p); > + ELSE usesSetjmp := TRUE; > + result := Compile3 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -302,7 +317,9 @@ > l := CG.Next_label (3); > CG.Set_label (l, barrier := TRUE); > Marker.PushFrame (frame, M3RT.HandlerClass.Finally); > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > Marker.PushFinally (l, l+1, frame); > Index: src/stmts/TryStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v > retrieving revision 1.3 > diff -u -r1.3 TryStmt.m3 > --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 > +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -22,6 +22,7 @@ > hasElse : BOOLEAN; > elseBody : Stmt.T; > handled : ESet.T; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -153,6 +154,14 @@ > p.handles := h3; > END ReverseHandlers; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + IF (p.handles = NIL) AND (NOT p.hasElse) THEN > + RETURN FALSE; > + END; > + RETURN NOT Target.Has_stack_walker; > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR h: Handler; handled: ESet.T; > BEGIN > @@ -182,6 +191,10 @@ > WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; > Stmt.TypeCheck (p.elseBody, cs); > PopHandler (); > + > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = > @@ -245,14 +258,19 @@ > END CheckHandler; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > IF (p.handles = NIL) AND (NOT p.hasElse) THEN > - RETURN Stmt.Compile (p.body); > - END; > - IF Target.Has_stack_walker > - THEN RETURN Compile1 (p); > - ELSE RETURN Compile2 (p); > + result := Stmt.Compile (p.body); > + ELSIF Target.Has_stack_walker THEN > + result := Compile1 (p); > + ELSE > + usesSetjmp := TRUE; > + result := Compile2 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -423,7 +441,9 @@ > END; > > (* capture the machine state *) > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > oc := Stmt.Compile (p.body); > > Thanks, > - Jay > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Wed Feb 9 23:00:57 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Wed, 09 Feb 2011 14:00:57 -0800 Subject: [M3devel] threading on Windows? Message-ID: <20110209220057.790671A2078@async.async.caltech.edu> Hi m3devel, I'm just curious if anyone out there who's running CM3 on Windows has had a chance to try my thread testing program yet. Current status on Unix (Linux, FreeBSD) appears to be that there are problems in pthreads (but I think only under heavy load) and that user threading works perfectly. So I now wonder how things are on the third threading platform (Windows). The thread testing program is at m3-libs/m3core/tests/thread . (Note however that there is a memory leak in TRY-EXCEPT in the current head so don't try to update everything to the CVS head. The previous release should be OK.) Also has anyone else noticed that debugging information has recently broken? m3gdb is very unhappy on all platforms for me... Mika From rcolebur at SCIRES.COM Wed Feb 9 23:25:20 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Wed, 9 Feb 2011 17:25:20 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <20110209220057.790671A2078@async.async.caltech.edu> References: <20110209220057.790671A2078@async.async.caltech.edu> Message-ID: Mika: Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. As soon as I can get the build problem resolved, I'll try out your test program. Regards, Randy Coleburn -----Original Message----- From: Mika Nystrom [mailto:mika at async.caltech.edu] Sent: Wednesday, February 09, 2011 5:01 PM To: m3devel at elegosoft.com Subject: [M3devel] threading on Windows? Hi m3devel, I'm just curious if anyone out there who's running CM3 on Windows has had a chance to try my thread testing program yet. Current status on Unix (Linux, FreeBSD) appears to be that there are problems in pthreads (but I think only under heavy load) and that user threading works perfectly. So I now wonder how things are on the third threading platform (Windows). The thread testing program is at m3-libs/m3core/tests/thread . (Note however that there is a memory leak in TRY-EXCEPT in the current head so don't try to update everything to the CVS head. The previous release should be OK.) Also has anyone else noticed that debugging information has recently broken? m3gdb is very unhappy on all platforms for me... Mika From jay.krell at cornell.edu Wed Feb 9 23:58:30 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 9 Feb 2011 22:58:30 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, Message-ID: Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Thu Feb 10 00:22:56 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, Message-ID: I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 10 03:01:22 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 10 Feb 2011 02:01:22 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , Message-ID: In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Thu Feb 10 04:28:45 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Wed, 9 Feb 2011 22:28:45 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , Message-ID: Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 10 05:40:03 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 10 Feb 2011 04:40:03 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , , Message-ID: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -0500 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Feb 10 06:48:20 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 10 Feb 2011 00:48:20 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , , Message-ID: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: > Of course not. > It is a made-up function that the frontend generates a call to. > That it never previously did. > Using the general function call interface. > You need to add a special case, in the general function call code, to do something different and specific > for this specific function. > > > - Jay > > From: rcolebur at SCIRES.COM > To: jay.krell at cornell.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 22:28:45 -0500 > Subject: Re: [M3devel] threading on Windows? > > Jay: > > I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. > > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 9:01 PM > To: Coleburn, Randy; m3devel > Subject: RE: [M3devel] threading on Windows? > > In m3back/src/M3x86.m3. > > - Jay > > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 18:22:56 -0500 > Subject: Re: [M3devel] threading on Windows? > > I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. > All I know is that everything was working fine until I checked out the HEAD repository. > When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? > Are we talking Modula-3 code, C code, Assembler, what? What package/module? > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 5:59 PM > To: Coleburn, Randy; Mika Nystrom; m3devel > Subject: RE: [M3devel] threading on Windows? > > Maybe someone else can do it? > The fix is: special case calls to the function "m3_alloca". > Change it to call "_chkstk" (or maybe "chkstk", whatever works). > The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. > To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. > > - Jay > > > From: rcolebur at SCIRES.COM > > To: mika at async.caltech.edu; m3devel at elegosoft.com > > Date: Wed, 9 Feb 2011 17:25:20 -0500 > > Subject: Re: [M3devel] threading on Windows? > > > > Mika: > > > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > > > As soon as I can get the build problem resolved, I'll try out your test program. > > > > Regards, > > Randy Coleburn > > > > -----Original Message----- > > From: Mika Nystrom [mailto:mika at async.caltech.edu] > > Sent: Wednesday, February 09, 2011 5:01 PM > > To: m3devel at elegosoft.com > > Subject: [M3devel] threading on Windows? > > > > Hi m3devel, > > > > I'm just curious if anyone out there who's running CM3 on Windows has had > > a chance to try my thread testing program yet. Current status on Unix > > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > > I think only under heavy load) and that user threading works perfectly. > > So I now wonder how things are on the third threading platform (Windows). > > The thread testing program is at m3-libs/m3core/tests/thread . > > > > (Note however that there is a memory leak in TRY-EXCEPT in the current > > head so don't try to update everything to the CVS head. The previous > > release should be OK.) > > > > Also has anyone else noticed that debugging information has recently > > broken? m3gdb is very unhappy on all platforms for me... > > > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 10 12:01:12 2011 From: jay.krell at cornell.edu (jay.krell at cornell.edu) Date: Thu, 10 Feb 2011 03:01:12 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu> Message-ID: <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: > What happened to fixing this with runtime hooks? > > > On Feb 9, 2011, at 11:40 PM, Jay K wrote: > >> Of course not. >> It is a made-up function that the frontend generates a call to. >> That it never previously did. >> Using the general function call interface. >> You need to add a special case, in the general function call code, to do something different and specific >> for this specific function. >> >> >> - Jay >> >> From: rcolebur at SCIRES.COM >> To: jay.krell at cornell.edu; m3devel at elegosoft.com >> Date: Wed, 9 Feb 2011 22:28:45 -050 >> Subject: Re: [M3devel] threading on Windows? >> >> Jay: >> >> I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. >> >> Regards, >> Randy >> >> From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K >> Sent: Wednesday, February 09, 2011 9:01 PM >> To: Coleburn, Randy; m3devel >> Subject: RE: [M3devel] threading on Windows? >> >> In m3back/src/M3x86.m3. >> >> - Jay >> >> >> From: rcolebur at SCIRES.COM >> To: m3devel at elegosoft.com >> Date: Wed, 9 Feb 2011 18:22:56 -0500 >> Subject: Re: [M3devel] threading on Windows? >> >> I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. >> All I know is that everything was working fine until I checked out the HEAD repository. >> When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? >> Are we talking Modula-3 code, C code, Assembler, what? What package/module? >> Regards, >> Randy >> >> From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K >> Sent: Wednesday, February 09, 2011 5:59 PM >> To: Coleburn, Randy; Mika Nystrom; m3devel >> Subject: RE: [M3devel] threading on Windows? >> >> Maybe someone else can do it? >> The fix is: special case calls to the function "m3_alloca". >> Change it to call "_chkstk" (or maybe "chkstk", whatever works). >> The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. >> To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. >> >> - Jay >> >> > From: rcolebur at SCIRES.COM >> > To: mika at async.caltech.edu; m3devel at elegosoft.com >> > Date: Wed, 9 Feb 2011 17:25:20 -0500 >> > Subject: Re: [M3devel] threading on Windows? >> > >> > Mika: >> > >> > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. >> > >> > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. >> > >> > As soon as I can get the build problem resolved, I'll try out your test program. >> > >> > Regards, >> > Randy Coleburn >> > >> > -----Original Message----- >> > From: Mika Nystrom [mailto:mika at async.caltech.edu] >> > Sent: Wednesday, February 09, 2011 5:01 PM >> > To: m3devel at elegosoft.com >> > Subject: [M3devel] threading on Windows? >> > >> > Hi m3devel, >> > >> > I'm just curious if anyone out there who's running CM3 on Windows has had >> > a chance to try my thread testing program yet. Current status on Unix >> > (Linux, FreeBSD) appears to be that there are problems in pthreads (but >> > I think only under heavy load) and that user threading works perfectly. >> > So I now wonder how things are on the third threading platform (Windows). >> > The thread testing program is at m3-libs/m3core/tests/thread . >> > >> > (Note however that there is a memory leak in TRY-EXCEPT in the current >> > head so don't try to update everything to the CVS head. The previous >> > release should be OK.) >> > >> > Also has anyone else noticed that debugging information has recently >> > broken? m3gdb is very unhappy on all platforms for me... >> > >> > Mika > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Feb 10 21:32:41 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 10 Feb 2011 15:32:41 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> Message-ID: <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: > 1. I never understood why this any better. > 2. It how things are currently. > 3. Why not m3cg.alloca? (but see #2) > > - Jay/iPad > > On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: > >> What happened to fixing this with runtime hooks? >> >> >> On Feb 9, 2011, at 11:40 PM, Jay K wrote: >> >>> Of course not. >>> It is a made-up function that the frontend generates a call to. >>> That it never previously did. >>> Using the general function call interface. >>> You need to add a special case, in the general function call code, to do something different and specific >>> for this specific function. >>> >>> >>> - Jay >>> >>> From: rcolebur at SCIRES.COM >>> To: jay.krell at cornell.edu; m3devel at elegosoft.com >>> Date: Wed, 9 Feb 2011 22:28:45 -050 >>> Subject: Re: [M3devel] threading on Windows? >>> >>> Jay: >>> >>> I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. >>> >>> Regards, >>> Randy >>> >>> From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K >>> Sent: Wednesday, February 09, 2011 9:01 PM >>> To: Coleburn, Randy; m3devel >>> Subject: RE: [M3devel] threading on Windows? >>> >>> In m3back/src/M3x86.m3. >>> >>> - Jay >>> >>> >>> From: rcolebur at SCIRES.COM >>> To: m3devel at elegosoft.com >>> Date: Wed, 9 Feb 2011 18:22:56 -0500 >>> Subject: Re: [M3devel] threading on Windows? >>> >>> I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. >>> All I know is that everything was working fine until I checked out the HEAD repository. >>> When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? >>> Are we talking Modula-3 code, C code, Assembler, what? What package/module? >>> Regards, >>> Randy >>> >>> From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K >>> Sent: Wednesday, February 09, 2011 5:59 PM >>> To: Coleburn, Randy; Mika Nystrom; m3devel >>> Subject: RE: [M3devel] threading on Windows? >>> >>> Maybe someone else can do it? >>> The fix is: special case calls to the function "m3_alloca". >>> Change it to call "_chkstk" (or maybe "chkstk", whatever works). >>> The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. >>> To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. >>> >>> - Jay >>> >>> > From: rcolebur at SCIRES.COM >>> > To: mika at async.caltech.edu; m3devel at elegosoft.com >>> > Date: Wed, 9 Feb 2011 17:25:20 -0500 >>> > Subject: Re: [M3devel] threading on Windows? >>> > >>> > Mika: >>> > >>> > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. >>> > >>> > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. >>> > >>> > As soon as I can get the build problem resolved, I'll try out your test program. >>> > >>> > Regards, >>> > Randy Coleburn >>> > >>> > -----Original Message----- >>> > From: Mika Nystrom [mailto:mika at async.caltech.edu] >>> > Sent: Wednesday, February 09, 2011 5:01 PM >>> > To: m3devel at elegosoft.com >>> > Subject: [M3devel] threading on Windows? >>> > >>> > Hi m3devel, >>> > >>> > I'm just curious if anyone out there who's running CM3 on Windows has had >>> > a chance to try my thread testing program yet. Current status on Unix >>> > (Linux, FreeBSD) appears to be that there are problems in pthreads (but >>> > I think only under heavy load) and that user threading works perfectly. >>> > So I now wonder how things are on the third threading platform (Windows). >>> > The thread testing program is at m3-libs/m3core/tests/thread . >>> > >>> > (Note however that there is a memory leak in TRY-EXCEPT in the current >>> > head so don't try to update everything to the CVS head. The previous >>> > release should be OK.) >>> > >>> > Also has anyone else noticed that debugging information has recently >>> > broken? m3gdb is very unhappy on all platforms for me... >>> > >>> > Mika >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Thu Feb 10 21:59:25 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 10 Feb 2011 15:59:25 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> Message-ID: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am "in the dark", both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I'm seeing several different symbols that don't resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking > wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Feb 10 22:25:51 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 10 Feb 2011 16:25:51 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> Message-ID: <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu> Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: > Jay / Tony: > > I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. > > I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. > > I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. > > BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. > > Regards, > Randy > > From: Tony Hosking [mailto:hosking at cs.purdue.edu] > Sent: Thursday, February 10, 2011 3:33 PM > To: jay.krell at cornell.edu > Cc: Coleburn, Randy; m3devel > Subject: Re: [M3devel] threading on Windows? > > Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. > > On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: > > > 1. I never understood why this any better. > 2. It how things are currently. > 3. Why not m3cg.alloca? (but see #2) > > - Jay/iPad > > On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: > > What happened to fixing this with runtime hooks? > > > On Feb 9, 2011, at 11:40 PM, Jay K wrote: > > > Of course not. > It is a made-up function that the frontend generates a call to. > That it never previously did. > Using the general function call interface. > You need to add a special case, in the general function call code, to do something different and specific > for this specific function. > > > - Jay > > From: rcolebur at SCIRES.COM > To: jay.krell at cornell.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 22:28:45 -050 > Subject: Re: [M3devel] threading on Windows? > > Jay: > > I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. > > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 9:01 PM > To: Coleburn, Randy; m3devel > Subject: RE: [M3devel] threading on Windows? > > In m3back/src/M3x86.m3. > > - Jay > > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 18:22:56 -0500 > Subject: Re: [M3devel] threading on Windows? > > I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. > All I know is that everything was working fine until I checked out the HEAD repository. > When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? > Are we talking Modula-3 code, C code, Assembler, what? What package/module? > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 5:59 PM > To: Coleburn, Randy; Mika Nystrom; m3devel > Subject: RE: [M3devel] threading on Windows? > > Maybe someone else can do it? > The fix is: special case calls to the function "m3_alloca". > Change it to call "_chkstk" (or maybe "chkstk", whatever works). > The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. > To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. > > - Jay > > > From: rcolebur at SCIRES.COM > > To: mika at async.caltech.edu; m3devel at elegosoft.com > > Date: Wed, 9 Feb 2011 17:25:20 -0500 > > Subject: Re: [M3devel] threading on Windows? > > > > Mika: > > > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > > > As soon as I can get the build problem resolved, I'll try out your test program. > > > > Regards, > > Randy Coleburn > > > > -----Original Message----- > > From: Mika Nystrom [mailto:mika at async.caltech.edu] > > Sent: Wednesday, February 09, 2011 5:01 PM > > To: m3devel at elegosoft.com > > Subject: [M3devel] threading on Windows? > > > > Hi m3devel, > > > > I'm just curious if anyone out there who's running CM3 on Windows has had > > a chance to try my thread testing program yet. Current status on Unix > > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > > I think only under heavy load) and that user threading works perfectly. > > So I now wonder how things are on the third threading platform (Windows). > > The thread testing program is at m3-libs/m3core/tests/thread . > > > > (Note however that there is a memory leak in TRY-EXCEPT in the current > > head so don't try to update everything to the CVS head. The previous > > release should be OK.) > > > > Also has anyone else noticed that debugging information has recently > > broken? m3gdb is very unhappy on all platforms for me... > > > > Mika > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 10 23:11:55 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 10 Feb 2011 22:11:55 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> Message-ID: Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 11 00:13:57 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Thu, 10 Feb 2011 23:13:57 +0000 (GMT) Subject: [M3devel] threading on Windows? In-Reply-To: <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> Message-ID: <295326.94632.qm@web29715.mail.ird.yahoo.com> Hi all: by the way this must be a top one if you want cross platform compatibility rather than built-in platform specific fixes, why just we can make another thread runtime and put everything on it, e.g make this threading behavior consistent with the rest of pthread unsupported platforms (say DECthreads, or any other thread safe interface, was this ever supported i.e for SPIN trends threads or as now Pthreads are as thread safe?), at lower layer level, i.e like it was with the user threads, perhaps it is like the counter example you would need to test more deeply in the implementation not other ones' implementations, perhaps? Is there any test invalid currently, or any of it correct, if there isn't a "safe" pattern in the code why do you test unsafe threaded code (i.e all behave odly bad or all behave the same bad? This is to make another system thread safe implementation, like it was SPIN, sadly uniprocessor platform, this is better to test in an uniprocessor rather than multiprocessor indeed as its a simulation of the real code is running on it), still very impressive bench marks time in the user level or kernel level for DEC system SPIN , is like the same? for both user level ones and besides it behaves like it was a Mach 3.0 kernel user or a DECThreads client, as it had it underneath it layered, this proves a better runtime makes even better for correctness and efficiency, at least is what I see in the benchmarks, who knows with SMP systems, but just a point on it. Besides this SPIN trends threads were more abstract and more specialized e.g allow to select scheduling policy (currently at compile time in pthreads) and timing selected values. I wonder how anything of this would be manageable on a SMP machine, still worth the value to check as it is really faster than anything on the market for those days, see: http://qstream.org/~krasic/cs508-2006/summaries/paper17/SpinOS.ppt http://www.cs.cornell.edu/People/egs/papers/spin-strands.pdf Yet another hint for your thread testing, there is a static checker for threading, it used thread states in object oriented programs written in path expression constraints and proved to check threads synchronization consistency, one would be able to compile it to thread primitives to check for semantic analysis and optimization, just in case the references see below at the end of this paragraph, I must say this looks very interesting so your work would be to create annotations not trace them, or trace them if needed for create them, but still very promising if one is capable of creating the abstract annotations to some extent ahead of time or later if in execution time, as it was in some intelligent debuggers, at some extent should be the call graphs make the path expressions, perhaps abstract as through a Trestle Locking level abstraction and a thread safe condition? http://www.informatik.fh-luebeck.de/Seehusen/Publications/OS92/osna.ps http://130.203.133.150/viewdoc/versions;jsessionid=6B086982E4C53F40A01EDA48A003EE3A?doi=10.1.1.34.1092 (let it http://www.cs.rug.nl/~wim/pub/whh217.ps.gz ) and http://www.cs.rug.nl/~wim/pub/whh233.ps.gz and so http://books.google.com/books?id=Hb_pOpI8mJ8C&lpg=PP1&dq=Concurrent%20programming&pg=PA221#v=onepage&q&f=false http://www.cs.brown.edu/people/spr/fieldbook/fieldbook.html Thanks in advance --- El jue, 10/2/11, jay.krell at cornell.edu escribi?: De: jay.krell at cornell.edu Asunto: Re: [M3devel] threading on Windows? Para: "Tony Hosking" CC: "m3devel" , "Jay K" Fecha: jueves, 10 de febrero, 2011 06:01 1. I never understood why this any better. 2. It how things are currently.3. Why not m3cg.alloca? (but see #2) ?- Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. ? ? ?- Jay? ? From:?rcolebur at SCIRES.COM To:?jay.krell at cornell.edu;?m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay:?I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file.?Regards,Randy?From:?jayk123 at hotmail.com?[mailto:jayk123 at hotmail.com]?On Behalf Of?Jay K Sent:?Wednesday, February 09, 2011 9:01 PM To:?Coleburn, Randy; m3devel Subject:?RE: [M3devel] threading on Windows??In m3back/src/M3x86.m3. ? ?- Jay ?From:?rcolebur at SCIRES.COM To:?m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows?I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve.All I know is that everything was working fine until I checked out the HEAD repository.When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case??Are we talking Modula-3 code, C code, Assembler, what?? What package/module?Regards,Randy?From:?jayk123 at hotmail.com?[mailto:jayk123 at hotmail.com]?On Behalf Of?Jay K Sent:?Wednesday, February 09, 2011 5:59 PM To:?Coleburn, Randy; Mika Nystrom; m3devel Subject:?RE: [M3devel] threading on Windows??Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed?in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. ? ?- Jay ? > From:?rcolebur at SCIRES.COM > To:?mika at async.caltech.edu;?m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? >? > Mika: >? > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. >? > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. >? > As soon as I can get the build problem resolved, I'll try out your test program. >? > Regards, > Randy Coleburn >? > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu]? > Sent: Wednesday, February 09, 2011 5:01 PM > To:?m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? >? > Hi m3devel, >? > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . >? > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) >? > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... >? > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 11 00:40:48 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Thu, 10 Feb 2011 23:40:48 +0000 (GMT) Subject: [M3devel] threading on Windows? In-Reply-To: Message-ID: <161929.78691.qm@web29701.mail.ird.yahoo.com> Hi all: is useful to say here that the compiler linker approach at some extent was done in the same way as was in Modula-3, now Open Watcom, used same runtime structure and required as far as I know no better enlightment in the original versions of it, I read it some quite time ago in its sources, hard to tell you where it was, but if OpenWatcom sold it as a Windows solution in its runtime why not in the Modula-3 one, indeed how it would be to port its linker to work for Modula-3 and tune it for the? required? works of it. Is better to adapt to a better portable (Linux, etc) solution than? to adapt to a? non-portable still solution as I believe. Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] threading on Windows? Para: "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 17:11 Why is it better? ?Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) ?- Jay/iPad ? You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. ? ? Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. ? ? As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. ? ? Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. ? ? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Feb 11 01:36:12 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 11 Feb 2011 00:36:12 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <161929.78691.qm@web29701.mail.ird.yahoo.com> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> Message-ID: Daniel, you generally make very little sense to me. Please consider additional English training. "Modula-3" defines and implements its own threading library, it's own I/O library, it's own traced and untraced memory allocator (traced == garbage collected), it's own GUI library (Trestle). It is its own portabilty layer, akin to various other options, e.g. using pthreads and then implementing pthreads on Windows, using Tk or Qt, etc., and depending on them to be ported. etc. Yes, I am deliberately confusing "language" and "library". Which are people interested in? One, the other, or both? Now, granted, we are portable between essentially only two platforms Win32/NT and "Posix", where "Posix" is defined as including pthreads and X Windows, and even in a sense gcc. (Even if Modula-3 predates Posix, the dependencies are pretty light, such that they have mostly always been almost consistent across the board -- open/read/write/close, socket/accept/connect/recv/send/listen, a few variations for signals). (Personally I would like to change the backend to generate C...) One could "get out of this (library) business" and target another intermediate portability layer, i.e. write "bindings" for Tk, Qt, etc., and then use Modula-3 "just" as a language, and not an associated large set of libraries. In general, it is desired for portability layers to be thin in terms of performance and possibly state. Ideally, given the choice of writing to Win32 threads and getting the smallest fastest program that runs only on Wondows, or pthreads and getting the smallest fastest program that runs on Posix, nobody should shy away from using Modula-3 threads, since they run pretty much just as well as Win32 and pthreads. Pretty much. Actually condition variables can't be done super well on Win32 prior to Vista, but they are done in a pretty ok way now. Still, threads and mutexes at least are very thin layers. Implementing threading more completely yourself, on many systems, precludes you from actually running on more than one processor. Win7's "user mode scheduling" offers an option with more control and still the ability to run on multiple processors. But I suspect we aren't very interested, at least not currently. - Jay Date: Thu, 10 Feb 2011 23:40:48 +0000 From: dabenavidesd at yahoo.es To: hosking at cs.purdue.edu; jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Hi all: is useful to say here that the compiler linker approach at some extent was done in the same way as was in Modula-3, now Open Watcom, used same runtime structure and required as far as I know no better enlightment in the original versions of it, I read it some quite time ago in its sources, hard to tell you where it was, but if OpenWatcom sold it as a Windows solution in its runtime why not in the Modula-3 one, indeed how it would be to port its linker to work for Modula-3 and tune it for the required works of it. Is better to adapt to a better portable (Linux, etc) solution than to adapt to a non-portable still solution as I believe. Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] threading on Windows? Para: "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 17:11 Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 11 02:08:31 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Fri, 11 Feb 2011 01:08:31 +0000 (GMT) Subject: [M3devel] threading on Windows? In-Reply-To: Message-ID: <355223.44215.qm@web29702.mail.ird.yahoo.com> Hi all: thanks for your suggestion yet I learn also from you and others so I try to keep improving it, I may read some English grammar too, this is the best I can do for now, sorry for that and thanks for being open about it (other suggestions are most welcome). Please think about this, I believe this is transforming very quickly and we don't know pretty much yet where it would be in some not short time, see my point, 1.2 billions of smart phones are being ship, 3 million more than those of PCs, if you see this, this is a turn break point, you need to be able to see also this market becomes more important than some may think. Still about the PCs, the current trend is too much research being done at the GPGPU thing, with yet another revolution of this days in computation, this is my point precisely, will be able to tackle the opportunity of the change, or be overwhelmed by it, and just wait for another time. I just say in behalf of the things I have said that could be more common than we thought, yet there are good opportunities, who thought this was another revolution to come, I did not believe it but I'm more turned to think this is another "trend" in computing. And I think you would not believe but in a country like mine they thought to construct hardware assisted graphics chips, it didn't turn out to be real, but many things came after this, we have self configurable hardware, what else you one want? Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: RE: [M3devel] threading on Windows? Para: dabenavidesd at yahoo.es, "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 19:36 Daniel, you generally make very little sense to me. Please consider?additional?English training. ? ? "Modula-3" defines?and implements its?own threading library,?it's own I/O library, it's own traced and untraced memory allocator (traced == garbage collected), it's own GUI library (Trestle). ? It is its own portabilty layer, akin to various other options, e.g. using pthreads and?then implementing pthreads on Windows, using Tk or Qt, etc., and depending on them to be ported. etc. Yes, I am deliberately confusing "language" and "library". Which are people interested in? One, the other, or both? ? ? Now, granted, we are portable between essentially only two platforms Win32/NT and "Posix", where "Posix" is defined as including pthreads and X Windows, and even in a sense gcc. ? (Even if Modula-3 predates Posix, the dependencies are pretty light, such that they have mostly always been almost consistent across the board -- open/read/write/close, socket/accept/connect/recv/send/listen, a few variations for signals). ? (Personally I would like to change the backend to generate C...) ? ? One could "get out of this (library) business" and target another intermediate portability layer, i.e. write "bindings" for Tk, Qt, etc., and then use Modula-3 "just" as a language, and not an?associated large set of libraries. ? ? In general, it is desired for portability layers to be thin in terms of performance and possibly state. Ideally, given the choice of writing to Win32 threads and getting the smallest fastest program that runs only on Wondows, or pthreads and getting the smallest fastest program that runs on Posix, nobody should shy away from using Modula-3 threads, since they run pretty much just as well as Win32 and pthreads. Pretty much. Actually condition variables can't be done super well on Win32 prior to Vista, but they are done in a pretty ok way now. Still, threads and mutexes at least are very thin layers. ? ? Implementing threading more completely yourself, on many systems, precludes you from actually running on more than one processor. Win7's "user mode scheduling" offers an option with more control and still the ability to run on multiple processors. But I suspect we aren't very interested, at least not currently. ? ? ? ?- Jay ? Date: Thu, 10 Feb 2011 23:40:48 +0000 From: dabenavidesd at yahoo.es To: hosking at cs.purdue.edu; jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Hi all: is useful to say here that the compiler linker approach at some extent was done in the same way as was in Modula-3, now Open Watcom, used same runtime structure and required as far as I know no better enlightment in the original versions of it, I read it some quite time ago in its sources, hard to tell you where it was, but if OpenWatcom sold it as a Windows solution in its runtime why not in the Modula-3 one, indeed how it would be to port its linker to work for Modula-3 and tune it for the? required? works of it. Is better to adapt to a better portable (Linux, etc) solution than? to adapt to a? non-portable still solution as I believe. Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] threading on Windows? Para: "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 17:11 #yiv1956618167 .yiv1956618167ExternalClass #yiv1956618167ecxyiv295862095 .yiv1956618167ecxyiv295862095hmmessage P {padding:0px;} #yiv1956618167 .yiv1956618167ExternalClass #yiv1956618167ecxyiv295862095 .yiv1956618167ecxyiv295862095hmmessage {font-size:10pt;font-family:Tahoma;} Why is it better? ?Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) ?- Jay/iPad ? You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. ? ? Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. ? ? As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. ? ? Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. ? ? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 11 03:51:10 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Fri, 11 Feb 2011 02:51:10 +0000 (GMT) Subject: [M3devel] threading on Windows? In-Reply-To: Message-ID: <332041.74062.qm@web29718.mail.ird.yahoo.com> Hi all: Jay let me make a statement, perhaps if some can follow me to agree in this, a claim from a? Professor here, very pro FOSS anyway: "The cost of hardware are always descending and growing up in technology capabilities, this is not the case in software terms, there is little improvement in the techniques and yet they are more excessive expensive each time it has a new version". I would agree if one looks to some companies, like in M$ fields, this is the case certainly, as Win ships in every machine human has seen almost, ok but in other responsible houses this was not the case, Digital Equipment Corporation if I may say so, they had very good machines, Professor told you could specify very precisely what do you would want of it and they shipped and were very endurable. Of software though they didn't make a revolution they did the best achievements in the software I have seen so far, runtime capabilities of this compiler I have see so far anything like this and specially static analysis tools, very impressive, even some people would tell and have told certainly already and certainly they are greater than I am. Thanks in advance Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: RE: [M3devel] threading on Windows? Para: dabenavidesd at yahoo.es, "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 19:36 Daniel, you generally make very little sense to me. Please consider?additional?English training. ? ? "Modula-3" defines?and implements its?own threading library,?it's own I/O library, it's own traced and untraced memory allocator (traced == garbage collected), it's own GUI library (Trestle). ? It is its own portabilty layer, akin to various other options, e.g. using pthreads and?then implementing pthreads on Windows, using Tk or Qt, etc., and depending on them to be ported. etc. Yes, I am deliberately confusing "language" and "library". Which are people interested in? One, the other, or both? ? ? Now, granted, we are portable between essentially only two platforms Win32/NT and "Posix", where "Posix" is defined as including pthreads and X Windows, and even in a sense gcc. ? (Even if Modula-3 predates Posix, the dependencies are pretty light, such that they have mostly always been almost consistent across the board -- open/read/write/close, socket/accept/connect/recv/send/listen, a few variations for signals). ? (Personally I would like to change the backend to generate C...) ? ? One could "get out of this (library) business" and target another intermediate portability layer, i.e. write "bindings" for Tk, Qt, etc., and then use Modula-3 "just" as a language, and not an?associated large set of libraries. ? ? In general, it is desired for portability layers to be thin in terms of performance and possibly state. Ideally, given the choice of writing to Win32 threads and getting the smallest fastest program that runs only on Wondows, or pthreads and getting the smallest fastest program that runs on Posix, nobody should shy away from using Modula-3 threads, since they run pretty much just as well as Win32 and pthreads. Pretty much. Actually condition variables can't be done super well on Win32 prior to Vista, but they are done in a pretty ok way now. Still, threads and mutexes at least are very thin layers. ? ? Implementing threading more completely yourself, on many systems, precludes you from actually running on more than one processor. Win7's "user mode scheduling" offers an option with more control and still the ability to run on multiple processors. But I suspect we aren't very interested, at least not currently. ? ? ? ?- Jay ? Date: Thu, 10 Feb 2011 23:40:48 +0000 From: dabenavidesd at yahoo.es To: hosking at cs.purdue.edu; jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Hi all: is useful to say here that the compiler linker approach at some extent was done in the same way as was in Modula-3, now Open Watcom, used same runtime structure and required as far as I know no better enlightment in the original versions of it, I read it some quite time ago in its sources, hard to tell you where it was, but if OpenWatcom sold it as a Windows solution in its runtime why not in the Modula-3 one, indeed how it would be to port its linker to work for Modula-3 and tune it for the? required? works of it. Is better to adapt to a better portable (Linux, etc) solution than? to adapt to a? non-portable still solution as I believe. Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] threading on Windows? Para: "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 17:11 #yiv863291047 .yiv863291047ExternalClass #yiv863291047ecxyiv295862095 .yiv863291047ecxyiv295862095hmmessage P {padding:0px;} #yiv863291047 .yiv863291047ExternalClass #yiv863291047ecxyiv295862095 .yiv863291047ecxyiv295862095hmmessage {font-size:10pt;font-family:Tahoma;} Why is it better? ?Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) ?- Jay/iPad ? You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. ? ? Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. ? ? As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. ? ? Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. ? ? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Feb 11 22:26:52 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Fri, 11 Feb 2011 16:26:52 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu> Message-ID: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new "heap-based jmpbuf allocation" have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate "alloca" routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I "do" anything, I'll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the "heap based jumpbuf allocation scheme" was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am "in the dark", both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I'm seeing several different symbols that don't resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking > wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Feb 11 22:29:43 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 11 Feb 2011 16:29:43 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu> Message-ID: <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: > Tony: > > Thank you very much for this explanation. > > So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: > 1. Space leak problem. > 2. Binding to appropriate ?alloca? routine on Windows platform. > > Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. > > I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. > > Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. > > Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. > > Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. > > If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. > > Regards, > Randy Coleburn > > From: Tony Hosking [mailto:hosking at cs.purdue.edu] > Sent: Thursday, February 10, 2011 4:26 PM > To: Coleburn, Randy > Cc: m3devel > Subject: Re: [M3devel] threading on Windows? > > Hi Randy, > > A far as I know, the changes are as follows. > > TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. > > The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. > > To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. > > -- Tony > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: > > > Jay / Tony: > > I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. > > I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. > > I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. > > BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. > > Regards, > Randy > > From: Tony Hosking [mailto:hosking at cs.purdue.edu] > Sent: Thursday, February 10, 2011 3:33 PM > To: jay.krell at cornell.edu > Cc: Coleburn, Randy; m3devel > Subject: Re: [M3devel] threading on Windows? > > Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. > > On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: > > > > 1. I never understood why this any better. > 2. It how things are currently. > 3. Why not m3cg.alloca? (but see #2) > > - Jay/iPad > > On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: > > What happened to fixing this with runtime hooks? > > > On Feb 9, 2011, at 11:40 PM, Jay K wrote: > > > > Of course not. > It is a made-up function that the frontend generates a call to. > That it never previously did. > Using the general function call interface. > You need to add a special case, in the general function call code, to do something different and specific > for this specific function. > > > - Jay > > From: rcolebur at SCIRES.COM > To: jay.krell at cornell.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 22:28:45 -050 > Subject: Re: [M3devel] threading on Windows? > > Jay: > > I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. > > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 9:01 PM > To: Coleburn, Randy; m3devel > Subject: RE: [M3devel] threading on Windows? > > In m3back/src/M3x86.m3. > > - Jay > > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 18:22:56 -0500 > Subject: Re: [M3devel] threading on Windows? > > I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. > All I know is that everything was working fine until I checked out the HEAD repository. > When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? > Are we talking Modula-3 code, C code, Assembler, what? What package/module? > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 5:59 PM > To: Coleburn, Randy; Mika Nystrom; m3devel > Subject: RE: [M3devel] threading on Windows? > > Maybe someone else can do it? > The fix is: special case calls to the function "m3_alloca". > Change it to call "_chkstk" (or maybe "chkstk", whatever works). > The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. > To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. > > - Jay > > > From: rcolebur at SCIRES.COM > > To: mika at async.caltech.edu; m3devel at elegosoft.com > > Date: Wed, 9 Feb 2011 17:25:20 -0500 > > Subject: Re: [M3devel] threading on Windows? > > > > Mika: > > > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > > > As soon as I can get the build problem resolved, I'll try out your test program. > > > > Regards, > > Randy Coleburn > > > > -----Original Message----- > > From: Mika Nystrom [mailto:mika at async.caltech.edu] > > Sent: Wednesday, February 09, 2011 5:01 PM > > To: m3devel at elegosoft.com > > Subject: [M3devel] threading on Windows? > > > > Hi m3devel, > > > > I'm just curious if anyone out there who's running CM3 on Windows has had > > a chance to try my thread testing program yet. Current status on Unix > > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > > I think only under heavy load) and that user threading works perfectly. > > So I now wonder how things are on the third threading platform (Windows). > > The thread testing program is at m3-libs/m3core/tests/thread . > > > > (Note however that there is a memory leak in TRY-EXCEPT in the current > > head so don't try to update everything to the CVS head. The previous > > release should be OK.) > > > > Also has anyone else noticed that debugging information has recently > > broken? m3gdb is very unhappy on all platforms for me... > > > > Mika > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Feb 11 22:51:37 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 11 Feb 2011 21:51:37 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> Message-ID: It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - Jay From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote:Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely:1. Space leak problem.2. Binding to appropriate ?alloca? routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards,Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- TonyAntony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. Regards,Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler.On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently.3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote:What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows?Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows?I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve.All I know is that everything was working fine until I checked out the HEAD repository.When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case??Are we talking Modula-3 code, C code, Assembler, what? What package/module?Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 00:09:17 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 11 Feb 2011 23:09:17 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> Message-ID: > me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows It is totally compatible. Someone just has to write a few lines of code to implement it. - Jay From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate ?alloca? routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Sat Feb 12 00:17:15 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Fri, 11 Feb 2011 18:17:15 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> Message-ID: Jay: Ok, I don't mean to start a new issue or upset anyone. I see you checked in some new changes. I updated to latest HEAD and tried to build, but again I get unresolved symbols beginning with m3-libs\sysutils (see attached sysutils.lst) So, not sure what you mean by "it should be ok now". Are you saying you do, or do not, want me to work further on the solving the "problem," regardless of how we describe it ? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, February 11, 2011 4:52 PM To: Tony; Coleburn, Randy Cc: m3devel Subject: RE: [M3devel] threading on Windows? It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - Jay ________________________________ From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new "heap-based jmpbuf allocation" have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate "alloca" routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I "do" anything, I'll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the "heap based jumpbuf allocation scheme" was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am "in the dark", both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I'm seeing several different symbols that don't resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking > wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sysutils.lst Type: application/octet-stream Size: 4059 bytes Desc: sysutils.lst URL: From mika at async.caltech.edu Sat Feb 12 02:57:15 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 17:57:15 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> Message-ID: <20110212015715.A43471A2078@async.async.caltech.edu> Hello m3devel, I'm attempting to test the CVS head as of Jay's latest checkins with the thread tester. I'm afraid there seems to be something wrong. I get a seg fault when running a few threads that do the following: PROCEDURE TApply(cl : Closure) : REFANY = BEGIN Thread.Pause(InitPause); LOOP TRY WITH now = Time.Now() DO times1[cl.id]:= FLOOR(now); IF now < 0.0d0 THEN RAISE X END END EXCEPT X => <*ASSERT FALSE*> END END END TApply; The crash is as follows: (gdb) run -tests tryexcept Starting program: /home/mika/cm3-cvs/cm3/m3-libs/m3core/tests/thread/LINUXLIBC6/threadtest -tests tryexcept [Thread debugging using libthread_db enabled] Writing file...done Creating tryexcept threads...done running...printing oldest/median age/newest . Program received signal SIGSEGV, Segmentation fault. 0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 53 { (gdb) where #0 0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 #1 0x0808cb57 in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 #2 0x0804b875 in Main__TApply (cl=) at ../src/Main.m3:205 #3 0x08086c5e in ThreadPosix__RunThread () at ../src/thread/POSIX/ThreadPosix.m3:993 #4 0x400c343b in makecontext () from /lib/libc.so.6 #5 0x00000000 in ?? () (gdb) Note I am using user threads since pthreads are known to be buggy at the moment. I would like to find one configuration (any configuration!!) that lets me run a multithreaded program under AMD64_LINUX. I don't care what version of the compiler it is, whether the threads are user or pthreads, how many things I have to hack. Does anyone know what I need to do? Mika From mika at async.caltech.edu Sat Feb 12 03:08:36 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 18:08:36 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: <20110212015715.A43471A2078@async.async.caltech.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu> Message-ID: <20110212020836.97F7A1A207A@async.async.caltech.edu> I forgot to add that I was running that program on the platform LINUXLIBC6. On FreeBSD4 I get the following bizarre behavior (program hangs): (gdb) run -tests tryexcept Starting program: /big/home/mika/cm3-writable/cm3/m3-libs/m3core/tests/thread/FreeBSD4/threadtest -tests tryexcept Writing file...done Creating tryexcept threads...done running...printing oldest/median age/newest .^C Program received signal SIGINT, Interrupt. [Switching to Thread 1 (LWP 100315)] 0x681093f3 in pthread_testcancel () from /usr/lib/libpthread.so.1 (gdb) where #0 0x681093f3 in pthread_testcancel () from /usr/lib/libpthread.so.1 #1 0x680f917c in sigaction () from /usr/lib/libpthread.so.1 #2 0x680f9d03 in sigaction () from /usr/lib/libpthread.so.1 #3 0x680f850b in sigaction () from /usr/lib/libpthread.so.1 #4 0x680f828c in sigaction () from /usr/lib/libpthread.so.1 #5 0x6810192e in pthread_mutexattr_init () from /usr/lib/libpthread.so.1 #6 0x00000000 in ?? () (gdb) Even though I am using ThreadPosix, not ThreadPthread... Also on AMD64_LINUX, the following: (gdb) run -tests tryexcept Starting program: /home/mika/cm3/cm3/m3-libs/m3core/tests/thread/AMD64_LINUX/threadtest -tests tryexcept [Thread debugging using libthread_db enabled] Writing file...done Creating tryexcept threads...done running...printing oldest/median age/newest .[New Thread 0x2b7877844620 (LWP 24086)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x2b7877844620 (LWP 24086)] 0x00007fff44dff5b1 in ?? () (gdb) where #0 0x00007fff44dff5b1 in ?? () #1 0x00007fff44dff6a5 in gettimeofday () #2 0x00002b787589690a in gettimeofday () from /lib/libc.so.6 #3 0x000000000045148f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:58 #4 0x000000000045139e in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 #5 0x0000000000404da7 in Main__TApply (cl=Cannot access memory at address 0x2b7974014e68 ) at ../src/Main.m3:205 #6 0x000000000044b3fb in ThreadPosix__RunThread () at ../src/thread/POSIX/ThreadPosix.m3:993 #7 0x00002b787584b7b0 in ?? () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () (gdb) AMD64_FREEBSD seems to work, oddly enough, but slowly. I wonder if my compiler buidling has failed somehow (I am using the Python script following a recipe provided by Jay after modifying m3core/src/thread/m3makefile ) Mika Mika Nystrom writes: > >Hello m3devel, > >I'm attempting to test the CVS head as of Jay's latest checkins with >the thread tester. > >I'm afraid there seems to be something wrong. > >I get a seg fault when running a few threads that do the following: > >PROCEDURE TApply(cl : Closure) : REFANY = > BEGIN > Thread.Pause(InitPause); > LOOP > TRY > WITH now = Time.Now() DO > times1[cl.id]:= FLOOR(now); > IF now < 0.0d0 THEN RAISE X END > END > EXCEPT X => <*ASSERT FALSE*> > END > END > END TApply; > >The crash is as follows: > >(gdb) run -tests tryexcept >Starting program: /home/mika/cm3-cvs/cm3/m3-libs/m3core/tests/thread/LINUXLIBC >6/threadtest -tests tryexcept >[Thread debugging using libthread_db enabled] >Writing file...done >Creating tryexcept threads...done >running...printing oldest/median age/newest >. >Program received signal SIGSEGV, Segmentation fault. >0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 >53 { >(gdb) where >#0 0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 >#1 0x0808cb57 in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 >#2 0x0804b875 in Main__TApply (cl=) > at ../src/Main.m3:205 >#3 0x08086c5e in ThreadPosix__RunThread () > at ../src/thread/POSIX/ThreadPosix.m3:993 >#4 0x400c343b in makecontext () from /lib/libc.so.6 >#5 0x00000000 in ?? () >(gdb) > >Note I am using user threads since pthreads are known to be buggy at the >moment. > >I would like to find one configuration (any configuration!!) that lets >me run a multithreaded program under AMD64_LINUX. I don't care what >version of the compiler it is, whether the threads are user or pthreads, >how many things I have to hack. Does anyone know what I need to do? > > Mika From mika at async.caltech.edu Sat Feb 12 03:50:13 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 18:50:13 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: <20110212020836.97F7A1A207A@async.async.caltech.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu> <20110212020836.97F7A1A207A@async.async.caltech.edu> Message-ID: <20110212025013.916EA1A2078@async.async.caltech.edu> Another odd datapoint is that even rather aggressive testing seems not to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user threads, of course. So status right now of user threads seems to be: FreeBSD4 : strange failure in libpthread as reported by gdb LINUXLIBC6 : seg fault AMD64_LINUX : seg fault AMD64_FREEBSD : OK I386_DARWIN : OK I am never, ever 100% certain of my bootstrapping procedure so it's possible I've done something wrong... however I'm using upgrade.py and when I look for Thread\*.*o files I only see the ThreadPosix file and not ThreadPThread. I'll try bootstrapping again :-) Mika Mika Nystrom writes: > >I forgot to add that I was running that program on the platform LINUXLIBC6. > >On FreeBSD4 I get the following bizarre behavior (program hangs): > >(gdb) run -tests tryexcept >Starting program: /big/home/mika/cm3-writable/cm3/m3-libs/m3core/tests/thread/ >FreeBSD4/threadtest -tests tryexcept >Writing file...done >Creating tryexcept threads...done >running...printing oldest/median age/newest >.^C >Program received signal SIGINT, Interrupt. >[Switching to Thread 1 (LWP 100315)] >0x681093f3 in pthread_testcancel () from /usr/lib/libpthread.so.1 > >(gdb) where >#0 0x681093f3 in pthread_testcancel () from /usr/lib/libpthread.so.1 >#1 0x680f917c in sigaction () from /usr/lib/libpthread.so.1 >#2 0x680f9d03 in sigaction () from /usr/lib/libpthread.so.1 >#3 0x680f850b in sigaction () from /usr/lib/libpthread.so.1 >#4 0x680f828c in sigaction () from /usr/lib/libpthread.so.1 >#5 0x6810192e in pthread_mutexattr_init () from /usr/lib/libpthread.so.1 >#6 0x00000000 in ?? () >(gdb) > >Even though I am using ThreadPosix, not ThreadPthread... > >Also on AMD64_LINUX, the following: > >(gdb) run -tests tryexcept >Starting program: /home/mika/cm3/cm3/m3-libs/m3core/tests/thread/AMD64_LINUX/t >hreadtest -tests tryexcept >[Thread debugging using libthread_db enabled] >Writing file...done >Creating tryexcept threads...done >running...printing oldest/median age/newest >.[New Thread 0x2b7877844620 (LWP 24086)] > >Program received signal SIGSEGV, Segmentation fault. >[Switching to Thread 0x2b7877844620 (LWP 24086)] >0x00007fff44dff5b1 in ?? () >(gdb) where >#0 0x00007fff44dff5b1 in ?? () >#1 0x00007fff44dff6a5 in gettimeofday () >#2 0x00002b787589690a in gettimeofday () from /lib/libc.so.6 >#3 0x000000000045148f in TimePosix__Now () > at ../src/time/POSIX/TimePosixC.c:58 >#4 0x000000000045139e in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 >#5 0x0000000000404da7 in Main__TApply (cl=Cannot access memory at address 0x2 >b7974014e68 >) at ../src/Main.m3:205 >#6 0x000000000044b3fb in ThreadPosix__RunThread () > at ../src/thread/POSIX/ThreadPosix.m3:993 >#7 0x00002b787584b7b0 in ?? () from /lib/libc.so.6 >#8 0x0000000000000000 in ?? () >(gdb) > >AMD64_FREEBSD seems to work, oddly enough, but slowly. > >I wonder if my compiler buidling has failed somehow (I am using the >Python script following a recipe provided by Jay after modifying >m3core/src/thread/m3makefile ) > > Mika > > >Mika Nystrom writes: >> >>Hello m3devel, >> >>I'm attempting to test the CVS head as of Jay's latest checkins with >>the thread tester. >> >>I'm afraid there seems to be something wrong. >> >>I get a seg fault when running a few threads that do the following: >> >>PROCEDURE TApply(cl : Closure) : REFANY = >> BEGIN >> Thread.Pause(InitPause); >> LOOP >> TRY >> WITH now = Time.Now() DO >> times1[cl.id]:= FLOOR(now); >> IF now < 0.0d0 THEN RAISE X END >> END >> EXCEPT X => <*ASSERT FALSE*> >> END >> END >> END TApply; >> >>The crash is as follows: >> >>(gdb) run -tests tryexcept >>Starting program: /home/mika/cm3-cvs/cm3/m3-libs/m3core/tests/thread/LINUXLIB >C >>6/threadtest -tests tryexcept >>[Thread debugging using libthread_db enabled] >>Writing file...done >>Creating tryexcept threads...done >>running...printing oldest/median age/newest >>. >>Program received signal SIGSEGV, Segmentation fault. >>0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 >>53 { >>(gdb) where >>#0 0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 >>#1 0x0808cb57 in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 >>#2 0x0804b875 in Main__TApply (cl=) >> at ../src/Main.m3:205 >>#3 0x08086c5e in ThreadPosix__RunThread () >> at ../src/thread/POSIX/ThreadPosix.m3:993 >>#4 0x400c343b in makecontext () from /lib/libc.so.6 >>#5 0x00000000 in ?? () >>(gdb) >> >>Note I am using user threads since pthreads are known to be buggy at the >>moment. >> >>I would like to find one configuration (any configuration!!) that lets >>me run a multithreaded program under AMD64_LINUX. I don't care what >>version of the compiler it is, whether the threads are user or pthreads, >>how many things I have to hack. Does anyone know what I need to do? >> >> Mika From mika at async.caltech.edu Sat Feb 12 04:05:25 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 19:05:25 -0800 Subject: [M3devel] pthread_atfork Message-ID: <20110212030525.822411A2078@async.async.caltech.edu> Tony, What's RTProcessC.c doing? I note the following cryptic comment: /* NOTE: Even userthreads now depends * on availability of pthreads. * This can be fixed if need be. */ now further down we can read: #if defined(_WIN32) \ || defined(__vms) \ || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) return 0; #else while (1) { int i = pthread_atfork(prepare, parent, child); if (i != EAGAIN) return i; sleep(0); } #endif so on FreeBSD 5 (what I'm running on my "FreeBSD4" system), RTProcess.RegisterForkHandlers does nothing? Hmmm..... Mika From mika at async.caltech.edu Sat Feb 12 07:48:12 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 22:48:12 -0800 Subject: [M3devel] threading on Windows? Message-ID: <20110212064812.286E61A2078@async.async.caltech.edu> Well, with the following changes: re-bootstrapping on the two Linux systems and removal of the -pthread flag on FreeBSD4, all is well. User threading works on all the architectures listed below, at least as far as the thread testing program can test it. (It seems that because of the way pthread_atfork is coded, pthread is not needed at all on FreeBSD4.) Mika Mika Nystrom writes: >Another odd datapoint is that even rather aggressive testing seems not >to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user >threads, of course. > >So status right now of user threads seems to be: > >FreeBSD4 : strange failure in libpthread as reported by gdb >LINUXLIBC6 : seg fault >AMD64_LINUX : seg fault >AMD64_FREEBSD : OK >I386_DARWIN : OK From jay.krell at cornell.edu Sat Feb 12 11:24:01 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 10:24:01 +0000 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212030525.822411A2078@async.async.caltech.edu> References: <20110212030525.822411A2078@async.async.caltech.edu> Message-ID: I wrote that. As I recall, FreeBSD < 6 doesn't have pthread_atfork. I think that was based on reading the man pages. So, yes. What would you suggest? All this atfork stuff came about as part of getting cvsupd to work. With pthreads/kernel threads. It historically worked, with Modula-3 user threads. Modula-3 user threads have this funny property that all threads survive work. With pthreads/kernel threads, fork gives you a new process with just one thread. (Solaris has fork1 and forkall, and fork maybe used to be forkall, but now it is fork1). cvsupd is one of those slightly unusual programs that does "fork and then do more work" as supposed to the more typical "fork and exec". Which is to say..I don't know. Maybe, don't expect pthreads to work on FreeBSD < 6? At least not in a process that ever calls fork? Or does "fork and then more work"? In reality, i think a lot of libraries have trouble with "fork and then do more work", but I don't know. I recall vague notices in the Apple/Darwin manpages not to assume this works -- the obvious implication that there is plenty of use of pthreads e.g. mutexes, but relatively little use of pthread_atfork. You could perhaps replace all uses of fork with a function that called the fork handlers, then fork, then the other handlers. That is probably what I meant in the comment about this being easy to fix. You might though then run into the problem that callers of fork can't do much before exec...oh, that's vfork I bet actually. - Jay > To: hosking at cs.purdue.edu > Date: Fri, 11 Feb 2011 19:05:25 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: [M3devel] pthread_atfork > > Tony, > > What's RTProcessC.c doing? > > I note the following cryptic comment: > > /* NOTE: Even userthreads now depends > * on availability of pthreads. > * This can be fixed if need be. > */ > > now further down we can read: > > #if defined(_WIN32) \ > || defined(__vms) \ > || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > return 0; > #else > while (1) > { > int i = pthread_atfork(prepare, parent, child); > if (i != EAGAIN) > return i; > sleep(0); > } > #endif > > so on FreeBSD 5 (what I'm running on my "FreeBSD4" system), > RTProcess.RegisterForkHandlers does nothing? > > Hmmm..... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 11:39:06 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 10:39:06 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , , , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, , <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , , , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , , , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu>, , Message-ID: TextUtils.mo : error LNK2001: unresolved external symbol __alloca TextUtils.mo : error LNK2001: unresolved external symbol _Csetjmp__Jumpbuf_size sysutils.dll : fatal error LNK1120: 2 unresolved externals Most likely you didn't build it correctly. Go back to a working cm3/m3core/libm3 and run upgrade.py. There should be no longer any references to alloca of Csetjmp__Jumpbuf_size. Really, Csetjmp__Jumpbuf_size should never have been a problem. That likely also indicates some sort of incorrect bootstrap. > Are you saying you do, or do not, want me to work further on the solving the ?problem,? regardless of how we describe it ? I still want alloca to work, either via a function call or a specific m3cg method. (I still don't see that RTHooks.Alloca makes better sense; I still don't see a way to avoid hacking the compiler, given the custom calling convention.) It will be difficult to test the fix until/unless m3front is changed back. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Fri, 11 Feb 2011 18:17:15 -0500 Subject: Re: [M3devel] threading on Windows? Jay: Ok, I don?t mean to start a new issue or upset anyone. I see you checked in some new changes. I updated to latest HEAD and tried to build, but again I get unresolved symbols beginning with m3-libs\sysutils (see attached sysutils.lst) So, not sure what you mean by ?it should be ok now?. Are you saying you do, or do not, want me to work further on the solving the ?problem,? regardless of how we describe it ? Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, February 11, 2011 4:52 PM To: Tony; Coleburn, Randy Cc: m3devel Subject: RE: [M3devel] threading on Windows? It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - JayFrom: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely:1. Space leak problem.2. Binding to appropriate ?alloca? routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards,Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- TonyAntony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. Regards,Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler.On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently.3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote:What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows?Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows?I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve.All I know is that everything was working fine until I checked out the HEAD repository.When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case??Are we talking Modula-3 code, C code, Assembler, what? What package/module?Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Sat Feb 12 13:02:11 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Sat, 12 Feb 2011 13:02:11 +0100 Subject: [M3devel] pthread_atfork In-Reply-To: References: <20110212030525.822411A2078@async.async.caltech.edu> Message-ID: <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com> I think it would be acceptable to say that pthread threading is broken on FreeBSD6 (as it is not officially supported any more). All the stuff below should be documented in the 'known problems' section. Especially the different behaviours of fork for user and pthreads. This should also be mentioned in the M3 interfaces (source code)! Also that to use use threads with FreeBSD requires to remove the -pthread flag, as Mika noted. BTW, do we have a chapter explaining what to do to use user threads instead of the default pthreads? Olaf PS: Hint: this would be a great task for anybody lurking on the lists and trying to get involved :-) Quoting Jay K : > > I wrote that. > > As I recall, FreeBSD < 6 doesn't have pthread_atfork. > > I think that was based on reading the man pages. > > So, yes. What would you suggest? > > > > > > All this atfork stuff came about as part of getting cvsupd to work. > > With pthreads/kernel threads. It historically worked, with > Modula-3 user threads. > > Modula-3 user threads have this funny property that all threads > survive work. > > With pthreads/kernel threads, fork gives you a new process with > just one thread. > > (Solaris has fork1 and forkall, and fork maybe used to be > forkall, but now it is fork1). > > > > > > cvsupd is one of those slightly unusual programs that does "fork and > then do more work" > > as supposed to the more typical "fork and exec". > > > > Which is to say..I don't know. Maybe, don't expect pthreads to work > on FreeBSD < 6? > > At least not in a process that ever calls fork? Or does "fork and > then more work"? > > > > > > In reality, i think a lot of libraries have trouble with "fork and > then do more work", but > > I don't know. I recall vague notices in the Apple/Darwin manpages > not to assume > > this works -- the obvious implication that there is plenty of use of > pthreads e.g. mutexes, > > but relatively little use of pthread_atfork. > > > > > > You could perhaps replace all uses of fork with a function that > called the fork handlers, > > then fork, then the other handlers. That is probably what I meant in > the comment > > about this being easy to fix. > > You might though then run into the problem that callers of fork can't do much > > before exec...oh, that's vfork I bet actually. > > > > > > - Jay > > > > > > >> To: hosking at cs.purdue.edu >> Date: Fri, 11 Feb 2011 19:05:25 -0800 >> From: mika at async.caltech.edu >> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >> Subject: [M3devel] pthread_atfork >> >> Tony, >> >> What's RTProcessC.c doing? >> >> I note the following cryptic comment: >> >> /* NOTE: Even userthreads now depends >> * on availability of pthreads. >> * This can be fixed if need be. >> */ >> >> now further down we can read: >> >> #if defined(_WIN32) \ >> || defined(__vms) \ >> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) >> return 0; >> #else >> while (1) >> { >> int i = pthread_atfork(prepare, parent, child); >> if (i != EAGAIN) >> return i; >> sleep(0); >> } >> #endif >> >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system), >> RTProcess.RegisterForkHandlers does nothing? >> >> Hmmm..... >> >> Mika > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Sat Feb 12 14:38:33 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 13:38:33 +0000 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com> References: <20110212030525.822411A2078@async.async.caltech.edu>, , <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com> Message-ID: > Especially the different behaviours of fork for user and pthreads The behavior has been unified. User threads no longer survive fork. Except perhaps on FreeBSD 5? (Off by one somewhere in the discussion!) It looks like it is ok on FreeBSD 6. I think we could fix it on FreeBSD 5. At least if we changed all of our fork() to be RTProcess.Fork() and called the handlers in there, at least on FreeBSD 5. I believe the difference is only noticable if you fork-and-do-work. Not if you fork-and-exec. I am very tempted to fix this, so that nobody would be discussing where it did or didn't work. (Though, fork() would still vary in behavior, but we'd never use it directly..) Threads not surviving fork is new with using pthreads. Threads never survived fork on Windows. For user threads, the change was a bit late, didn't occur until we worked on getting cvsupd to work with pthreads. - Jay > Date: Sat, 12 Feb 2011 13:02:11 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] pthread_atfork > > I think it would be acceptable to say that pthread threading is > broken on FreeBSD6 (as it is not officially supported any more). > All the stuff below should be documented in the 'known problems' > section. Especially the different behaviours of fork for > user and pthreads. This should also be mentioned in the M3 > interfaces (source code)! > > Also that to use use threads with FreeBSD requires to remove the > -pthread flag, as Mika noted. > > BTW, do we have a chapter explaining what to do to use user threads > instead of the default pthreads? > > Olaf > > PS: Hint: this would be a great task for anybody lurking on the lists > and trying to get involved :-) > > Quoting Jay K : > > > > > I wrote that. > > > > As I recall, FreeBSD < 6 doesn't have pthread_atfork. > > > > I think that was based on reading the man pages. > > > > So, yes. What would you suggest? > > > > > > > > > > > > All this atfork stuff came about as part of getting cvsupd to work. > > > > With pthreads/kernel threads. It historically worked, with > > Modula-3 user threads. > > > > Modula-3 user threads have this funny property that all threads > > survive work. > > > > With pthreads/kernel threads, fork gives you a new process with > > just one thread. > > > > (Solaris has fork1 and forkall, and fork maybe used to be > > forkall, but now it is fork1). > > > > > > > > > > > > cvsupd is one of those slightly unusual programs that does "fork and > > then do more work" > > > > as supposed to the more typical "fork and exec". > > > > > > > > Which is to say..I don't know. Maybe, don't expect pthreads to work > > on FreeBSD < 6? > > > > At least not in a process that ever calls fork? Or does "fork and > > then more work"? > > > > > > > > > > > > In reality, i think a lot of libraries have trouble with "fork and > > then do more work", but > > > > I don't know. I recall vague notices in the Apple/Darwin manpages > > not to assume > > > > this works -- the obvious implication that there is plenty of use of > > pthreads e.g. mutexes, > > > > but relatively little use of pthread_atfork. > > > > > > > > > > > > You could perhaps replace all uses of fork with a function that > > called the fork handlers, > > > > then fork, then the other handlers. That is probably what I meant in > > the comment > > > > about this being easy to fix. > > > > You might though then run into the problem that callers of fork can't do much > > > > before exec...oh, that's vfork I bet actually. > > > > > > > > > > > > - Jay > > > > > > > > > > > > > >> To: hosking at cs.purdue.edu > >> Date: Fri, 11 Feb 2011 19:05:25 -0800 > >> From: mika at async.caltech.edu > >> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > >> Subject: [M3devel] pthread_atfork > >> > >> Tony, > >> > >> What's RTProcessC.c doing? > >> > >> I note the following cryptic comment: > >> > >> /* NOTE: Even userthreads now depends > >> * on availability of pthreads. > >> * This can be fixed if need be. > >> */ > >> > >> now further down we can read: > >> > >> #if defined(_WIN32) \ > >> || defined(__vms) \ > >> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > >> return 0; > >> #else > >> while (1) > >> { > >> int i = pthread_atfork(prepare, parent, child); > >> if (i != EAGAIN) > >> return i; > >> sleep(0); > >> } > >> #endif > >> > >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system), > >> RTProcess.RegisterForkHandlers does nothing? > >> > >> Hmmm..... > >> > >> Mika > > > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 15:09:59 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 14:09:59 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <20110212064812.286E61A2078@async.async.caltech.edu> References: <20110212064812.286E61A2078@async.async.caltech.edu> Message-ID: > >LINUXLIBC6 : seg fault I've seen this now. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb3200b90 (LWP 18154)] 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) at ../src/runtime/common/RTCollector.m3:409 409 IF hdr.typecode = RT0.TextLitTypecode THEN RETURN END; 1: x/i $pc 0xb6dee58f : mov (%eax),%eax (gdb) bt 9 #0 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) at ../src/runtime/common/RTCollector.m3:409 #1 0xb6deaa46 in RTHeapMap__Walk (x=0xb6a20014, pc=0xb7738648, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:202 #2 0xb6dea341 in RTHeapMap__DoWalkRef (t=0xb7738694, a=0xb6a20014, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:62 #3 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739c54, a=0xb6a2000c, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 #4 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739e68, a=0xb6a2000c, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 #5 0xb6dea2bf in RTHeapMap__WalkRef (h=0xb6a20008, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:47 #6 0xb6df0636 in RTCollector__CleanBetween (h=0xb6a20008, he=0xb6a30000, clean=0 '\0') at ../src/runtime/common/RTCollector.m3:1091 #7 0xb6df0473 in RTCollector__CleanPage (page=0xb6a20000) at ../src/runtime/common/RTCollector.m3:1064 #8 0xb6defc96 in RTCollector__CollectSomeInStateZero () at ../src/runtime/common/RTCollector.m3:885 (More stack frames follow...) (gdb) p $eax $2 = 2097148 (gdb) p/x $eax $3 = 0x1ffffc (gdb) x/x $eax+4 0x200000: Cannot access memory at address 0x200000 (gdb) x/x $eax 0x1ffffc: Cannot access memory at address 0x1ffffc Maybe I should try with a correctly sized jmpbuf to assure it isn't stack overflow? But notice that the next page doesn't work either. Actually I tried adding a lot to $eax and still not accessible. Maybe maybe maybe maybe I'll dig into this. But I'm very busy lately and maybe indefinitely into future.. Thanks, - Jay > To: jay.krell at cornell.edu; hosking at cs.purdue.edu > Date: Fri, 11 Feb 2011 22:48:12 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] threading on Windows? > > Well, with the following changes: re-bootstrapping on the two Linux > systems and removal of the -pthread flag on FreeBSD4, all is well. > User threading works on all the architectures listed below, at least > as far as the thread testing program can test it. (It seems that because > of the way pthread_atfork is coded, pthread is not needed at all on > FreeBSD4.) > > Mika > > Mika Nystrom writes: > >Another odd datapoint is that even rather aggressive testing seems not > >to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user > >threads, of course. > > > >So status right now of user threads seems to be: > > > >FreeBSD4 : strange failure in libpthread as reported by gdb > >LINUXLIBC6 : seg fault > >AMD64_LINUX : seg fault > >AMD64_FREEBSD : OK > >I386_DARWIN : OK -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 15:10:13 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 14:10:13 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110212064812.286E61A2078@async.async.caltech.edu>, Message-ID: (ps: using kernel/pthreads) From: jay.krell at cornell.edu To: mika at async.caltech.edu; hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] threading on Windows? Date: Sat, 12 Feb 2011 14:09:59 +0000 > >LINUXLIBC6 : seg fault I've seen this now. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb3200b90 (LWP 18154)] 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) at ../src/runtime/common/RTCollector.m3:409 409 IF hdr.typecode = RT0.TextLitTypecode THEN RETURN END; 1: x/i $pc 0xb6dee58f : mov (%eax),%eax (gdb) bt 9 #0 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) at ../src/runtime/common/RTCollector.m3:409 #1 0xb6deaa46 in RTHeapMap__Walk (x=0xb6a20014, pc=0xb7738648, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:202 #2 0xb6dea341 in RTHeapMap__DoWalkRef (t=0xb7738694, a=0xb6a20014, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:62 #3 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739c54, a=0xb6a2000c, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 #4 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739e68, a=0xb6a2000c, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 #5 0xb6dea2bf in RTHeapMap__WalkRef (h=0xb6a20008, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:47 #6 0xb6df0636 in RTCollector__CleanBetween (h=0xb6a20008, he=0xb6a30000, clean=0 '\0') at ../src/runtime/common/RTCollector.m3:1091 #7 0xb6df0473 in RTCollector__CleanPage (page=0xb6a20000) at ../src/runtime/common/RTCollector.m3:1064 #8 0xb6defc96 in RTCollector__CollectSomeInStateZero () at ../src/runtime/common/RTCollector.m3:885 (More stack frames follow...) (gdb) p $eax $2 = 2097148 (gdb) p/x $eax $3 = 0x1ffffc (gdb) x/x $eax+4 0x200000: Cannot access memory at address 0x200000 (gdb) x/x $eax 0x1ffffc: Cannot access memory at address 0x1ffffc Maybe I should try with a correctly sized jmpbuf to assure it isn't stack overflow? But notice that the next page doesn't work either. Actually I tried adding a lot to $eax and still not accessible. Maybe maybe maybe maybe I'll dig into this. But I'm very busy lately and maybe indefinitely into future.. Thanks, - Jay > To: jay.krell at cornell.edu; hosking at cs.purdue.edu > Date: Fri, 11 Feb 2011 22:48:12 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] threading on Windows? > > Well, with the following changes: re-bootstrapping on the two Linux > systems and removal of the -pthread flag on FreeBSD4, all is well. > User threading works on all the architectures listed below, at least > as far as the thread testing program can test it. (It seems that because > of the way pthread_atfork is coded, pthread is not needed at all on > FreeBSD4.) > > Mika > > Mika Nystrom writes: > >Another odd datapoint is that even rather aggressive testing seems not > >to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user > >threads, of course. > > > >So status right now of user threads seems to be: > > > >FreeBSD4 : strange failure in libpthread as reported by gdb > >LINUXLIBC6 : seg fault > >AMD64_LINUX : seg fault > >AMD64_FREEBSD : OK > >I386_DARWIN : OK -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 12 17:59:30 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 08:59:30 -0800 Subject: [M3devel] pthread_atfork In-Reply-To: References: <20110212030525.822411A2078@async.async.caltech.edu> Message-ID: <20110212165930.DC32C1A2078@async.async.caltech.edu> Hi Jay, I think "fork and do work" qualifies as a bug in the application, if the application wants to be portable. Why can't it fork and exec instead? If it wants to do more work it should just fork and exec another copy of itself. fork() is also not part of Modula-3... Process.Create() is, however. Don't tell me you broke that as part of "fixing" things for cvsupd... There is still one aspect of your comment I don't understand. What does the call to pthread_atfork actually accomplish when running with user threads? You say below that "don't expect pthreads to work on FreeBSD < 6 [using the fork-and-do-work pattern]". Fair enough. But why would user threads work? We're skipping pthread_atfork for them too. Do we not need pthread_atfork with user threads? What you're saying suggests that FreeBSD >= 6 and all other platforms need pthread_atfork with user threads. Is that really what you mean? At the moment I'm just concerned with getting user threads to work as I know there are bugs with pthreads. I also think we should not waste time and effort making the "fork and do work" pattern work unless it's really, really necessary (which I really don't see why it would be). Fix the application instead of introducing bugs in the libraries and language implementation! I've been trying for a long, long time to switch from PM3 to CM3 (almost ten years, I think). These bugs from "enhancements" have stopped me every time. Maybe things are finally working well enough... (some comments inline in your quoted text below) Mika Jay K writes: >--_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >I wrote that. > >As I recall=2C FreeBSD < 6 doesn't have pthread_atfork. > >I think that was based on reading the man pages. > >So=2C yes. What would you suggest? > > > > > >All this atfork stuff came about as part of getting cvsupd to work. > > With pthreads/kernel threads. It historically worked=2C with Modula-3 use= >r threads. > > Modula-3 user threads have this funny property that all threads survive w= >ork. you mean survive fork don't you? > > With pthreads/kernel threads=2C fork gives you a new process with just on= >e thread. > > (Solaris has fork1 and forkall=2C and fork maybe used to be forkall=2C b= >ut now it is fork1). > > > > > >cvsupd is one of those slightly unusual programs that does "fork and then d= >o more work" > >as supposed to the more typical "fork and exec". > > > >Which is to say..I don't know. Maybe=2C don't expect pthreads to work on Fr= >eeBSD < 6? > >At least not in a process that ever calls fork? Or does "fork and then more= > work"? > > > > > >In reality=2C i think a lot of libraries have trouble with "fork and then d= >o more work"=2C but > >I don't know. I recall vague notices in the Apple/Darwin manpages not to as= >sume > >this works -- the obvious implication that there is plenty of use of pthrea= >ds e.g. mutexes=2C > >but relatively little use of pthread_atfork. > > > > > >You could perhaps replace all uses of fork with a function that called the = >fork handlers=2C > >then fork=2C then the other handlers. That is probably what I meant in the = >comment > >about this being easy to fix. > >You might though then run into the problem that callers of fork can't do mu= >ch > >before exec...oh=2C that's vfork I bet actually. I don't understand this paragraph either :( > > > > > > - Jay > > > > > > >> To: hosking at cs.purdue.edu >> Date: Fri=2C 11 Feb 2011 19:05:25 -0800 >> From: mika at async.caltech.edu >> CC: m3devel at elegosoft.com=3B jay.krell at cornell.edu >> Subject: [M3devel] pthread_atfork >>=20 >> Tony=2C >>=20 >> What's RTProcessC.c doing? >>=20 >> I note the following cryptic comment: >>=20 >> /* NOTE: Even userthreads now depends >> * on availability of pthreads. >> * This can be fixed if need be. >> */ >>=20 >> now further down we can read: >>=20 >> #if defined(_WIN32) \ >> || defined(__vms) \ >> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) >> return 0=3B >> #else >> while (1) >> { >> int i =3D pthread_atfork(prepare=2C parent=2C child)=3B >> if (i !=3D EAGAIN) >> return i=3B >> sleep(0)=3B >> } >> #endif >>=20 >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=2C >> RTProcess.RegisterForkHandlers does nothing? >>=20 >> Hmmm..... >>=20 >> Mika > = > >--_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >I wrote that.
>As I recall=2C FreeBSD <=3B 6 doesn't have pthread_atfork.
>I think that was based on reading the man pages.
>So=2C yes. What would you suggest?
>
>
>All this atfork stuff came about as part of getting cvsupd to work.
> =3B With pthreads/kernel threads. It historically worked=2C with Modul= >a-3 user threads.
> =3B Modula-3 user threads have this funny property that all threads su= >rvive work.
> =3B With pthreads/kernel threads=2C fork gives you a new process with = >just one thread.
> =3B =3B (Solaris has fork1 and forkall=2C and fork maybe used to b= >e forkall=2C but now it is fork1).
>
>
>cvsupd is one of those slightly unusual programs that does "fork and then d= >o more work"
>as supposed to the more typical "fork and exec".
>
>Which is to say..I don't know. Maybe=2C don't expect pthreads to work on Fr= >eeBSD <=3B 6?
>At least not in a process that ever calls fork? Or does "fork and then more= > work"?
>
>
>In reality=2C i think a lot of libraries have trouble with "fork and then d= >o more work"=2C but
>I don't know. I recall vague notices in the Apple/Darwin manpages not to as= >sume
>this works -- the obvious implication that there is plenty of use of pthrea= >ds e.g. mutexes=2C
>but relatively little use of pthread_atfork.
>
>
>You could perhaps replace all uses of fork with a function that called the = >fork handlers=2C
>then fork=2C then the other handlers. That is probably what I meant in the = >comment
>about this being easy to fix.
>You might though then run into the problem that callers of fork can't do mu= >ch
>before exec...oh=2C that's vfork I bet actually.
>
>
> =3B- Jay






>=3B To: hosking at cs.purdue.edur>>=3B Date: Fri=2C 11 Feb 2011 19:05:25 -0800
>=3B From: mika at async= >.caltech.edu
>=3B CC: m3devel at elegosoft.com=3B jay.krell at cornell.edur>>=3B Subject: [M3devel] pthread_atfork
>=3B
>=3B Tony=2C
= >>=3B
>=3B What's RTProcessC.c doing?
>=3B
>=3B I note th= >e following cryptic comment:
>=3B
>=3B /* NOTE: Even userthreads= > now depends
>=3B * on availability of pthreads.
>=3B * This ca= >n be fixed if need be.
>=3B */
>=3B
>=3B now further down = >we can read:
>=3B
>=3B #if defined(_WIN32) \
>=3B |= >| defined(__vms) \
>=3B || (defined(__FreeBSD__) &=3B&= >=3B (__FreeBSD__ <=3B 6))
>=3B return 0=3B
>=3B #else
&g= >t=3B while (1)
>=3B {
>=3B int i =3D pthread_atfork= >(prepare=2C parent=2C child)=3B
>=3B if (i !=3D EAGAIN)
>= >=3B return i=3B
>=3B sleep(0)=3B
>=3B }
>= >=3B #endif
>=3B
>=3B so on FreeBSD 5 (what I'm running on my "Fr= >eeBSD4" system)=2C
>=3B RTProcess.RegisterForkHandlers does nothing?r>>=3B
>=3B Hmmm.....
>=3B
>=3B Mika
> = > >= > >--_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_-- From wagner at elegosoft.com Sat Feb 12 20:55:27 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Sat, 12 Feb 2011 20:55:27 +0100 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212165930.DC32C1A2078@async.async.caltech.edu> References: <20110212030525.822411A2078@async.async.caltech.edu> <20110212165930.DC32C1A2078@async.async.caltech.edu> Message-ID: <20110212205527.e6xetojo0ssgocg4@mail.elegosoft.com> Well, I think actually the fork-and-do-work pattern has been the default for a long time with the M3 user threads implementation. It wasn't specified though, and I think no application except CVSup did use it. Jay is not responsible for any breakage here, he just tried to fix CVSup and establish a consistent behaviour after our change to system pthreads on most platforms. Olaf Quoting Mika Nystrom : > Hi Jay, > > I think "fork and do work" qualifies as a bug in the application, if > the application wants to be portable. Why can't it fork and exec instead? > If it wants to do more work it should just fork and exec another copy > of itself. > > fork() is also not part of Modula-3... Process.Create() is, however. > Don't tell me you broke that as part of "fixing" things for cvsupd... > > There is still one aspect of your comment I don't understand. What does > the call to pthread_atfork actually accomplish when running with user > threads? You say below that "don't expect pthreads to work on FreeBSD < > 6 [using the fork-and-do-work pattern]". Fair enough. But why would > user threads work? We're skipping pthread_atfork for them too. > > Do we not need pthread_atfork with user threads? What you're saying > suggests that FreeBSD >= 6 and all other platforms need pthread_atfork > with user threads. Is that really what you mean? > > At the moment I'm just concerned with getting user threads to work as > I know there are bugs with pthreads. > > I also think we should not waste time and effort making the "fork and do > work" pattern work unless it's really, really necessary (which I really > don't see why it would be). Fix the application instead of introducing > bugs in the libraries and language implementation! > > I've been trying for a long, long time to switch from PM3 to CM3 (almost > ten years, I think). These bugs from "enhancements" have stopped me > every time. Maybe things are finally working well enough... > > (some comments inline in your quoted text below) > > Mika > > > Jay K writes: >> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >> Content-Type: text/plain; charset="iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> >> I wrote that. >> >> As I recall=2C FreeBSD < 6 doesn't have pthread_atfork. >> >> I think that was based on reading the man pages. >> >> So=2C yes. What would you suggest? >> >> >> >> >> >> All this atfork stuff came about as part of getting cvsupd to work. >> >> With pthreads/kernel threads. It historically worked=2C with Modula-3 use= >> r threads. >> >> Modula-3 user threads have this funny property that all threads survive w= >> ork. > > you mean survive fork don't you? > >> >> With pthreads/kernel threads=2C fork gives you a new process with just on= >> e thread. >> >> (Solaris has fork1 and forkall=2C and fork maybe used to be forkall=2C b= >> ut now it is fork1). >> >> >> >> >> >> cvsupd is one of those slightly unusual programs that does "fork and then d= >> o more work" >> >> as supposed to the more typical "fork and exec". >> >> >> >> Which is to say..I don't know. Maybe=2C don't expect pthreads to work on Fr= >> eeBSD < 6? >> >> At least not in a process that ever calls fork? Or does "fork and then more= >> work"? >> >> >> >> >> >> In reality=2C i think a lot of libraries have trouble with "fork and then d= >> o more work"=2C but >> >> I don't know. I recall vague notices in the Apple/Darwin manpages not to as= >> sume >> >> this works -- the obvious implication that there is plenty of use of pthrea= >> ds e.g. mutexes=2C >> >> but relatively little use of pthread_atfork. >> >> >> >> >> >> You could perhaps replace all uses of fork with a function that called the = >> fork handlers=2C >> >> then fork=2C then the other handlers. That is probably what I meant in the = >> comment >> >> about this being easy to fix. >> >> You might though then run into the problem that callers of fork can't do mu= >> ch >> >> before exec...oh=2C that's vfork I bet actually. > > I don't understand this paragraph either :( > >> >> >> >> >> >> - Jay >> >> >> >> >> >> >>> To: hosking at cs.purdue.edu >>> Date: Fri=2C 11 Feb 2011 19:05:25 -0800 >>> From: mika at async.caltech.edu >>> CC: m3devel at elegosoft.com=3B jay.krell at cornell.edu >>> Subject: [M3devel] pthread_atfork >>> =20 >>> Tony=2C >>> =20 >>> What's RTProcessC.c doing? >>> =20 >>> I note the following cryptic comment: >>> =20 >>> /* NOTE: Even userthreads now depends >>> * on availability of pthreads. >>> * This can be fixed if need be. >>> */ >>> =20 >>> now further down we can read: >>> =20 >>> #if defined(_WIN32) \ >>> || defined(__vms) \ >>> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) >>> return 0=3B >>> #else >>> while (1) >>> { >>> int i =3D pthread_atfork(prepare=2C parent=2C child)=3B >>> if (i !=3D EAGAIN) >>> return i=3B >>> sleep(0)=3B >>> } >>> #endif >>> =20 >>> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=2C >>> RTProcess.RegisterForkHandlers does nothing? >>> =20 >>> Hmmm..... >>> =20 >>> Mika >> = >> >> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >> Content-Type: text/html; charset="iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> >> >> >> >> >> I wrote that.
>> As I recall=2C FreeBSD <=3B 6 doesn't have pthread_atfork.
>> I think that was based on reading the man pages.
>> So=2C yes. What would you suggest?
>>
>>
>> All this atfork stuff came about as part of getting cvsupd to work.
>>  =3B With pthreads/kernel threads. It historically worked=2C with Modul= >> a-3 user threads.
>>  =3B Modula-3 user threads have this funny property that all threads su= >> rvive work.
>>  =3B With pthreads/kernel threads=2C fork gives you a new process with = >> just one thread.
>>  =3B =3B (Solaris has fork1 and forkall=2C and fork maybe used to b= >> e forkall=2C but now it is fork1).
>>
>>
>> cvsupd is one of those slightly unusual programs that does "fork and then d= >> o more work"
>> as supposed to the more typical "fork and exec".
>>
>> Which is to say..I don't know. Maybe=2C don't expect pthreads to work on Fr= >> eeBSD <=3B 6?
>> At least not in a process that ever calls fork? Or does "fork and then more= >> work"?
>>
>>
>> In reality=2C i think a lot of libraries have trouble with "fork and then d= >> o more work"=2C but
>> I don't know. I recall vague notices in the Apple/Darwin manpages not to as= >> sume
>> this works -- the obvious implication that there is plenty of use of pthrea= >> ds e.g. mutexes=2C
>> but relatively little use of pthread_atfork.
>>
>>
>> You could perhaps replace all uses of fork with a function that called the = >> fork handlers=2C
>> then fork=2C then the other handlers. That is probably what I meant in the = >> comment
>> about this being easy to fix.
>> You might though then run into the problem that callers of fork can't do mu= >> ch
>> before exec...oh=2C that's vfork I bet actually.
>>
>>
>>  =3B- Jay






>=3B To: hosking at cs.purdue.edu> r>>=3B Date: Fri=2C 11 Feb 2011 19:05:25 -0800
>=3B From: mika at async= >> .caltech.edu
>=3B CC: m3devel at elegosoft.com=3B jay.krell at cornell.edu> r>>=3B Subject: [M3devel] pthread_atfork
>=3B
>=3B Tony=2C
= >> >=3B
>=3B What's RTProcessC.c doing?
>=3B
>=3B I note th= >> e following cryptic comment:
>=3B
>=3B /* NOTE: Even userthreads= >> now depends
>=3B * on availability of pthreads.
>=3B * This ca= >> n be fixed if need be.
>=3B */
>=3B
>=3B now further down = >> we can read:
>=3B
>=3B #if defined(_WIN32) \
>=3B |= >> | defined(__vms) \
>=3B || (defined(__FreeBSD__) &=3B&= >> =3B (__FreeBSD__ <=3B 6))
>=3B return 0=3B
>=3B #else
&g= >> t=3B while (1)
>=3B {
>=3B int i =3D pthread_atfork= >> (prepare=2C parent=2C child)=3B
>=3B if (i !=3D EAGAIN)
>= >> =3B return i=3B
>=3B sleep(0)=3B
>=3B }
>= >> =3B #endif
>=3B
>=3B so on FreeBSD 5 (what I'm running on my "Fr= >> eeBSD4" system)=2C
>=3B RTProcess.RegisterForkHandlers does nothing?> r>>=3B
>=3B Hmmm.....
>=3B
>=3B Mika
>> = >> >> = >> >> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_-- > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Sat Feb 12 21:25:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 12:25:50 -0800 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212205527.e6xetojo0ssgocg4@mail.elegosoft.com> References: <20110212030525.822411A2078@async.async.caltech.edu> <20110212165930.DC32C1A2078@async.async.caltech.edu> <20110212205527.e6xetojo0ssgocg4@mail.elegosoft.com> Message-ID: <20110212202550.542D41A2078@async.async.caltech.edu> How do you do "fork-and-do-work" from Modula-3? All my programs that create new Unix processes do it with Process.Create, which as far as I know does a fork-and-exec... I'm not sure what if anything is broken here, either. But then again I didn't fully understand Jay's last mail on the subject. Mika Olaf Wagner writes: >Well, I think actually the fork-and-do-work pattern has been >the default for a long time with the M3 user threads implementation. >It wasn't specified though, and I think no application except >CVSup did use it. > >Jay is not responsible for any breakage here, he just tried to fix >CVSup and establish a consistent behaviour after our change to >system pthreads on most platforms. > >Olaf > >Quoting Mika Nystrom : > >> Hi Jay, >> >> I think "fork and do work" qualifies as a bug in the application, if >> the application wants to be portable. Why can't it fork and exec instead? >> If it wants to do more work it should just fork and exec another copy >> of itself. >> >> fork() is also not part of Modula-3... Process.Create() is, however. >> Don't tell me you broke that as part of "fixing" things for cvsupd... >> >> There is still one aspect of your comment I don't understand. What does >> the call to pthread_atfork actually accomplish when running with user >> threads? You say below that "don't expect pthreads to work on FreeBSD < >> 6 [using the fork-and-do-work pattern]". Fair enough. But why would >> user threads work? We're skipping pthread_atfork for them too. >> >> Do we not need pthread_atfork with user threads? What you're saying >> suggests that FreeBSD >=3D 6 and all other platforms need pthread_atfork >> with user threads. Is that really what you mean? >> >> At the moment I'm just concerned with getting user threads to work as >> I know there are bugs with pthreads. >> >> I also think we should not waste time and effort making the "fork and do >> work" pattern work unless it's really, really necessary (which I really >> don't see why it would be). Fix the application instead of introducing >> bugs in the libraries and language implementation! >> >> I've been trying for a long, long time to switch from PM3 to CM3 (almost >> ten years, I think). These bugs from "enhancements" have stopped me >> every time. Maybe things are finally working well enough... >> >> (some comments inline in your quoted text below) >> >> Mika >> >> >> Jay K writes: >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >>> Content-Type: text/plain; charset=3D"iso-8859-1" >>> Content-Transfer-Encoding: quoted-printable >>> >>> >>> I wrote that. >>> >>> As I recall=3D2C FreeBSD < 6 doesn't have pthread_atfork. >>> >>> I think that was based on reading the man pages. >>> >>> So=3D2C yes. What would you suggest? >>> >>> >>> >>> >>> >>> All this atfork stuff came about as part of getting cvsupd to work. >>> >>> With pthreads/kernel threads. It historically worked=3D2C with Modula-3 = >use=3D >>> r threads. >>> >>> Modula-3 user threads have this funny property that all threads survive = >w=3D >>> ork. >> >> you mean survive fork don't you? >> >>> >>> With pthreads/kernel threads=3D2C fork gives you a new process with just= > on=3D >>> e thread. >>> >>> (Solaris has fork1 and forkall=3D2C and fork maybe used to be forkall= >=3D2C b=3D >>> ut now it is fork1). >>> >>> >>> >>> >>> >>> cvsupd is one of those slightly unusual programs that does "fork and then= > d=3D >>> o more work" >>> >>> as supposed to the more typical "fork and exec". >>> >>> >>> >>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o= >n Fr=3D >>> eeBSD < 6? >>> >>> At least not in a process that ever calls fork? Or does "fork and then mo= >re=3D >>> work"? >>> >>> >>> >>> >>> >>> In reality=3D2C i think a lot of libraries have trouble with "fork and th= >en d=3D >>> o more work"=3D2C but >>> >>> I don't know. I recall vague notices in the Apple/Darwin manpages not to = >as=3D >>> sume >>> >>> this works -- the obvious implication that there is plenty of use of pthr= >ea=3D >>> ds e.g. mutexes=3D2C >>> >>> but relatively little use of pthread_atfork. >>> >>> >>> >>> >>> >>> You could perhaps replace all uses of fork with a function that called th= >e =3D >>> fork handlers=3D2C >>> >>> then fork=3D2C then the other handlers. That is probably what I meant in = >the =3D >>> comment >>> >>> about this being easy to fix. >>> >>> You might though then run into the problem that callers of fork can't do = >mu=3D >>> ch >>> >>> before exec...oh=3D2C that's vfork I bet actually. >> >> I don't understand this paragraph either :( >> >>> >>> >>> >>> >>> >>> - Jay >>> >>> >>> >>> >>> >>> >>>> To: hosking at cs.purdue.edu >>>> Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800 >>>> From: mika at async.caltech.edu >>>> CC: m3devel at elegosoft.com=3D3B jay.krell at cornell.edu >>>> Subject: [M3devel] pthread_atfork >>>> =3D20 >>>> Tony=3D2C >>>> =3D20 >>>> What's RTProcessC.c doing? >>>> =3D20 >>>> I note the following cryptic comment: >>>> =3D20 >>>> /* NOTE: Even userthreads now depends >>>> * on availability of pthreads. >>>> * This can be fixed if need be. >>>> */ >>>> =3D20 >>>> now further down we can read: >>>> =3D20 >>>> #if defined(_WIN32) \ >>>> || defined(__vms) \ >>>> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) >>>> return 0=3D3B >>>> #else >>>> while (1) >>>> { >>>> int i =3D3D pthread_atfork(prepare=3D2C parent=3D2C child)=3D3B >>>> if (i !=3D3D EAGAIN) >>>> return i=3D3B >>>> sleep(0)=3D3B >>>> } >>>> #endif >>>> =3D20 >>>> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=3D2C >>>> RTProcess.RegisterForkHandlers does nothing? >>>> =3D20 >>>> Hmmm..... >>>> =3D20 >>>> Mika >>> =09=09 =09 =09=09 =3D >>> >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >>> Content-Type: text/html; charset=3D"iso-8859-1" >>> Content-Transfer-Encoding: quoted-printable >>> >>> >>> >>> >>> >>> >>> I wrote that.
>>> As I recall=3D2C FreeBSD <=3D3B 6 doesn't have pthread_atfork.
>>> I think that was based on reading the man pages.
>>> So=3D2C yes. What would you suggest?
>>>
>>>
>>> All this atfork stuff came about as part of getting cvsupd to work.
>>>  =3D3B With pthreads/kernel threads. It historically worked=3D2C with= > Modul=3D >>> a-3 user threads.
>>>  =3D3B Modula-3 user threads have this funny property that all thread= >s su=3D >>> rvive work.
>>>  =3D3B With pthreads/kernel threads=3D2C fork gives you a new process= > with =3D >>> just one thread.
>>>  =3D3B =3D3B (Solaris has fork1 and forkall=3D2C and fork maybe u= >sed to b=3D >>> e forkall=3D2C but now it is fork1).
>>>
>>>
>>> cvsupd is one of those slightly unusual programs that does "fork and then= > d=3D >>> o more work"
>>> as supposed to the more typical "fork and exec".
>>>
>>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o= >n Fr=3D >>> eeBSD <=3D3B 6?
>>> At least not in a process that ever calls fork? Or does "fork and then mo= >re=3D >>> work"?
>>>
>>>
>>> In reality=3D2C i think a lot of libraries have trouble with "fork and th= >en d=3D >>> o more work"=3D2C but
>>> I don't know. I recall vague notices in the Apple/Darwin manpages not to = >as=3D >>> sume
>>> this works -- the obvious implication that there is plenty of use of pthr= >ea=3D >>> ds e.g. mutexes=3D2C
>>> but relatively little use of pthread_atfork.
>>>
>>>
>>> You could perhaps replace all uses of fork with a function that called th= >e =3D >>> fork handlers=3D2C
>>> then fork=3D2C then the other handlers. That is probably what I meant in = >the =3D >>> comment
>>> about this being easy to fix.
>>> You might though then run into the problem that callers of fork can't do = >mu=3D >>> ch
>>> before exec...oh=3D2C that's vfork I bet actually.
>>>
>>>
>>>  =3D3B- Jay






>=3D3B To: hosking at cs.purdue= >.edu>> r>>=3D3B Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800
>=3D3B From: mi= >ka at async=3D >>> .caltech.edu
>=3D3B CC: m3devel at elegosoft.com=3D3B jay.krell at cornell= >.edu>> r>>=3D3B Subject: [M3devel] pthread_atfork
>=3D3B
>=3D3B Ton= >y=3D2C
=3D >>> >=3D3B
>=3D3B What's RTProcessC.c doing?
>=3D3B
>=3D3B= > I note th=3D >>> e following cryptic comment:
>=3D3B
>=3D3B /* NOTE: Even usert= >hreads=3D >>> now depends
>=3D3B * on availability of pthreads.
>=3D3B * Th= >is ca=3D >>> n be fixed if need be.
>=3D3B */
>=3D3B
>=3D3B now furth= >er down =3D >>> we can read:
>=3D3B
>=3D3B #if defined(_WIN32) \
>=3D3B = > |=3D >>> | defined(__vms) \
>=3D3B || (defined(__FreeBSD__) &=3D3B= >&=3D >>> =3D3B (__FreeBSD__ <=3D3B 6))
>=3D3B return 0=3D3B
>=3D3B= > #else
&g=3D >>> t=3D3B while (1)
>=3D3B {
>=3D3B int i =3D3D pthr= >ead_atfork=3D >>> (prepare=3D2C parent=3D2C child)=3D3B
>=3D3B if (i !=3D3D EAGA= >IN)
>=3D >>> =3D3B return i=3D3B
>=3D3B sleep(0)=3D3B
>=3D3B = > }
>=3D >>> =3D3B #endif
>=3D3B
>=3D3B so on FreeBSD 5 (what I'm running o= >n my "Fr=3D >>> eeBSD4" system)=3D2C
>=3D3B RTProcess.RegisterForkHandlers does noth= >ing?>> r>>=3D3B
>=3D3B Hmmm.....
>=3D3B
>=3D3B Mika
>>> =09 =09 =09=09 =3D >>> >>> =3D >>> >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_-- >> > > > >--=20 >Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa= >ny >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz= >: Berlin >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Sat Feb 12 21:43:02 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 12:43:02 -0800 Subject: [M3devel] Performance issues with Process.Wait under userthreads Message-ID: <20110212204302.19C781A2078@async.async.caltech.edu> Hi again m3devel (especially Tony), I have finally taken a bite of a problem that has been annoying me for a very, very long time. Under user threads, the code for Process.Wait is as follows (see ThreadPosix.m3): PROCEDURE WaitProcess (pid: int; VAR status: int): int = (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) CONST Delay = 0.1D0; BEGIN LOOP WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO IF r # 0 THEN RETURN r END; END; Pause(Delay); END; END WaitProcess; It inserts a 0.1 second delay after each failed waitpid. This is extremely annoying for programs that start a long sequence of child processes and wait for them in sequence. Namely, the compiler itself. As a result the cm3 compiler (and PM3's m3build) are normally very very slow when using user threads. For about the last ten years, I've had a hacked up m3build (for my PM3 installation) that skips the Pause and busy-waits instead. Note there is another problem here. Since the Modula-3 runtime ignores SIGCHLD, no zombie processes are created since the Unix system automatically reaps the child processes. I can see this would be a problem since PIDs are eventually reused and ... couldn't Uexec.waitpid wind up referring to the wrong process?? I will further note that the comment in Process.i3 reads as follows: PROCEDURE Wait(p: T): ExitCode; Wait until the process with handle p terminates, then free the operating system resources associated with the process and return an exit code indicating the reason for its termination. It is a checked runtime error to call Wait twice on the same process handle. I am going to take this as fair warning that Process.Create *may* use resources that are not going to be released until Process.Wait has been called. I have modified (in my local copy of CM3) the system as follows. I have come up with a semi-general mechanism for immediately unblocking a thread on the receipt of a unix signal. 1. The system relies on changing ThreadPosix.XPause such that if a signal is allowed to wake up a threads, that fact is recorded in a new field in the thread's descriptor record (of type ThreadPosix.T). 2. On receipt of a waited-for unix signal, a mask is set and control is passed to the thread scheduler which maintains the non-zero mask for exactly one iteration through the thread ring. 3. If a thread is paused and waiting for EITHER a signal or some time, the thread is released for running and the thread's waiting state is cleared. The changes are more or less as follows: 1. I have added a new field of type "int" to ThreadPosix.T: (* if state = pausing, the time at which we can restart *) waitingForTime: Time.T; + (* if state = pausing, the signal that truncates the pause *) + waitingForSig: int := -1; + (* true if we are waiting during an AlertWait or AlertJoin or AlertPause *) alertable: BOOLEAN := FALSE; 2. Modifications to pause: + PROCEDURE SigPause(n: LONGREAL; sig: int)= + <*FATAL Alerted*> + VAR until := n + Time.Now (); + BEGIN + XPause(until, FALSE, sig); + END SigPause; + PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= VAR until := n + Time.Now (); BEGIN XPause(until, TRUE); END AlertPause; ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) ! RAISES {Alerted} = BEGIN INC (inCritical); self.waitingForTime := until; self.alertable := alertable; + IF sig # -1 THEN + self.waitingForSig := sig + END; ICannotRun (State.pausing); DEC (inCritical); InternalYield (); 3. The received-signals mask: ! CONST MaxSigs = 64; ! TYPE Sig = [ 0..MaxSigs-1 ]; ! ! (* in order to listen to other signals, they have to be enabled in ! allow_sigvtalrm as well *) ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); ! ! gotSigs := SET OF Sig { }; ! ValueOfSIGCHLD() is a C function used to get the value of the SIGCHLD constant without guessing at it (in ThreadPosixC.c). 4. changes to the signal handler: ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); ! ! INC(inCritical); ! (* mark signal as being delivered *) ! IF sig >= 0 AND sig < MaxSigs THEN ! gotSigs := gotSigs + SET OF Sig { sig } ! END; ! DEC(inCritical); ! ! IF inCritical = 0 AND heapState.inCritical = 0 THEN ! InternalYield () ! END; END switch_thread; Note that I don't know if INC/DEC(inCritical) does exactly the right thing here. 5. changes to the scheduler: a. thread wakeup IF t.alertable AND t.alertPending THEN CanRun (t); EXIT; + + ELSIF t.waitingForSig IN gotSigs THEN + t.waitingForSig := -1; + CanRun(t); + EXIT; ELSIF t.waitingForTime <= now THEN CanRun (t); EXIT; ! b. clearing the mask END; END; + gotSigs := SET OF Sig {}; + IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN IF perfOn THEN PerfRunning (t.id); END; (* At least one thread wants to run; transfer to it *) 6. changes to WaitProcess (Process.Wait): PROCEDURE WaitProcess (pid: int; VAR status: int): int = (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) ! CONST Delay = 10.0D0; BEGIN LOOP WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO IF r # 0 THEN RETURN r END; END; ! SigPause(Delay,SIGCHLD); END; END WaitProcess; 7. install signal handler even if program is single-threaded: BEGIN + (* we need to call set up the signal handler for other reasons than + just thread switching now *) + setup_sigvtalrm (switch_thread); END ThreadPosix. 8. modify signal handler in ThreadPosixC.c to catch SIGCHLD: sigemptyset(&ThreadSwitchSignal); sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); + sigaddset(&ThreadSwitchSignal, SIGCHLD); act.sa_handler = handler; act.sa_flags = SA_RESTART; sigemptyset(&(act.sa_mask)); if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); + if (sigaction (SIGCHLD, &act, NULL)) abort(); I'll send the complete diff in a separate message for those who want to study it more closely. I propose the above changes for inclusion in the current CM3 repository. Mika From mika at async.caltech.edu Sat Feb 12 21:44:06 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 12:44:06 -0800 Subject: [M3devel] SIGCHLD diff Message-ID: <20110212204406.88A661A2078@async.async.caltech.edu> Index: ThreadPosix.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v retrieving revision 1.16 diff -c -r1.16 ThreadPosix.i3 *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 *************** *** 39,42 **** --- 39,45 ---- (*---------------------------------------------------------------------------*) + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> + PROCEDURE ValueOfSIGCHLD(): int; + END ThreadPosix. Index: ThreadPosix.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v retrieving revision 1.73 diff -c -r1.73 ThreadPosix.m3 *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 *************** *** 97,102 **** --- 97,105 ---- (* if state = pausing, the time at which we can restart *) waitingForTime: Time.T; + (* if state = pausing, the signal that truncates the pause *) + waitingForSig: int := -1; + (* true if we are waiting during an AlertWait or AlertJoin or AlertPause *) alertable: BOOLEAN := FALSE; *************** *** 147,152 **** --- 150,160 ---- defaultStackSize := 3000; + (* note that even though the "heavy machinery" is only used for + multipleThreads, we still need to set up the signal handler so + that we can catch signals from other sources than thread switching. + e.g., we use SIGCHLD to speed up Process.Wait *) + VAR stats: RECORD n_forks := 0; *************** *** 511,527 **** XPause(until, FALSE); END Pause; PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= VAR until := n + Time.Now (); BEGIN XPause(until, TRUE); END AlertPause; ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE) RAISES {Alerted} = BEGIN INC (inCritical); self.waitingForTime := until; self.alertable := alertable; ICannotRun (State.pausing); DEC (inCritical); InternalYield (); --- 519,546 ---- XPause(until, FALSE); END Pause; + PROCEDURE SigPause(n: LONGREAL; sig: int)= + <*FATAL Alerted*> + VAR until := n + Time.Now (); + BEGIN + XPause(until, FALSE, sig); + END SigPause; + PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= VAR until := n + Time.Now (); BEGIN XPause(until, TRUE); END AlertPause; ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) ! RAISES {Alerted} = BEGIN INC (inCritical); self.waitingForTime := until; self.alertable := alertable; + IF sig # -1 THEN + self.waitingForSig := sig + END; ICannotRun (State.pausing); DEC (inCritical); InternalYield (); *************** *** 703,712 **** END; END StartSwitching; ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); ! IF inCritical = 0 AND heapState.inCritical = 0 THEN InternalYield () END; END switch_thread; (*------------------------------------------------------------- scheduler ---*) --- 722,750 ---- END; END StartSwitching; ! CONST MaxSigs = 64; ! TYPE Sig = [ 0..MaxSigs-1 ]; ! ! (* in order to listen to other signals, they have to be enabled in ! allow_sigvtalrm as well *) ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); ! ! gotSigs := SET OF Sig { }; ! ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); ! ! INC(inCritical); ! (* mark signal as being delivered *) ! IF sig >= 0 AND sig < MaxSigs THEN ! gotSigs := gotSigs + SET OF Sig { sig } ! END; ! DEC(inCritical); ! ! IF inCritical = 0 AND heapState.inCritical = 0 THEN ! InternalYield () ! END; END switch_thread; (*------------------------------------------------------------- scheduler ---*) *************** *** 782,792 **** IF t.alertable AND t.alertPending THEN CanRun (t); EXIT; ELSIF t.waitingForTime <= now THEN CanRun (t); EXIT; ! ELSIF NOT somePausing THEN earliest := t.waitingForTime; somePausing := TRUE; --- 820,835 ---- IF t.alertable AND t.alertPending THEN CanRun (t); EXIT; + + ELSIF t.waitingForSig IN gotSigs THEN + t.waitingForSig := -1; + CanRun(t); + EXIT; ELSIF t.waitingForTime <= now THEN CanRun (t); EXIT; ! ELSIF NOT somePausing THEN earliest := t.waitingForTime; somePausing := TRUE; *************** *** 886,891 **** --- 929,936 ---- END; END; + gotSigs := SET OF Sig {}; + IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN IF perfOn THEN PerfRunning (t.id); END; (* At least one thread wants to run; transfer to it *) *************** *** 948,960 **** PROCEDURE WaitProcess (pid: int; VAR status: int): int = (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) ! CONST Delay = 0.1D0; BEGIN LOOP WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO IF r # 0 THEN RETURN r END; END; ! Pause(Delay); END; END WaitProcess; --- 993,1005 ---- PROCEDURE WaitProcess (pid: int; VAR status: int): int = (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) ! CONST Delay = 10.0D0; BEGIN LOOP WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO IF r # 0 THEN RETURN r END; END; ! SigPause(Delay,SIGCHLD); END; END WaitProcess; *************** *** 1361,1364 **** --- 1406,1412 ---- VAR debug := RTParams.IsPresent ("debugthreads"); BEGIN + (* we need to call set up the signal handler for other reasons than + just thread switching now *) + setup_sigvtalrm (switch_thread); END ThreadPosix. Index: ThreadPosixC.c =================================================================== RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v retrieving revision 1.59 diff -c -r1.59 ThreadPosixC.c *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 *************** *** 78,88 **** --- 78,97 ---- sigemptyset(&ThreadSwitchSignal); sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); + sigaddset(&ThreadSwitchSignal, SIGCHLD); act.sa_handler = handler; act.sa_flags = SA_RESTART; sigemptyset(&(act.sa_mask)); if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); + if (sigaction (SIGCHLD, &act, NULL)) abort(); + } + + int + __cdecl + ThreadPosix__value_of_SIGCHLD(void) + { + return SIGCHLD; } void From dabenavidesd at yahoo.es Sat Feb 12 22:08:13 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sat, 12 Feb 2011 21:08:13 +0000 (GMT) Subject: [M3devel] pthread_atfork In-Reply-To: <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com> Message-ID: <651549.53394.qm@web29714.mail.ird.yahoo.com> Hi all: by the way you told, do user space interfaces need to be linked differently from user space threads library than the one in kernel threads, so the behaviour can be selected better at runtime if more than one available processor (i.e a virtualized one, perhaps two if there are virtualized GPU, GPGPU, etc) is available on demand (in example the collector has its own user space thread allocated control and one in kernel space that user can control from the Runtime), could it work in such a way to allow a Thread to live in spaces that are accesible through like an NetObj.T, thus allowing it to communciate them through such an abstraction? Perhaps this could allow to run the kernel threads with Modula-3 governance and if so put them to work or switch to kernel them to accelerate its process according to its demands and such debug them in a special API like the one in the SPIN OS, fairly more comprenhensive thread implementation of Mach, Cthreads (aka C Pthreads calls) and own strends. Also scheduling could be handled to manage both user/kernel level synchronization, the threads and also allow to change from having a specialized hardware-assisted virtualized processor to a non hardware-assissted virtualized processor (like an NetObj.T again), which is like the same situation for Modula-3 with just one processor so in case it needs, it does so. In any case thread and object migration are safely defined over Obliq. Granted, this is same situation like was before the enter of virtualization hardware but again the whole idea of hardware-assisted virtualization was not new, even there is an article in which a PDP-10 is claimed has requirements to make it virtual in hardware (and I believe in software too). So SPIN could be a platform, like for hardware assissted virtualization and/or operating-system level virtualization, if so, we would be able to trace system calls for a given OS client running on it, rather than barely, so it makes the implementation with hardware-assisted source-level debugger ready, which is related to where programs are segfaulting now (no system-level breakpoints, that is, no hardware assisted source-level debugger for virtualization user, different from Modula-3 situation in current user-level space threading or SPIN gdb DEC TTD Topaz Tele Debugger at source-level service and not to forget the ldb cross-platform source-level remote debugger) in pthread current implementations, likewise the whole idea of customizable scheduler and kernel allocator weren't and aren't new nor bad at all, neither for us nor were for SPINers; remember the CSA Occam2 TRAC sourced OS SuperSetPlus machine which had from 16 up to 256 individual processors independently executable as according to their product info add it could be connected over LAN up to 16 Mac, PC, and Sun workstations, pretty much like a compilation and execution engine in one, pretty and good mussle to have to see running. They offered besides Occam2, C, C++, Modula-2, Pascal and FORTRAN, prolog and Ada with source level debuggers and Operating Systems as their Software products too. I hope this would be an scenario in some years to come, at least for any of us, if solet's see where and what this takes to/from us. The main question will be whether this guys will allow enogh detail to drive their GPGPU or CPU, I hope this history doesn't repeat like it is and wasn't in the past of machine OSes sources, i.e braking traditional development etc. I'm sorry for the long of the message. Thanks in advance --- El s?b, 12/2/11, Olaf Wagner escribi?: > De: Olaf Wagner > Asunto: Re: [M3devel] pthread_atfork > Para: m3devel at elegosoft.com > Fecha: s?bado, 12 de febrero, 2011 07:02 > I think it would be acceptable to say > that pthread threading is > broken on FreeBSD6 (as it is not officially supported any > more). > All the stuff below should be documented in the 'known > problems' > section. Especially the different behaviours of fork for > user and pthreads. This should also be mentioned in the M3 > interfaces (source code)! > > Also that to use use threads with FreeBSD requires to > remove the > -pthread flag, as Mika noted. > > BTW, do we have a chapter explaining what to do to use user > threads > instead of the default pthreads? > > Olaf > > PS: Hint: this would be a great task for anybody lurking on > the lists > and trying to get involved :-) > > Quoting Jay K : > > > > > I wrote that. > > > > As I recall, FreeBSD < 6 doesn't have > pthread_atfork. > > > > I think that was based on reading the man pages. > > > > So, yes. What would you suggest? > > > > > > > > > > > > All this atfork stuff came about as part of getting > cvsupd to work. > > > > With pthreads/kernel threads. It > historically worked, with > > Modula-3 user threads. > > > > Modula-3 user threads have this funny > property that all threads > > survive work. > > > > With pthreads/kernel threads, fork > gives you a new process with > > just one thread. > > > > (Solaris has fork1 and forkall, and fork > maybe used to be > > forkall, but now it is fork1). > > > > > > > > > > > > cvsupd is one of those slightly unusual programs that > does "fork and > > then do more work" > > > > as supposed to the more typical "fork and exec". > > > > > > > > Which is to say..I don't know. Maybe, don't expect > pthreads to work > > on FreeBSD < 6? > > > > At least not in a process that ever calls fork? Or > does "fork and > > then more work"? > > > > > > > > > > > > In reality, i think a lot of libraries have trouble > with "fork and > > then do more work", but > > > > I don't know. I recall vague notices in the > Apple/Darwin manpages > > not to assume > > > > this works -- the obvious implication that there is > plenty of use of > > pthreads e.g. mutexes, > > > > but relatively little use of pthread_atfork. > > > > > > > > > > > > You could perhaps replace all uses of fork with a > function that > > called the fork handlers, > > > > then fork, then the other handlers. That is probably > what I meant in > > the comment > > > > about this being easy to fix. > > > > You might though then run into the problem that > callers of fork can't do much > > > > before exec...oh, that's vfork I bet actually. > > > > > > > > > > > > - Jay > > > > > > > > > > > > > >> To: hosking at cs.purdue.edu > >> Date: Fri, 11 Feb 2011 19:05:25 -0800 > >> From: mika at async.caltech.edu > >> CC: m3devel at elegosoft.com; > jay.krell at cornell.edu > >> Subject: [M3devel] pthread_atfork > >> > >> Tony, > >> > >> What's RTProcessC.c doing? > >> > >> I note the following cryptic comment: > >> > >> /* NOTE: Even userthreads now depends > >> * on availability of pthreads. > >> * This can be fixed if need be. > >> */ > >> > >> now further down we can read: > >> > >> #if defined(_WIN32) \ > >> || > defined(__vms) \ > >> || > (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > >> return 0; > >> #else > >> while (1) > >> { > >> int i = > pthread_atfork(prepare, parent, child); > >> if (i != EAGAIN) > >> return i; > >> sleep(0); > >> } > >> #endif > >> > >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" > system), > >> RTProcess.RegisterForkHandlers does nothing? > >> > >> Hmmm..... > >> > >> Mika > > > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 > 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | > Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | > USt-IdNr: DE163214194 > > From jay.krell at cornell.edu Sat Feb 12 22:45:18 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 21:45:18 +0000 Subject: [M3devel] pthread_atfork In-Reply-To: <651549.53394.qm@web29714.mail.ird.yahoo.com> References: <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com>, <651549.53394.qm@web29714.mail.ird.yahoo.com> Message-ID: > so the behaviour can be selected better at runtime if more than one available processor I would like user vs. kernel threads to be selectable at link and/or run-time, but Tony has nixed the small changes required to make it so. It has also been suggested you could copy around different m3core.so files to switch, but that isn't the case either. Thread.T has a different typeid in the two systems. Anyway, the better behavior is almost always kernel threads. You should really just generally stick with them. You do forever ongoing debate on these matters, and the "pendulum" has seemingly swung many times, so I can't really say everyone agrees in kernel threads always. Windows 7 adds "user mode scheduling", which is still kernel threads -- there is still a one-to-one mapping between some usermode state and kernelmode state, but the scheduling decisions are made in usermode. I believe pthreads on OpenVMS use the "m:n" approach. But Linux, Solaris by default, Darwin, I suspect but am not sure FreeBSD and NetBSD, Win32 all use the simple "1:1" model. Where every thread is a kernel thread. All scheduling is done by the kernel. Win32 also has "fibers" but these are fairly...surprisingly difficult to use, but in a way that Tony et. al. wouldn't be surprised by. When you use fibers, you have to write your own scheduler, and your own locking primitives. Normal Win32 functions like GetCurrentThreadId(), EnterCriticalSection() become sort of useless/broken in the face of fibers. Basically the same as what it takes to implement Modula-3 user threads. Though you can run on multiple processors I believe fairly easily. Hm. The "m:n" model is actually not difficult to implement on Win32 and w/ pthreads. But still, probably not a good idea. - Jay > Date: Sat, 12 Feb 2011 21:08:13 +0000 > From: dabenavidesd at yahoo.es > To: m3devel at elegosoft.com; wagner at elegosoft.com > Subject: Re: [M3devel] pthread_atfork > > Hi all: > by the way you told, do user space interfaces need to be linked differently from user space threads library than the one in kernel threads, so the behaviour can be selected better at runtime if more than one available processor (i.e a virtualized one, perhaps two if there are virtualized GPU, GPGPU, etc) is available on demand (in example the collector has its own user space thread allocated control and one in kernel space that user can control from the Runtime), could it work in such a way to allow a Thread to live in spaces that are accesible through like an NetObj.T, thus allowing it to communciate them through such an abstraction? > Perhaps this could allow to run the kernel threads with Modula-3 governance and if so put them to work or switch to kernel them to accelerate its process according to its demands and such debug them in a special API like the one in the SPIN OS, fairly more comprenhensive thread implementation of Mach, Cthreads (aka C Pthreads calls) and own strends. Also scheduling could be handled to manage both user/kernel level synchronization, the threads and also allow to change from having a specialized hardware-assisted virtualized processor to a non hardware-assissted virtualized processor (like an NetObj.T again), which is like the same situation for Modula-3 with just one processor so in case it needs, it does so. In any case thread and object migration are safely defined over Obliq. > Granted, this is same situation like was before the enter of virtualization hardware but again the whole idea of hardware-assisted virtualization was not new, even there is an article in which a PDP-10 is claimed has requirements to make it virtual in hardware (and I believe in software too). So SPIN could be a platform, like for hardware assissted virtualization and/or operating-system level virtualization, if so, we would be able to trace system calls for a given OS client running on it, rather than barely, so it makes the implementation with hardware-assisted source-level debugger ready, which is related to where programs are segfaulting now (no system-level breakpoints, that is, no hardware assisted source-level debugger for virtualization user, different from Modula-3 situation in current user-level space threading or SPIN gdb DEC TTD Topaz Tele Debugger at source-level service and not to forget the ldb cross-platform source-level remote debugger) > in pthread current implementations, likewise the whole idea of customizable scheduler and kernel allocator weren't and aren't new nor bad at all, neither for us nor were for SPINers; remember the CSA Occam2 TRAC sourced OS SuperSetPlus machine which had from 16 up to 256 individual processors independently executable as according to their product info add it could be connected over LAN up to 16 Mac, PC, and Sun workstations, pretty much like a compilation and execution engine in one, pretty and good mussle to have to see running. They offered besides Occam2, C, C++, Modula-2, Pascal and FORTRAN, prolog and Ada with source level debuggers and Operating Systems as their Software products too. I hope this would be an scenario in some years to come, at least for any of us, if solet's see where and what this takes to/from us. > The main question will be whether this guys will allow enogh detail to drive their GPGPU or CPU, I hope this history doesn't repeat like it is and wasn't in the past of machine OSes sources, i.e braking traditional development etc. > I'm sorry for the long of the message. > Thanks in advance > > --- El s?b, 12/2/11, Olaf Wagner escribi?: > > > De: Olaf Wagner > > Asunto: Re: [M3devel] pthread_atfork > > Para: m3devel at elegosoft.com > > Fecha: s?bado, 12 de febrero, 2011 07:02 > > I think it would be acceptable to say > > that pthread threading is > > broken on FreeBSD6 (as it is not officially supported any > > more). > > All the stuff below should be documented in the 'known > > problems' > > section. Especially the different behaviours of fork for > > user and pthreads. This should also be mentioned in the M3 > > interfaces (source code)! > > > > Also that to use use threads with FreeBSD requires to > > remove the > > -pthread flag, as Mika noted. > > > > BTW, do we have a chapter explaining what to do to use user > > threads > > instead of the default pthreads? > > > > Olaf > > > > PS: Hint: this would be a great task for anybody lurking on > > the lists > > and trying to get involved :-) > > > > Quoting Jay K : > > > > > > > > I wrote that. > > > > > > As I recall, FreeBSD < 6 doesn't have > > pthread_atfork. > > > > > > I think that was based on reading the man pages. > > > > > > So, yes. What would you suggest? > > > > > > > > > > > > > > > > > > All this atfork stuff came about as part of getting > > cvsupd to work. > > > > > > With pthreads/kernel threads. It > > historically worked, with > > > Modula-3 user threads. > > > > > > Modula-3 user threads have this funny > > property that all threads > > > survive work. > > > > > > With pthreads/kernel threads, fork > > gives you a new process with > > > just one thread. > > > > > > (Solaris has fork1 and forkall, and fork > > maybe used to be > > > forkall, but now it is fork1). > > > > > > > > > > > > > > > > > > cvsupd is one of those slightly unusual programs that > > does "fork and > > > then do more work" > > > > > > as supposed to the more typical "fork and exec". > > > > > > > > > > > > Which is to say..I don't know. Maybe, don't expect > > pthreads to work > > > on FreeBSD < 6? > > > > > > At least not in a process that ever calls fork? Or > > does "fork and > > > then more work"? > > > > > > > > > > > > > > > > > > In reality, i think a lot of libraries have trouble > > with "fork and > > > then do more work", but > > > > > > I don't know. I recall vague notices in the > > Apple/Darwin manpages > > > not to assume > > > > > > this works -- the obvious implication that there is > > plenty of use of > > > pthreads e.g. mutexes, > > > > > > but relatively little use of pthread_atfork. > > > > > > > > > > > > > > > > > > You could perhaps replace all uses of fork with a > > function that > > > called the fork handlers, > > > > > > then fork, then the other handlers. That is probably > > what I meant in > > > the comment > > > > > > about this being easy to fix. > > > > > > You might though then run into the problem that > > callers of fork can't do much > > > > > > before exec...oh, that's vfork I bet actually. > > > > > > > > > > > > > > > > > > - Jay > > > > > > > > > > > > > > > > > > > > >> To: hosking at cs.purdue.edu > > >> Date: Fri, 11 Feb 2011 19:05:25 -0800 > > >> From: mika at async.caltech.edu > > >> CC: m3devel at elegosoft.com; > > jay.krell at cornell.edu > > >> Subject: [M3devel] pthread_atfork > > >> > > >> Tony, > > >> > > >> What's RTProcessC.c doing? > > >> > > >> I note the following cryptic comment: > > >> > > >> /* NOTE: Even userthreads now depends > > >> * on availability of pthreads. > > >> * This can be fixed if need be. > > >> */ > > >> > > >> now further down we can read: > > >> > > >> #if defined(_WIN32) \ > > >> || > > defined(__vms) \ > > >> || > > (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > > >> return 0; > > >> #else > > >> while (1) > > >> { > > >> int i = > > pthread_atfork(prepare, parent, child); > > >> if (i != EAGAIN) > > >> return i; > > >> sleep(0); > > >> } > > >> #endif > > >> > > >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" > > system), > > >> RTProcess.RegisterForkHandlers does nothing? > > >> > > >> Hmmm..... > > >> > > >> Mika > > > > > > > > > > > -- > > Olaf Wagner -- elego Software Solutions GmbH > > > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > > phone: +49 30 23 45 86 96 mobile: +49 177 2345 > > 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | > > Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > > Handelregister: Amtsgericht Charlottenburg HRB 77719 | > > USt-IdNr: DE163214194 > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 22:50:28 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 21:50:28 +0000 Subject: [M3devel] Performance issues with Process.Wait under userthreads In-Reply-To: <20110212204302.19C781A2078@async.async.caltech.edu> References: <20110212204302.19C781A2078@async.async.caltech.edu> Message-ID: I don't believe there is a good solution here, other than using pthreads. Is there? Maybe tune the wait down smaller? I understand it stinks either way. I don't understand the SIGCHLD stuff, at a quick glance. Nor your changes. However you might be the only user of user threads. Is there any downside to your change? (Other than more code, ok.) Any loss of performance in any situation? Any loss of portability? - Jay > To: m3devel at elegosoft.com > Date: Sat, 12 Feb 2011 12:43:02 -0800 > From: mika at async.caltech.edu > Subject: [M3devel] Performance issues with Process.Wait under userthreads > > Hi again m3devel (especially Tony), > > I have finally taken a bite of a problem that has been annoying me > for a very, very long time. > > Under user threads, the code for Process.Wait is as follows (see > ThreadPosix.m3): > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > CONST Delay = 0.1D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > Pause(Delay); > END; > END WaitProcess; > > It inserts a 0.1 second delay after each failed waitpid. This is extremely > annoying for programs that start a long sequence of child processes and > wait for them in sequence. Namely, the compiler itself. As a result > the cm3 compiler (and PM3's m3build) are normally very very slow when > using user threads. For about the last ten years, I've had a hacked > up m3build (for my PM3 installation) that skips the Pause and busy-waits > instead. > > Note there is another problem here. Since the Modula-3 runtime ignores > SIGCHLD, no zombie processes are created since the Unix system automatically > reaps the child processes. I can see this would be a problem since PIDs > are eventually reused and ... couldn't Uexec.waitpid wind up referring to > the wrong process?? > > I will further note that the comment in Process.i3 reads as follows: > > PROCEDURE Wait(p: T): ExitCode; > > Wait until the process with handle p terminates, then free the operating system resources associated with the process and return an exit code indicating the reason for its termination. It is a checked runtime error to call Wait twice on the same process handle. > > I am going to take this as fair warning that Process.Create *may* use > resources that are not going to be released until Process.Wait has been > called. > > I have modified (in my local copy of CM3) the system as follows. > I have come up with a semi-general mechanism for immediately unblocking > a thread on the receipt of a unix signal. > > 1. The system relies on changing > ThreadPosix.XPause such that if a signal is allowed to wake up a threads, > that fact is recorded in a new field in the thread's descriptor > record (of type ThreadPosix.T). > > 2. On receipt of a waited-for unix signal, a mask is set and control > is passed to the thread scheduler which maintains the non-zero mask for > exactly one iteration through the thread ring. > > 3. If a thread is paused and waiting for EITHER a signal or some time, > the thread is released for running and the thread's waiting state is > cleared. > > The changes are more or less as follows: > > 1. I have added a new field of type "int" to ThreadPosix.T: > > (* if state = pausing, the time at which we can restart *) > waitingForTime: Time.T; > > + (* if state = pausing, the signal that truncates the pause *) > + waitingForSig: int := -1; > + > (* true if we are waiting during an AlertWait or AlertJoin > or AlertPause *) > alertable: BOOLEAN := FALSE; > > > 2. Modifications to pause: > > + PROCEDURE SigPause(n: LONGREAL; sig: int)= > + <*FATAL Alerted*> > + VAR until := n + Time.Now (); > + BEGIN > + XPause(until, FALSE, sig); > + END SigPause; > + > PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= > VAR until := n + Time.Now (); > BEGIN > XPause(until, TRUE); > END AlertPause; > > ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) > ! RAISES {Alerted} = > BEGIN > INC (inCritical); > self.waitingForTime := until; > self.alertable := alertable; > + IF sig # -1 THEN > + self.waitingForSig := sig > + END; > ICannotRun (State.pausing); > DEC (inCritical); > InternalYield (); > > 3. The received-signals mask: > > ! CONST MaxSigs = 64; > ! TYPE Sig = [ 0..MaxSigs-1 ]; > ! > ! (* in order to listen to other signals, they have to be enabled in > ! allow_sigvtalrm as well *) > ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); > ! > ! gotSigs := SET OF Sig { }; > ! > > ValueOfSIGCHLD() is a C function used to get the value of the SIGCHLD > constant without guessing at it (in ThreadPosixC.c). > > 4. changes to the signal handler: > > ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); > ! > ! INC(inCritical); > ! (* mark signal as being delivered *) > ! IF sig >= 0 AND sig < MaxSigs THEN > ! gotSigs := gotSigs + SET OF Sig { sig } > ! END; > ! DEC(inCritical); > ! > ! IF inCritical = 0 AND heapState.inCritical = 0 THEN > ! InternalYield () > ! END; > END switch_thread; > > Note that I don't know if INC/DEC(inCritical) does exactly the right > thing here. > > 5. changes to the scheduler: > > a. thread wakeup > IF t.alertable AND t.alertPending THEN > CanRun (t); > EXIT; > + > + ELSIF t.waitingForSig IN gotSigs THEN > + t.waitingForSig := -1; > + CanRun(t); > + EXIT; > > ELSIF t.waitingForTime <= now THEN > CanRun (t); > EXIT; > ! > > b. clearing the mask > > END; > END; > > + gotSigs := SET OF Sig {}; > + > IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN > IF perfOn THEN PerfRunning (t.id); END; > (* At least one thread wants to run; transfer to it *) > > 6. changes to WaitProcess (Process.Wait): > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > ! CONST Delay = 10.0D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > ! SigPause(Delay,SIGCHLD); > END; > END WaitProcess; > > 7. install signal handler even if program is single-threaded: > > BEGIN > + (* we need to call set up the signal handler for other reasons than > + just thread switching now *) > + setup_sigvtalrm (switch_thread); > END ThreadPosix. > > 8. modify signal handler in ThreadPosixC.c to catch SIGCHLD: > > sigemptyset(&ThreadSwitchSignal); > sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); > + sigaddset(&ThreadSwitchSignal, SIGCHLD); > > act.sa_handler = handler; > act.sa_flags = SA_RESTART; > sigemptyset(&(act.sa_mask)); > if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); > + if (sigaction (SIGCHLD, &act, NULL)) abort(); > > I'll send the complete diff in a separate message for those who want > to study it more closely. > > I propose the above changes for inclusion in the current CM3 repository. > > Mika > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 23:08:42 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 22:08:42 +0000 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212202550.542D41A2078@async.async.caltech.edu> References: <20110212030525.822411A2078@async.async.caltech.edu>, , <20110212165930.DC32C1A2078@async.async.caltech.edu>, <20110212205527.e6xetojo0ssgocg4@mail.elegosoft.com>, <20110212202550.542D41A2078@async.async.caltech.edu> Message-ID: "fork() and do more work" is portable, to Posix, which is many systems. It is easily done in Modula-3 by calling fork(). Given a choice between fixing fork() and do more work, vs. changing cvsupd to not depend on it, and given what I know about fixing fork() and do more work, I chose fork() and do more work. Or just abandon cvsupd. cvsupd is far from the only user of this pattern, in the wider non-Modula-3 world. > But why would user threads work? We're skipping pthread_atfork for them too. It is quite possible that fork and do more work + user threads + FreeBSD < 6, doesn't work. It is quite possible that it works intermittently. Let me try to explain what pthread_atfork is for. Though the opengroup spec does discuss it in detail and is clear (Bing for "opengroup pthread_atfork"; the first hit is it). In a multithreaded system, arbitrary mutexes are held at arbitrary times. Aribrary libraries spawn arbitrary worker threads, and use mutexes therein. As well, such programs might fork and do more work. There is not necessarily any central arbitrator among worker threads and the thread that calls fork(). That is, when you fork, you might be holding multiple mutexes. There isn't a way for the forking thread to request all other threads to enter some quiescent state in which no mutexes are held. If you fork and do more work, the new process has only one thread, the thread that called fork(). But any mutex ownership at the time of fork() is inherited. As a result, attemps to use mutexes in the new child process will have a high propensity to hang/deadlock. Therefore...go back..and do introduce an arbitrator. The implementation of fork() is therefore integrated with the implementation of pthreads. pthread_atfork establishes a set of callbacks. 3 sets actually. fork() calls them at the appropriate times -- before fork in the parent, and after fork in both the parent and the child. What the callbacks are meant to do is acquire all locks in the parent before the fork, and release them all in the parent and child after the work. And possibly reinitialize globals in the child after the fork. In the Modula-3 user thread system, threads exist by virtue of having memory allocated for their stack, small memory allocated for their processor context/registers/etc., and by being on a global linked list. Scheduling is triggered by an occasional timer. Historically, upon fork(), all that memory and the global linked list were unchanged. Therefore the child process would continue to have all the same threads as the parent. But compare this to pthreads. Solaris has an option to be either way, but in everything else I've seen -- OpenBSD, FreeBSD, NetBSD, Darwin, Linux -- and I think Cygwin, HP-UX, Irix, AIX, and probably the Posix spec -- fork() is specified as creating a child process with only one thread. Solaris has fork1() and forkall() and the behavior of plain fork() has varied from release to release. So, again, upon fork(), historical Modula-3 user threads retain all threads. If a thread held a mutex at fork() time, that's ok, it will continue running in the child and eventually release the mutex. No hang. If fork() is followed shortly by exec, well, then indeed, even with user threads, alll threads are gone, globals get reinitialized (to zero), there is a new thread. I guess you just have to be a bit careful what you do between fork() and exec(). Notice that things like Process.Create() suspend scheduling and probably after that avoid allocating any memory and possibly avoid touching garbage collected memory and probably avoid taking any locks. That is, upon suspending scheduling, they don't depend on any mutex to need to be released by any other thread. The various threads/mutexes can be in an arbitrary state. In the child they will go away shortly (upon exec). In the parent, scheduling will shortly resume. Does some of this make sense? > It is quite possible that fork and do more work + user threads + FreeBSD < 6, doesn't work. Historically it did work -- child process retained all the threads. These days I expect..I expect you still get that same behavior. We could fix it, I believe, by never calling fork() directly. Have a wrapper Process.Fork(). On most systems, it would just call fork. But on OpenVMS and FreeBSD < 6...there is also the Register function. On most systems, it'd just call pthread_atfork. On OpenVMS and FreeBSD < 6, it should maintain the lists itself. And then Process.Fork on those systems, make the call outs. Indeed, the OpenVMS/FreeBSD<6 implementation could be the one and only implementation. If fork() was disallowed and Process.Fork() the only allowed replacement. Or, it could be tested that way, and then actually commited the other way. I very well might do this. - Jay > To: wagner at elegosoft.com > Date: Sat, 12 Feb 2011 12:25:50 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] pthread_atfork > > How do you do "fork-and-do-work" from Modula-3? > > All my programs that create new Unix processes do it with Process.Create, > which as far as I know does a fork-and-exec... > > I'm not sure what if anything is broken here, either. But then again I > didn't fully understand Jay's last mail on the subject. > > Mika > > Olaf Wagner writes: > >Well, I think actually the fork-and-do-work pattern has been > >the default for a long time with the M3 user threads implementation. > >It wasn't specified though, and I think no application except > >CVSup did use it. > > > >Jay is not responsible for any breakage here, he just tried to fix > >CVSup and establish a consistent behaviour after our change to > >system pthreads on most platforms. > > > >Olaf > > > >Quoting Mika Nystrom : > > > >> Hi Jay, > >> > >> I think "fork and do work" qualifies as a bug in the application, if > >> the application wants to be portable. Why can't it fork and exec instead? > >> If it wants to do more work it should just fork and exec another copy > >> of itself. > >> > >> fork() is also not part of Modula-3... Process.Create() is, however. > >> Don't tell me you broke that as part of "fixing" things for cvsupd... > >> > >> There is still one aspect of your comment I don't understand. What does > >> the call to pthread_atfork actually accomplish when running with user > >> threads? You say below that "don't expect pthreads to work on FreeBSD < > >> 6 [using the fork-and-do-work pattern]". Fair enough. But why would > >> user threads work? We're skipping pthread_atfork for them too. > >> > >> Do we not need pthread_atfork with user threads? What you're saying > >> suggests that FreeBSD >=3D 6 and all other platforms need pthread_atfork > >> with user threads. Is that really what you mean? > >> > >> At the moment I'm just concerned with getting user threads to work as > >> I know there are bugs with pthreads. > >> > >> I also think we should not waste time and effort making the "fork and do > >> work" pattern work unless it's really, really necessary (which I really > >> don't see why it would be). Fix the application instead of introducing > >> bugs in the libraries and language implementation! > >> > >> I've been trying for a long, long time to switch from PM3 to CM3 (almost > >> ten years, I think). These bugs from "enhancements" have stopped me > >> every time. Maybe things are finally working well enough... > >> > >> (some comments inline in your quoted text below) > >> > >> Mika > >> > >> > >> Jay K writes: > >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ > >>> Content-Type: text/plain; charset=3D"iso-8859-1" > >>> Content-Transfer-Encoding: quoted-printable > >>> > >>> > >>> I wrote that. > >>> > >>> As I recall=3D2C FreeBSD < 6 doesn't have pthread_atfork. > >>> > >>> I think that was based on reading the man pages. > >>> > >>> So=3D2C yes. What would you suggest? > >>> > >>> > >>> > >>> > >>> > >>> All this atfork stuff came about as part of getting cvsupd to work. > >>> > >>> With pthreads/kernel threads. It historically worked=3D2C with Modula-3 = > >use=3D > >>> r threads. > >>> > >>> Modula-3 user threads have this funny property that all threads survive = > >w=3D > >>> ork. > >> > >> you mean survive fork don't you? > >> > >>> > >>> With pthreads/kernel threads=3D2C fork gives you a new process with just= > > on=3D > >>> e thread. > >>> > >>> (Solaris has fork1 and forkall=3D2C and fork maybe used to be forkall= > >=3D2C b=3D > >>> ut now it is fork1). > >>> > >>> > >>> > >>> > >>> > >>> cvsupd is one of those slightly unusual programs that does "fork and then= > > d=3D > >>> o more work" > >>> > >>> as supposed to the more typical "fork and exec". > >>> > >>> > >>> > >>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o= > >n Fr=3D > >>> eeBSD < 6? > >>> > >>> At least not in a process that ever calls fork? Or does "fork and then mo= > >re=3D > >>> work"? > >>> > >>> > >>> > >>> > >>> > >>> In reality=3D2C i think a lot of libraries have trouble with "fork and th= > >en d=3D > >>> o more work"=3D2C but > >>> > >>> I don't know. I recall vague notices in the Apple/Darwin manpages not to = > >as=3D > >>> sume > >>> > >>> this works -- the obvious implication that there is plenty of use of pthr= > >ea=3D > >>> ds e.g. mutexes=3D2C > >>> > >>> but relatively little use of pthread_atfork. > >>> > >>> > >>> > >>> > >>> > >>> You could perhaps replace all uses of fork with a function that called th= > >e =3D > >>> fork handlers=3D2C > >>> > >>> then fork=3D2C then the other handlers. That is probably what I meant in = > >the =3D > >>> comment > >>> > >>> about this being easy to fix. > >>> > >>> You might though then run into the problem that callers of fork can't do = > >mu=3D > >>> ch > >>> > >>> before exec...oh=3D2C that's vfork I bet actually. > >> > >> I don't understand this paragraph either :( > >> > >>> > >>> > >>> > >>> > >>> > >>> - Jay > >>> > >>> > >>> > >>> > >>> > >>> > >>>> To: hosking at cs.purdue.edu > >>>> Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800 > >>>> From: mika at async.caltech.edu > >>>> CC: m3devel at elegosoft.com=3D3B jay.krell at cornell.edu > >>>> Subject: [M3devel] pthread_atfork > >>>> =3D20 > >>>> Tony=3D2C > >>>> =3D20 > >>>> What's RTProcessC.c doing? > >>>> =3D20 > >>>> I note the following cryptic comment: > >>>> =3D20 > >>>> /* NOTE: Even userthreads now depends > >>>> * on availability of pthreads. > >>>> * This can be fixed if need be. > >>>> */ > >>>> =3D20 > >>>> now further down we can read: > >>>> =3D20 > >>>> #if defined(_WIN32) \ > >>>> || defined(__vms) \ > >>>> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > >>>> return 0=3D3B > >>>> #else > >>>> while (1) > >>>> { > >>>> int i =3D3D pthread_atfork(prepare=3D2C parent=3D2C child)=3D3B > >>>> if (i !=3D3D EAGAIN) > >>>> return i=3D3B > >>>> sleep(0)=3D3B > >>>> } > >>>> #endif > >>>> =3D20 > >>>> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=3D2C > >>>> RTProcess.RegisterForkHandlers does nothing? > >>>> =3D20 > >>>> Hmmm..... > >>>> =3D20 > >>>> Mika > >>> =09=09 =09 =09=09 =3D > >>> > >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ > >>> Content-Type: text/html; charset=3D"iso-8859-1" > >>> Content-Transfer-Encoding: quoted-printable > >>> > >>> > >>> > >>> > >>> > >>> > >>> I wrote that.
> >>> As I recall=3D2C FreeBSD <=3D3B 6 doesn't have pthread_atfork.
> >>> I think that was based on reading the man pages.
> >>> So=3D2C yes. What would you suggest?
> >>>
> >>>
> >>> All this atfork stuff came about as part of getting cvsupd to work.
> >>>  =3D3B With pthreads/kernel threads. It historically worked=3D2C with= > > Modul=3D > >>> a-3 user threads.
> >>>  =3D3B Modula-3 user threads have this funny property that all thread= > >s su=3D > >>> rvive work.
> >>>  =3D3B With pthreads/kernel threads=3D2C fork gives you a new process= > > with =3D > >>> just one thread.
> >>>  =3D3B =3D3B (Solaris has fork1 and forkall=3D2C and fork maybe u= > >sed to b=3D > >>> e forkall=3D2C but now it is fork1).
> >>>
> >>>
> >>> cvsupd is one of those slightly unusual programs that does "fork and then= > > d=3D > >>> o more work"
> >>> as supposed to the more typical "fork and exec".
> >>>
> >>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o= > >n Fr=3D > >>> eeBSD <=3D3B 6?
> >>> At least not in a process that ever calls fork? Or does "fork and then mo= > >re=3D > >>> work"?
> >>>
> >>>
> >>> In reality=3D2C i think a lot of libraries have trouble with "fork and th= > >en d=3D > >>> o more work"=3D2C but
> >>> I don't know. I recall vague notices in the Apple/Darwin manpages not to = > >as=3D > >>> sume
> >>> this works -- the obvious implication that there is plenty of use of pthr= > >ea=3D > >>> ds e.g. mutexes=3D2C
> >>> but relatively little use of pthread_atfork.
> >>>
> >>>
> >>> You could perhaps replace all uses of fork with a function that called th= > >e =3D > >>> fork handlers=3D2C
> >>> then fork=3D2C then the other handlers. That is probably what I meant in = > >the =3D > >>> comment
> >>> about this being easy to fix.
> >>> You might though then run into the problem that callers of fork can't do = > >mu=3D > >>> ch
> >>> before exec...oh=3D2C that's vfork I bet actually.
> >>>
> >>>
> >>>  =3D3B- Jay






>=3D3B To: hosking at cs.purdue= > >.edu >>> r>>=3D3B Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800
>=3D3B From: mi= > >ka at async=3D > >>> .caltech.edu
>=3D3B CC: m3devel at elegosoft.com=3D3B jay.krell at cornell= > >.edu >>> r>>=3D3B Subject: [M3devel] pthread_atfork
>=3D3B
>=3D3B Ton= > >y=3D2C
=3D > >>> >=3D3B
>=3D3B What's RTProcessC.c doing?
>=3D3B
>=3D3B= > > I note th=3D > >>> e following cryptic comment:
>=3D3B
>=3D3B /* NOTE: Even usert= > >hreads=3D > >>> now depends
>=3D3B * on availability of pthreads.
>=3D3B * Th= > >is ca=3D > >>> n be fixed if need be.
>=3D3B */
>=3D3B
>=3D3B now furth= > >er down =3D > >>> we can read:
>=3D3B
>=3D3B #if defined(_WIN32) \
>=3D3B = > > |=3D > >>> | defined(__vms) \
>=3D3B || (defined(__FreeBSD__) &=3D3B= > >&=3D > >>> =3D3B (__FreeBSD__ <=3D3B 6))
>=3D3B return 0=3D3B
>=3D3B= > > #else
&g=3D > >>> t=3D3B while (1)
>=3D3B {
>=3D3B int i =3D3D pthr= > >ead_atfork=3D > >>> (prepare=3D2C parent=3D2C child)=3D3B
>=3D3B if (i !=3D3D EAGA= > >IN)
>=3D > >>> =3D3B return i=3D3B
>=3D3B sleep(0)=3D3B
>=3D3B = > > }
>=3D > >>> =3D3B #endif
>=3D3B
>=3D3B so on FreeBSD 5 (what I'm running o= > >n my "Fr=3D > >>> eeBSD4" system)=3D2C
>=3D3B RTProcess.RegisterForkHandlers does noth= > >ing? >>> r>>=3D3B
>=3D3B Hmmm.....
>=3D3B
>=3D3B Mika
> >>> =09 =09 =09=09 =3D > >>> > >>> =3D > >>> > >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_-- > >> > > > > > > > >--=20 > >Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa= > >ny > >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz= > >: Berlin > >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sun Feb 13 00:27:43 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 12 Feb 2011 18:27:43 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <20110212015715.A43471A2078@async.async.caltech.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu> Message-ID: <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> Mika, I have no idea what the issue is here. But I do note that AMD64_LINUX with pthreads used to run just fine. I ran a bunch or stress tests when first ported to that target and all seemed good. What is preventing you using pthreads on this target? -- Tony On Feb 11, 2011, at 8:57 PM, Mika Nystrom wrote: > I would like to find one configuration (any configuration!!) that lets > me run a multithreaded program under AMD64_LINUX. I don't care what > version of the compiler it is, whether the threads are user or pthreads, > how many things I have to hack. Does anyone know what I need to do? From jay.krell at cornell.edu Sun Feb 13 00:37:12 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 23:37:12 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu>, <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> Message-ID: I assume he means "the thread tester should work"? But agreed, the vast vast vast vast majority of things are ok and have been ok for the vast vast vast majority of Modula-3's existance. Even in recent years. pthreads and user threads. Infinite loops with TRY in January 2011 would use infinite stack. Understood. Finite loops would merely use more than they should and reclaim upon function return. cvsupd didn't work for a while with pthreads. Because of its use of "fork and do more work" and its assumption that all threads survive fork. Relatively few programs do that and they were all broken by pthreads. The new thread tester probably needs more attention. - Jay > Subject: Re: [M3devel] threading on Windows? > From: hosking at cs.purdue.edu > Date: Sat, 12 Feb 2011 18:27:43 -0500 > CC: jay.krell at cornell.edu; m3devel at elegosoft.com > To: mika at async.caltech.edu > > Mika, > > I have no idea what the issue is here. But I do note that AMD64_LINUX with pthreads used to run just fine. I ran a bunch or stress tests when first ported to that target and all seemed good. What is preventing you using pthreads on this target? > > -- Tony > > On Feb 11, 2011, at 8:57 PM, Mika Nystrom wrote: > > > I would like to find one configuration (any configuration!!) that lets > > me run a multithreaded program under AMD64_LINUX. I don't care what > > version of the compiler it is, whether the threads are user or pthreads, > > how many things I have to hack. Does anyone know what I need to do? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 13 00:39:15 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 15:39:15 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu> <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> Message-ID: <20110212233915.A8EEE1A2078@async.async.caltech.edu> Tony, I have yet to find a single system on which my thread testing program actually runs properly under pthreads. I have however managed to get it to work everywhere with user threading. My last email to you on the issue said the following: read: segfault after a long time fork: OK alloc: deadlock creat: appears OK for a long time then hangs hard (can't get into debugger) lock: OK I think Jay has now fixed the "read" problem, which had nothing to do with threading but was a TRY-EXCEPT issue. The fact that the runtime deadlocks when NEWing objects I think is the most serious problem. Could you try it, please? If you run "threadtest -n 10 -tests all" you should see something like this (the following is AMD64_FREEBSD with user threading---the numbers denote how much threads are lagging, in seconds). The program is utterly standard Modula-3. /big/home/mika/cm3-writable/cm3/m3-libs/m3core/tests/thread/AMD64_FREEBSD/threadtest -n 10 -tests all Writing file...done Creating read threads...done Creating nxread threads...done Creating tryexcept threads...done Creating fork threads...done Creating alloc threads...done Creating creat threads...done Creating lock threads...done running...printing oldest/median age/newest ..........laziest thread is 56/13/0 (tests: read 1/1/0 nxread 3/2/2 tryexcept 12/8/3 fork 13/13/13 alloc 17/15/13 creat 38/17/17 lock 56/18/17) ..........laziest thread is 57/13/0 (tests: read 2/1/0 nxread 3/2/2 tryexcept 12/8/3 fork 13/13/13 alloc 17/15/13 creat 36/17/17 lock 57/36/17) ..........laziest thread is 40/13/0 (tests: read 1/1/0 nxread 3/2/1 tryexcept 12/8/3 fork 13/13/13 alloc 17/15/13 creat 37/17/17 lock 40/37/17) ..........laziest thread is 34/14/0 (tests: read 1/0/0 nxread 3/2/1 tryexcept 13/9/3 fork 14/14/14 alloc 17/15/14 creat 31/17/17 lock 34/31/17) ..........laziest thread is 56/13/0 (tests: read 2/1/0 nxread 3/2/2 tryexcept 12/8/3 fork 13/13/13 alloc 16/15/13 creat 37/16/16 lock 56/18/16) ..........laziest thread is 37/14/0 (tests: read 1/0/0 nxread 3/2/1 tryexcept 13/8/3 fork 14/14/14 alloc 18/15/14 creat 37/19/19 lock 37/19/19) ..........laziest thread is 34/15/0 (tests: read 2/1/0 nxread 3/3/2 tryexcept 14/9/3 fork 15/15/15 alloc 16/16/15 creat 34/17/17 lock 34/17/17) ..........laziest thread is 34/14/0 (tests: read 1/1/0 nxread 3/2/1 tryexcept 13/8/3 fork 14/14/14 alloc 16/15/14 creat 34/16/16 lock 34/16/16) ..........laziest thread is 35/14/0 (tests: read 2/1/0 nxread 3/2/2 tryexcept 13/9/3 fork 14/14/14 alloc 17/16/14 creat 35/17/17 lock 35/17/17) ..........laziest thread is 37/14/0 (tests: read 2/1/0 nxread 3/2/2 tryexcept 13/8/3 fork 14/14/14 alloc 16/15/14 creat 37/17/17 lock 37/17/17) All tests complete. Congratulations. Mika Tony Hosking writes: >Mika, > >I have no idea what the issue is here. But I do note that AMD64_LINUX = >with pthreads used to run just fine. I ran a bunch or stress tests when = >first ported to that target and all seemed good. What is preventing you = >using pthreads on this target? > >-- Tony > >On Feb 11, 2011, at 8:57 PM, Mika Nystrom wrote: > >> I would like to find one configuration (any configuration!!) that lets >> me run a multithreaded program under AMD64_LINUX. I don't care what >> version of the compiler it is, whether the threads are user or = >pthreads, >> how many things I have to hack. Does anyone know what I need to do? From mika at async.caltech.edu Sun Feb 13 01:33:57 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 16:33:57 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu>, <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> Message-ID: <20110213003358.00D2A1A2078@async.async.caltech.edu> Jay K writes: >--_879a340b-b2bb-415e-936c-be8d5cdd855c_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >I assume he means "the thread tester should work"? >But agreed=2C the vast vast vast vast majority of things are ok and have be= >en ok for the vast vast vast majority of Modula-3's existance. > Even in recent years. pthreads and user threads. > Infinite loops with TRY in January 2011 would use infinite stack. Underst= >ood. > Finite loops would merely use more than they should and reclaim upon func= >tion return. >cvsupd didn't work for a while with pthreads. Because of its use of "fork a= >nd do more work" and its assumption >that all threads survive fork. Relatively few programs do that and they wer= >e all broken by pthreads. >=20 >The new thread tester probably needs more attention. >=20 > - Jay My thread tester wasn't developed "ex nihilo". It's based on analyzing the failure modes of a larger application that I am prevented from distributing the source code for, both for reasons of confidentiality and practicality. I have purposely written the tester to be "obviously correct" and I would like nothing more than for someone to show me where I have made a mistake and that pthreads are working as they should. For example, here is the "alloc" thread: PROCEDURE AApply(cl : Closure) : REFANY = BEGIN Thread.Pause(InitPause); LOOP VAR arr := NEW(REF ARRAY OF INTEGER, 1025); BEGIN FOR i := FIRST(arr^)+1 TO LAST(arr^) DO arr[i] := arr[i] - arr[i-1] END; times1[cl.id] := FLOOR(Time.Now()) END END END AApply; Is there anything here that is wrong or bad? When I tried pthreads just now on AMD64_LINUX, running two instances of AApply at the same time caused the runtime to deadlock. [BTW, I have never seen a problem with user threads except for building issues and the TRY-EXCEPT thing you just fixed.] Unfortunately, since just about every Modula-3 program allocates memory, my analysis is that because it appears you can deadlock the runtime by allocating memory, nothing is *really* OK with pthreads. Please tell me I'm wrong! Mika P.S. backtraces of deadlock: Thread 3 (Thread 0x40862950 (LWP 19974)): #0 0x00002b9d49f730e1 in nanosleep () from /lib/libpthread.so.0 #1 0x000000000045806d in ThreadPThread__Nanosleep (req=Cannot access memory at address 0x140861b08 ) at ../src/thread/PTHREAD/ThreadPThreadC.c:500 #2 0x0000000000454294 in ThreadPThread__CommonSleep () at ../src/thread/PTHREAD/ThreadPThread.m3:551 #3 0x0000000000456438 in ThreadPThread__StopWorld () at ../src/thread/PTHREAD/ThreadPThread.m3:1086 #4 0x0000000000455787 in RTThread__SuspendOthers () at ../src/thread/PTHREAD/ThreadPThread.m3:812 #5 0x000000000043970c in RTCollector__CollectSomeInStateZero () at ../src/runtime/common/RTCollector.m3:746 #6 0x00000000004396b8 in RTCollector__CollectSome () at ../src/runtime/common/RTCollector.m3:720 #7 0x00000000004390b9 in RTHeapRep__CollectEnough () at ../src/runtime/common/RTCollector.m3:654 #8 0x0000000000436111 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=Cannot access memory at address 0x140861dc8 ) at ../src/runtime/common/RTAllocator.m3:363 #9 0x000000000043596f in RTAllocator__GetOpenArray (M3_Eic7CK_def=Cannot access memory at address 0x140861eb8 ) at ../src/runtime/common/RTAllocator.m3:292 #10 0x0000000000434ab1 in RTHooks__AllocateOpenArray (M3_AJWxb1_defn=Cannot access memory at address 0x140861f38 ) at ../src/runtime/common/RTAllocator.m3:141 #11 0x0000000000407030 in Main__AApply (M3_AP7a1g_cl=Cannot access memory at address 0x140861f88 ) at ../src/Main.m3:234 #12 0x0000000000453b3f in ThreadPThread__RunThread (M3_DMxDjQ_me=Cannot access memory at address 0x140862028 ) at ../src/thread/PTHREAD/ThreadPThread.m3:475 #13 0x00000000004537d3 in ThreadPThread__ThreadBase (M3_AJWxb1_param=Cannot access memory at address 0x1408620d8 ) at ../src/thread/PTHREAD/ThreadPThread.m3:447 #14 0x00002b9d49f6bfc7 in start_thread () from /lib/libpthread.so.0 #15 0x00002b9d4b3bd64d in clone () from /lib/libc.so.6 #16 0x0000000000000000 in ?? () Thread 2 (Thread 0x40661950 (LWP 19973)): #0 0x00002b9d49f72384 in __lll_lock_wait () from /lib/libpthread.so.0 #1 0x00002b9d49f6dbf0 in _L_lock_102 () from /lib/libpthread.so.0 #2 0x00002b9d49f6d4fe in pthread_mutex_lock () from /lib/libpthread.so.0 #3 0x000000000045815e in ThreadPThread__pthread_mutex_lock (m=Cannot access memory at address 0x140660d68 ) at ../src/thread/PTHREAD/ThreadPThreadC.c:557 #4 0x00000000004575af in RTOS__LockHeap () at ../src/thread/PTHREAD/ThreadPThread.m3:1434 #5 0x0000000000436107 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=Cannot access memory at address 0x140660dc8 ) at ../src/runtime/common/RTAllocator.m3:361 #6 0x000000000043596f in RTAllocator__GetOpenArray (M3_Eic7CK_def=Cannot access memory at address 0x140660eb8 ) at ../src/runtime/common/RTAllocator.m3:292 #7 0x0000000000434ab1 in RTHooks__AllocateOpenArray (M3_AJWxb1_defn=Cannot access memory at address 0x140660f38 ) at ../src/runtime/common/RTAllocator.m3:141 #8 0x0000000000407030 in Main__AApply (M3_AP7a1g_cl=Cannot access memory at address 0x140660f88 ) at ../src/Main.m3:234 #9 0x0000000000453b3f in ThreadPThread__RunThread (M3_DMxDjQ_me=Cannot access memory at address 0x140661028 ) at ../src/thread/PTHREAD/ThreadPThread.m3:475 #10 0x00000000004537d3 in ThreadPThread__ThreadBase (M3_AJWxb1_param=Cannot access memory at address 0x1406610d8 ) at ../src/thread/PTHREAD/ThreadPThread.m3:447 #11 0x00002b9d49f6bfc7 in start_thread () from /lib/libpthread.so.0 #12 0x00002b9d4b3bd64d in clone () from /lib/libc.so.6 #13 0x0000000000000000 in ?? () Thread 1 (Thread 0x2b9d4d32a1a0 (LWP 19965)): #0 0x00002b9d4b3202a2 in sigsuspend () from /lib/libc.so.6 #1 0x0000000000457ba9 in ThreadPThread__sigsuspend () at ../src/thread/PTHREAD/ThreadPThreadC.c:141 #2 0x0000000000456a0d in ThreadPThread__SignalHandler (M3_DLS2Hj_sig=Cannot access memory at address 0x80001dde057c ) at ../src/thread/PTHREAD/ThreadPThread.m3:1205 #3 #4 0x00002b9d49f6ffad in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #5 0x00000000004580b3 in ThreadPThread__pthread_cond_timedwait (cond=Cannot access memory at address 0x80001dde0a88 ) at ../src/thread/PTHREAD/ThreadPThreadC.c:515 #6 0x00000000004544cf in ThreadPThread__XPause (M3_DMxDjQ_self=Cannot access memory at address 0x80001dde0ab8 ) at ../src/thread/PTHREAD/ThreadPThread.m3:580 #7 0x00000000004545b2 in Thread__Pause (M3_CtKayy_n=Cannot access memory at address 0x80001dde0b08 ) at ../src/thread/PTHREAD/ThreadPThread.m3:595 #8 0x00000000004097ee in Main_M3 (M3_AcxOUs_mode=Cannot access memory at address 0x80001dde0bc8 ) at ../src/Main.m3:449 #9 0x0000000000445799 in RTLinker__RunMainBody (M3_DjPxE3_m=Cannot access memory at address 0x80001dde0f58 ) at ../src/runtime/common/RTLinker.m3:406 #10 0x00000000004448f2 in RTLinker__AddUnitI (M3_DjPxE3_m=Cannot access memory at address 0x80001dde0ff8 ) at ../src/runtime/common/RTLinker.m3:113 #11 0x0000000000444990 in RTLinker__AddUnit (M3_DjPxE5_b=Cannot access memory at address 0x80001dde1018 ) at ../src/runtime/common/RTLinker.m3:122 #12 0x000000000040652c in main (argc=Cannot access memory at address 0x80001dde1068 ) at _m3main.mc:4 From mika at async.caltech.edu Sun Feb 13 07:29:55 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 22:29:55 -0800 Subject: [M3devel] Performance issues with Process.Wait under userthreads In-Reply-To: References: <20110212204302.19C781A2078@async.async.caltech.edu> Message-ID: <20110213062955.4F84B1A2078@async.async.caltech.edu> Jay K writes: >--_5ec25af1-29a7-4f8a-b7ad-e61068941159_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >I don't believe there is a good solution here=2C other than using pthreads. >Is there? Maybe tune the wait down smaller? >I understand it stinks either way. Well as I keep saying, I'm looking for a reliable runtime. I really need this---and soon. I can't keep futzing around with PM3 forever. I'd hope pthreads is it but for now the only thing that passes my tests is user threading. We (group at Caltech) determined many years ago that tuning the wait does nothing, unless you set it to zero (busy waiting). Possibly this has to do with the time quantum of the user threads scheduler, I don't know. >=20 >=20 >I don't understand the SIGCHLD stuff=2C at a quick glance. The idea is this. We have SIGVTALRM coming in already to do threadswitching. So I am enabling SIGCHLD and then taking SIGCHLD the way SIGVTALRM is taken, that is, in the scheduler. The other change is that a paused thread can wait for a condition such as "either after X delay OR after Y signal" and then we allow the paused thread to become runnable on the Y signal rather than after the X delay. >Nor your changes. >However you might be the only user of user threads. >Is there any downside to your change? > (Other than more code=2C ok.) >Any loss of performance in any situation? >Any loss of portability? I am not sure. Is SIGCHLD completely standard? I am worried about one thing. The thread switcher turns signals back on as its first action. If a large number of children exit at the same time, I fear that we get a very deeply nested signal stack, which doesn't sound good. Perhaps signals should not be turned back on immediately. Or SIGCHLD should be set to SIG_IGN and then re-enabled? (Simulating old-fashioned "unreliable" SysV signals!) I'm not sure. In any case I was just trying to parallelize the compiler and ran into some very odd things related to having lots of child processes running around... The SIGCHLD change speeds up the CM3 compiler by about a factor of 3 when using user threads. Mika >=20 >=20 > - Jay >=20 >> To: m3devel at elegosoft.com >> Date: Sat=2C 12 Feb 2011 12:43:02 -0800 >> From: mika at async.caltech.edu >> Subject: [M3devel] Performance issues with Process.Wait under userthreads >>=20 >> Hi again m3devel (especially Tony)=2C=20 >>=20 >> I have finally taken a bite of a problem that has been annoying me >> for a very=2C very long time. >>=20 >> Under user threads=2C the code for Process.Wait is as follows (see=20 >> ThreadPosix.m3): >>=20 >> PROCEDURE WaitProcess (pid: int=3B VAR status: int): int =3D >> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >> CONST Delay =3D 0.1D0=3B >> BEGIN >> LOOP >> WITH r =3D Uexec.waitpid(pid=2C ADR(status)=2C Uexec.WNOHANG) DO >> IF r # 0 THEN RETURN r END=3B >> END=3B >> Pause(Delay)=3B >> END=3B >> END WaitProcess=3B >>=20 >> It inserts a 0.1 second delay after each failed waitpid. This is extremel= >y >> annoying for programs that start a long sequence of child processes and >> wait for them in sequence. Namely=2C the compiler itself. As a result >> the cm3 compiler (and PM3's m3build) are normally very very slow when >> using user threads. For about the last ten years=2C I've had a hacked >> up m3build (for my PM3 installation) that skips the Pause and busy-waits >> instead.=20 >>=20 >> Note there is another problem here. Since the Modula-3 runtime ignores >> SIGCHLD=2C no zombie processes are created since the Unix system automati= >cally >> reaps the child processes. I can see this would be a problem since PIDs >> are eventually reused and ... couldn't Uexec.waitpid wind up referring to >> the wrong process?? >>=20 >> I will further note that the comment in Process.i3 reads as follows: >>=20 >> PROCEDURE Wait(p: T): ExitCode=3B >>=20 >> Wait until the process with handle p terminates=2C then free the operatin= >g system resources associated with the process and return an exit code indi= >cating the reason for its termination. It is a checked runtime error to cal= >l Wait twice on the same process handle.=20 >>=20 >> I am going to take this as fair warning that Process.Create *may* use >> resources that are not going to be released until Process.Wait has been >> called. >>=20 >> I have modified (in my local copy of CM3) the system as follows. >> I have come up with a semi-general mechanism for immediately unblocking >> a thread on the receipt of a unix signal.=20 >>=20 >> 1. The system relies on changing >> ThreadPosix.XPause such that if a signal is allowed to wake up a threads= >=2C >> that fact is recorded in a new field in the thread's descriptor >> record (of type ThreadPosix.T).=20 >>=20 >> 2. On receipt of a waited-for unix signal=2C a mask is set and control >> is passed to the thread scheduler which maintains the non-zero mask for >> exactly one iteration through the thread ring. >>=20 >> 3. If a thread is paused and waiting for EITHER a signal or some time=2C >> the thread is released for running and the thread's waiting state is=20 >> cleared. >>=20 >> The changes are more or less as follows: >>=20 >> 1. I have added a new field of type "int" to ThreadPosix.T: >>=20 >> (* if state =3D pausing=2C the time at which we can restart *) >> waitingForTime: Time.T=3B >>=20 >> + (* if state =3D pausing=2C the signal that truncates the pause *) >> + waitingForSig: int :=3D -1=3B >> + >> (* true if we are waiting during an AlertWait or AlertJoin >> or AlertPause *) >> alertable: BOOLEAN :=3D FALSE=3B >>=20 >>=20 >> 2. Modifications to pause: >>=20 >> + PROCEDURE SigPause(n: LONGREAL=3B sig: int)=3D >> + <*FATAL Alerted*> >> + VAR until :=3D n + Time.Now ()=3B >> + BEGIN >> + XPause(until=2C FALSE=2C sig)=3B >> + END SigPause=3B >> + >> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >> VAR until :=3D n + Time.Now ()=3B >> BEGIN >> XPause(until=2C TRUE)=3B >> END AlertPause=3B >>=20 >> ! PROCEDURE XPause (READONLY until: Time.T=3B alertable :=3D FALSE=3B sig= >:int :=3D -1) >> ! RAISES {Alerted} =3D >> BEGIN >> INC (inCritical)=3B >> self.waitingForTime :=3D until=3B >> self.alertable :=3D alertable=3B >> + IF sig # -1 THEN >> + self.waitingForSig :=3D sig >> + END=3B >> ICannotRun (State.pausing)=3B >> DEC (inCritical)=3B >> InternalYield ()=3B >>=20 >> 3. The received-signals mask: >>=20 >> ! CONST MaxSigs =3D 64=3B >> ! TYPE Sig =3D [ 0..MaxSigs-1 ]=3B >> ! >> ! (* in order to listen to other signals=2C they have to be enabled in >> ! allow_sigvtalrm as well *) >> ! VAR (*CONST*) SIGCHLD :=3D ValueOfSIGCHLD()=3B >> ! >> ! gotSigs :=3D SET OF Sig { }=3B >> ! >>=20 >> ValueOfSIGCHLD() is a C function used to get the value of the SIGCHLD >> constant without guessing at it (in ThreadPosixC.c). >>=20 >> 4. changes to the signal handler: >>=20 >> ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >> BEGIN >> allow_sigvtalrm ()=3B >> ! >> ! INC(inCritical)=3B >> ! (* mark signal as being delivered *) >> ! IF sig >=3D 0 AND sig < MaxSigs THEN >> ! gotSigs :=3D gotSigs + SET OF Sig { sig } >> ! END=3B >> ! DEC(inCritical)=3B >> ! >> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >> ! InternalYield () >> ! END=3B >> END switch_thread=3B >>=20 >> Note that I don't know if INC/DEC(inCritical) does exactly the right >> thing here. >>=20 >> 5. changes to the scheduler: >>=20 >> a. thread wakeup >> IF t.alertable AND t.alertPending THEN >> CanRun (t)=3B >> EXIT=3B >> + >> + ELSIF t.waitingForSig IN gotSigs THEN >> + t.waitingForSig :=3D -1=3B >> + CanRun(t)=3B >> + EXIT=3B >>=20 >> ELSIF t.waitingForTime <=3D now THEN >> CanRun (t)=3B >> EXIT=3B >> ! >>=20 >> b. clearing the mask >>=20 >> END=3B >> END=3B >>=20 >> + gotSigs :=3D SET OF Sig {}=3B >> + >> IF t.state =3D State.alive AND (scanned OR NOT someBlocking) THEN >> IF perfOn THEN PerfRunning (t.id)=3B END=3B >> (* At least one thread wants to run=3B transfer to it *) >>=20 >> 6. changes to WaitProcess (Process.Wait): >>=20 >> PROCEDURE WaitProcess (pid: int=3B VAR status: int): int =3D >> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >> ! CONST Delay =3D 10.0D0=3B >> BEGIN >> LOOP >> WITH r =3D Uexec.waitpid(pid=2C ADR(status)=2C Uexec.WNOHANG) DO >> IF r # 0 THEN RETURN r END=3B >> END=3B >> ! SigPause(Delay=2CSIGCHLD)=3B >> END=3B >> END WaitProcess=3B >>=20 >> 7. install signal handler even if program is single-threaded: >>=20 >> BEGIN >> + (* we need to call set up the signal handler for other reasons than >> + just thread switching now *) >> + setup_sigvtalrm (switch_thread)=3B >> END ThreadPosix. >>=20 >> 8. modify signal handler in ThreadPosixC.c to catch SIGCHLD: >>=20 >> sigemptyset(&ThreadSwitchSignal)=3B >> sigaddset(&ThreadSwitchSignal=2C SIG_TIMESLICE)=3B >> + sigaddset(&ThreadSwitchSignal=2C SIGCHLD)=3B >>=20 >> act.sa_handler =3D handler=3B >> act.sa_flags =3D SA_RESTART=3B >> sigemptyset(&(act.sa_mask))=3B >> if (sigaction (SIG_TIMESLICE=2C &act=2C NULL)) abort()=3B >> + if (sigaction (SIGCHLD=2C &act=2C NULL)) abort()=3B >>=20 >> I'll send the complete diff in a separate message for those who want >> to study it more closely. >>=20 >> I propose the above changes for inclusion in the current CM3 repository. >>=20 >> Mika >>=20 > = > >--_5ec25af1-29a7-4f8a-b7ad-e61068941159_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >I don't believe there is a good solution here=2C other than using pthreads.= >
>Is there? Maybe tune the wait down smaller?
>I understand it stinks either way.
> =3B
> =3B
>I don't =3Bunderstand the SIGCHLD stuff=2C at a quick glance.
>Nor your changes.
>However you might be the only user of user threads.
>Is there any downside to your change?
> =3B(Other than more code=2C ok.)
>Any loss of performance in any situation?
>Any loss of portability?
> =3B
> =3B
> =3B- Jay
 =3B
>>=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 12 Feb 2011 12:43:0= >2 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > Performance issues with Process.Wait under userthreads
>=3B
>= >=3B Hi again m3devel (especially Tony)=2C
>=3B
>=3B I have fina= >lly taken a bite of a problem that has been annoying me
>=3B for a ver= >y=2C very long time.
>=3B
>=3B Under user threads=2C the code fo= >r Process.Wait is as follows (see
>=3B ThreadPosix.m3):
>=3B >>=3B PROCEDURE WaitProcess (pid: int=3B VAR status: int): int =3D
>= >=3B (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *)
>=3B C= >ONST Delay =3D 0.1D0=3B
>=3B BEGIN
>=3B LOOP
>=3B WITH r =3D= > Uexec.waitpid(pid=2C ADR(status)=2C Uexec.WNOHANG) DO
>=3B IF r # 0 T= >HEN RETURN r END=3B
>=3B END=3B
>=3B Pause(Delay)=3B
>=3B EN= >D=3B
>=3B END WaitProcess=3B
>=3B
>=3B It inserts a 0.1 sec= >ond delay after each failed waitpid. This is extremely
>=3B annoying f= >or programs that start a long sequence of child processes and
>=3B wai= >t for them in sequence. Namely=2C the compiler itself. As a result
>= >=3B the cm3 compiler (and PM3's m3build) are normally very very slow whenR>>=3B using user threads. For about the last ten years=2C I've had a hac= >ked
>=3B up m3build (for my PM3 installation) that skips the Pause and= > busy-waits
>=3B instead.
>=3B
>=3B Note there is another = >problem here. Since the Modula-3 runtime ignores
>=3B SIGCHLD=2C no zo= >mbie processes are created since the Unix system automatically
>=3B re= >aps the child processes. I can see this would be a problem since PIDs
&g= >t=3B are eventually reused and ... couldn't Uexec.waitpid wind up referring= > to
>=3B the wrong process??
>=3B
>=3B I will further note = >that the comment in Process.i3 reads as follows:
>=3B
>=3B PROCE= >DURE Wait(p: T): ExitCode=3B
>=3B
>=3B Wait until the process wi= >th handle p terminates=2C then free the operating system resources associat= >ed with the process and return an exit code indicating the reason for its t= >ermination. It is a checked runtime error to call Wait twice on the same pr= >ocess handle.
>=3B
>=3B I am going to take this as fair warning= > that Process.Create *may* use
>=3B resources that are not going to be= > released until Process.Wait has been
>=3B called.
>=3B
>= >=3B I have modified (in my local copy of CM3) the system as follows.
>= >=3B I have come up with a semi-general mechanism for immediately unblocking= >
>=3B a thread on the receipt of a unix signal.
>=3B
>=3B = >1. The system relies on changing
>=3B ThreadPosix.XPause such that if = >a signal is allowed to wake up a threads=2C
>=3B that fact is recorded= > in a new field in the thread's descriptor
>=3B record (of type Thread= >Posix.T).
>=3B
>=3B 2. On receipt of a waited-for unix signal= >=2C a mask is set and control
>=3B is passed to the thread scheduler w= >hich maintains the non-zero mask for
>=3B exactly one iteration throug= >h the thread ring.
>=3B
>=3B 3. If a thread is paused and waitin= >g for EITHER a signal or some time=2C
>=3B the thread is released for = >running and the thread's waiting state is
>=3B cleared.
>=3B >>=3B The changes are more or less as follows:
>=3B
>=3B 1. I = >have added a new field of type "int" to ThreadPosix.T:
>=3B
>=3B= > (* if state =3D pausing=2C the time at which we can restart *)
>=3B w= >aitingForTime: Time.T=3B
>=3B
>=3B + (* if state =3D pausing=2C = >the signal that truncates the pause *)
>=3B + waitingForSig: int :=3D = >-1=3B
>=3B +
>=3B (* true if we are waiting during an AlertWait o= >r AlertJoin
>=3B or AlertPause *)
>=3B alertable: BOOLEAN :=3D FA= >LSE=3B
>=3B
>=3B
>=3B 2. Modifications to pause:
>=3B= >
>=3B + PROCEDURE SigPause(n: LONGREAL=3B sig: int)=3D
>=3B + &l= >t=3B*FATAL Alerted*>=3B
>=3B + VAR until :=3D n + Time.Now ()=3B
= >>=3B + BEGIN
>=3B + XPause(until=2C FALSE=2C sig)=3B
>=3B + END= > SigPause=3B
>=3B +
>=3B PROCEDURE AlertPause(n: LONGREAL) RAISES= > {Alerted}=3D
>=3B VAR until :=3D n + Time.Now ()=3B
>=3B BEGINR>>=3B XPause(until=2C TRUE)=3B
>=3B END AlertPause=3B
>=3B >>=3B ! PROCEDURE XPause (READONLY until: Time.T=3B alertable :=3D FALSE= >=3B sig:int :=3D -1)
>=3B ! RAISES {Alerted} =3D
>=3B BEGIN
&g= >t=3B INC (inCritical)=3B
>=3B self.waitingForTime :=3D until=3B
>= >=3B self.alertable :=3D alertable=3B
>=3B + IF sig # -1 THEN
>=3B= > + self.waitingForSig :=3D sig
>=3B + END=3B
>=3B ICannotRun (Sta= >te.pausing)=3B
>=3B DEC (inCritical)=3B
>=3B InternalYield ()=3B<= >BR>>=3B
>=3B 3. The received-signals mask:
>=3B
>=3B ! C= >ONST MaxSigs =3D 64=3B
>=3B ! TYPE Sig =3D [ 0..MaxSigs-1 ]=3B
>= >=3B !
>=3B ! (* in order to listen to other signals=2C they have to be= > enabled in
>=3B ! allow_sigvtalrm as well *)
>=3B ! VAR (*CONST*= >) SIGCHLD :=3D ValueOfSIGCHLD()=3B
>=3B !
>=3B ! gotSigs :=3D SET= > OF Sig { }=3B
>=3B !
>=3B
>=3B ValueOfSIGCHLD() is a C fun= >ction used to get the value of the SIGCHLD
>=3B constant without guess= >ing at it (in ThreadPosixC.c).
>=3B
>=3B 4. changes to the signa= >l handler:
>=3B
>=3B ! PROCEDURE switch_thread (sig: int) RAISES= > {Alerted} =3D
>=3B BEGIN
>=3B allow_sigvtalrm ()=3B
>=3B !<= >BR>>=3B ! INC(inCritical)=3B
>=3B ! (* mark signal as being delivere= >d *)
>=3B ! IF sig >=3B=3D 0 AND sig <=3B MaxSigs THEN
>=3B != > gotSigs :=3D gotSigs + SET OF Sig { sig }
>=3B ! END=3B
>=3B ! D= >EC(inCritical)=3B
>=3B !
>=3B ! IF inCritical =3D 0 AND heapState= >.inCritical =3D 0 THEN
>=3B ! InternalYield ()
>=3B ! END=3B
&= >gt=3B END switch_thread=3B
>=3B
>=3B Note that I don't know if I= >NC/DEC(inCritical) does exactly the right
>=3B thing here.
>=3B <= >BR>>=3B 5. changes to the scheduler:
>=3B
>=3B a. thread wakeu= >p
>=3B IF t.alertable AND t.alertPending THEN
>=3B CanRun (t)=3B<= >BR>>=3B EXIT=3B
>=3B +
>=3B + ELSIF t.waitingForSig IN gotSigs = >THEN
>=3B + t.waitingForSig :=3D -1=3B
>=3B + CanRun(t)=3B
>= >=3B + EXIT=3B
>=3B
>=3B ELSIF t.waitingForTime <=3B=3D now THE= >N
>=3B CanRun (t)=3B
>=3B EXIT=3B
>=3B !
>=3B
>= >=3B b. clearing the mask
>=3B
>=3B END=3B
>=3B END=3B
&g= >t=3B
>=3B + gotSigs :=3D SET OF Sig {}=3B
>=3B +
>=3B IF t.= >state =3D State.alive AND (scanned OR NOT someBlocking) THEN
>=3B IF p= >erfOn THEN PerfRunning (t.id)=3B END=3B
>=3B (* At least one thread wa= >nts to run=3B transfer to it *)
>=3B
>=3B 6. changes to WaitProc= >ess (Process.Wait):
>=3B
>=3B PROCEDURE WaitProcess (pid: int=3B= > VAR status: int): int =3D
>=3B (* ThreadPThread.m3 and ThreadPosix.m3= > are very similar. *)
>=3B ! CONST Delay =3D 10.0D0=3B
>=3B BEGIN= >
>=3B LOOP
>=3B WITH r =3D Uexec.waitpid(pid=2C ADR(status)=2C Ue= >xec.WNOHANG) DO
>=3B IF r # 0 THEN RETURN r END=3B
>=3B END=3B>>=3B ! SigPause(Delay=2CSIGCHLD)=3B
>=3B END=3B
>=3B END WaitP= >rocess=3B
>=3B
>=3B 7. install signal handler even if program is= > single-threaded:
>=3B
>=3B BEGIN
>=3B + (* we need to call= > set up the signal handler for other reasons than
>=3B + just thread s= >witching now *)
>=3B + setup_sigvtalrm (switch_thread)=3B
>=3B EN= >D ThreadPosix.
>=3B
>=3B 8. modify signal handler in ThreadPosix= >C.c to catch SIGCHLD:
>=3B
>=3B sigemptyset(&=3BThreadSwitchS= >ignal)=3B
>=3B sigaddset(&=3BThreadSwitchSignal=2C SIG_TIMESLICE)= >=3B
>=3B + sigaddset(&=3BThreadSwitchSignal=2C SIGCHLD)=3B
>= >=3B
>=3B act.sa_handler =3D handler=3B
>=3B act.sa_flags =3D SA_= >RESTART=3B
>=3B sigemptyset(&=3B(act.sa_mask))=3B
>=3B if (sig= >action (SIG_TIMESLICE=2C &=3Bact=2C NULL)) abort()=3B
>=3B + if (si= >gaction (SIGCHLD=2C &=3Bact=2C NULL)) abort()=3B
>=3B
>=3B I'= >ll send the complete diff in a separate message for those who want
>= >=3B to study it more closely.
>=3B
>=3B I propose the above chan= >ges for inclusion in the current CM3 repository.
>=3B
>=3B Mika<= >BR>>=3B
>= > >--_5ec25af1-29a7-4f8a-b7ad-e61068941159_-- From mika at async.caltech.edu Sun Feb 13 08:04:14 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 23:04:14 -0800 Subject: [M3devel] More on threading Message-ID: <20110213070414.DD0341A2078@async.async.caltech.edu> Hi again m3devel, Well I thought I would relate another story having to do with threading and CM3. It's half-baked at the moment, but it only reflects a few hours' work on my part, sorry... I'm about to give up now because I don't know if the problem is in my code, or in threading, or what... maybe it is the atfork problem Jay talked about? Programmers spend a lot of time waiting for their compilers to run, so it would be nice if the compilation process were faster. After perusing the source code for the CM3 I realized that there is no feedback at all from the back-end and assembler steps to the rest of the compiler, and I believe that with the GCC back-end the vast majority of the runtime of the compiler is spent in these stages. So.... how hard could this be to parallelize? I decided to try making the two stages (cm3cg1 and as) into "promises" rather than running them on the spot. The QMachine adds a promise to call the back-end to a RefSeq.T called "promises", to which the Builder can also add promises. Then, after making all the promises, collect them and run them ("force them" a true Schemer would say) in parallel. Of course something goes wrong somewhere. With 10 threads I am able to run 31 seconds of CPU time in 19 seconds of wallclock time (using an average 160% of CPU) but then something goes wrong and my process is stuck in state "uxmxn". This is using user threads on AMD64_FREEBSD on a machine with four CPUs. Mika P.S. diff attached, it's a bit ugly, but maybe it works on some other system.... the code works just fine if you run one thread at a time. I.e., if you change IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END; to IF threads.size() > 0 THEN EVAL Thread.Join(threads.remlo()) END; Index: cm3/src/Builder.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/cm3/src/Builder.m3,v retrieving revision 1.36 diff -c -r1.36 Builder.m3 *** cm3/src/Builder.m3 24 Aug 2010 05:24:24 -0000 1.36 --- cm3/src/Builder.m3 13 Feb 2011 06:47:40 -0000 *************** *** 15,20 **** --- 15,22 ---- IMPORT QIdent; FROM Target IMPORT M3BackendMode_t, BackendAssembly, BackendModeStrings; FROM M3Path IMPORT OSKind, OSKindStrings; + IMPORT Pathname; + IMPORT RefSeq; TYPE UK = M3Unit.Kind; *************** *** 131,136 **** --- 133,139 ---- link_coverage : TEXT; (* coverage library *) m3_front_flags: Arg.List; (* configuration options for the front *) m3_options : Arg.List; (* misc. user options for the frontend *) + delayBackend := FALSE; END; TYPE *************** *** 932,937 **** --- 935,956 ---- (*------------------------------------------------------------ compilation --*) + TYPE MarkerPromise = QMachine.Promise BRANDED OBJECT OVERRIDES fulfil := FulfilNothing END; + + PROCEDURE FulfilNothing(<*UNUSED*>mp : MarkerPromise) = BEGIN END FulfilNothing; + + TYPE SeqClosure = Thread.Closure OBJECT seq : RefSeq.T; OVERRIDES apply := SeqApply END; + + PROCEDURE SeqApply(cl : SeqClosure) : REFANY = + BEGIN + FOR i := 0 TO cl.seq.size()-1 DO + WITH p = NARROW(cl.seq.get(i),QMachine.Promise) DO + p.fulfil() + END + END; + RETURN NIL + END SeqApply; + PROCEDURE CompileEverything (s: State; schedule: SourceList) = VAR u: M3Unit.T; BEGIN *************** *** 941,948 **** (* compile all the sources using the initial schedule *) FOR i := 0 TO LAST (schedule^) DO ! CompileOne (s, schedule[i]); END; FlushPending (s); (* recompile any interfaces where we goofed on the exports *) --- 960,1000 ---- (* compile all the sources using the initial schedule *) FOR i := 0 TO LAST (schedule^) DO ! s.delayBackend := TRUE; ! TRY ! CompileOne (s, schedule[i]); ! FINALLY ! s.delayBackend := FALSE; ! END; ! ! s.machine.promises.addhi(NEW(MarkerPromise)); ! ! END; ! ! VAR ! curSeq := NEW(RefSeq.T).init(); ! threads := NEW(RefSeq.T).init(); ! BEGIN ! FOR i := 0 TO s.machine.promises.size()-1 DO ! WITH p = s.machine.promises.get(i) DO ! curSeq.addhi(p); ! IF i = s.machine.promises.size()-1 OR ISTYPE(p,MarkerPromise) THEN ! WITH cl = NEW(SeqClosure, seq := curSeq) DO ! threads.addhi (Thread.Fork(cl)); ! ! IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END; ! ! curSeq := NEW(RefSeq.T).init() ! END ! END ! END ! END; ! WHILE threads.size() > 0 DO EVAL Thread.Join(threads.remlo()) END; END; + + EVAL s.machine.promises.init(); + + FlushPending (s); (* recompile any interfaces where we goofed on the exports *) *************** *** 1151,1156 **** --- 1203,1227 ---- END; END CompileM3; + TYPE + NotePromise = QMachine.Promise OBJECT + nam : Pathname.T; + OVERRIDES + fulfil := FulfilNP; + END; + + RemovePromise = QMachine.Promise OBJECT + nam : Pathname.T; + OVERRIDES + fulfil := FulfilRP; + END; + + PROCEDURE FulfilNP(np : NotePromise) = + BEGIN Utils.NoteTempFile(np.nam) END FulfilNP; + + PROCEDURE FulfilRP(rp : RemovePromise) = + BEGIN Utils.Remove(rp.nam) END FulfilRP; + PROCEDURE PushOneM3 (s: State; u: M3Unit.T): BOOLEAN = VAR tmpC, tmpS: TEXT; *************** *** 1191,1208 **** | 3 => (* -bootstrap, +m3back, +asm *) tmpC := TempCName (u); tmpS := TempSName (u); IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; IF RunM3 (s, u, tmpC) THEN ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) ! AND RunAsm (s, tmpS, u.object) THEN END; need_merge := TRUE; END; IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; | 6, (* +bootstrap, +m3back, -asm *) 7 => (* +bootstrap, +m3back, +asm *) --- 1262,1318 ---- | 3 => (* -bootstrap, +m3back, +asm *) + IF s.delayBackend THEN tmpC := TempCName (u); tmpS := TempSName (u); + (* IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; + *) + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(NotePromise, nam := tmpC)) + END; + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(NotePromise, nam := tmpS)) + END; + IF RunM3 (s, u, tmpC) THEN ! s.machine.record(TRUE); ! TRY ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) ! AND RunAsm (s, tmpS, u.object) THEN ! END; ! FINALLY ! s.machine.record(FALSE) END; + need_merge := TRUE; END; + (* IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; + *) + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpC)) + END; + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpS)) + END; + + ELSE + tmpC := TempCName (u); + tmpS := TempSName (u); + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; + IF RunM3 (s, u, tmpC) THEN + IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) + AND RunAsm (s, tmpS, u.object) THEN + END; + need_merge := TRUE; + END; + IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; + IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; + END | 6, (* +bootstrap, +m3back, -asm *) 7 => (* +bootstrap, +m3back, +asm *) Index: cm3/src/m3makefile =================================================================== RCS file: /usr/cvs/cm3/m3-sys/cm3/src/m3makefile,v retrieving revision 1.24 diff -c -r1.24 m3makefile *** cm3/src/m3makefile 8 Dec 2010 07:30:57 -0000 1.24 --- cm3/src/m3makefile 13 Feb 2011 06:47:42 -0000 *************** *** 29,34 **** --- 29,35 ---- end module ("Arg") module ("Builder") module ("Dirs") module ("M3Build") module ("M3Loc") Index: m3quake/src/QMachine.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.i3,v retrieving revision 1.6 diff -c -r1.6 QMachine.i3 *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -0000 1.6 --- m3quake/src/QMachine.i3 13 Feb 2011 06:47:43 -0000 *************** *** 8,13 **** --- 8,14 ---- IMPORT Thread, Wr, QValue, QCode; FROM Quake IMPORT Machine, Error, ID, IDMap; + IMPORT RefSeq; REVEAL T <: T_; *************** *** 15,20 **** --- 16,22 ---- T = Machine; T_ = OBJECT map: IDMap := NIL; (* READONLY *) + promises : RefSeq.T; METHODS init (map: IDMap): T; evaluate (s: QCode.Stream) RAISES {Error, Thread.Alerted}; *************** *** 37,42 **** --- 39,46 ---- set_wr (wr: Wr.T); exec_echo (b: BOOLEAN): BOOLEAN; trace (b: BOOLEAN); + + record(on : BOOLEAN); (* instead of performing certain acts, promise *) END; PROCEDURE PushBool (t: T; b: BOOLEAN); *************** *** 51,54 **** --- 55,63 ---- PROCEDURE GetEnv (default, v0, v1, v2, v3, v4: TEXT := NIL): TEXT; + TYPE Promise = OBJECT METHODS fulfil() RAISES { Error } END; + END QMachine. + + + Index: m3quake/src/QMachine.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3,v retrieving revision 1.35 diff -c -r1.35 QMachine.m3 *** m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35 --- m3quake/src/QMachine.m3 13 Feb 2011 06:47:44 -0000 *************** *** 16,21 **** --- 16,22 ---- IMPORT TextUtils, FSUtils, System, DirStack; (* sysutils *) IMPORT Compiler; IMPORT M3Path; + IMPORT RefSeq; CONST OnUnix = (Compiler.ThisOS = Compiler.OS.POSIX); *************** *** 44,49 **** --- 45,52 ---- shell : TEXT := NIL; sh_option : TEXT := NIL; tmp_dir : TEXT := NIL; + + doRecord := FALSE; OVERRIDES init := Init; evaluate := Evaluate; *************** *** 66,73 **** --- 69,81 ---- set_wr := SetWr; exec_echo := ExecEcho; trace := Trace; + + record := Record; END; + PROCEDURE Record(t : T; on : BOOLEAN) = + BEGIN t.doRecord := on END Record; + TYPE Registers = RECORD cp : QCode.Stream := NIL; (* code pointer *) *************** *** 139,144 **** --- 147,154 ---- t.globals := NEW (IntRefTbl.Default).init (); t.default_wr := Stdio.stdout; + t.promises := NEW(RefSeq.T).init(); + InitOSEnv (t); InitBuiltins (t); *************** *** 1555,1564 **** END; ELSE FlushIO (); ! Process.GetStandardFileHandles (stdin, stdout, stderr); ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), ! stdin := stdin, stdout := stdout, ! stderr := stderr); END; EXCEPT | Thread.Alerted => --- 1565,1594 ---- END; ELSE FlushIO (); ! IF t.doRecord THEN ! handle := NIL; ! WITH a = NEW(REF ARRAY OF TEXT, n_shell_args) DO ! a^ := SUBARRAY(args,0,n_shell_args); ! VAR wrx : Wr.T; BEGIN ! IF echo OR t.do_echo THEN ! wrx := wr ! ELSE ! wrx := NIL ! END; ! t.promises.addhi(NEW(ExecPromise, ! cmd := t.shell, ! wr := wrx, ! args := a, ! t := t, ! ignore_errors := ignore_errors)) ! END ! END ! ELSE ! Process.GetStandardFileHandles (stdin, stdout, stderr); ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), ! stdin := stdin, stdout := stdout, ! stderr := stderr); ! END; END; EXCEPT | Thread.Alerted => *************** *** 1573,1579 **** END; (* wait for everything to shutdown... *) ! exit_code := Process.Wait (handle); END; IF onlyTry THEN --- 1603,1613 ---- END; (* wait for everything to shutdown... *) ! IF handle = NIL THEN ! exit_code := 0 ! ELSE ! exit_code := Process.Wait (handle); ! END; (* else we're only promising *) END; IF onlyTry THEN *************** *** 1589,1594 **** --- 1623,1664 ---- END ExecCommand; + TYPE + ExecPromise = Promise OBJECT + cmd : TEXT; + args : REF ARRAY OF TEXT; + t : T; + wr : Wr.T; + ignore_errors : BOOLEAN; + OVERRIDES + fulfil := FulfilExecPromise; + END; + + PROCEDURE FulfilExecPromise(ep : ExecPromise) RAISES { Error } = + VAR + stdin, stdout, stderr: File.T; + BEGIN + Process.GetStandardFileHandles (stdin, stdout, stderr); + TRY + IF ep.wr # NIL THEN + Wr.PutText (ep.wr, ep.args[1]); + Wr.PutText (ep.wr, Wr.EOL); + FlushIO (); + END; + WITH handle = Process.Create (ep.cmd, ep.args^, + stdin := stdin, stdout := stdout, + stderr := stderr), + exit_code = Process.Wait(handle) DO + IF exit_code # 0 AND NOT ep.ignore_errors THEN + Err (ep.t, Fmt.F("exit %s: %s", Fmt.Int(exit_code), ep.cmd)) + END + END + EXCEPT + OSError.E (ec) => + Err (ep.t, Fmt.F ("exec failed%s *** %s", OSErr (ec), ep.cmd)); + END + END FulfilExecPromise; + PROCEDURE KillProcess (handle: Process.T) = BEGIN IF (handle # NIL) THEN From jay.krell at cornell.edu Sun Feb 13 08:41:23 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 13 Feb 2011 07:41:23 +0000 Subject: [M3devel] More on threading In-Reply-To: <20110213070414.DD0341A2078@async.async.caltech.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> Message-ID: Yes, gcc is slow and using as is wasteful. Yes this should all be multi-threaded. However it seems to me that m3front has many globals and therefore really making things multi-threaded isn't going to be easy. I think you are mostly wasting your time on user threads though. Have you tried the NT/x86 system? It is very fast. In the gcc system, each .mc file is run through m3cgc1 to produce an .ms file, which is run through as to produce an object file .mo. In the NT/x86 system, cm3 outputs .obj files directly. No .mc files. No m3cgc1 or as. Granted, I'd just as soon output .c files, and hope for a C compiler that doesn't run a separate assembler. My timing might loosen up a bit and I hope to spend some on the thread stress tester using pthreads. Next weekend. Am I right though that Darwin works fine? Solaris? FreeBSD? NetBSD? OpenBSD? Linux? I know, not Linux. You generally don't need any of these systems. All of the x86/AMD64 systems can run in a virtual machine. Solaris/sparc we have good access. - Jay > To: m3devel at elegosoft.com > Date: Sat, 12 Feb 2011 23:04:14 -0800 > From: mika at async.caltech.edu > CC: jay.krell at cornell.edu > Subject: [M3devel] More on threading > > > Hi again m3devel, > > Well I thought I would relate another story having to do with threading > and CM3. It's half-baked at the moment, but it only reflects a few hours' > work on my part, sorry... I'm about to give up now because I don't know > if the problem is in my code, or in threading, or what... maybe it is > the atfork problem Jay talked about? > > Programmers spend a lot of time waiting for their compilers to run, so > it would be nice if the compilation process were faster. After perusing > the source code for the CM3 I realized that there is no feedback at all > from the back-end and assembler steps to the rest of the compiler, and > I believe that with the GCC back-end the vast majority of the runtime > of the compiler is spent in these stages. > > So.... how hard could this be to parallelize? I decided to try making > the two stages (cm3cg1 and as) into "promises" rather than running them > on the spot. The QMachine adds a promise to call the back-end to a > RefSeq.T called "promises", to which the Builder can also add promises. > Then, after making all the promises, collect them and run them ("force > them" a true Schemer would say) in parallel. > > Of course something goes wrong somewhere. With 10 threads I am able to > run 31 seconds of CPU time in 19 seconds of wallclock time (using an > average 160% of CPU) but then something goes wrong and my process > is stuck in state "uxmxn". This is using user threads on AMD64_FREEBSD > on a machine with four CPUs. > > Mika > > P.S. diff attached, it's a bit ugly, but maybe it works on some other > system.... the code works just fine if you run one thread at a time. > I.e., if you change > > IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END; > > to > > IF threads.size() > 0 THEN EVAL Thread.Join(threads.remlo()) END; > > > > Index: cm3/src/Builder.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/cm3/src/Builder.m3,v > retrieving revision 1.36 > diff -c -r1.36 Builder.m3 > *** cm3/src/Builder.m3 24 Aug 2010 05:24:24 -0000 1.36 > --- cm3/src/Builder.m3 13 Feb 2011 06:47:40 -0000 > *************** > *** 15,20 **** > --- 15,22 ---- > IMPORT QIdent; > FROM Target IMPORT M3BackendMode_t, BackendAssembly, BackendModeStrings; > FROM M3Path IMPORT OSKind, OSKindStrings; > + IMPORT Pathname; > + IMPORT RefSeq; > > TYPE > UK = M3Unit.Kind; > *************** > *** 131,136 **** > --- 133,139 ---- > link_coverage : TEXT; (* coverage library *) > m3_front_flags: Arg.List; (* configuration options for the front *) > m3_options : Arg.List; (* misc. user options for the frontend *) > + delayBackend := FALSE; > END; > > TYPE > *************** > *** 932,937 **** > --- 935,956 ---- > > (*------------------------------------------------------------ compilation --*) > > + TYPE MarkerPromise = QMachine.Promise BRANDED OBJECT OVERRIDES fulfil := FulfilNothing END; > + > + PROCEDURE FulfilNothing(<*UNUSED*>mp : MarkerPromise) = BEGIN END FulfilNothing; > + > + TYPE SeqClosure = Thread.Closure OBJECT seq : RefSeq.T; OVERRIDES apply := SeqApply END; > + > + PROCEDURE SeqApply(cl : SeqClosure) : REFANY = > + BEGIN > + FOR i := 0 TO cl.seq.size()-1 DO > + WITH p = NARROW(cl.seq.get(i),QMachine.Promise) DO > + p.fulfil() > + END > + END; > + RETURN NIL > + END SeqApply; > + > PROCEDURE CompileEverything (s: State; schedule: SourceList) = > VAR u: M3Unit.T; > BEGIN > *************** > *** 941,948 **** > > (* compile all the sources using the initial schedule *) > FOR i := 0 TO LAST (schedule^) DO > ! CompileOne (s, schedule[i]); > END; > FlushPending (s); > > (* recompile any interfaces where we goofed on the exports *) > --- 960,1000 ---- > > (* compile all the sources using the initial schedule *) > FOR i := 0 TO LAST (schedule^) DO > ! s.delayBackend := TRUE; > ! TRY > ! CompileOne (s, schedule[i]); > ! FINALLY > ! s.delayBackend := FALSE; > ! END; > ! > ! s.machine.promises.addhi(NEW(MarkerPromise)); > ! > ! END; > ! > ! VAR > ! curSeq := NEW(RefSeq.T).init(); > ! threads := NEW(RefSeq.T).init(); > ! BEGIN > ! FOR i := 0 TO s.machine.promises.size()-1 DO > ! WITH p = s.machine.promises.get(i) DO > ! curSeq.addhi(p); > ! IF i = s.machine.promises.size()-1 OR ISTYPE(p,MarkerPromise) THEN > ! WITH cl = NEW(SeqClosure, seq := curSeq) DO > ! threads.addhi (Thread.Fork(cl)); > ! > ! IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END; > ! > ! curSeq := NEW(RefSeq.T).init() > ! END > ! END > ! END > ! END; > ! WHILE threads.size() > 0 DO EVAL Thread.Join(threads.remlo()) END; > END; > + > + EVAL s.machine.promises.init(); > + > + > FlushPending (s); > > (* recompile any interfaces where we goofed on the exports *) > *************** > *** 1151,1156 **** > --- 1203,1227 ---- > END; > END CompileM3; > > + TYPE > + NotePromise = QMachine.Promise OBJECT > + nam : Pathname.T; > + OVERRIDES > + fulfil := FulfilNP; > + END; > + > + RemovePromise = QMachine.Promise OBJECT > + nam : Pathname.T; > + OVERRIDES > + fulfil := FulfilRP; > + END; > + > + PROCEDURE FulfilNP(np : NotePromise) = > + BEGIN Utils.NoteTempFile(np.nam) END FulfilNP; > + > + PROCEDURE FulfilRP(rp : RemovePromise) = > + BEGIN Utils.Remove(rp.nam) END FulfilRP; > + > PROCEDURE PushOneM3 (s: State; u: M3Unit.T): BOOLEAN = > VAR > tmpC, tmpS: TEXT; > *************** > *** 1191,1208 **** > > > | 3 => (* -bootstrap, +m3back, +asm *) > tmpC := TempCName (u); > tmpS := TempSName (u); > IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; > IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; > IF RunM3 (s, u, tmpC) THEN > ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) > ! AND RunAsm (s, tmpS, u.object) THEN > END; > need_merge := TRUE; > END; > IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; > IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; > > | 6, (* +bootstrap, +m3back, -asm *) > 7 => (* +bootstrap, +m3back, +asm *) > --- 1262,1318 ---- > > > | 3 => (* -bootstrap, +m3back, +asm *) > + IF s.delayBackend THEN > tmpC := TempCName (u); > tmpS := TempSName (u); > + (* > IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; > IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; > + *) > + IF (NOT s.keep_files) THEN > + s.machine.promises.addhi(NEW(NotePromise, nam := tmpC)) > + END; > + IF (NOT s.keep_files) THEN > + s.machine.promises.addhi(NEW(NotePromise, nam := tmpS)) > + END; > + > IF RunM3 (s, u, tmpC) THEN > ! s.machine.record(TRUE); > ! TRY > ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) > ! AND RunAsm (s, tmpS, u.object) THEN > ! END; > ! FINALLY > ! s.machine.record(FALSE) > END; > + > need_merge := TRUE; > END; > + (* > IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; > IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; > + *) > + IF (NOT s.keep_files) THEN > + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpC)) > + END; > + IF (NOT s.keep_files) THEN > + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpS)) > + END; > + > + ELSE > + tmpC := TempCName (u); > + tmpS := TempSName (u); > + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; > + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; > + IF RunM3 (s, u, tmpC) THEN > + IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) > + AND RunAsm (s, tmpS, u.object) THEN > + END; > + need_merge := TRUE; > + END; > + IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; > + IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; > + END > > | 6, (* +bootstrap, +m3back, -asm *) > 7 => (* +bootstrap, +m3back, +asm *) > Index: cm3/src/m3makefile > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/cm3/src/m3makefile,v > retrieving revision 1.24 > diff -c -r1.24 m3makefile > *** cm3/src/m3makefile 8 Dec 2010 07:30:57 -0000 1.24 > --- cm3/src/m3makefile 13 Feb 2011 06:47:42 -0000 > *************** > *** 29,34 **** > --- 29,35 ---- > end > module ("Arg") > module ("Builder") > module ("Dirs") > module ("M3Build") > module ("M3Loc") > Index: m3quake/src/QMachine.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.i3,v > retrieving revision 1.6 > diff -c -r1.6 QMachine.i3 > *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -0000 1.6 > --- m3quake/src/QMachine.i3 13 Feb 2011 06:47:43 -0000 > *************** > *** 8,13 **** > --- 8,14 ---- > > IMPORT Thread, Wr, QValue, QCode; > FROM Quake IMPORT Machine, Error, ID, IDMap; > + IMPORT RefSeq; > > REVEAL > T <: T_; > *************** > *** 15,20 **** > --- 16,22 ---- > T = Machine; > T_ = OBJECT > map: IDMap := NIL; (* READONLY *) > + promises : RefSeq.T; > METHODS > init (map: IDMap): T; > evaluate (s: QCode.Stream) RAISES {Error, Thread.Alerted}; > *************** > *** 37,42 **** > --- 39,46 ---- > set_wr (wr: Wr.T); > exec_echo (b: BOOLEAN): BOOLEAN; > trace (b: BOOLEAN); > + > + record(on : BOOLEAN); (* instead of performing certain acts, promise *) > END; > > PROCEDURE PushBool (t: T; b: BOOLEAN); > *************** > *** 51,54 **** > --- 55,63 ---- > > PROCEDURE GetEnv (default, v0, v1, v2, v3, v4: TEXT := NIL): TEXT; > > + TYPE Promise = OBJECT METHODS fulfil() RAISES { Error } END; > + > END QMachine. > + > + > + > Index: m3quake/src/QMachine.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3,v > retrieving revision 1.35 > diff -c -r1.35 QMachine.m3 > *** m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35 > --- m3quake/src/QMachine.m3 13 Feb 2011 06:47:44 -0000 > *************** > *** 16,21 **** > --- 16,22 ---- > IMPORT TextUtils, FSUtils, System, DirStack; (* sysutils *) > IMPORT Compiler; > IMPORT M3Path; > + IMPORT RefSeq; > > CONST > OnUnix = (Compiler.ThisOS = Compiler.OS.POSIX); > *************** > *** 44,49 **** > --- 45,52 ---- > shell : TEXT := NIL; > sh_option : TEXT := NIL; > tmp_dir : TEXT := NIL; > + > + doRecord := FALSE; > OVERRIDES > init := Init; > evaluate := Evaluate; > *************** > *** 66,73 **** > --- 69,81 ---- > set_wr := SetWr; > exec_echo := ExecEcho; > trace := Trace; > + > + record := Record; > END; > > + PROCEDURE Record(t : T; on : BOOLEAN) = > + BEGIN t.doRecord := on END Record; > + > TYPE > Registers = RECORD > cp : QCode.Stream := NIL; (* code pointer *) > *************** > *** 139,144 **** > --- 147,154 ---- > t.globals := NEW (IntRefTbl.Default).init (); > t.default_wr := Stdio.stdout; > > + t.promises := NEW(RefSeq.T).init(); > + > InitOSEnv (t); > InitBuiltins (t); > > *************** > *** 1555,1564 **** > END; > ELSE > FlushIO (); > ! Process.GetStandardFileHandles (stdin, stdout, stderr); > ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), > ! stdin := stdin, stdout := stdout, > ! stderr := stderr); > END; > EXCEPT > | Thread.Alerted => > --- 1565,1594 ---- > END; > ELSE > FlushIO (); > ! IF t.doRecord THEN > ! handle := NIL; > ! WITH a = NEW(REF ARRAY OF TEXT, n_shell_args) DO > ! a^ := SUBARRAY(args,0,n_shell_args); > ! VAR wrx : Wr.T; BEGIN > ! IF echo OR t.do_echo THEN > ! wrx := wr > ! ELSE > ! wrx := NIL > ! END; > ! t.promises.addhi(NEW(ExecPromise, > ! cmd := t.shell, > ! wr := wrx, > ! args := a, > ! t := t, > ! ignore_errors := ignore_errors)) > ! END > ! END > ! ELSE > ! Process.GetStandardFileHandles (stdin, stdout, stderr); > ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), > ! stdin := stdin, stdout := stdout, > ! stderr := stderr); > ! END; > END; > EXCEPT > | Thread.Alerted => > *************** > *** 1573,1579 **** > END; > > (* wait for everything to shutdown... *) > ! exit_code := Process.Wait (handle); > END; > > IF onlyTry THEN > --- 1603,1613 ---- > END; > > (* wait for everything to shutdown... *) > ! IF handle = NIL THEN > ! exit_code := 0 > ! ELSE > ! exit_code := Process.Wait (handle); > ! END; (* else we're only promising *) > END; > > IF onlyTry THEN > *************** > *** 1589,1594 **** > --- 1623,1664 ---- > > END ExecCommand; > > + TYPE > + ExecPromise = Promise OBJECT > + cmd : TEXT; > + args : REF ARRAY OF TEXT; > + t : T; > + wr : Wr.T; > + ignore_errors : BOOLEAN; > + OVERRIDES > + fulfil := FulfilExecPromise; > + END; > + > + PROCEDURE FulfilExecPromise(ep : ExecPromise) RAISES { Error } = > + VAR > + stdin, stdout, stderr: File.T; > + BEGIN > + Process.GetStandardFileHandles (stdin, stdout, stderr); > + TRY > + IF ep.wr # NIL THEN > + Wr.PutText (ep.wr, ep.args[1]); > + Wr.PutText (ep.wr, Wr.EOL); > + FlushIO (); > + END; > + WITH handle = Process.Create (ep.cmd, ep.args^, > + stdin := stdin, stdout := stdout, > + stderr := stderr), > + exit_code = Process.Wait(handle) DO > + IF exit_code # 0 AND NOT ep.ignore_errors THEN > + Err (ep.t, Fmt.F("exit %s: %s", Fmt.Int(exit_code), ep.cmd)) > + END > + END > + EXCEPT > + OSError.E (ec) => > + Err (ep.t, Fmt.F ("exec failed%s *** %s", OSErr (ec), ep.cmd)); > + END > + END FulfilExecPromise; > + > PROCEDURE KillProcess (handle: Process.T) = > BEGIN > IF (handle # NIL) THEN > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 13 19:17:17 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 10:17:17 -0800 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu> Message-ID: <20110213181717.4F0E91A2078@async.async.caltech.edu> Jay K writes: >--_b09106d1-2a84-4ffe-876c-0b2e4cdfd0b0_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Yes=2C gcc is slow and using as is wasteful. >Yes this should all be multi-threaded. >However it seems to me that m3front has many globals and therefore >really making things multi-threaded isn't going to be easy. OK but as long as the back-end is taking up most of the cycles the property that the output of the front-end is used only by cmcg1 (m3back/m3cgc/whatever you call it) and the assembler means that what I did works for a pretty big speedup. In fact you can see that it works just fine (from the point of view of generating the correct output) since you can re-order running the backend with respect to all the steps done by the front-end. What you lose is (perhaps) some error messages, and of course, in general, any implicit dependencies between the error messages. I don't understand why the program gets stuck in uxmxn though. Maybe a problem with, sigh, user threads... >I think you are mostly wasting your time on user threads though. I hope so... I too would really like to use pthreads. > > >Have you tried the NT/x86 system? >It is very fast. > > >In the gcc system=2C each .mc file is run through m3cgc1 to produce an .ms = >file=2C >which is run through as to produce an object file .mo. > > >In the NT/x86 system=2C cm3 outputs .obj files directly. >No .mc files. No m3cgc1 or as. >Granted=2C I'd just as soon output .c files=2C and hope for a C compiler th= >at doesn't run a separate assembler. The NT compiler sounds faster but of course it's easier to parallelize the unix one using the method I outlined :-) Well unless you're using pthreads of course, then you can just fork a new thread for the integrated NT backend as long as it doesn't share too many variables with the front-end. But no I don't use Windows. I was asking on here how to get started with Modula-3 on Windows once but my machine was/is a Windows 2000 system and apparently CM3 no longer works on win2k..? I used to use PM3/Klagenfurt a lot on Windows (Cygwin) but thankfully my clients on that project let me switch the system to Unix so I don't have to touch that anymore. > > >My timing might loosen up a bit and I hope to spend some on the thread stre= >ss tester using pthreads. Next weekend. >Am I right though that Darwin works fine? >Solaris? FreeBSD? NetBSD? OpenBSD? Linux? I know=2C not Linux. >You generally don't need any of these systems. That would be great. I don't think pthreads works properly anywhere at the moment. My guess is that it never has, if you stress it hard enough. I first ran into problems on Darwin and then switched to AMD64 simply because I have better access to faster machines with AMD64. At first there were some obvious problems that Tony fixed quickly but then I kept running into odd behavior with my application and I added various other tests until I could reproduce everything I'd seen go wrong with my application, and I think at that point I was seeing problems on every platform (except user threads). The uxmxn problem I see with user threads on FreeBSD suggests that perhaps I need to add some more tests, however. Mika >All of the x86/AMD64 systems can run in a virtual machine. >Solaris/sparc we have good access. > > > - Jay > > >> To: m3devel at elegosoft.com >> Date: Sat=2C 12 Feb 2011 23:04:14 -0800 >> From: mika at async.caltech.edu >> CC: jay.krell at cornell.edu >> Subject: [M3devel] More on threading >>=20 >>=20 >> Hi again m3devel=2C >>=20 >> Well I thought I would relate another story having to do with threading >> and CM3. It's half-baked at the moment=2C but it only reflects a few hou= >rs' >> work on my part=2C sorry... I'm about to give up now because I don't know >> if the problem is in my code=2C or in threading=2C or what... maybe it is >> the atfork problem Jay talked about? >>=20 >> Programmers spend a lot of time waiting for their compilers to run=2C so >> it would be nice if the compilation process were faster. After perusing >> the source code for the CM3 I realized that there is no feedback at all >> from the back-end and assembler steps to the rest of the compiler=2C and >> I believe that with the GCC back-end the vast majority of the runtime >> of the compiler is spent in these stages. >>=20 >> So.... how hard could this be to parallelize? I decided to try making >> the two stages (cm3cg1 and as) into "promises" rather than running them >> on the spot. The QMachine adds a promise to call the back-end to a >> RefSeq.T called "promises"=2C to which the Builder can also add promises. >> Then=2C after making all the promises=2C collect them and run them ("forc= >e >> them" a true Schemer would say) in parallel. >>=20 >> Of course something goes wrong somewhere. With 10 threads I am able to >> run 31 seconds of CPU time in 19 seconds of wallclock time (using an >> average 160% of CPU) but then something goes wrong and my process >> is stuck in state "uxmxn". This is using user threads on AMD64_FREEBSD >> on a machine with four CPUs. >>=20 >> Mika >>=20 >> P.S. diff attached=2C it's a bit ugly=2C but maybe it works on some other >> system.... the code works just fine if you run one thread at a time. >> I.e.=2C if you change >>=20 >> IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END=3B >>=20 >> to >>=20 >> IF threads.size() > 0 THEN EVAL Thread.Join(threads.remlo()) END=3B >>=20 >>=20 >>=20 >> Index: cm3/src/Builder.m3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: /usr/cvs/cm3/m3-sys/cm3/src/Builder.m3=2Cv >> retrieving revision 1.36 >> diff -c -r1.36 Builder.m3 >> *** cm3/src/Builder.m3 24 Aug 2010 05:24:24 -0000 1.36 >> --- cm3/src/Builder.m3 13 Feb 2011 06:47:40 -0000 >> *************** >> *** 15=2C20 **** >> --- 15=2C22 ---- >> IMPORT QIdent=3B >> FROM Target IMPORT M3BackendMode_t=2C BackendAssembly=2C BackendModeStr= >ings=3B >> FROM M3Path IMPORT OSKind=2C OSKindStrings=3B >> + IMPORT Pathname=3B >> + IMPORT RefSeq=3B >> =20 >> TYPE >> UK =3D M3Unit.Kind=3B >> *************** >> *** 131=2C136 **** >> --- 133=2C139 ---- >> link_coverage : TEXT=3B (* coverage library *) >> m3_front_flags: Arg.List=3B (* configuration options for = >the front *) >> m3_options : Arg.List=3B (* misc. user options for the= > frontend *) >> + delayBackend :=3D FALSE=3B >> END=3B >> =20 >> TYPE >> *************** >> *** 932=2C937 **** >> --- 935=2C956 ---- >> =20 >> (*------------------------------------------------------------ compilat= >ion --*) >> =20 >> + TYPE MarkerPromise =3D QMachine.Promise BRANDED OBJECT OVERRIDES fulfil= > :=3D FulfilNothing END=3B >> +=20 >> + PROCEDURE FulfilNothing(<*UNUSED*>mp : MarkerPromise) =3D BEGIN END Ful= >filNothing=3B >> +=20 >> + TYPE SeqClosure =3D Thread.Closure OBJECT seq : RefSeq.T=3B OVERRIDES a= >pply :=3D SeqApply END=3B >> +=20 >> + PROCEDURE SeqApply(cl : SeqClosure) : REFANY =3D >> + BEGIN >> + FOR i :=3D 0 TO cl.seq.size()-1 DO >> + WITH p =3D NARROW(cl.seq.get(i)=2CQMachine.Promise) DO >> + p.fulfil() >> + END >> + END=3B >> + RETURN NIL >> + END SeqApply=3B >> +=20 >> PROCEDURE CompileEverything (s: State=3B schedule: SourceList) =3D >> VAR u: M3Unit.T=3B >> BEGIN >> *************** >> *** 941=2C948 **** >> =20 >> (* compile all the sources using the initial schedule *) >> FOR i :=3D 0 TO LAST (schedule^) DO >> ! CompileOne (s=2C schedule[i])=3B >> END=3B >> FlushPending (s)=3B >> =20 >> (* recompile any interfaces where we goofed on the exports *) >> --- 960=2C1000 ---- >> =20 >> (* compile all the sources using the initial schedule *) >> FOR i :=3D 0 TO LAST (schedule^) DO >> ! s.delayBackend :=3D TRUE=3B >> ! TRY >> ! CompileOne (s=2C schedule[i])=3B >> ! FINALLY >> ! s.delayBackend :=3D FALSE=3B >> ! END=3B >> !=20 >> ! s.machine.promises.addhi(NEW(MarkerPromise))=3B >> !=20 >> ! END=3B >> !=20 >> ! VAR >> ! curSeq :=3D NEW(RefSeq.T).init()=3B >> ! threads :=3D NEW(RefSeq.T).init()=3B >> ! BEGIN >> ! FOR i :=3D 0 TO s.machine.promises.size()-1 DO >> ! WITH p =3D s.machine.promises.get(i) DO >> ! curSeq.addhi(p)=3B >> ! IF i =3D s.machine.promises.size()-1 OR ISTYPE(p=2CMarkerProm= >ise) THEN >> ! WITH cl =3D NEW(SeqClosure=2C seq :=3D curSeq) DO >> ! threads.addhi (Thread.Fork(cl))=3B >> !=20 >> ! IF threads.size() > 10 THEN EVAL Thread.Join(threads.reml= >o()) END=3B >> ! =20 >> ! curSeq :=3D NEW(RefSeq.T).init() >> ! END >> ! END >> ! END >> ! END=3B >> ! WHILE threads.size() > 0 DO EVAL Thread.Join(threads.remlo()) END= >=3B >> END=3B >> + =20 >> + EVAL s.machine.promises.init()=3B >> +=20 >> +=20 >> FlushPending (s)=3B >> =20 >> (* recompile any interfaces where we goofed on the exports *) >> *************** >> *** 1151=2C1156 **** >> --- 1203=2C1227 ---- >> END=3B >> END CompileM3=3B >> =20 >> + TYPE >> + NotePromise =3D QMachine.Promise OBJECT >> + nam : Pathname.T=3B >> + OVERRIDES >> + fulfil :=3D FulfilNP=3B >> + END=3B >> +=20 >> + RemovePromise =3D QMachine.Promise OBJECT >> + nam : Pathname.T=3B >> + OVERRIDES >> + fulfil :=3D FulfilRP=3B >> + END=3B >> +=20 >> + PROCEDURE FulfilNP(np : NotePromise) =3D=20 >> + BEGIN Utils.NoteTempFile(np.nam) END FulfilNP=3B >> +=20 >> + PROCEDURE FulfilRP(rp : RemovePromise) =3D=20 >> + BEGIN Utils.Remove(rp.nam) END FulfilRP=3B >> +=20 >> PROCEDURE PushOneM3 (s: State=3B u: M3Unit.T): BOOLEAN =3D >> VAR >> tmpC=2C tmpS: TEXT=3B >> *************** >> *** 1191=2C1208 **** >> =20 >> =20 >> | 3 =3D> (* -bootstrap=2C +m3back=2C +asm *) >> tmpC :=3D TempCName (u)=3B >> tmpS :=3D TempSName (u)=3B >> IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3B >> IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END=3B >> IF RunM3 (s=2C u=2C tmpC) THEN >> ! IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.optimize) >> ! AND RunAsm (s=2C tmpS=2C u.object) THEN >> END=3B >> need_merge :=3D TRUE=3B >> END=3B >> IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END=3B >> IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END=3B >> =20 >> | 6=2C (* +bootstrap=2C +m3back=2C -asm *) >> 7 =3D> (* +bootstrap=2C +m3back=2C +asm *) >> --- 1262=2C1318 ---- >> =20 >> =20 >> | 3 =3D> (* -bootstrap=2C +m3back=2C +asm *) >> + IF s.delayBackend THEN >> tmpC :=3D TempCName (u)=3B >> tmpS :=3D TempSName (u)=3B >> + (* >> IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3B >> IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END=3B >> + *) >> + IF (NOT s.keep_files) THEN=20 >> + s.machine.promises.addhi(NEW(NotePromise=2C nam :=3D tmpC))=20 >> + END=3B >> + IF (NOT s.keep_files) THEN=20 >> + s.machine.promises.addhi(NEW(NotePromise=2C nam :=3D tmpS))=20 >> + END=3B >> +=20 >> IF RunM3 (s=2C u=2C tmpC) THEN >> ! s.machine.record(TRUE)=3B >> ! TRY >> ! IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.optimize) >> ! AND RunAsm (s=2C tmpS=2C u.object) THEN >> ! END=3B >> ! FINALLY >> ! s.machine.record(FALSE) >> END=3B >> +=20 >> need_merge :=3D TRUE=3B >> END=3B >> + (* >> IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END=3B >> IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END=3B >> + *) >> + IF (NOT s.keep_files) THEN=20 >> + s.machine.promises.addhi(NEW(RemovePromise=2C nam :=3D tmpC))= >=20 >> + END=3B >> + IF (NOT s.keep_files) THEN=20 >> + s.machine.promises.addhi(NEW(RemovePromise=2C nam :=3D tmpS))= >=20 >> + END=3B >> +=20 >> + ELSE >> + tmpC :=3D TempCName (u)=3B >> + tmpS :=3D TempSName (u)=3B >> + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3B >> + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END=3B >> + IF RunM3 (s=2C u=2C tmpC) THEN >> + IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.optimize) >> + AND RunAsm (s=2C tmpS=2C u.object) THEN >> + END=3B >> + need_merge :=3D TRUE=3B >> + END=3B >> + IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END=3B >> + IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END=3B >> + END >> =20 >> | 6=2C (* +bootstrap=2C +m3back=2C -asm *) >> 7 =3D> (* +bootstrap=2C +m3back=2C +asm *) >> Index: cm3/src/m3makefile >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: /usr/cvs/cm3/m3-sys/cm3/src/m3makefile=2Cv >> retrieving revision 1.24 >> diff -c -r1.24 m3makefile >> *** cm3/src/m3makefile 8 Dec 2010 07:30:57 -0000 1.24 >> --- cm3/src/m3makefile 13 Feb 2011 06:47:42 -0000 >> *************** >> *** 29=2C34 **** >> --- 29=2C35 ---- >> end >> module ("Arg") >> module ("Builder") >> module ("Dirs") >> module ("M3Build") >> module ("M3Loc") >> Index: m3quake/src/QMachine.i3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.i3=2Cv >> retrieving revision 1.6 >> diff -c -r1.6 QMachine.i3 >> *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -0000 1.6 >> --- m3quake/src/QMachine.i3 13 Feb 2011 06:47:43 -0000 >> *************** >> *** 8=2C13 **** >> --- 8=2C14 ---- >> =20 >> IMPORT Thread=2C Wr=2C QValue=2C QCode=3B >> FROM Quake IMPORT Machine=2C Error=2C ID=2C IDMap=3B >> + IMPORT RefSeq=3B >> =20 >> REVEAL >> T <: T_=3B >> *************** >> *** 15=2C20 **** >> --- 16=2C22 ---- >> T =3D Machine=3B >> T_ =3D OBJECT >> map: IDMap :=3D NIL=3B (* READONLY *) >> + promises : RefSeq.T=3B >> METHODS >> init (map: IDMap): T=3B >> evaluate (s: QCode.Stream) RAISES {Error=2C Th= >read.Alerted}=3B >> *************** >> *** 37=2C42 **** >> --- 39=2C46 ---- >> set_wr (wr: Wr.T)=3B >> exec_echo (b: BOOLEAN): BOOLEAN=3B >> trace (b: BOOLEAN)=3B >> + =20 >> + record(on : BOOLEAN)=3B (* instead of performing certain acts= >=2C promise *) >> END=3B >> =20 >> PROCEDURE PushBool (t: T=3B b: BOOLEAN)=3B >> *************** >> *** 51=2C54 **** >> --- 55=2C63 ---- >> =20 >> PROCEDURE GetEnv (default=2C v0=2C v1=2C v2=2C v3=2C v4: TEXT :=3D NIL)= >: TEXT=3B >> =20 >> + TYPE Promise =3D OBJECT METHODS fulfil() RAISES { Error } END=3B >> +=20 >> END QMachine. >> +=20 >> +=20 >> +=20 >> Index: m3quake/src/QMachine.m3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3=2Cv >> retrieving revision 1.35 >> diff -c -r1.35 QMachine.m3 >> *** m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35 >> --- m3quake/src/QMachine.m3 13 Feb 2011 06:47:44 -0000 >> *************** >> *** 16=2C21 **** >> --- 16=2C22 ---- >> IMPORT TextUtils=2C FSUtils=2C System=2C DirStack=3B (* sysutils *) >> IMPORT Compiler=3B >> IMPORT M3Path=3B >> + IMPORT RefSeq=3B >> =20 >> CONST >> OnUnix =3D (Compiler.ThisOS =3D Compiler.OS.POSIX)=3B >> *************** >> *** 44=2C49 **** >> --- 45=2C52 ---- >> shell : TEXT :=3D NIL=3B >> sh_option : TEXT :=3D NIL=3B >> tmp_dir : TEXT :=3D NIL=3B >> +=20 >> + doRecord :=3D FALSE=3B >> OVERRIDES >> init :=3D Init=3B >> evaluate :=3D Evaluate=3B >> *************** >> *** 66=2C73 **** >> --- 69=2C81 ---- >> set_wr :=3D SetWr=3B >> exec_echo :=3D ExecEcho=3B >> trace :=3D Trace=3B >> +=20 >> + record :=3D Record=3B >> END=3B >> =20 >> + PROCEDURE Record(t : T=3B on : BOOLEAN) =3D=20 >> + BEGIN t.doRecord :=3D on END Record=3B >> +=20 >> TYPE >> Registers =3D RECORD >> cp : QCode.Stream :=3D NIL=3B (* code pointer *) >> *************** >> *** 139=2C144 **** >> --- 147=2C154 ---- >> t.globals :=3D NEW (IntRefTbl.Default).init ()=3B >> t.default_wr :=3D Stdio.stdout=3B >> =20 >> + t.promises :=3D NEW(RefSeq.T).init()=3B >> +=20 >> InitOSEnv (t)=3B >> InitBuiltins (t)=3B >> =20 >> *************** >> *** 1555=2C1564 **** >> END=3B >> ELSE >> FlushIO ()=3B >> ! Process.GetStandardFileHandles (stdin=2C stdout=2C stderr)=3B >> ! handle :=3D Process.Create (t.shell=2C SUBARRAY (args=2C 0=2C= > n_shell_args)=2C >> ! stdin :=3D stdin=2C stdout :=3D std= >out=2C >> ! stderr :=3D stderr)=3B >> END=3B >> EXCEPT >> | Thread.Alerted =3D> >> --- 1565=2C1594 ---- >> END=3B >> ELSE >> FlushIO ()=3B >> ! IF t.doRecord THEN >> ! handle :=3D NIL=3B >> ! WITH a =3D NEW(REF ARRAY OF TEXT=2C n_shell_args) DO >> ! a^ :=3D SUBARRAY(args=2C0=2Cn_shell_args)=3B >> ! VAR wrx : Wr.T=3B BEGIN >> ! IF echo OR t.do_echo THEN >> ! wrx :=3D wr >> ! ELSE >> ! wrx :=3D NIL >> ! END=3B >> ! t.promises.addhi(NEW(ExecPromise=2C >> ! cmd :=3D t.shell=2C >> ! wr :=3D wrx=2C >> ! args :=3D a=2C >> ! t :=3D t=2C >> ! ignore_errors :=3D ignore_errors)) >> ! END >> ! END >> ! ELSE >> ! Process.GetStandardFileHandles (stdin=2C stdout=2C stderr)= >=3B >> ! handle :=3D Process.Create (t.shell=2C SUBARRAY (args=2C 0= >=2C n_shell_args)=2C >> ! stdin :=3D stdin=2C stdout :=3D s= >tdout=2C >> ! stderr :=3D stderr)=3B >> ! END=3B >> END=3B >> EXCEPT >> | Thread.Alerted =3D> >> *************** >> *** 1573=2C1579 **** >> END=3B >> =20 >> (* wait for everything to shutdown... *) >> ! exit_code :=3D Process.Wait (handle)=3B >> END=3B >> =20 >> IF onlyTry THEN >> --- 1603=2C1613 ---- >> END=3B >> =20 >> (* wait for everything to shutdown... *) >> ! IF handle =3D NIL THEN >> ! exit_code :=3D 0=20 >> ! ELSE >> ! exit_code :=3D Process.Wait (handle)=3B >> ! END=3B (* else we're only promising *) >> END=3B >> =20 >> IF onlyTry THEN >> *************** >> *** 1589=2C1594 **** >> --- 1623=2C1664 ---- >> =20 >> END ExecCommand=3B >> =20 >> + TYPE=20 >> + ExecPromise =3D Promise OBJECT >> + cmd : TEXT=3B >> + args : REF ARRAY OF TEXT=3B >> + t : T=3B >> + wr : Wr.T=3B >> + ignore_errors : BOOLEAN=3B >> + OVERRIDES >> + fulfil :=3D FulfilExecPromise=3B >> + END=3B >> +=20 >> + PROCEDURE FulfilExecPromise(ep : ExecPromise) RAISES { Error } =3D=20 >> + VAR >> + stdin=2C stdout=2C stderr: File.T=3B >> + BEGIN >> + Process.GetStandardFileHandles (stdin=2C stdout=2C stderr)=3B >> + TRY >> + IF ep.wr # NIL THEN >> + Wr.PutText (ep.wr=2C ep.args[1])=3B >> + Wr.PutText (ep.wr=2C Wr.EOL)=3B >> + FlushIO ()=3B >> + END=3B >> + WITH handle =3D Process.Create (ep.cmd=2C ep.args^=2C >> + stdin :=3D stdin=2C stdout :=3D std= >out=2C >> + stderr :=3D stderr)=2C >> + exit_code =3D Process.Wait(handle) DO >> + IF exit_code # 0 AND NOT ep.ignore_errors THEN >> + Err (ep.t=2C Fmt.F("exit %s: %s"=2C Fmt.Int(exit_code)=2C ep.= >cmd)) >> + END >> + END >> + EXCEPT >> + OSError.E (ec) =3D> >> + Err (ep.t=2C Fmt.F ("exec failed%s *** %s"=2C OSErr (ec)=2C e= >p.cmd))=3B >> + END >> + END FulfilExecPromise=3B >> +=20 >> PROCEDURE KillProcess (handle: Process.T) =3D >> BEGIN >> IF (handle # NIL) THEN >>=20 > = > >--_b09106d1-2a84-4ffe-876c-0b2e4cdfd0b0_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >Yes=2C gcc is slow and using as is wasteful.
Yes this should all be mult= >i-threaded.
However it seems to me that m3front has many globals and the= >refore
really making things multi-threaded isn't going to be easy.
I = >think you are mostly wasting your time on user threads though.


H= >ave you tried the NT/x86 system?
It is very fast.


In the gcc = >system=2C each .mc file is run through m3cgc1 to produce an .ms file=2C
= >which is run through as to produce an object file .mo.


In the NT= >/x86 system=2C cm3 outputs .obj files directly.
No .mc files. No m3cgc1 = >or as.
Granted=2C I'd just as soon output .c files=2C and hope for a C c= >ompiler that doesn't run a separate assembler.


My timing might l= >oosen up a bit and I hope to spend some on the thread stress tester using p= >threads. Next weekend.
Am I right though that Darwin works fine?
Sola= >ris? FreeBSD? NetBSD? OpenBSD? Linux? I know=2C not Linux.
You generally= > don't need any of these systems.
All of the x86/AMD64 systems can run i= >n a virtual machine.
Solaris/sparc we have good access.


 = >=3B- Jay


>=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C= > 12 Feb 2011 23:04:14 -0800
>=3B From: mika at async.caltech.edu
>= >=3B CC: jay.krell at cornell.edu
>=3B Subject: [M3devel] More on threadin= >g
>=3B
>=3B
>=3B Hi again m3devel=2C
>=3B
>=3B = >Well I thought I would relate another story having to do with threading
= >>=3B and CM3. It's half-baked at the moment=2C but it only reflects a fe= >w hours'
>=3B work on my part=2C sorry... I'm about to give up now bec= >ause I don't know
>=3B if the problem is in my code=2C or in threading= >=2C or what... maybe it is
>=3B the atfork problem Jay talked about?r>>=3B
>=3B Programmers spend a lot of time waiting for their compi= >lers to run=2C so
>=3B it would be nice if the compilation process wer= >e faster. After perusing
>=3B the source code for the CM3 I realized = >that there is no feedback at all
>=3B from the back-end and assembler = >steps to the rest of the compiler=2C and
>=3B I believe that with the = >GCC back-end the vast majority of the runtime
>=3B of the compiler is = >spent in these stages.
>=3B
>=3B So.... how hard could this be t= >o parallelize? I decided to try making
>=3B the two stages (cm3cg1 an= >d as) into "promises" rather than running them
>=3B on the spot. The = >QMachine adds a promise to call the back-end to a
>=3B RefSeq.T called= > "promises"=2C to which the Builder can also add promises.
>=3B Then= >=2C after making all the promises=2C collect them and run them ("force
&= >gt=3B them" a true Schemer would say) in parallel.
>=3B
>=3B Of = >course something goes wrong somewhere. With 10 threads I am able to
>= >=3B run 31 seconds of CPU time in 19 seconds of wallclock time (using an>>=3B average 160% of CPU) but then something goes wrong and my processr>>=3B is stuck in state "uxmxn". This is using user threads on AMD64_FR= >EEBSD
>=3B on a machine with four CPUs.
>=3B
>=3B Mika= >
>=3B
>=3B P.S. diff attached=2C it's a bit ugly=2C but maybe it= > works on some other
>=3B system.... the code works just fine if you = >run one thread at a time.
>=3B I.e.=2C if you change
>=3B
>= >=3B IF threads.size() >=3B 10 THEN EVAL Thread.Join(threads.remlo())= > END=3B
>=3B
>=3B to
>=3B
>=3B IF threads.size()= > >=3B 0 THEN EVAL Thread.Join(threads.remlo()) END=3B
>=3B
>= >=3B
>=3B
>=3B Index: cm3/src/Builder.m3
>=3B =3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=3B RCS file: /usr/cvs/cm3/m3= >-sys/cm3/src/Builder.m3=2Cv
>=3B retrieving revision 1.36
>=3B di= >ff -c -r1.36 Builder.m3
>=3B *** cm3/src/Builder.m3 24 Aug 2010 05: >24:= >24 -0000 1.36
>=3B --- cm3/src/Builder.m3 13 Feb 2011 06:47:40 -0 >000r>>=3B ***************
>=3B *** 15=2C20 ****
>=3B --- 15=2C22 -= >---
>=3B IMPORT QIdent=3B
>=3B FROM Target IMPORT M3BackendMo= >de_t=2C BackendAssembly=2C BackendModeStrings=3B
>=3B FROM M3Path IM= >PORT OSKind=2C OSKindStrings=3B
>=3B + IMPORT Pathname=3B
>=3B + = >IMPORT RefSeq=3B
>=3B
>=3B TYPE
>=3B UK =3D M3Unit.= >Kind=3B
>=3B ***************
>=3B *** 131=2C136 ****
>=3B --= >- 133=2C139 ----
>=3B link_coverage : TEXT=3B (* c= >overage library *)
>=3B m3_front_flags: Arg.List=3B (*= > configuration options for the front *)
>=3B m3_options : Arg= >.List=3B (* misc. user options for the frontend *)
>=3B + = > delayBackend :=3D FALSE=3B
>=3B END=3B
>=3B
>=3B = >TYPE
>=3B ***************
>=3B *** 932=2C937 ****
>=3B --- 9= >35=2C956 ----
>=3B
>=3B (*----------------------------------= >-------------------------- compilation --*)
>=3B
>=3B + TYPE M= >arkerPromise =3D QMachine.Promise BRANDED OBJECT OVERRIDES fulfil :=3D Fulf= >ilNothing END=3B
>=3B +
>=3B + PROCEDURE FulfilNothing(<=3B*UN= >USED*>=3Bmp : MarkerPromise) =3D BEGIN END FulfilNothing=3B
>=3B + <= >br>>=3B + TYPE SeqClosure =3D Thread.Closure OBJECT seq : RefSeq.T=3B OVE= >RRIDES apply :=3D SeqApply END=3B
>=3B +
>=3B + PROCEDURE SeqApp= >ly(cl : SeqClosure) : REFANY =3D
>=3B + BEGIN
>=3B + FOR i = >:=3D 0 TO cl.seq.size()-1 DO
>=3B + WITH p =3D NARROW(cl.seq.get= >(i)=2CQMachine.Promise) DO
>=3B + p.fulfil()
>=3B + = > END
>=3B + END=3B
>=3B + RETURN NIL
>=3B + END Se= >qApply=3B
>=3B +
>=3B PROCEDURE CompileEverything (s: State=3B= > schedule: SourceList) =3D
>=3B VAR u: M3Unit.T=3B
>=3B = >BEGIN
>=3B ***************
>=3B *** 941=2C948 ****
>=3B r>>=3B (* compile all the sources using the initial schedule *)
= >>=3B FOR i :=3D 0 TO LAST (schedule^) DO
>=3B ! CompileO= >ne (s=2C schedule[i])=3B
>=3B END=3B
>=3B FlushPendin= >g (s)=3B
>=3B
>=3B (* recompile any interfaces where we = >goofed on the exports *)
>=3B --- 960=2C1000 ----
>=3B
>= >=3B (* compile all the sources using the initial schedule *)
>= >=3B FOR i :=3D 0 TO LAST (schedule^) DO
>=3B ! s.delayBack= >end :=3D TRUE=3B
>=3B ! TRY
>=3B ! CompileOne (s=2C= > schedule[i])=3B
>=3B ! FINALLY
>=3B ! s.delayBacke= >nd :=3D FALSE=3B
>=3B ! END=3B
>=3B !
>=3B ! s.= >machine.promises.addhi(NEW(MarkerPromise))=3B
>=3B !
>=3B ! = >END=3B
>=3B !
>=3B ! VAR
>=3B ! curSeq :=3D NEW(= >RefSeq.T).init()=3B
>=3B ! threads :=3D NEW(RefSeq.T).init()=3B<= >br>>=3B ! BEGIN
>=3B ! FOR i :=3D 0 TO s.machine.promises.= >size()-1 DO
>=3B ! WITH p =3D s.machine.promises.get(i) DO
= >>=3B ! curSeq.addhi(p)=3B
>=3B ! IF i =3D s.mach= >ine.promises.size()-1 OR ISTYPE(p=2CMarkerPromise) THEN
>=3B ! = > WITH cl =3D NEW(SeqClosure=2C seq :=3D curSeq) DO
>=3B ! = > threads.addhi (Thread.Fork(cl))=3B
>=3B !
>=3B ! = > IF threads.size() >=3B 10 THEN EVAL Thread.Join(threads.remlo()) END= >=3B
>=3B !
>=3B ! curSeq :=3D NEW(Re= >fSeq.T).init()
>=3B ! END
>=3B ! END
>= >=3B ! END
>=3B ! END=3B
>=3B ! WHILE threads.= >size() >=3B 0 DO EVAL Thread.Join(threads.remlo()) END=3B
>=3B = > END=3B
>=3B +
>=3B + EVAL s.machine.promises.in= >it()=3B
>=3B +
>=3B +
>=3B FlushPending (s)=3B
&g= >t=3B
>=3B (* recompile any interfaces where we goofed on the = >exports *)
>=3B ***************
>=3B *** 1151=2C1156 ****
>= >=3B --- 1203=2C1227 ----
>=3B END=3B
>=3B END CompileM3= >=3B
>=3B
>=3B + TYPE
>=3B + NotePromise =3D QMachine.Pr= >omise OBJECT
>=3B + nam : Pathname.T=3B
>=3B + OVERRIDES>>=3B + fulfil :=3D FulfilNP=3B
>=3B + END=3B
>=3B +
= >>=3B + RemovePromise =3D QMachine.Promise OBJECT
>=3B + nam : = >Pathname.T=3B
>=3B + OVERRIDES
>=3B + fulfil :=3D FulfilRP= >=3B
>=3B + END=3B
>=3B +
>=3B + PROCEDURE FulfilNP(np : N= >otePromise) =3D
>=3B + BEGIN Utils.NoteTempFile(np.nam) END FulfilN= >P=3B
>=3B +
>=3B + PROCEDURE FulfilRP(rp : RemovePromise) =3D r>>=3B + BEGIN Utils.Remove(rp.nam) END FulfilRP=3B
>=3B +
>= >=3B PROCEDURE PushOneM3 (s: State=3B u: M3Unit.T): BOOLEAN =3D
>=3B= > VAR
>=3B tmpC=2C tmpS: TEXT=3B
>=3B ***************>>=3B *** 1191=2C1208 ****
>=3B
>=3B
>=3B | 3 = >=3D>=3B (* -bootstrap=2C +m3back=2C +asm *)
>=3B tmpC := >=3D TempCName (u)=3B
>=3B tmpS :=3D TempSName (u)=3B
>= >=3B IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3Br>>=3B IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END= >=3B
>=3B IF RunM3 (s=2C u=2C tmpC) THEN
>=3B ! = > IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.optimize)
>=3B ! = > AND RunAsm (s=2C tmpS=2C u.object) THEN
>=3B END= >=3B
>=3B need_merge :=3D TRUE=3B
>=3B END= >=3B
>=3B IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END= >=3B
>=3B IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END= >=3B
>=3B
>=3B | 6=2C (* +bootstrap=2C +m3back=2C -asm= > *)
>=3B 7 =3D>=3B (* +bootstrap=2C +m3back=2C +asm *)
&= >gt=3B --- 1262=2C1318 ----
>=3B
>=3B
>=3B | 3 = >=3D>=3B (* -bootstrap=2C +m3back=2C +asm *)
>=3B + IF s.delay= >Backend THEN
>=3B tmpC :=3D TempCName (u)=3B
>=3B = > tmpS :=3D TempSName (u)=3B
>=3B + (*
>=3B IF (NOT = >s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3B
>=3B IF = >(NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END=3B
>=3B + *)
= >>=3B + IF (NOT s.keep_files) THEN
>=3B + s.machin= >e.promises.addhi(NEW(NotePromise=2C nam :=3D tmpC))
>=3B + EN= >D=3B
>=3B + IF (NOT s.keep_files) THEN
>=3B + = >s.machine.promises.addhi(NEW(NotePromise=2C nam :=3D tmpS))
>=3B + = > END=3B
>=3B +
>=3B IF RunM3 (s=2C u=2C tmpC) THE= >N
>=3B ! s.machine.record(TRUE)=3B
>=3B ! TRY= >
>=3B ! IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.o= >ptimize)
>=3B ! AND RunAsm (s=2C tmpS=2C u.object) THEN>>=3B ! END=3B
>=3B ! FINALLY
>=3B ! = > s.machine.record(FALSE)
>=3B END=3B
>=3B + r>>=3B need_merge :=3D TRUE=3B
>=3B END=3B
= >>=3B + (*
>=3B IF (NOT s.keep_files) THEN Utils.Remove (tm= >pC) END=3B
>=3B IF (NOT s.keep_files) THEN Utils.Remove (tmp= >S) END=3B
>=3B + *)
>=3B + IF (NOT s.keep_files) THEN >>=3B + s.machine.promises.addhi(NEW(RemovePromise=2C nam :=3D = >tmpC))
>=3B + END=3B
>=3B + IF (NOT s.keep_files= >) THEN
>=3B + s.machine.promises.addhi(NEW(RemovePromise=2C= > nam :=3D tmpS))
>=3B + END=3B
>=3B +
>=3B + = > ELSE
>=3B + tmpC :=3D TempCName (u)=3B
>=3B + = > tmpS :=3D TempSName (u)=3B
>=3B + IF (NOT s.keep_files) TH= >EN Utils.NoteTempFile (tmpC) END=3B
>=3B + IF (NOT s.keep_fi= >les) THEN Utils.NoteTempFile (tmpS) END=3B
>=3B + IF RunM3 (= >s=2C u=2C tmpC) THEN
>=3B + IF RunM3Back (s=2C tmpC=2C tm= >pS=2C u.debug=2C u.optimize)
>=3B + AND RunAsm (s=2C tmpS= >=2C u.object) THEN
>=3B + END=3B
>=3B + n= >eed_merge :=3D TRUE=3B
>=3B + END=3B
>=3B + I= >F (NOT s.keep_files) THEN Utils.Remove (tmpC) END=3B
>=3B + = >IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END=3B
>=3B + END= >
>=3B
>=3B | 6=2C (* +bootstrap=2C +m3back=2C -asm *)= >
>=3B 7 =3D>=3B (* +bootstrap=2C +m3back=2C +asm *)
>= >=3B Index: cm3/src/m3makefile
>=3B =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D
>=3B RCS file: /usr/cvs/cm3/m3-sys/cm3/src/m3makefi= >le=2Cv
>=3B retrieving revision 1.24
>=3B diff -c -r1.24 m3makefi= >le
>=3B *** cm3/src/m3makefile 8 Dec 2010 07:30:57 -0000 1.24
>>= >=3B --- cm3/src/m3makefile 13 Feb 2011 06:47:42 -0000
>=3B ********** >*= >****
>=3B *** 29=2C34 ****
>=3B --- 29=2C35 ----
>=3B end<= >br>>=3B module ("Arg")
>=3B module ("Builder")
>=3B modul= >e ("Dirs")
>=3B module ("M3Build")
>=3B module ("M3Loc")
&= >gt=3B Index: m3quake/src/QMachine.i3
>=3B =3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D
>=3B RCS file: /usr/cvs/cm3/m3-sys/m3quake/sr= >c/QMachine.i3=2Cv
>=3B retrieving revision 1.6
>=3B diff -c -r1.6= > QMachine.i3
>=3B *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -00 >0= >0 1.6
>=3B --- m3quake/src/QMachine.i3 13 Feb 2011 06:47:43 -0 >000
&= >gt=3B ***************
>=3B *** 8=2C13 ****
>=3B --- 8=2C14 ----r>>=3B
>=3B IMPORT Thread=2C Wr=2C QValue=2C QCode=3B
>=3B= > FROM Quake IMPORT Machine=2C Error=2C ID=2C IDMap=3B
>=3B + IMPORT = >RefSeq=3B
>=3B
>=3B REVEAL
>=3B T <=3B: T_=3B
= >>=3B ***************
>=3B *** 15=2C20 ****
>=3B --- 16=2C22 ---= >-
>=3B T =3D Machine=3B
>=3B T_ =3D OBJECT
>=3B = > map: IDMap :=3D NIL=3B (* READONLY *)
>=3B + promises : RefSeq.T= >=3B
>=3B METHODS
>=3B init (map: IDMap): T=3B
&= >gt=3B evaluate (s: QCode.Stream) RAISES {Error= >=2C Thread.Alerted}=3B
>=3B ***************
>=3B *** 37=2C42 ****= >
>=3B --- 39=2C46 ----
>=3B set_wr (wr: Wr.T)=3B
>= >=3B exec_echo (b: BOOLEAN): BOOLEAN=3B
>=3B trace (b: = >BOOLEAN)=3B
>=3B +
>=3B + record(on : BOOLEAN)=3B (*= > instead of performing certain acts=2C promise *)
>=3B END=3B
&= >gt=3B
>=3B PROCEDURE PushBool (t: T=3B b: BOOLEAN)=3B
>=3B = >***************
>=3B *** 51=2C54 ****
>=3B --- 55=2C63 ----
&g= >t=3B
>=3B PROCEDURE GetEnv (default=2C v0=2C v1=2C v2=2C v3=2C v4= >: TEXT :=3D NIL): TEXT=3B
>=3B
>=3B + TYPE Promise =3D OBJECT = >METHODS fulfil() RAISES { Error } END=3B
>=3B +
>=3B END QMac= >hine.
>=3B +
>=3B +
>=3B +
>=3B Index: m3quake/src/Q= >Machine.m3
>=3B =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D>>=3B RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3=2Cv
>=3B= > retrieving revision 1.35
>=3B diff -c -r1.35 QMachine.m3
>=3B **= >* m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35
>=3B >--- m3qu= >ake/src/QMachine.m3 13 Feb 2011 06:47:44 -0000
>=3B ***************
= >>>=3B *** 16=2C21 ****
>=3B --- 16=2C22 ----
>=3B IMPORT Text= >Utils=2C FSUtils=2C System=2C DirStack=3B (* sysutils *)
>=3B IMPORT= > Compiler=3B
>=3B IMPORT M3Path=3B
>=3B + IMPORT RefSeq=3B
&= >gt=3B
>=3B CONST
>=3B OnUnix =3D (Compiler.ThisOS =3D Co= >mpiler.OS.POSIX)=3B
>=3B ***************
>=3B *** 44=2C49 ****>>=3B --- 45=2C52 ----
>=3B shell : TEXT :=3D NIL= >=3B
>=3B sh_option : TEXT :=3D NIL=3B
>=3B tm= >p_dir : TEXT :=3D NIL=3B
>=3B +
>=3B + doRecord := >=3D FALSE=3B
>=3B OVERRIDES
>=3B init :=3D Init=3B= >
>=3B evaluate :=3D Evaluate=3B
>=3B ***************
&g= >t=3B *** 66=2C73 ****
>=3B --- 69=2C81 ----
>=3B set_wr = >:=3D SetWr=3B
>=3B exec_echo :=3D ExecEcho=3B
>=3B tr= >ace :=3D Trace=3B
>=3B +
>=3B + record :=3D Record=3B>>=3B END=3B
>=3B
>=3B + PROCEDURE Record(t : T=3B on : = >BOOLEAN) =3D
>=3B + BEGIN t.doRecord :=3D on END Record=3B
>= >=3B +
>=3B TYPE
>=3B Registers =3D RECORD
>=3B = >cp : QCode.Stream :=3D NIL=3B (* code pointer *)
>=3B **************= >*
>=3B *** 139=2C144 ****
>=3B --- 147=2C154 ----
>=3B = > t.globals :=3D NEW (IntRefTbl.Default).init ()=3B
>=3B t.def= >ault_wr :=3D Stdio.stdout=3B
>=3B
>=3B + t.promises :=3D N= >EW(RefSeq.T).init()=3B
>=3B +
>=3B InitOSEnv (t)=3B
>= >=3B InitBuiltins (t)=3B
>=3B
>=3B ***************
>= >=3B *** 1555=2C1564 ****
>=3B END=3B
>=3B E= >LSE
>=3B FlushIO ()=3B
>=3B ! Process.GetSt= >andardFileHandles (stdin=2C stdout=2C stderr)=3B
>=3B ! hand= >le :=3D Process.Create (t.shell=2C SUBARRAY (args=2C 0=2C n_shell_args)=2C<= >br>>=3B ! stdin :=3D stdin=2C stdout = >:=3D stdout=2C
>=3B ! stderr :=3D = >stderr)=3B
>=3B END=3B
>=3B EXCEPT
>=3B = > | Thread.Alerted =3D>=3B
>=3B --- 1565=2C1594 ----
>=3B = > END=3B
>=3B ELSE
>=3B FlushIO ()= >=3B
>=3B ! IF t.doRecord THEN
>=3B ! handle= > :=3D NIL=3B
>=3B ! WITH a =3D NEW(REF ARRAY OF TEXT=2C n_= >shell_args) DO
>=3B ! a^ :=3D SUBARRAY(args=2C0=2Cn_shel= >l_args)=3B
>=3B ! VAR wrx : Wr.T=3B BEGIN
>=3B ! = > IF echo OR t.do_echo THEN
>=3B ! wrx := >=3D wr
>=3B ! ELSE
>=3B ! wrx := >=3D NIL
>=3B ! END=3B
>=3B ! t.pr= >omises.addhi(NEW(ExecPromise=2C
>=3B ! = > cmd :=3D t.shell=2C
>=3B ! = >wr :=3D wrx=2C
>=3B ! args :=3D a= >=2C
>=3B ! t :=3D t=2C
>=3B = >! ignore_errors :=3D ignore_errors))r>>=3B ! END
>=3B ! END
>=3B ! = > ELSE
>=3B ! Process.GetStandardFileHandles (stdin=2C s= >tdout=2C stderr)=3B
>=3B ! handle :=3D Process.Create (t.s= >hell=2C SUBARRAY (args=2C 0=2C n_shell_args)=2C
>=3B ! = > stdin :=3D stdin=2C stdout :=3D stdout=2C
>=3B = >! stderr :=3D stderr)=3B
>=3B ! = > END=3B
>=3B END=3B
>=3B EXCEPT
>= >=3B | Thread.Alerted =3D>=3B
>=3B ***************
>=3B = >*** 1573=2C1579 ****
>=3B END=3B
>=3B
>=3B = > (* wait for everything to shutdown... *)
>=3B ! exit_code :=3D= > Process.Wait (handle)=3B
>=3B END=3B
>=3B
>=3B = > IF onlyTry THEN
>=3B --- 1603=2C1613 ----
>=3B END=3B<= >br>>=3B
>=3B (* wait for everything to shutdown... *)
= >>=3B ! IF handle =3D NIL THEN
>=3B ! exit_code :=3D 0 = >
>=3B ! ELSE
>=3B ! exit_code :=3D Process.Wait (ha= >ndle)=3B
>=3B ! END=3B (* else we're only promising *)
>=3B= > END=3B
>=3B
>=3B IF onlyTry THEN
>=3B ******= >*********
>=3B *** 1589=2C1594 ****
>=3B --- 1623=2C1664 ----
= >>=3B
>=3B END ExecCommand=3B
>=3B
>=3B + TYPE r>>=3B + ExecPromise =3D Promise OBJECT
>=3B + cmd : TEXT=3Br>>=3B + args : REF ARRAY OF TEXT=3B
>=3B + t : T=3B
>= >=3B + wr : Wr.T=3B
>=3B + ignore_errors : BOOLEAN=3B
>=3B= > + OVERRIDES
>=3B + fulfil :=3D FulfilExecPromise=3B
>=3B += > END=3B
>=3B +
>=3B + PROCEDURE FulfilExecPromise(ep : ExecPro= >mise) RAISES { Error } =3D
>=3B + VAR
>=3B + stdin=2C stdo= >ut=2C stderr: File.T=3B
>=3B + BEGIN
>=3B + Process.GetStan= >dardFileHandles (stdin=2C stdout=2C stderr)=3B
>=3B + TRY
>= >=3B + IF ep.wr # NIL THEN
>=3B + Wr.PutText (ep.wr=2C ep= >.args[1])=3B
>=3B + Wr.PutText (ep.wr=2C Wr.EOL)=3B
>=3B = >+ FlushIO ()=3B
>=3B + END=3B
>=3B + WITH han= >dle =3D Process.Create (ep.cmd=2C ep.args^=2C
>=3B + = > stdin :=3D stdin=2C stdout :=3D stdout=2C
>=3B + = > stderr :=3D stderr)=2C
>=3B + = > exit_code =3D Process.Wait(handle) DO
>=3B + IF exit_code= > # 0 AND NOT ep.ignore_errors THEN
>=3B + Err (ep.t=2C Fmt.F= >("exit %s: %s"=2C Fmt.Int(exit_code)=2C ep.cmd))
>=3B + END>>=3B + END
>=3B + EXCEPT
>=3B + OSError.E (ec)= > =3D>=3B
>=3B + Err (ep.t=2C Fmt.F ("exec failed%s *** %s"= >=2C OSErr (ec)=2C ep.cmd))=3B
>=3B + END
>=3B + END FulfilE= >xecPromise=3B
>=3B +
>=3B PROCEDURE KillProcess (handle: Proce= >ss.T) =3D
>=3B BEGIN
>=3B IF (handle # NIL) THEN
>= >=3B
>= > >--_b09106d1-2a84-4ffe-876c-0b2e4cdfd0b0_-- From hosking at cs.purdue.edu Sun Feb 13 21:18:03 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:18:03 -0500 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110212204406.88A661A2078@async.async.caltech.edu> References: <20110212204406.88A661A2078@async.async.caltech.edu> Message-ID: Hi Mika, Why was this change needed (I haven't looked closely)? I don't think we diverged much from previous implementations of user threads, which presumably used to work for you. On Feb 12, 2011, at 3:44 PM, Mika Nystrom wrote: > > Index: ThreadPosix.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v > retrieving revision 1.16 > diff -c -r1.16 ThreadPosix.i3 > *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 > --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 > *************** > *** 39,42 **** > --- 39,45 ---- > > (*---------------------------------------------------------------------------*) > > + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> > + PROCEDURE ValueOfSIGCHLD(): int; > + > END ThreadPosix. > Index: ThreadPosix.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v > retrieving revision 1.73 > diff -c -r1.73 ThreadPosix.m3 > *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 > --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 > *************** > *** 97,102 **** > --- 97,105 ---- > (* if state = pausing, the time at which we can restart *) > waitingForTime: Time.T; > > + (* if state = pausing, the signal that truncates the pause *) > + waitingForSig: int := -1; > + > (* true if we are waiting during an AlertWait or AlertJoin > or AlertPause *) > alertable: BOOLEAN := FALSE; > *************** > *** 147,152 **** > --- 150,160 ---- > > defaultStackSize := 3000; > > + (* note that even though the "heavy machinery" is only used for > + multipleThreads, we still need to set up the signal handler so > + that we can catch signals from other sources than thread switching. > + e.g., we use SIGCHLD to speed up Process.Wait *) > + > VAR > stats: RECORD > n_forks := 0; > *************** > *** 511,527 **** > XPause(until, FALSE); > END Pause; > > PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= > VAR until := n + Time.Now (); > BEGIN > XPause(until, TRUE); > END AlertPause; > > ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE) RAISES {Alerted} = > BEGIN > INC (inCritical); > self.waitingForTime := until; > self.alertable := alertable; > ICannotRun (State.pausing); > DEC (inCritical); > InternalYield (); > --- 519,546 ---- > XPause(until, FALSE); > END Pause; > > + PROCEDURE SigPause(n: LONGREAL; sig: int)= > + <*FATAL Alerted*> > + VAR until := n + Time.Now (); > + BEGIN > + XPause(until, FALSE, sig); > + END SigPause; > + > PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= > VAR until := n + Time.Now (); > BEGIN > XPause(until, TRUE); > END AlertPause; > > ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) > ! RAISES {Alerted} = > BEGIN > INC (inCritical); > self.waitingForTime := until; > self.alertable := alertable; > + IF sig # -1 THEN > + self.waitingForSig := sig > + END; > ICannotRun (State.pausing); > DEC (inCritical); > InternalYield (); > *************** > *** 703,712 **** > END; > END StartSwitching; > > ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); > ! IF inCritical = 0 AND heapState.inCritical = 0 THEN InternalYield () END; > END switch_thread; > > (*------------------------------------------------------------- scheduler ---*) > --- 722,750 ---- > END; > END StartSwitching; > > ! CONST MaxSigs = 64; > ! TYPE Sig = [ 0..MaxSigs-1 ]; > ! > ! (* in order to listen to other signals, they have to be enabled in > ! allow_sigvtalrm as well *) > ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); > ! > ! gotSigs := SET OF Sig { }; > ! > ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); > ! > ! INC(inCritical); > ! (* mark signal as being delivered *) > ! IF sig >= 0 AND sig < MaxSigs THEN > ! gotSigs := gotSigs + SET OF Sig { sig } > ! END; > ! DEC(inCritical); > ! > ! IF inCritical = 0 AND heapState.inCritical = 0 THEN > ! InternalYield () > ! END; > END switch_thread; > > (*------------------------------------------------------------- scheduler ---*) > *************** > *** 782,792 **** > IF t.alertable AND t.alertPending THEN > CanRun (t); > EXIT; > > ELSIF t.waitingForTime <= now THEN > CanRun (t); > EXIT; > ! > ELSIF NOT somePausing THEN > earliest := t.waitingForTime; > somePausing := TRUE; > --- 820,835 ---- > IF t.alertable AND t.alertPending THEN > CanRun (t); > EXIT; > + > + ELSIF t.waitingForSig IN gotSigs THEN > + t.waitingForSig := -1; > + CanRun(t); > + EXIT; > > ELSIF t.waitingForTime <= now THEN > CanRun (t); > EXIT; > ! > ELSIF NOT somePausing THEN > earliest := t.waitingForTime; > somePausing := TRUE; > *************** > *** 886,891 **** > --- 929,936 ---- > END; > END; > > + gotSigs := SET OF Sig {}; > + > IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN > IF perfOn THEN PerfRunning (t.id); END; > (* At least one thread wants to run; transfer to it *) > *************** > *** 948,960 **** > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > ! CONST Delay = 0.1D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > ! Pause(Delay); > END; > END WaitProcess; > > --- 993,1005 ---- > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > ! CONST Delay = 10.0D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > ! SigPause(Delay,SIGCHLD); > END; > END WaitProcess; > > *************** > *** 1361,1364 **** > --- 1406,1412 ---- > VAR debug := RTParams.IsPresent ("debugthreads"); > > BEGIN > + (* we need to call set up the signal handler for other reasons than > + just thread switching now *) > + setup_sigvtalrm (switch_thread); > END ThreadPosix. > Index: ThreadPosixC.c > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v > retrieving revision 1.59 > diff -c -r1.59 ThreadPosixC.c > *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 > --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 > *************** > *** 78,88 **** > --- 78,97 ---- > > sigemptyset(&ThreadSwitchSignal); > sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); > + sigaddset(&ThreadSwitchSignal, SIGCHLD); > > act.sa_handler = handler; > act.sa_flags = SA_RESTART; > sigemptyset(&(act.sa_mask)); > if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); > + if (sigaction (SIGCHLD, &act, NULL)) abort(); > + } > + > + int > + __cdecl > + ThreadPosix__value_of_SIGCHLD(void) > + { > + return SIGCHLD; > } > > void From hosking at cs.purdue.edu Sun Feb 13 21:22:13 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:22:13 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110212064812.286E61A2078@async.async.caltech.edu> Message-ID: <35125A4E-2BB1-48B9-87EC-58CFCFA5D5AF@cs.purdue.edu> Looks like stack overflow to me. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 12, 2011, at 9:09 AM, Jay K wrote: > > >LINUXLIBC6 : seg fault > > I've seen this now. > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0xb3200b90 (LWP 18154)] > 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) > at ../src/runtime/common/RTCollector.m3:409 > 409 IF hdr.typecode = RT0.TextLitTypecode THEN RETURN END; > 1: x/i $pc > 0xb6dee58f : mov (%eax),%eax > (gdb) bt 9 > #0 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) > at ../src/runtime/common/RTCollector.m3:409 > #1 0xb6deaa46 in RTHeapMap__Walk (x=0xb6a20014, pc=0xb7738648, v=0xb6a9000c) > at ../src/runtime/common/RTHeapMap.m3:202 > #2 0xb6dea341 in RTHeapMap__DoWalkRef (t=0xb7738694, a=0xb6a20014, > v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:62 > #3 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739c54, a=0xb6a2000c, > v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 > #4 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739e68, a=0xb6a2000c, > v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 > #5 0xb6dea2bf in RTHeapMap__WalkRef (h=0xb6a20008, v=0xb6a9000c) > at ../src/runtime/common/RTHeapMap.m3:47 > #6 0xb6df0636 in RTCollector__CleanBetween (h=0xb6a20008, he=0xb6a30000, > clean=0 '\0') at ../src/runtime/common/RTCollector.m3:1091 > #7 0xb6df0473 in RTCollector__CleanPage (page=0xb6a20000) > at ../src/runtime/common/RTCollector.m3:1064 > #8 0xb6defc96 in RTCollector__CollectSomeInStateZero () > at ../src/runtime/common/RTCollector.m3:885 > (More stack frames follow...) > (gdb) p $eax > $2 = 2097148 > (gdb) p/x $eax > $3 = 0x1ffffc > (gdb) x/x $eax+4 > 0x200000: Cannot access memory at address 0x200000 > (gdb) x/x $eax > 0x1ffffc: Cannot access memory at address 0x1ffffc > > > Maybe I should try with a correctly sized jmpbuf to assure it isn't stack overflow? > But notice that the next page doesn't work either. Actually I tried adding > a lot to $eax and still not accessible. > > > Maybe maybe maybe maybe I'll dig into this. But I'm very busy lately and maybe indefinitely into future.. > > > Thanks, > - Jay > > > > To: jay.krell at cornell.edu; hosking at cs.purdue.edu > > Date: Fri, 11 Feb 2011 22:48:12 -0800 > > From: mika at async.caltech.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] threading on Windows? > > > > Well, with the following changes: re-bootstrapping on the two Linux > > systems and removal of the -pthread flag on FreeBSD4, all is well. > > User threading works on all the architectures listed below, at least > > as far as the thread testing program can test it. (It seems that because > > of the way pthread_atfork is coded, pthread is not needed at all on > > FreeBSD4.) > > > > Mika > > > > Mika Nystrom writes: > > >Another odd datapoint is that even rather aggressive testing seems not > > >to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user > > >threads, of course. > > > > > >So status right now of user threads seems to be: > > > > > >FreeBSD4 : strange failure in libpthread as reported by gdb > > >LINUXLIBC6 : seg fault > > >AMD64_LINUX : seg fault > > >AMD64_FREEBSD : OK > > >I386_DARWIN : OK -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sun Feb 13 21:32:05 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:32:05 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <20110213003358.00D2A1A2078@async.async.caltech.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu>, <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> <20110213003358.00D2A1A2078@async.async.caltech.edu> Message-ID: I am confused. This backtrace looks like it is from the release, which we know has problems. On Feb 12, 2011, at 7:33 PM, Mika Nystrom wrote: > P.S. backtraces of deadlock: > > Thread 3 (Thread 0x40862950 (LWP 19974)): > #0 0x00002b9d49f730e1 in nanosleep () from /lib/libpthread.so.0 > #1 0x000000000045806d in ThreadPThread__Nanosleep (req=Cannot access memory at address 0x140861b08 > ) at ../src/thread/PTHREAD/ThreadPThreadC.c:500 > #2 0x0000000000454294 in ThreadPThread__CommonSleep () at ../src/thread/PTHREAD/ThreadPThread.m3:551 > #3 0x0000000000456438 in ThreadPThread__StopWorld () at ../src/thread/PTHREAD/ThreadPThread.m3:1086 > #4 0x0000000000455787 in RTThread__SuspendOthers () at ../src/thread/PTHREAD/ThreadPThread.m3:812 > #5 0x000000000043970c in RTCollector__CollectSomeInStateZero () at ../src/runtime/common/RTCollector.m3:746 > #6 0x00000000004396b8 in RTCollector__CollectSome () at ../src/runtime/common/RTCollector.m3:720 > #7 0x00000000004390b9 in RTHeapRep__CollectEnough () at ../src/runtime/common/RTCollector.m3:654 > #8 0x0000000000436111 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=Cannot access memory at address 0x140861dc8 > ) at ../src/runtime/common/RTAllocator.m3:363 > #9 0x000000000043596f in RTAllocator__GetOpenArray (M3_Eic7CK_def=Cannot access memory at address 0x140861eb8 > ) at ../src/runtime/common/RTAllocator.m3:292 > #10 0x0000000000434ab1 in RTHooks__AllocateOpenArray (M3_AJWxb1_defn=Cannot access memory at address 0x140861f38 > ) at ../src/runtime/common/RTAllocator.m3:141 > #11 0x0000000000407030 in Main__AApply (M3_AP7a1g_cl=Cannot access memory at address 0x140861f88 > ) at ../src/Main.m3:234 > #12 0x0000000000453b3f in ThreadPThread__RunThread (M3_DMxDjQ_me=Cannot access memory at address 0x140862028 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:475 > #13 0x00000000004537d3 in ThreadPThread__ThreadBase (M3_AJWxb1_param=Cannot access memory at address 0x1408620d8 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:447 > #14 0x00002b9d49f6bfc7 in start_thread () from /lib/libpthread.so.0 > #15 0x00002b9d4b3bd64d in clone () from /lib/libc.so.6 > #16 0x0000000000000000 in ?? () > > Thread 2 (Thread 0x40661950 (LWP 19973)): > #0 0x00002b9d49f72384 in __lll_lock_wait () from /lib/libpthread.so.0 > #1 0x00002b9d49f6dbf0 in _L_lock_102 () from /lib/libpthread.so.0 > #2 0x00002b9d49f6d4fe in pthread_mutex_lock () from /lib/libpthread.so.0 > #3 0x000000000045815e in ThreadPThread__pthread_mutex_lock (m=Cannot access memory at address 0x140660d68 > ) at ../src/thread/PTHREAD/ThreadPThreadC.c:557 > #4 0x00000000004575af in RTOS__LockHeap () at ../src/thread/PTHREAD/ThreadPThread.m3:1434 > #5 0x0000000000436107 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=Cannot access memory at address 0x140660dc8 > ) at ../src/runtime/common/RTAllocator.m3:361 > #6 0x000000000043596f in RTAllocator__GetOpenArray (M3_Eic7CK_def=Cannot access memory at address 0x140660eb8 > ) at ../src/runtime/common/RTAllocator.m3:292 > #7 0x0000000000434ab1 in RTHooks__AllocateOpenArray (M3_AJWxb1_defn=Cannot access memory at address 0x140660f38 > ) at ../src/runtime/common/RTAllocator.m3:141 > #8 0x0000000000407030 in Main__AApply (M3_AP7a1g_cl=Cannot access memory at address 0x140660f88 > ) at ../src/Main.m3:234 > #9 0x0000000000453b3f in ThreadPThread__RunThread (M3_DMxDjQ_me=Cannot access memory at address 0x140661028 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:475 > #10 0x00000000004537d3 in ThreadPThread__ThreadBase (M3_AJWxb1_param=Cannot access memory at address 0x1406610d8 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:447 > #11 0x00002b9d49f6bfc7 in start_thread () from /lib/libpthread.so.0 > #12 0x00002b9d4b3bd64d in clone () from /lib/libc.so.6 > #13 0x0000000000000000 in ?? () > > Thread 1 (Thread 0x2b9d4d32a1a0 (LWP 19965)): > #0 0x00002b9d4b3202a2 in sigsuspend () from /lib/libc.so.6 > #1 0x0000000000457ba9 in ThreadPThread__sigsuspend () at ../src/thread/PTHREAD/ThreadPThreadC.c:141 > #2 0x0000000000456a0d in ThreadPThread__SignalHandler (M3_DLS2Hj_sig=Cannot access memory at address 0x80001dde057c > ) at ../src/thread/PTHREAD/ThreadPThread.m3:1205 > #3 > #4 0x00002b9d49f6ffad in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 > #5 0x00000000004580b3 in ThreadPThread__pthread_cond_timedwait (cond=Cannot access memory at address 0x80001dde0a88 > ) at ../src/thread/PTHREAD/ThreadPThreadC.c:515 > #6 0x00000000004544cf in ThreadPThread__XPause (M3_DMxDjQ_self=Cannot access memory at address 0x80001dde0ab8 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:580 > #7 0x00000000004545b2 in Thread__Pause (M3_CtKayy_n=Cannot access memory at address 0x80001dde0b08 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:595 > #8 0x00000000004097ee in Main_M3 (M3_AcxOUs_mode=Cannot access memory at address 0x80001dde0bc8 > ) at ../src/Main.m3:449 > #9 0x0000000000445799 in RTLinker__RunMainBody (M3_DjPxE3_m=Cannot access memory at address 0x80001dde0f58 > ) at ../src/runtime/common/RTLinker.m3:406 > #10 0x00000000004448f2 in RTLinker__AddUnitI (M3_DjPxE3_m=Cannot access memory at address 0x80001dde0ff8 > ) at ../src/runtime/common/RTLinker.m3:113 > #11 0x0000000000444990 in RTLinker__AddUnit (M3_DjPxE5_b=Cannot access memory at address 0x80001dde1018 > ) at ../src/runtime/common/RTLinker.m3:122 > #12 0x000000000040652c in main (argc=Cannot access memory at address 0x80001dde1068 > ) at _m3main.mc:4 From mika at async.caltech.edu Sun Feb 13 21:37:26 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 12:37:26 -0800 Subject: [M3devel] SIGCHLD diff In-Reply-To: References: <20110212204406.88A661A2078@async.async.caltech.edu> Message-ID: <20110213203726.82ADB1A2078@async.async.caltech.edu> Tony, The old user threads has an old problem with Process.Wait. It has a Thread.Pause(0.1d0) in it. This causes the compiler to run three times slower than it should. The m3build I've been using for years has that Thread.Pause removed (so it uses 100% CPU instead, busy-waiting...) But yeah user threads works the same as it used to (except for now calling pthread_atfork, which I'll return to in another email...) Mika Tony Hosking writes: >Hi Mika, Why was this change needed (I haven't looked closely)? I don't = >think we diverged much from previous implementations of user threads, = >which presumably used to work for you. > >On Feb 12, 2011, at 3:44 PM, Mika Nystrom wrote: > >>=20 >> Index: ThreadPosix.i3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: = >/usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v >> retrieving revision 1.16 >> diff -c -r1.16 ThreadPosix.i3 >> *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 >> --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 >> *************** >> *** 39,42 **** >> --- 39,45 ---- >>=20 >> = >(*------------------------------------------------------------------------= >---*) >>=20 >> + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> >> + PROCEDURE ValueOfSIGCHLD(): int; >> +=20 >> END ThreadPosix. >> Index: ThreadPosix.m3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: = >/usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v >> retrieving revision 1.73 >> diff -c -r1.73 ThreadPosix.m3 >> *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 >> --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 >> *************** >> *** 97,102 **** >> --- 97,105 ---- >> (* if state =3D pausing, the time at which we can restart *) >> waitingForTime: Time.T; >>=20 >> + (* if state =3D pausing, the signal that truncates the pause *) >> + waitingForSig: int :=3D -1; >> +=20 >> (* true if we are waiting during an AlertWait or AlertJoin=20 >> or AlertPause *) >> alertable: BOOLEAN :=3D FALSE; >> *************** >> *** 147,152 **** >> --- 150,160 ---- >>=20 >> defaultStackSize :=3D 3000; >>=20 >> + (* note that even though the "heavy machinery" is only used for=20 >> + multipleThreads, we still need to set up the signal handler so >> + that we can catch signals from other sources than thread = >switching. >> + e.g., we use SIGCHLD to speed up Process.Wait *) >> +=20 >> VAR >> stats: RECORD >> n_forks :=3D 0; >> *************** >> *** 511,527 **** >> XPause(until, FALSE); >> END Pause; >>=20 >> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >> VAR until :=3D n + Time.Now (); >> BEGIN >> XPause(until, TRUE); >> END AlertPause; >>=20 >> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D FALSE) = >RAISES {Alerted} =3D >> BEGIN >> INC (inCritical); >> self.waitingForTime :=3D until; >> self.alertable :=3D alertable; >> ICannotRun (State.pausing); >> DEC (inCritical); >> InternalYield (); >> --- 519,546 ---- >> XPause(until, FALSE); >> END Pause; >>=20 >> + PROCEDURE SigPause(n: LONGREAL; sig: int)=3D >> + <*FATAL Alerted*> >> + VAR until :=3D n + Time.Now (); >> + BEGIN >> + XPause(until, FALSE, sig); >> + END SigPause; >> +=20 >> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >> VAR until :=3D n + Time.Now (); >> BEGIN >> XPause(until, TRUE); >> END AlertPause; >>=20 >> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D FALSE; = >sig:int :=3D -1)=20 >> ! RAISES {Alerted} =3D >> BEGIN >> INC (inCritical); >> self.waitingForTime :=3D until; >> self.alertable :=3D alertable; >> + IF sig # -1 THEN >> + self.waitingForSig :=3D sig >> + END; >> ICannotRun (State.pausing); >> DEC (inCritical); >> InternalYield (); >> *************** >> *** 703,712 **** >> END; >> END StartSwitching; >>=20 >> ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} =3D >> BEGIN >> allow_sigvtalrm (); >> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN = >InternalYield () END; >> END switch_thread; >>=20 >> (*------------------------------------------------------------- = >scheduler ---*) >> --- 722,750 ---- >> END; >> END StartSwitching; >>=20 >> ! CONST MaxSigs =3D 64; >> ! TYPE Sig =3D [ 0..MaxSigs-1 ]; >> !=20 >> ! (* in order to listen to other signals, they have to be enabled in >> ! allow_sigvtalrm as well *) >> ! VAR (*CONST*) SIGCHLD :=3D ValueOfSIGCHLD(); >> ! =20 >> ! gotSigs :=3D SET OF Sig { }; >> !=20 >> ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >> BEGIN >> allow_sigvtalrm (); >> !=20 >> ! INC(inCritical); >> ! (* mark signal as being delivered *) >> ! IF sig >=3D 0 AND sig < MaxSigs THEN >> ! gotSigs :=3D gotSigs + SET OF Sig { sig } >> ! END; >> ! DEC(inCritical); >> !=20 >> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN=20 >> ! InternalYield ()=20 >> ! END; >> END switch_thread; >>=20 >> (*------------------------------------------------------------- = >scheduler ---*) >> *************** >> *** 782,792 **** >> IF t.alertable AND t.alertPending THEN >> CanRun (t); >> EXIT; >>=20 >> ELSIF t.waitingForTime <=3D now THEN >> CanRun (t); >> EXIT; >> ! =20 >> ELSIF NOT somePausing THEN >> earliest :=3D t.waitingForTime; >> somePausing :=3D TRUE; >> --- 820,835 ---- >> IF t.alertable AND t.alertPending THEN >> CanRun (t); >> EXIT; >> + =20 >> + ELSIF t.waitingForSig IN gotSigs THEN >> + t.waitingForSig :=3D -1; >> + CanRun(t); >> + EXIT; >>=20 >> ELSIF t.waitingForTime <=3D now THEN >> CanRun (t); >> EXIT; >> !=20 >> ELSIF NOT somePausing THEN >> earliest :=3D t.waitingForTime; >> somePausing :=3D TRUE; >> *************** >> *** 886,891 **** >> --- 929,936 ---- >> END; >> END; >>=20 >> + gotSigs :=3D SET OF Sig {}; >> +=20 >> IF t.state =3D State.alive AND (scanned OR NOT someBlocking) THEN >> IF perfOn THEN PerfRunning (t.id); END; >> (* At least one thread wants to run; transfer to it *) >> *************** >> *** 948,960 **** >>=20 >> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D >> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >> ! CONST Delay =3D 0.1D0; >> BEGIN >> LOOP >> WITH r =3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >> IF r # 0 THEN RETURN r END; >> END; >> ! Pause(Delay); >> END; >> END WaitProcess; >>=20 >> --- 993,1005 ---- >>=20 >> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D >> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >> ! CONST Delay =3D 10.0D0; >> BEGIN >> LOOP >> WITH r =3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >> IF r # 0 THEN RETURN r END; >> END; >> ! SigPause(Delay,SIGCHLD); >> END; >> END WaitProcess; >>=20 >> *************** >> *** 1361,1364 **** >> --- 1406,1412 ---- >> VAR debug :=3D RTParams.IsPresent ("debugthreads"); >>=20 >> BEGIN >> + (* we need to call set up the signal handler for other reasons = >than >> + just thread switching now *) >> + setup_sigvtalrm (switch_thread); >> END ThreadPosix. >> Index: ThreadPosixC.c >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: = >/usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v >> retrieving revision 1.59 >> diff -c -r1.59 ThreadPosixC.c >> *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 >> --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 >> *************** >> *** 78,88 **** >> --- 78,97 ---- >>=20 >> sigemptyset(&ThreadSwitchSignal); >> sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); >> + sigaddset(&ThreadSwitchSignal, SIGCHLD); >>=20 >> act.sa_handler =3D handler; >> act.sa_flags =3D SA_RESTART; >> sigemptyset(&(act.sa_mask)); >> if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); >> + if (sigaction (SIGCHLD, &act, NULL)) abort(); >> + } >> +=20 >> + int >> + __cdecl >> + ThreadPosix__value_of_SIGCHLD(void) >> + { >> + return SIGCHLD; >> } >>=20 >> void From hosking at cs.purdue.edu Sun Feb 13 21:37:32 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:37:32 -0500 Subject: [M3devel] Performance issues with Process.Wait under userthreads In-Reply-To: <20110212204302.19C781A2078@async.async.caltech.edu> References: <20110212204302.19C781A2078@async.async.caltech.edu> Message-ID: <80DB3538-EAFF-4C03-A117-75EE49BB5D2C@cs.purdue.edu> This all seems pretty reasonable. On Feb 12, 2011, at 3:43 PM, Mika Nystrom wrote: > Hi again m3devel (especially Tony), > > I have finally taken a bite of a problem that has been annoying me > for a very, very long time. > > Under user threads, the code for Process.Wait is as follows (see > ThreadPosix.m3): > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > CONST Delay = 0.1D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > Pause(Delay); > END; > END WaitProcess; > > It inserts a 0.1 second delay after each failed waitpid. This is extremely > annoying for programs that start a long sequence of child processes and > wait for them in sequence. Namely, the compiler itself. As a result > the cm3 compiler (and PM3's m3build) are normally very very slow when > using user threads. For about the last ten years, I've had a hacked > up m3build (for my PM3 installation) that skips the Pause and busy-waits > instead. > > Note there is another problem here. Since the Modula-3 runtime ignores > SIGCHLD, no zombie processes are created since the Unix system automatically > reaps the child processes. I can see this would be a problem since PIDs > are eventually reused and ... couldn't Uexec.waitpid wind up referring to > the wrong process?? > > I will further note that the comment in Process.i3 reads as follows: > > PROCEDURE Wait(p: T): ExitCode; > > Wait until the process with handle p terminates, then free the operating system resources associated with the process and return an exit code indicating the reason for its termination. It is a checked runtime error to call Wait twice on the same process handle. > > I am going to take this as fair warning that Process.Create *may* use > resources that are not going to be released until Process.Wait has been > called. > > I have modified (in my local copy of CM3) the system as follows. > I have come up with a semi-general mechanism for immediately unblocking > a thread on the receipt of a unix signal. > > 1. The system relies on changing > ThreadPosix.XPause such that if a signal is allowed to wake up a threads, > that fact is recorded in a new field in the thread's descriptor > record (of type ThreadPosix.T). > > 2. On receipt of a waited-for unix signal, a mask is set and control > is passed to the thread scheduler which maintains the non-zero mask for > exactly one iteration through the thread ring. > > 3. If a thread is paused and waiting for EITHER a signal or some time, > the thread is released for running and the thread's waiting state is > cleared. > > The changes are more or less as follows: > > 1. I have added a new field of type "int" to ThreadPosix.T: > > (* if state = pausing, the time at which we can restart *) > waitingForTime: Time.T; > > + (* if state = pausing, the signal that truncates the pause *) > + waitingForSig: int := -1; > + > (* true if we are waiting during an AlertWait or AlertJoin > or AlertPause *) > alertable: BOOLEAN := FALSE; > > > 2. Modifications to pause: > > + PROCEDURE SigPause(n: LONGREAL; sig: int)= > + <*FATAL Alerted*> > + VAR until := n + Time.Now (); > + BEGIN > + XPause(until, FALSE, sig); > + END SigPause; > + > PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= > VAR until := n + Time.Now (); > BEGIN > XPause(until, TRUE); > END AlertPause; > > ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) > ! RAISES {Alerted} = > BEGIN > INC (inCritical); > self.waitingForTime := until; > self.alertable := alertable; > + IF sig # -1 THEN > + self.waitingForSig := sig > + END; > ICannotRun (State.pausing); > DEC (inCritical); > InternalYield (); > > 3. The received-signals mask: > > ! CONST MaxSigs = 64; > ! TYPE Sig = [ 0..MaxSigs-1 ]; > ! > ! (* in order to listen to other signals, they have to be enabled in > ! allow_sigvtalrm as well *) > ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); > ! > ! gotSigs := SET OF Sig { }; > ! > > ValueOfSIGCHLD() is a C function used to get the value of the SIGCHLD > constant without guessing at it (in ThreadPosixC.c). > > 4. changes to the signal handler: > > ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); > ! > ! INC(inCritical); > ! (* mark signal as being delivered *) > ! IF sig >= 0 AND sig < MaxSigs THEN > ! gotSigs := gotSigs + SET OF Sig { sig } > ! END; > ! DEC(inCritical); > ! > ! IF inCritical = 0 AND heapState.inCritical = 0 THEN > ! InternalYield () > ! END; > END switch_thread; > > Note that I don't know if INC/DEC(inCritical) does exactly the right > thing here. > > 5. changes to the scheduler: > > a. thread wakeup > IF t.alertable AND t.alertPending THEN > CanRun (t); > EXIT; > + > + ELSIF t.waitingForSig IN gotSigs THEN > + t.waitingForSig := -1; > + CanRun(t); > + EXIT; > > ELSIF t.waitingForTime <= now THEN > CanRun (t); > EXIT; > ! > > b. clearing the mask > > END; > END; > > + gotSigs := SET OF Sig {}; > + > IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN > IF perfOn THEN PerfRunning (t.id); END; > (* At least one thread wants to run; transfer to it *) > > 6. changes to WaitProcess (Process.Wait): > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > ! CONST Delay = 10.0D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > ! SigPause(Delay,SIGCHLD); > END; > END WaitProcess; > > 7. install signal handler even if program is single-threaded: > > BEGIN > + (* we need to call set up the signal handler for other reasons than > + just thread switching now *) > + setup_sigvtalrm (switch_thread); > END ThreadPosix. > > 8. modify signal handler in ThreadPosixC.c to catch SIGCHLD: > > sigemptyset(&ThreadSwitchSignal); > sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); > + sigaddset(&ThreadSwitchSignal, SIGCHLD); > > act.sa_handler = handler; > act.sa_flags = SA_RESTART; > sigemptyset(&(act.sa_mask)); > if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); > + if (sigaction (SIGCHLD, &act, NULL)) abort(); > > I'll send the complete diff in a separate message for those who want > to study it more closely. > > I propose the above changes for inclusion in the current CM3 repository. > > Mika From hosking at cs.purdue.edu Sun Feb 13 21:44:11 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:44:11 -0500 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110213203726.82ADB1A2078@async.async.caltech.edu> References: <20110212204406.88A661A2078@async.async.caltech.edu> <20110213203726.82ADB1A2078@async.async.caltech.edu> Message-ID: Sorry, I missed you e-mail describing this change. Looks good. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: > Tony, > > The old user threads has an old problem with Process.Wait. It has a > Thread.Pause(0.1d0) in it. This causes the compiler to run three times > slower than it should. The m3build I've been using for years has > that Thread.Pause removed (so it uses 100% CPU instead, busy-waiting...) > > But yeah user threads works the same as it used to (except for > now calling pthread_atfork, which I'll return to in another email...) > > Mika > > Tony Hosking writes: >> Hi Mika, Why was this change needed (I haven't looked closely)? I don't = >> think we diverged much from previous implementations of user threads, = >> which presumably used to work for you. >> >> On Feb 12, 2011, at 3:44 PM, Mika Nystrom wrote: >> >>> =20 >>> Index: ThreadPosix.i3 >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> RCS file: = >> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v >>> retrieving revision 1.16 >>> diff -c -r1.16 ThreadPosix.i3 >>> *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 >>> --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 >>> *************** >>> *** 39,42 **** >>> --- 39,45 ---- >>> =20 >>> = >> (*------------------------------------------------------------------------= >> ---*) >>> =20 >>> + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> >>> + PROCEDURE ValueOfSIGCHLD(): int; >>> +=20 >>> END ThreadPosix. >>> Index: ThreadPosix.m3 >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> RCS file: = >> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v >>> retrieving revision 1.73 >>> diff -c -r1.73 ThreadPosix.m3 >>> *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 >>> --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 >>> *************** >>> *** 97,102 **** >>> --- 97,105 ---- >>> (* if state =3D pausing, the time at which we can restart *) >>> waitingForTime: Time.T; >>> =20 >>> + (* if state =3D pausing, the signal that truncates the pause *) >>> + waitingForSig: int :=3D -1; >>> +=20 >>> (* true if we are waiting during an AlertWait or AlertJoin=20 >>> or AlertPause *) >>> alertable: BOOLEAN :=3D FALSE; >>> *************** >>> *** 147,152 **** >>> --- 150,160 ---- >>> =20 >>> defaultStackSize :=3D 3000; >>> =20 >>> + (* note that even though the "heavy machinery" is only used for=20 >>> + multipleThreads, we still need to set up the signal handler so >>> + that we can catch signals from other sources than thread = >> switching. >>> + e.g., we use SIGCHLD to speed up Process.Wait *) >>> +=20 >>> VAR >>> stats: RECORD >>> n_forks :=3D 0; >>> *************** >>> *** 511,527 **** >>> XPause(until, FALSE); >>> END Pause; >>> =20 >>> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >>> VAR until :=3D n + Time.Now (); >>> BEGIN >>> XPause(until, TRUE); >>> END AlertPause; >>> =20 >>> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D FALSE) = >> RAISES {Alerted} =3D >>> BEGIN >>> INC (inCritical); >>> self.waitingForTime :=3D until; >>> self.alertable :=3D alertable; >>> ICannotRun (State.pausing); >>> DEC (inCritical); >>> InternalYield (); >>> --- 519,546 ---- >>> XPause(until, FALSE); >>> END Pause; >>> =20 >>> + PROCEDURE SigPause(n: LONGREAL; sig: int)=3D >>> + <*FATAL Alerted*> >>> + VAR until :=3D n + Time.Now (); >>> + BEGIN >>> + XPause(until, FALSE, sig); >>> + END SigPause; >>> +=20 >>> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >>> VAR until :=3D n + Time.Now (); >>> BEGIN >>> XPause(until, TRUE); >>> END AlertPause; >>> =20 >>> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D FALSE; = >> sig:int :=3D -1)=20 >>> ! RAISES {Alerted} =3D >>> BEGIN >>> INC (inCritical); >>> self.waitingForTime :=3D until; >>> self.alertable :=3D alertable; >>> + IF sig # -1 THEN >>> + self.waitingForSig :=3D sig >>> + END; >>> ICannotRun (State.pausing); >>> DEC (inCritical); >>> InternalYield (); >>> *************** >>> *** 703,712 **** >>> END; >>> END StartSwitching; >>> =20 >>> ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} =3D >>> BEGIN >>> allow_sigvtalrm (); >>> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN = >> InternalYield () END; >>> END switch_thread; >>> =20 >>> (*------------------------------------------------------------- = >> scheduler ---*) >>> --- 722,750 ---- >>> END; >>> END StartSwitching; >>> =20 >>> ! CONST MaxSigs =3D 64; >>> ! TYPE Sig =3D [ 0..MaxSigs-1 ]; >>> !=20 >>> ! (* in order to listen to other signals, they have to be enabled in >>> ! allow_sigvtalrm as well *) >>> ! VAR (*CONST*) SIGCHLD :=3D ValueOfSIGCHLD(); >>> ! =20 >>> ! gotSigs :=3D SET OF Sig { }; >>> !=20 >>> ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>> BEGIN >>> allow_sigvtalrm (); >>> !=20 >>> ! INC(inCritical); >>> ! (* mark signal as being delivered *) >>> ! IF sig >=3D 0 AND sig < MaxSigs THEN >>> ! gotSigs :=3D gotSigs + SET OF Sig { sig } >>> ! END; >>> ! DEC(inCritical); >>> !=20 >>> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN=20 >>> ! InternalYield ()=20 >>> ! END; >>> END switch_thread; >>> =20 >>> (*------------------------------------------------------------- = >> scheduler ---*) >>> *************** >>> *** 782,792 **** >>> IF t.alertable AND t.alertPending THEN >>> CanRun (t); >>> EXIT; >>> =20 >>> ELSIF t.waitingForTime <=3D now THEN >>> CanRun (t); >>> EXIT; >>> ! =20 >>> ELSIF NOT somePausing THEN >>> earliest :=3D t.waitingForTime; >>> somePausing :=3D TRUE; >>> --- 820,835 ---- >>> IF t.alertable AND t.alertPending THEN >>> CanRun (t); >>> EXIT; >>> + =20 >>> + ELSIF t.waitingForSig IN gotSigs THEN >>> + t.waitingForSig :=3D -1; >>> + CanRun(t); >>> + EXIT; >>> =20 >>> ELSIF t.waitingForTime <=3D now THEN >>> CanRun (t); >>> EXIT; >>> !=20 >>> ELSIF NOT somePausing THEN >>> earliest :=3D t.waitingForTime; >>> somePausing :=3D TRUE; >>> *************** >>> *** 886,891 **** >>> --- 929,936 ---- >>> END; >>> END; >>> =20 >>> + gotSigs :=3D SET OF Sig {}; >>> +=20 >>> IF t.state =3D State.alive AND (scanned OR NOT someBlocking) THEN >>> IF perfOn THEN PerfRunning (t.id); END; >>> (* At least one thread wants to run; transfer to it *) >>> *************** >>> *** 948,960 **** >>> =20 >>> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D >>> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >>> ! CONST Delay =3D 0.1D0; >>> BEGIN >>> LOOP >>> WITH r =3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >>> IF r # 0 THEN RETURN r END; >>> END; >>> ! Pause(Delay); >>> END; >>> END WaitProcess; >>> =20 >>> --- 993,1005 ---- >>> =20 >>> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D >>> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >>> ! CONST Delay =3D 10.0D0; >>> BEGIN >>> LOOP >>> WITH r =3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >>> IF r # 0 THEN RETURN r END; >>> END; >>> ! SigPause(Delay,SIGCHLD); >>> END; >>> END WaitProcess; >>> =20 >>> *************** >>> *** 1361,1364 **** >>> --- 1406,1412 ---- >>> VAR debug :=3D RTParams.IsPresent ("debugthreads"); >>> =20 >>> BEGIN >>> + (* we need to call set up the signal handler for other reasons = >> than >>> + just thread switching now *) >>> + setup_sigvtalrm (switch_thread); >>> END ThreadPosix. >>> Index: ThreadPosixC.c >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> RCS file: = >> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v >>> retrieving revision 1.59 >>> diff -c -r1.59 ThreadPosixC.c >>> *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 >>> --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 >>> *************** >>> *** 78,88 **** >>> --- 78,97 ---- >>> =20 >>> sigemptyset(&ThreadSwitchSignal); >>> sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); >>> + sigaddset(&ThreadSwitchSignal, SIGCHLD); >>> =20 >>> act.sa_handler =3D handler; >>> act.sa_flags =3D SA_RESTART; >>> sigemptyset(&(act.sa_mask)); >>> if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); >>> + if (sigaction (SIGCHLD, &act, NULL)) abort(); >>> + } >>> +=20 >>> + int >>> + __cdecl >>> + ThreadPosix__value_of_SIGCHLD(void) >>> + { >>> + return SIGCHLD; >>> } >>> =20 >>> void From hosking at cs.purdue.edu Sun Feb 13 21:46:26 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:46:26 -0500 Subject: [M3devel] More on threading In-Reply-To: <20110213181717.4F0E91A2078@async.async.caltech.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> Message-ID: <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> I know it worked properly at one point in time, when I stressed it very hard for some GC-related work I was doing. I need to see instances of failure (deadlock, crashes) captured in the debugger, running with the latest head code. I don't want to try and diagnose old versions of the system if at all possible. I may have some time to try your thread test program in the next couple of weeks. Please let me know workloads that reliably cause failure. On Feb 13, 2011, at 1:17 PM, Mika Nystrom wrote: > I don't think pthreads works properly anywhere at > the moment. My guess is that it never has, if you stress it hard enough. From mika at async.caltech.edu Sun Feb 13 22:15:51 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 13:15:51 -0800 Subject: [M3devel] More on threading In-Reply-To: <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> Message-ID: <20110213211551.2A60A1A2078@async.async.caltech.edu> I have bootstrapped so many versions of the compiler on so many systems in the past week I've probably lost track of where we are, yes. I'll do some testing on a bona fide head checkout and let you know what I find. Mika Tony Hosking writes: >I know it worked properly at one point in time, when I stressed it very = >hard for some GC-related work I was doing. >I need to see instances of failure (deadlock, crashes) captured in the = >debugger, running with the latest head code. >I don't want to try and diagnose old versions of the system if at all = >possible. >I may have some time to try your thread test program in the next couple = >of weeks. Please let me know workloads that reliably cause failure. > > >On Feb 13, 2011, at 1:17 PM, Mika Nystrom wrote: > >> I don't think pthreads works properly anywhere at >> the moment. My guess is that it never has, if you stress it hard = >enough. From mika at async.caltech.edu Sun Feb 13 23:08:04 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 14:08:04 -0800 Subject: [M3devel] SIGCHLD diff Message-ID: <20110213220804.7A9001A2078@async.async.caltech.edu> Hi Tony, I found a problem with the change. I would appreciate it if you and Jay could think about what I found and my solution (and anybody else is welcome to as well, of course). When fiddling with cm3 to get it to run in parallel, I pushed it up to fork up to 100 processes at once, just to see what happens. When I did that (with my SIGCHLD patch) I got a few errors such as illegal instruction and a bus error or two. I believe what was happening was that I was taking too many signals at once, because of the following code in ThreadPosix.m3: PROCEDURE switch_thread (sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); (* here *) INC(inCritical); (* mark signal as being delivered *) IF sig >= 0 AND sig < MaxSigs THEN gotSigs := gotSigs + SET OF Sig { sig } END; DEC(inCritical); IF inCritical = 0 AND heapState.inCritical = 0 THEN InternalYield () END; END switch_thread; I found it a bit odd that the first action of the signal handler is to enable signals. With VTALRM this is innocuous, but in my case, imagine what happens if you have a lot of exited child processes queued up. What happens is that your signal stack gets wound deeper and deeper, and each time you catch one SIGCHLD you ask for another. Eventually you run out of stack space. My change, which appears to fix this problem, is to split up the "allow" function as follows: PROCEDURE switch_thread (sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); (* VTALRM only *) INC(inCritical); IF inCritical = 1 THEN allow_othersigs (); END; (* SIGCHLD *) (* mark signal as being delivered *) IF sig >= 0 AND sig < MaxSigs THEN gotSigs := gotSigs + SET OF Sig { sig } END; DEC(inCritical); IF inCritical = 0 AND heapState.inCritical = 0 THEN InternalYield () END; END switch_thread; This way, SIGCHLD is not taken in nested signal handlers. But is there a problem here? Well, the fallback is that it takes 0.1 seconds to wake up on Process.Wait . Is there a chance that SIGCHLD signals are left pending? My argument is no, since in the "base" environment (i.e., not executing in a signal handler) SIGCHLD is enabled as a postcondition of allow_othersigs. All *other* occurrences of allow_sigvtalrm are of course replaced with allow_sigvtalrm(); allow_othersigs(); The goal is that SIGCHLD be enabled precisely when SIGVTALRM is except inside the signal handler itself. Mika Tony Hosking writes: >Sorry, I missed you e-mail describing this change. Looks good. > >Antony Hosking | Associate Professor | Computer Science | Purdue = >University >305 N. University Street | West Lafayette | IN 47907 | USA >Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > >On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: > >> Tony, >>=20 >> The old user threads has an old problem with Process.Wait. It has a=20= > >> Thread.Pause(0.1d0) in it. This causes the compiler to run three = >times >> slower than it should. The m3build I've been using for years has >> that Thread.Pause removed (so it uses 100% CPU instead, = >busy-waiting...) >>=20 >> But yeah user threads works the same as it used to (except for=20 >> now calling pthread_atfork, which I'll return to in another email...) >>=20 >> Mika >>=20 >> Tony Hosking writes: >>> Hi Mika, Why was this change needed (I haven't looked closely)? I = >don't =3D >>> think we diverged much from previous implementations of user threads, = >=3D >>> which presumably used to work for you. >>>=20 >>> On Feb 12, 2011, at 3:44 PM, Mika Nystrom wrote: >>>=20 >>>> =3D20 >>>> Index: ThreadPosix.i3 >>>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> =3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D= >3D=3D3D=3D3D=3D3D >>>> RCS file: =3D >>> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v >>>> retrieving revision 1.16 >>>> diff -c -r1.16 ThreadPosix.i3 >>>> *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 >>>> --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 >>>> *************** >>>> *** 39,42 **** >>>> --- 39,45 ---- >>>> =3D20 >>>> =3D >>> = >(*------------------------------------------------------------------------= >=3D >>> ---*) >>>> =3D20 >>>> + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> >>>> + PROCEDURE ValueOfSIGCHLD(): int; >>>> +=3D20 >>>> END ThreadPosix. >>>> Index: ThreadPosix.m3 >>>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> =3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D= >3D=3D3D=3D3D=3D3D >>>> RCS file: =3D >>> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v >>>> retrieving revision 1.73 >>>> diff -c -r1.73 ThreadPosix.m3 >>>> *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 >>>> --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 >>>> *************** >>>> *** 97,102 **** >>>> --- 97,105 ---- >>>> (* if state =3D3D pausing, the time at which we can restart *) >>>> waitingForTime: Time.T; >>>> =3D20 >>>> + (* if state =3D3D pausing, the signal that truncates the pause = >*) >>>> + waitingForSig: int :=3D3D -1; >>>> +=3D20 >>>> (* true if we are waiting during an AlertWait or AlertJoin=3D20 >>>> or AlertPause *) >>>> alertable: BOOLEAN :=3D3D FALSE; >>>> *************** >>>> *** 147,152 **** >>>> --- 150,160 ---- >>>> =3D20 >>>> defaultStackSize :=3D3D 3000; >>>> =3D20 >>>> + (* note that even though the "heavy machinery" is only used = >for=3D20 >>>> + multipleThreads, we still need to set up the signal handler = >so >>>> + that we can catch signals from other sources than thread =3D >>> switching. >>>> + e.g., we use SIGCHLD to speed up Process.Wait *) >>>> +=3D20 >>>> VAR >>>> stats: RECORD >>>> n_forks :=3D3D 0; >>>> *************** >>>> *** 511,527 **** >>>> XPause(until, FALSE); >>>> END Pause; >>>> =3D20 >>>> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D3D >>>> VAR until :=3D3D n + Time.Now (); >>>> BEGIN >>>> XPause(until, TRUE); >>>> END AlertPause; >>>> =3D20 >>>> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D3D FALSE) = >=3D >>> RAISES {Alerted} =3D3D >>>> BEGIN >>>> INC (inCritical); >>>> self.waitingForTime :=3D3D until; >>>> self.alertable :=3D3D alertable; >>>> ICannotRun (State.pausing); >>>> DEC (inCritical); >>>> InternalYield (); >>>> --- 519,546 ---- >>>> XPause(until, FALSE); >>>> END Pause; >>>> =3D20 >>>> + PROCEDURE SigPause(n: LONGREAL; sig: int)=3D3D >>>> + <*FATAL Alerted*> >>>> + VAR until :=3D3D n + Time.Now (); >>>> + BEGIN >>>> + XPause(until, FALSE, sig); >>>> + END SigPause; >>>> +=3D20 >>>> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D3D >>>> VAR until :=3D3D n + Time.Now (); >>>> BEGIN >>>> XPause(until, TRUE); >>>> END AlertPause; >>>> =3D20 >>>> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D3D FALSE; = >=3D >>> sig:int :=3D3D -1)=3D20 >>>> ! RAISES {Alerted} =3D3D >>>> BEGIN >>>> INC (inCritical); >>>> self.waitingForTime :=3D3D until; >>>> self.alertable :=3D3D alertable; >>>> + IF sig # -1 THEN >>>> + self.waitingForSig :=3D3D sig >>>> + END; >>>> ICannotRun (State.pausing); >>>> DEC (inCritical); >>>> InternalYield (); >>>> *************** >>>> *** 703,712 **** >>>> END; >>>> END StartSwitching; >>>> =3D20 >>>> ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} =3D3D= > >>>> BEGIN >>>> allow_sigvtalrm (); >>>> ! IF inCritical =3D3D 0 AND heapState.inCritical =3D3D 0 THEN =3D >>> InternalYield () END; >>>> END switch_thread; >>>> =3D20 >>>> (*------------------------------------------------------------- =3D >>> scheduler ---*) >>>> --- 722,750 ---- >>>> END; >>>> END StartSwitching; >>>> =3D20 >>>> ! CONST MaxSigs =3D3D 64; >>>> ! TYPE Sig =3D3D [ 0..MaxSigs-1 ]; >>>> !=3D20 >>>> ! (* in order to listen to other signals, they have to be enabled in >>>> ! allow_sigvtalrm as well *) >>>> ! VAR (*CONST*) SIGCHLD :=3D3D ValueOfSIGCHLD(); >>>> ! =3D20 >>>> ! gotSigs :=3D3D SET OF Sig { }; >>>> !=3D20 >>>> ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D3D >>>> BEGIN >>>> allow_sigvtalrm (); >>>> !=3D20 >>>> ! INC(inCritical); >>>> ! (* mark signal as being delivered *) >>>> ! IF sig >=3D3D 0 AND sig < MaxSigs THEN >>>> ! gotSigs :=3D3D gotSigs + SET OF Sig { sig } >>>> ! END; >>>> ! DEC(inCritical); >>>> !=3D20 >>>> ! IF inCritical =3D3D 0 AND heapState.inCritical =3D3D 0 THEN=3D20= > >>>> ! InternalYield ()=3D20 >>>> ! END; >>>> END switch_thread; >>>> =3D20 >>>> (*------------------------------------------------------------- =3D >>> scheduler ---*) >>>> *************** >>>> *** 782,792 **** >>>> IF t.alertable AND t.alertPending THEN >>>> CanRun (t); >>>> EXIT; >>>> =3D20 >>>> ELSIF t.waitingForTime <=3D3D now THEN >>>> CanRun (t); >>>> EXIT; >>>> ! =3D20 >>>> ELSIF NOT somePausing THEN >>>> earliest :=3D3D t.waitingForTime; >>>> somePausing :=3D3D TRUE; >>>> --- 820,835 ---- >>>> IF t.alertable AND t.alertPending THEN >>>> CanRun (t); >>>> EXIT; >>>> + =3D20 >>>> + ELSIF t.waitingForSig IN gotSigs THEN >>>> + t.waitingForSig :=3D3D -1; >>>> + CanRun(t); >>>> + EXIT; >>>> =3D20 >>>> ELSIF t.waitingForTime <=3D3D now THEN >>>> CanRun (t); >>>> EXIT; >>>> !=3D20 >>>> ELSIF NOT somePausing THEN >>>> earliest :=3D3D t.waitingForTime; >>>> somePausing :=3D3D TRUE; >>>> *************** >>>> *** 886,891 **** >>>> --- 929,936 ---- >>>> END; >>>> END; >>>> =3D20 >>>> + gotSigs :=3D3D SET OF Sig {}; >>>> +=3D20 >>>> IF t.state =3D3D State.alive AND (scanned OR NOT someBlocking) = >THEN >>>> IF perfOn THEN PerfRunning (t.id); END; >>>> (* At least one thread wants to run; transfer to it *) >>>> *************** >>>> *** 948,960 **** >>>> =3D20 >>>> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D3D >>>> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >>>> ! CONST Delay =3D3D 0.1D0; >>>> BEGIN >>>> LOOP >>>> WITH r =3D3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >>>> IF r # 0 THEN RETURN r END; >>>> END; >>>> ! Pause(Delay); >>>> END; >>>> END WaitProcess; >>>> =3D20 >>>> --- 993,1005 ---- >>>> =3D20 >>>> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D3D >>>> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >>>> ! CONST Delay =3D3D 10.0D0; >>>> BEGIN >>>> LOOP >>>> WITH r =3D3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >>>> IF r # 0 THEN RETURN r END; >>>> END; >>>> ! SigPause(Delay,SIGCHLD); >>>> END; >>>> END WaitProcess; >>>> =3D20 >>>> *************** >>>> *** 1361,1364 **** >>>> --- 1406,1412 ---- >>>> VAR debug :=3D3D RTParams.IsPresent ("debugthreads"); >>>> =3D20 >>>> BEGIN >>>> + (* we need to call set up the signal handler for other reasons =3D= > >>> than >>>> + just thread switching now *) >>>> + setup_sigvtalrm (switch_thread); >>>> END ThreadPosix. >>>> Index: ThreadPosixC.c >>>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> =3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D= >3D=3D3D=3D3D=3D3D >>>> RCS file: =3D >>> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v >>>> retrieving revision 1.59 >>>> diff -c -r1.59 ThreadPosixC.c >>>> *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 >>>> --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 >>>> *************** >>>> *** 78,88 **** >>>> --- 78,97 ---- >>>> =3D20 >>>> sigemptyset(&ThreadSwitchSignal); >>>> sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); >>>> + sigaddset(&ThreadSwitchSignal, SIGCHLD); >>>> =3D20 >>>> act.sa_handler =3D3D handler; >>>> act.sa_flags =3D3D SA_RESTART; >>>> sigemptyset(&(act.sa_mask)); >>>> if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); >>>> + if (sigaction (SIGCHLD, &act, NULL)) abort(); >>>> + } >>>> +=3D20 >>>> + int >>>> + __cdecl >>>> + ThreadPosix__value_of_SIGCHLD(void) >>>> + { >>>> + return SIGCHLD; >>>> } >>>> =3D20 >>>> void From mika at async.caltech.edu Sun Feb 13 23:27:51 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 14:27:51 -0800 Subject: [M3devel] More on threading In-Reply-To: <20110213181717.4F0E91A2078@async.async.caltech.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> Message-ID: <20110213222751.A98F31A2078@async.async.caltech.edu> All right, I have some more interesting observations on user threading. As well as on parallelizing CM3. They are connected because the latter is the source of the former. User threads on FreeBSD (FreeBSD7 /amd64) ======================= The fact that my process was getting stuck in "umtxn" (I think that is what top called it) turned out to be something inside fork() in libthr.so. Now, libthr.so is FreeBSD's name for pthreads, and this was a user threading program. This does not seem right. I tried the following: 1. remove the call to pthread_atfork No change in behavior. My parallelized CM3 still got stuck in umtxn. 2. remove the -pthread linking flag Success! Parallelized CM3 works! So, clearly -pthread is harmful to the health of Modula-3 programs running with user threading on FreeBSD. This is on FreeBSD 7.1, so it means pthread_atfork has to be removed as well. I'm not sure how to make the building of this as transparent as possible, but what's currently checked in is in any case wrong. Parallelizing CM3 ================= I found one bug in my earlier CM3 patch. Calls to Utils.* need to be monitored because there is a shared hash table in that module. New diff attached. With this diff the compiler runs correctly even when it is running 100 instances of the backend in parallel. Speedup is about 2.0 on a machine with four cores for a largish source directory. The max. number of subprocesses is set on line 985 of Builder.m3 . I am not claiming this patch is anywhere near "polished" enough to go into the CM3 tree at the moment. It's an experiment and starting point for further work... Mika Index: cm3/src/Builder.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/cm3/src/Builder.m3,v retrieving revision 1.36 diff -c -r1.36 Builder.m3 *** cm3/src/Builder.m3 24 Aug 2010 05:24:24 -0000 1.36 --- cm3/src/Builder.m3 13 Feb 2011 22:17:16 -0000 *************** *** 15,20 **** --- 15,22 ---- IMPORT QIdent; FROM Target IMPORT M3BackendMode_t, BackendAssembly, BackendModeStrings; FROM M3Path IMPORT OSKind, OSKindStrings; + IMPORT Pathname; + IMPORT RefSeq; TYPE UK = M3Unit.Kind; *************** *** 131,136 **** --- 133,139 ---- link_coverage : TEXT; (* coverage library *) m3_front_flags: Arg.List; (* configuration options for the front *) m3_options : Arg.List; (* misc. user options for the frontend *) + delayBackend := FALSE; END; TYPE *************** *** 932,937 **** --- 935,956 ---- (*------------------------------------------------------------ compilation --*) + TYPE MarkerPromise = QMachine.Promise BRANDED OBJECT OVERRIDES fulfil := FulfilNothing END; (* an empty promise, use as a marker *) + + PROCEDURE FulfilNothing(<*UNUSED*>mp : MarkerPromise) = BEGIN END FulfilNothing; + + TYPE SeqClosure = Thread.Closure OBJECT seq : RefSeq.T; OVERRIDES apply := SeqApply END; + + PROCEDURE SeqApply(cl : SeqClosure) : REFANY = + BEGIN (* force a Sequence of promises *) + FOR i := 0 TO cl.seq.size()-1 DO + WITH p = NARROW(cl.seq.get(i),QMachine.Promise) DO + p.fulfil() + END + END; + RETURN NIL + END SeqApply; + PROCEDURE CompileEverything (s: State; schedule: SourceList) = VAR u: M3Unit.T; BEGIN *************** *** 941,948 **** (* compile all the sources using the initial schedule *) FOR i := 0 TO LAST (schedule^) DO ! CompileOne (s, schedule[i]); END; FlushPending (s); (* recompile any interfaces where we goofed on the exports *) --- 960,1000 ---- (* compile all the sources using the initial schedule *) FOR i := 0 TO LAST (schedule^) DO ! s.delayBackend := TRUE; (* tell backend to promise instead of executing *) ! TRY ! CompileOne (s, schedule[i]); ! FINALLY ! s.delayBackend := FALSE; (* tell backend to go back to executing *) ! END; ! ! s.machine.promises.addhi(NEW(MarkerPromise)); (* promises, promises *) ! END; + + VAR + curSeq := NEW(RefSeq.T).init(); (* sequential chain *) + threads := NEW(RefSeq.T).init(); (* parallel chains *) + BEGIN + FOR i := 0 TO s.machine.promises.size()-1 DO + WITH p = s.machine.promises.get(i) DO + curSeq.addhi(p); + IF i = s.machine.promises.size()-1 OR ISTYPE(p,MarkerPromise) THEN (* mark as end of one sequential chain of promises *) + WITH cl = NEW(SeqClosure, seq := curSeq) DO + threads.addhi (Thread.Fork(cl)); (* start sequential chain *) + + IF threads.size() > 400 THEN EVAL Thread.Join(threads.remlo()) END; (* limit number of sequential chains executing in parallel *) + + curSeq := NEW(RefSeq.T).init() + END + END + END + END; + WHILE threads.size() > 0 DO EVAL Thread.Join(threads.remlo()) END; (* wait for all sequential chains to end *) + END; + + EVAL s.machine.promises.init(); (* clear out promises *) + + FlushPending (s); (* recompile any interfaces where we goofed on the exports *) *************** *** 1151,1156 **** --- 1203,1229 ---- END; END CompileM3; + TYPE + NotePromise = QMachine.Promise OBJECT + nam : Pathname.T; + OVERRIDES + fulfil := FulfilNP; + END; + + RemovePromise = QMachine.Promise OBJECT + nam : Pathname.T; + OVERRIDES + fulfil := FulfilRP; + END; + + VAR utilsMu := NEW(MUTEX); (* Utils.* fiddles with a table *) + + PROCEDURE FulfilNP(np : NotePromise) = + BEGIN LOCK utilsMu DO Utils.NoteTempFile(np.nam) END END FulfilNP; + + PROCEDURE FulfilRP(rp : RemovePromise) = + BEGIN LOCK utilsMu DO Utils.Remove(rp.nam) END END FulfilRP; + PROCEDURE PushOneM3 (s: State; u: M3Unit.T): BOOLEAN = VAR tmpC, tmpS: TEXT; *************** *** 1191,1208 **** | 3 => (* -bootstrap, +m3back, +asm *) tmpC := TempCName (u); tmpS := TempSName (u); IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; IF RunM3 (s, u, tmpC) THEN ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) ! AND RunAsm (s, tmpS, u.object) THEN END; need_merge := TRUE; END; IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; | 6, (* +bootstrap, +m3back, -asm *) 7 => (* +bootstrap, +m3back, +asm *) --- 1264,1320 ---- | 3 => (* -bootstrap, +m3back, +asm *) + IF s.delayBackend THEN tmpC := TempCName (u); tmpS := TempSName (u); + (* IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; + *) + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(NotePromise, nam := tmpC)) + END; + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(NotePromise, nam := tmpS)) + END; + IF RunM3 (s, u, tmpC) THEN ! s.machine.record(TRUE); ! TRY ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) ! AND RunAsm (s, tmpS, u.object) THEN ! END; ! FINALLY ! s.machine.record(FALSE) END; + need_merge := TRUE; END; + (* IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; + *) + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpC)) + END; + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpS)) + END; + + ELSE + tmpC := TempCName (u); + tmpS := TempSName (u); + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; + IF RunM3 (s, u, tmpC) THEN + IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) + AND RunAsm (s, tmpS, u.object) THEN + END; + need_merge := TRUE; + END; + IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; + IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; + END | 6, (* +bootstrap, +m3back, -asm *) 7 => (* +bootstrap, +m3back, +asm *) Index: m3quake/src/QMachine.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.i3,v retrieving revision 1.6 diff -c -r1.6 QMachine.i3 *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -0000 1.6 --- m3quake/src/QMachine.i3 13 Feb 2011 22:17:18 -0000 *************** *** 8,13 **** --- 8,14 ---- IMPORT Thread, Wr, QValue, QCode; FROM Quake IMPORT Machine, Error, ID, IDMap; + IMPORT RefSeq; REVEAL T <: T_; *************** *** 15,20 **** --- 16,22 ---- T = Machine; T_ = OBJECT map: IDMap := NIL; (* READONLY *) + promises : RefSeq.T; METHODS init (map: IDMap): T; evaluate (s: QCode.Stream) RAISES {Error, Thread.Alerted}; *************** *** 37,42 **** --- 39,46 ---- set_wr (wr: Wr.T); exec_echo (b: BOOLEAN): BOOLEAN; trace (b: BOOLEAN); + + record(on : BOOLEAN); (* instead of performing certain acts, promise *) END; PROCEDURE PushBool (t: T; b: BOOLEAN); *************** *** 51,54 **** --- 55,63 ---- PROCEDURE GetEnv (default, v0, v1, v2, v3, v4: TEXT := NIL): TEXT; + TYPE Promise = OBJECT METHODS fulfil() RAISES { Error } END; + END QMachine. + + + Index: m3quake/src/QMachine.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3,v retrieving revision 1.35 diff -c -r1.35 QMachine.m3 *** m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35 --- m3quake/src/QMachine.m3 13 Feb 2011 22:17:18 -0000 *************** *** 16,21 **** --- 16,22 ---- IMPORT TextUtils, FSUtils, System, DirStack; (* sysutils *) IMPORT Compiler; IMPORT M3Path; + IMPORT RefSeq; CONST OnUnix = (Compiler.ThisOS = Compiler.OS.POSIX); *************** *** 44,49 **** --- 45,52 ---- shell : TEXT := NIL; sh_option : TEXT := NIL; tmp_dir : TEXT := NIL; + + doRecord := FALSE; OVERRIDES init := Init; evaluate := Evaluate; *************** *** 66,73 **** --- 69,81 ---- set_wr := SetWr; exec_echo := ExecEcho; trace := Trace; + + record := Record; END; + PROCEDURE Record(t : T; on : BOOLEAN) = + BEGIN t.doRecord := on END Record; + TYPE Registers = RECORD cp : QCode.Stream := NIL; (* code pointer *) *************** *** 139,144 **** --- 147,154 ---- t.globals := NEW (IntRefTbl.Default).init (); t.default_wr := Stdio.stdout; + t.promises := NEW(RefSeq.T).init(); + InitOSEnv (t); InitBuiltins (t); *************** *** 1555,1564 **** END; ELSE FlushIO (); ! Process.GetStandardFileHandles (stdin, stdout, stderr); ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), ! stdin := stdin, stdout := stdout, ! stderr := stderr); END; EXCEPT | Thread.Alerted => --- 1565,1594 ---- END; ELSE FlushIO (); ! IF t.doRecord THEN ! handle := NIL; ! WITH a = NEW(REF ARRAY OF TEXT, n_shell_args) DO ! a^ := SUBARRAY(args,0,n_shell_args); ! VAR wrx : Wr.T; BEGIN ! IF echo OR t.do_echo THEN ! wrx := wr ! ELSE ! wrx := NIL ! END; ! t.promises.addhi(NEW(ExecPromise, ! cmd := t.shell, ! wr := wrx, ! args := a, ! t := t, ! ignore_errors := ignore_errors)) ! END ! END ! ELSE ! Process.GetStandardFileHandles (stdin, stdout, stderr); ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), ! stdin := stdin, stdout := stdout, ! stderr := stderr); ! END; END; EXCEPT | Thread.Alerted => *************** *** 1573,1579 **** END; (* wait for everything to shutdown... *) ! exit_code := Process.Wait (handle); END; IF onlyTry THEN --- 1603,1613 ---- END; (* wait for everything to shutdown... *) ! IF handle = NIL THEN ! exit_code := 0 ! ELSE ! exit_code := Process.Wait (handle); ! END; (* else we're only promising *) END; IF onlyTry THEN *************** *** 1589,1594 **** --- 1623,1664 ---- END ExecCommand; + TYPE + ExecPromise = Promise OBJECT + cmd : TEXT; + args : REF ARRAY OF TEXT; + t : T; + wr : Wr.T; + ignore_errors : BOOLEAN; + OVERRIDES + fulfil := FulfilExecPromise; + END; + + PROCEDURE FulfilExecPromise(ep : ExecPromise) RAISES { Error } = + VAR + stdin, stdout, stderr: File.T; + BEGIN + Process.GetStandardFileHandles (stdin, stdout, stderr); + TRY + IF ep.wr # NIL THEN + Wr.PutText (ep.wr, ep.args[1]); + Wr.PutText (ep.wr, Wr.EOL); + FlushIO (); + END; + WITH handle = Process.Create (ep.cmd, ep.args^, + stdin := stdin, stdout := stdout, + stderr := stderr), + exit_code = Process.Wait(handle) DO + IF exit_code # 0 AND NOT ep.ignore_errors THEN + Err (ep.t, Fmt.F("exit %s: %s", Fmt.Int(exit_code), ep.cmd)) + END + END + EXCEPT + OSError.E (ec) => + Err (ep.t, Fmt.F ("exec failed%s *** %s", OSErr (ec), ep.cmd)); + END + END FulfilExecPromise; + PROCEDURE KillProcess (handle: Process.T) = BEGIN IF (handle # NIL) THEN From mika at async.caltech.edu Mon Feb 14 00:15:02 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 15:15:02 -0800 Subject: [M3devel] bootstrapping problems In-Reply-To: References: <20110205215619.E65981A207A@async.async.caltech.edu> , <20110205234331.2F13A1A207A@async.async.caltech.edu> Message-ID: <20110213231502.7A5FA1A2078@async.async.caltech.edu> Jay sent me the following instructions for bootstrapping a while back: ./upgrade.py nocleangcc && ./do-cm3-all.py realclean skipgcc && ./do-cm3-all.py buildship skipgcc It seems reasonable enough for me. I unpacked the REL distribution, checked everything out, etc. However something is very wrong. The newer back-end does not seem compatible with the old m3front (not even backward-compatible), and it's getting called at the wrong time: ignoring ../src/m3overrides rm .M3SHIP rm .M3OVERRIDES rm .generic.tmp rm .generic.tmp inhale libsysutils.m3x inhale /usr/local/cm3/pkg/m3core/AMD64_LINUX/libm3core.a inhale /usr/local/cm3/pkg/libm3/AMD64_LINUX/libm3.a new source -> compiling EnvUtils.i3 m3front ../src/EnvUtils.i3 -w1 /home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 compilation terminated. m3_backend => 1 m3cc (aka cm3cg) failed compiling: EnvUtils.ic rm EnvUtils.is Why is it calling the new back end? This is the very first thing it does (it hasn't cleaned m3cc for some reason). The new back end doesn't like the output of the old compiler: (119)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>/home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 compilation terminated. But the old one in /usr/local/cm3/bin likes it just fine... (120)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is (121)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX> Is something going wrong with the bootstrapping process? I'm going to try to delete the newer back end and see if that fixes it. Mika From mika at async.caltech.edu Mon Feb 14 00:16:26 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 15:16:26 -0800 Subject: [M3devel] bootstrapping problems In-Reply-To: <20110213231502.7A5FA1A2078@async.async.caltech.edu> References: <20110205215619.E65981A207A@async.async.caltech.edu> , <20110205234331.2F13A1A207A@async.async.caltech.edu> <20110213231502.7A5FA1A2078@async.async.caltech.edu> Message-ID: <20110213231626.200DB1A2078@async.async.caltech.edu> I should probably not have "nocleangcc" and "skipgcc" in there huh... still a bit weird that the back-end isn't backward compatible? Mika Mika Nystrom writes: >Jay sent me the following instructions for bootstrapping a while back: > >./upgrade.py nocleangcc && > ./do-cm3-all.py realclean skipgcc && > ./do-cm3-all.py buildship skipgcc > >It seems reasonable enough for me. > >I unpacked the REL distribution, checked everything out, etc. > >However something is very wrong. The newer back-end does not seem >compatible with the old m3front (not even backward-compatible), and it's >getting called at the wrong time: > >ignoring ../src/m3overrides > >rm .M3SHIP >rm .M3OVERRIDES >rm .generic.tmp >rm .generic.tmp >inhale libsysutils.m3x >inhale /usr/local/cm3/pkg/m3core/AMD64_LINUX/libm3core.a >inhale /usr/local/cm3/pkg/libm3/AMD64_LINUX/libm3.a > >new source -> compiling EnvUtils.i3 >m3front ../src/EnvUtils.i3 -w1 >/home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-alig >n-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is >cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 >compilation terminated. > m3_backend => 1 >m3cc (aka cm3cg) failed compiling: EnvUtils.ic >rm EnvUtils.is > >Why is it calling the new back end? This is the very first thing it does (it >hasn't cleaned m3cc for some reason). > >The new back end doesn't like the output of the old compiler: > >(119)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>/home/mika/cm3/cm3/m3-sys >/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables >-quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is >cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 >compilation terminated. > >But the old one in /usr/local/cm3/bin likes it just fine... > >(120)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>cm3cg -gstabs+ -m64 -fPIC > -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o E >nvUtils.is >(121)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX> > >Is something going wrong with the bootstrapping process? I'm going to >try to delete the newer back end and see if that fixes it. > > Mika From jay.krell at cornell.edu Mon Feb 14 03:16:48 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 14 Feb 2011 02:16:48 +0000 Subject: [M3devel] bootstrapping problems In-Reply-To: <20110213231626.200DB1A2078@async.async.caltech.edu> References: <20110205215619.E65981A207A@async.async.caltech.edu>, , , <20110205234331.2F13A1A207A@async.async.caltech.edu>, , <20110213231502.7A5FA1A2078@async.async.caltech.edu>, <20110213231626.200DB1A2078@async.async.caltech.edu> Message-ID: /home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg That is the problem. For a long time, my config file were willing to run that. Instead of cm3cg in $PATH or cm3cg next to cm3. It bit me a bit too so I stopped doing it that way. The instructions are ok. Certainly the skipgcc. Nocleangcc depends on underlying correct incrementality, so might be a bit fragile. The backend has no need or obligation to be forward or backward compatible with any cm3/m3front. It need only be compatible with the one it goes with. The upgrading of cm3 and cm3cg is carefully coordinated to avoid any mismatch. Whenever the uprading has gone wrong, it has been fixed. Not any sort of compatibility restored. Granted, we could better use the version declaration and error right away clearly. I can make a change there. - Jay > To: m3devel at elegosoft.com > Date: Sun, 13 Feb 2011 15:16:26 -0800 > From: mika at async.caltech.edu > Subject: Re: [M3devel] bootstrapping problems > > I should probably not have "nocleangcc" and "skipgcc" in there huh... > > still a bit weird that the back-end isn't backward compatible? > > Mika > > Mika Nystrom writes: > >Jay sent me the following instructions for bootstrapping a while back: > > > >./upgrade.py nocleangcc && > > ./do-cm3-all.py realclean skipgcc && > > ./do-cm3-all.py buildship skipgcc > > > >It seems reasonable enough for me. > > > >I unpacked the REL distribution, checked everything out, etc. > > > >However something is very wrong. The newer back-end does not seem > >compatible with the old m3front (not even backward-compatible), and it's > >getting called at the wrong time: > > > >ignoring ../src/m3overrides > > > >rm .M3SHIP > >rm .M3OVERRIDES > >rm .generic.tmp > >rm .generic.tmp > >inhale libsysutils.m3x > >inhale /usr/local/cm3/pkg/m3core/AMD64_LINUX/libm3core.a > >inhale /usr/local/cm3/pkg/libm3/AMD64_LINUX/libm3.a > > > >new source -> compiling EnvUtils.i3 > >m3front ../src/EnvUtils.i3 -w1 > >/home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-alig > >n-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is > >cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 > >compilation terminated. > > m3_backend => 1 > >m3cc (aka cm3cg) failed compiling: EnvUtils.ic > >rm EnvUtils.is > > > >Why is it calling the new back end? This is the very first thing it does (it > >hasn't cleaned m3cc for some reason). > > > >The new back end doesn't like the output of the old compiler: > > > >(119)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>/home/mika/cm3/cm3/m3-sys > >/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables > >-quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is > >cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 > >compilation terminated. > > > >But the old one in /usr/local/cm3/bin likes it just fine... > > > >(120)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>cm3cg -gstabs+ -m64 -fPIC > > -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o E > >nvUtils.is > >(121)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX> > > > >Is something going wrong with the bootstrapping process? I'm going to > >try to delete the newer back end and see if that fixes it. > > > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Mon Feb 14 03:55:41 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 18:55:41 -0800 Subject: [M3devel] More on threading In-Reply-To: <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> Message-ID: <20110214025541.76AC51A2078@async.async.caltech.edu> All right, I have bootstrapped from the current CVS head with the PTHREAD thread implementation and obtained the following behavior. I'm on AMD64_LINUX here, but can repeat on AMD64_FREEBSD, LINUXLIBC6, I386_DARWIN if either of those is of any special interest. The AMD64_LINUX system reports as follows: Linux ukdsl204 2.6.26-2-amd64 #1 SMP Thu Sep 16 15:56:38 UTC 2010 x86_64 GNU/Linux Thread test results: all : 1. assert failed, ThreadPThread.m3, line 966 2. SEGV pc = 0x430f40 = Move + 0x6b in ../src/runtime/common/RTCollector.m3 3. ThreadPThreadC.c:372: ThreadPThread__pthread_mutex_delete: Assertion `e == 0' failed. read : OK nxread [read w/o try-except] : OK tryexcept : OK fork : OK forktoomuch [Process.Create w/o Process.Wait] : OK alloc : deadlock creat [ make new threads ] : deadlock lock : OK Now we can also try combinations, but I won't go tediously through all possibilites, just a few: alloc,nxread: SEGV pc = 0x430f40 = Move + 0x6b in ../src/runtime/common/RTCollector.m3 The "simplest" test that shows problem is "-tests alloc -n 2" (just two instances of alloc). Mika Tony Hosking writes: >I know it worked properly at one point in time, when I stressed it very = >hard for some GC-related work I was doing. >I need to see instances of failure (deadlock, crashes) captured in the = >debugger, running with the latest head code. >I don't want to try and diagnose old versions of the system if at all = >possible. >I may have some time to try your thread test program in the next couple = >of weeks. Please let me know workloads that reliably cause failure. > > >On Feb 13, 2011, at 1:17 PM, Mika Nystrom wrote: > >> I don't think pthreads works properly anywhere at >> the moment. My guess is that it never has, if you stress it hard = >enough. From rodney_bates at lcwb.coop Mon Feb 14 16:53:19 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 14 Feb 2011 09:53:19 -0600 Subject: [M3devel] threading on Windows? In-Reply-To: <20110209220057.790671A2078@async.async.caltech.edu> References: <20110209220057.790671A2078@async.async.caltech.edu> Message-ID: <4D594FEF.1030309@lcwb.coop> On 02/09/2011 04:00 PM, Mika Nystrom wrote: > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... Hmm, I'll try to look at that. I have been using the release branch for work on other projects. Even there, I have been seeing some problems in m3gdb not handling 64-bit processors. > > Mika > From mika at async.caltech.edu Mon Feb 14 22:18:29 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Mon, 14 Feb 2011 13:18:29 -0800 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110213220804.7A9001A2078@async.async.caltech.edu> References: <20110213220804.7A9001A2078@async.async.caltech.edu> Message-ID: <20110214211829.BA3761A2078@async.async.caltech.edu> Hi again m3devel, Does anyone understand the code in ThreadPosix.m3? After looking through it again I think my attached solution (sent out yesterday afternoon) is right, but I'm still just a little bit fuzzy on how the code works (is supposed to work) but I have some hypotheses. I realized it's been a long, long, loooong time since I coded anything with signals and signal handlers and I've forgotten much of the semantics. After all since Concurrent Pascal came out in '76 signals has always been the "old" way of doing things, right? (And I quit programming C in favor of M3 in 1999...) In ThreadPosixC.c, there's a function called setup_sigvtalrm that registers the Modula-3 procedure switch_thread as a signal handler. My understanding is that in C, signals are masked atomically on the call into the signal handler and unmasked atomically when the handler returns (unless the signal mask is changed, etc., while in the handler). However, switch_thread... doesn't return? Or? Or it does a longjmp somewhere? Which means signals may not be unmasked(?) Which is the reason for the calling "allow_sigvtalrm" as the first statement of the signal handler? Now I can't just turn on ALL signals in the signal handler because then I may get very deeply nested signal handlers, so I only call allow_allsignals (or whatever we call it) from the first-level signal handler. Hmm. Maybe that is not quite right? Do we eventually get back there? Yes I think we must because certain things depend on inCritical being zero... and when inCritical is zero SIGCHLD is again enabled. No I still don't fully understand how the code is supposed to work, especially in the presence of nested signal handling. Does anyone understand better? (I am still going to try to commit my patch later today because I don't think it could possibly make things worse and it does make a significant real-world improvement to the compiler performance under user threads.) Mika Mika Nystrom writes: >Hi Tony, > >I found a problem with the change. I would appreciate it if you and >Jay could think about what I found and my solution (and anybody else >is welcome to as well, of course). > >When fiddling with cm3 to get it to run in parallel, I pushed it up to >fork up to 100 processes at once, just to see what happens. When I did >that (with my SIGCHLD patch) I got a few errors such as illegal instruction >and a bus error or two. I believe what was happening was that I was >taking too many signals at once, because of the following code in >ThreadPosix.m3: > >PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); (* here *) > > INC(inCritical); > (* mark signal as being delivered *) > IF sig >= 0 AND sig < MaxSigs THEN > gotSigs := gotSigs + SET OF Sig { sig } > END; > DEC(inCritical); > > IF inCritical = 0 AND heapState.inCritical = 0 THEN > InternalYield () > END; > END switch_thread; > >I found it a bit odd that the first action of the signal handler is to >enable signals. With VTALRM this is innocuous, but in my case, imagine >what happens if you have a lot of exited child processes queued up. >What happens is that your signal stack gets wound deeper and deeper, and >each time you catch one SIGCHLD you ask for another. Eventually you >run out of stack space. > >My change, which appears to fix this problem, is to split up the "allow" >function as follows: > >PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); (* VTALRM only *) > > INC(inCritical); > IF inCritical = 1 THEN allow_othersigs (); END; (* SIGCHLD *) > > (* mark signal as being delivered *) > IF sig >= 0 AND sig < MaxSigs THEN > gotSigs := gotSigs + SET OF Sig { sig } > END; > DEC(inCritical); > > IF inCritical = 0 AND heapState.inCritical = 0 THEN > InternalYield () > END; > END switch_thread; > >This way, SIGCHLD is not taken in nested signal handlers. But is there a >problem here? Well, the fallback is that it takes 0.1 seconds to wake >up on Process.Wait . > >Is there a chance that SIGCHLD signals are left pending? My argument is >no, since in the "base" environment (i.e., not executing in a signal handler) >SIGCHLD is enabled as a postcondition of allow_othersigs. > >All *other* occurrences of allow_sigvtalrm are of course replaced with > >allow_sigvtalrm(); allow_othersigs(); > >The goal is that SIGCHLD be enabled precisely when SIGVTALRM is except inside >the signal handler itself. > > Mika > > >Tony Hosking writes: >>Sorry, I missed you e-mail describing this change. Looks good. >> >>Antony Hosking | Associate Professor | Computer Science | Purdue = >>University >>305 N. University Street | West Lafayette | IN 47907 | USA >>Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >>On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: >> >>> Tony, >>>=20 >>> The old user threads has an old problem with Process.Wait. It has a=20= >> >>> Thread.Pause(0.1d0) in it. This causes the compiler to run three = >>times >>> slower than it should. The m3build I've been using for years has >>> that Thread.Pause removed (so it uses 100% CPU instead, = >>busy-waiting...) >>>=20 >>> But yeah user threads works the same as it used to (except for=20 >>> now calling pthread_atfork, which I'll return to in another email...) >>>=20 >>> Mika ... From rcolebur at SCIRES.COM Tue Feb 15 00:01:14 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Mon, 14 Feb 2011 18:01:14 -0500 Subject: [M3devel] More on threading In-Reply-To: <20110213181717.4F0E91A2078@async.async.caltech.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> Message-ID: Mika wrote... >But no I don't use Windows. I was asking on here how to get started with >Modula-3 on Windows once but my machine was/is a Windows 2000 system and >apparently CM3 no longer works on win2k..? I use CM3 on Windows 2000/XP/7 and it works on all of these versions. --Randy Coleburn From rcolebur at SCIRES.COM Tue Feb 15 00:54:50 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Mon, 14 Feb 2011 18:54:50 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , , , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, , <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , , , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , , , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu>, , Message-ID: Jay: Up to now, I've been using my Windows CMD scripts to "upgrade", namely: scripts\dev\windows\ RCC_upgradeCM3.CMD But, based on your message, I am trying to do as you suggest, i.e., use your "upgrade.py". I am running Windows 7. I have just installed Python 2.7.1 via http://www.python.org/ftp/python/2.7.1/python-2.7.1.msi Here is error message I get: C:\cm3\Sandbox\scripts\python>upgrade.py Traceback (most recent call last): File "C:\cm3\Sandbox\scripts\python\upgrade.py", line 4, in import pylib File "C:\cm3\Sandbox\scripts\python\pylib.py", line 612, in if Target.startswith("NT386"): AttributeError: 'NoneType' object has no attribute 'startswith' Please advise. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Saturday, February 12, 2011 5:39 AM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? TextUtils.mo : error LNK2001: unresolved external symbol __alloca TextUtils.mo : error LNK2001: unresolved external symbol _Csetjmp__Jumpbuf_size sysutils.dll : fatal error LNK1120: 2 unresolved externals Most likely you didn't build it correctly. Go back to a working cm3/m3core/libm3 and run upgrade.py. There should be no longer any references to alloca of Csetjmp__Jumpbuf_size. Really, Csetjmp__Jumpbuf_size should never have been a problem. That likely also indicates some sort of incorrect bootstrap. > Are you saying you do, or do not, want me to work further on the solving the "problem," regardless of how we describe it ? I still want alloca to work, either via a function call or a specific m3cg method. (I still don't see that RTHooks.Alloca makes better sense; I still don't see a way to avoid hacking the compiler, given the custom calling convention.) It will be difficult to test the fix until/unless m3front is changed back. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Fri, 11 Feb 2011 18:17:15 -0500 Subject: Re: [M3devel] threading on Windows? Jay: Ok, I don't mean to start a new issue or upset anyone. I see you checked in some new changes. I updated to latest HEAD and tried to build, but again I get unresolved symbols beginning with m3-libs\sysutils (see attached sysutils.lst) So, not sure what you mean by "it should be ok now". Are you saying you do, or do not, want me to work further on the solving the "problem," regardless of how we describe it ? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, February 11, 2011 4:52 PM To: Tony; Coleburn, Randy Cc: m3devel Subject: RE: [M3devel] threading on Windows? It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - Jay ________________________________ From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new "heap-based jmpbuf allocation" have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate "alloca" routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I "do" anything, I'll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the "heap based jumpbuf allocation scheme" was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am "in the dark", both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I'm seeing several different symbols that don't resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking > wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 15 02:12:09 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 15 Feb 2011 01:12:09 +0000 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , <20110213181717.4F0E91A2078@async.async.caltech.edu>, Message-ID: Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 15 02:34:59 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 15 Feb 2011 01:34:59 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, ,,, ,,, ,,, ,,, ,,, ,,, ,,, ,,<39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, ,,<9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, ,,, ,,<74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, ,,, , , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu>, , , , , , Message-ID: I'll have to look into it. It is possible I dropped support for "NT386" in favor of "I386_NT". I've definitely used the Python a fair amount on Windows, but less lately. You can probably force it by first setting the environment variable CM3_TARGET to NT386 or I386_NT. Or by giving either of those parameters on the command line. But you shouldn't have to. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Mon, 14 Feb 2011 18:54:50 -0500 Subject: Re: [M3devel] threading on Windows? Jay: Up to now, I?ve been using my Windows CMD scripts to ?upgrade?, namely: scripts\dev\windows\ RCC_upgradeCM3.CMD But, based on your message, I am trying to do as you suggest, i.e., use your ?upgrade.py?. I am running Windows 7. I have just installed Python 2.7.1 via http://www.python.org/ftp/python/2.7.1/python-2.7.1.msi Here is error message I get: C:\cm3\Sandbox\scripts\python>upgrade.py Traceback (most recent call last): File "C:\cm3\Sandbox\scripts\python\upgrade.py", line 4, in import pylib File "C:\cm3\Sandbox\scripts\python\pylib.py", line 612, in if Target.startswith("NT386"): AttributeError: 'NoneType' object has no attribute 'startswith' Please advise. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Saturday, February 12, 2011 5:39 AM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? TextUtils.mo : error LNK2001: unresolved external symbol __alloca TextUtils.mo : error LNK2001: unresolved external symbol _Csetjmp__Jumpbuf_size sysutils.dll : fatal error LNK1120: 2 unresolved externals Most likely you didn't build it correctly. Go back to a working cm3/m3core/libm3 and run upgrade.py. There should be no longer any references to alloca of Csetjmp__Jumpbuf_size. Really, Csetjmp__Jumpbuf_size should never have been a problem. That likely also indicates some sort of incorrect bootstrap. > Are you saying you do, or do not, want me to work further on the solving the ?problem,? regardless of how we describe it ? I still want alloca to work, either via a function call or a specific m3cg method. (I still don't see that RTHooks.Alloca makes better sense; I still don't see a way to avoid hacking the compiler, given the custom calling convention.) It will be difficult to test the fix until/unless m3front is changed back. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Fri, 11 Feb 2011 18:17:15 -0500 Subject: Re: [M3devel] threading on Windows? Jay: Ok, I don?t mean to start a new issue or upset anyone. I see you checked in some new changes. I updated to latest HEAD and tried to build, but again I get unresolved symbols beginning with m3-libs\sysutils (see attached sysutils.lst) So, not sure what you mean by ?it should be ok now?. Are you saying you do, or do not, want me to work further on the solving the ?problem,? regardless of how we describe it ? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, February 11, 2011 4:52 PM To: Tony; Coleburn, Randy Cc: m3devel Subject: RE: [M3devel] threading on Windows? It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - Jay From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate ?alloca? routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Tue Feb 15 02:56:57 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Mon, 14 Feb 2011 17:56:57 -0800 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20110215013018.B4DC12474005@birch.elegosoft.com> Message-ID: <20110215015657.38D541A2078@async.async.caltech.edu> Jay K writes: >--_0b68c717-b1e6-43b7-b824-79e05d0f3a43_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >You might want to throttle the number of outstanding promises. Yeah that's in a different module. QMachine doesn't control what's actually done with the promises (how many in parallel are run etc). The idea is: 1. client of QMachine sets "doRecord" flag 2. QMachine writes all the back-end promises to a sequence. 3. client clears doRecord flag After that it's up to the client to launch the promises as it desires, one at a time or 100 at a time, as the case may be. I'm looking at how to specify this now. You're right it's annoying there's no automatic way of doing it. Default will be serial as always, the impatient can specify a flag or a setting somewhere. I'm leaning towards making it both a config variable and a command-line option... The code is kind of ugly but it sped up my compiles so much I immediately felt I couldn't live without it....I'm taking care that it won't change the behavior for anyone else (unless they ask for it). Mika >If only Posix provided good functions for e.g. finding the number of proces= >sors. >Or if NT or Posix made it easy to discover the number of spindles backing a= > path=2C >or to easily measure system utilization and backing down when it gets near = >100%. >You know=2C an idle machine should be maxed out=2C but no more. >A machine with an interactive user should reserve CPU/memory/I/O for forgro= >und activity.. >I don't I've ever seen this stuff done well though. >=20 > - Jay > >=20 >> Date: Tue=2C 15 Feb 2011 02:30:18 +0000 >> To: m3commit at elegosoft.com >> From: mika at elego.de >> Subject: [M3commit] CVS Update: cm3 >>=20 >> CVSROOT: /usr/cvs >> Changes by: mika at birch. 11/02/15 02:30:18 >>=20 >> Modified files: >> cm3/m3-sys/m3quake/src/: QMachine.i3 QMachine.m3 m3makefile=20 >> Added files: >> cm3/m3-sys/m3quake/src/: QPromise.i3 QPromise.m3=20 >>=20 >> Log message: >> Add support for the QMachine to be able to record (certain) Process.Creat= >es as "promises" rather than executing them on the fly. >>=20 >> The purpose of this modification is to allow parallelization of the CM3 c= >ompiler. The vast majority of the runtime of the compilation of a typical M= >odula-3 program is spent in the part "promised" here (when using the GCC ba= >ckend). >>=20 > = > >--_0b68c717-b1e6-43b7-b824-79e05d0f3a43_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >You might want to throttle the number of outstanding promises.
>If only Posix provided good functions for e.g. finding the number of proces= >sors.
>Or if =3BNT or Posix made it easy to discover the number of spindles ba= >cking a path=2C
>or to easily measure system utilization and backing down when it =3Bget= >s near 100%.
>You know=2C an idle machine should be maxed out=2C but no more.
>A machine with an interactive user should reserve =3BCPU/memory/I/O for= > forground activity..
>I don't I've ever seen this stuff done well though.
> =3B
> =3B- Jay

 =3B
>>=3B Date: Tue=2C 15 Feb 2011 02:30:18 +0000
>=3B To: m3commit at elego= >soft.com
>=3B From: mika at elego.de
>=3B Subject: [M3commit] CVS Up= >date: cm3
>=3B
>=3B CVSROOT: /usr/cvs
>=3B Changes by: mika= >@birch. 11/02/15 02:30:18
>=3B
>=3B Modified files:
>=3B cm= >3/m3-sys/m3quake/src/: QMachine.i3 QMachine.m3 m3makefile
>=3B Added = >files:
>=3B cm3/m3-sys/m3quake/src/: QPromise.i3 QPromise.m3
>= >=3B
>=3B Log message:
>=3B Add support for the QMachine to be ab= >le to record (certain) Process.Creates as "promises" rather than executing = >them on the fly.
>=3B
>=3B The purpose of this modification is t= >o allow parallelization of the CM3 compiler. The vast majority of the runti= >me of the compilation of a typical Modula-3 program is spent in the part "p= >romised" here (when using the GCC backend).
>=3B
> y> >= > >--_0b68c717-b1e6-43b7-b824-79e05d0f3a43_-- From wagner at elegosoft.com Tue Feb 15 08:53:37 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 15 Feb 2011 08:53:37 +0100 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110214211829.BA3761A2078@async.async.caltech.edu> References: <20110213220804.7A9001A2078@async.async.caltech.edu> <20110214211829.BA3761A2078@async.async.caltech.edu> Message-ID: <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> Yes, please commit changes that improve matters, even if they are not perfect. Perhaps Tony will review your latest suggestion again? Your thread test program should be integrated into the Hudson regression test framework for the cm3 current line; I don't think it is already. That way we would get an overview of the thread state for all platforms at least once a day. It would be great if the tests would show up in the cm3-current-test-m3tests-TARGET jobs; it shouldn't be too difficult. Basically, only one script should need to be extended, and some XML result generation added. IIRC it's in cm3/scripts/regression/defs.sh in test_m3tests(), or (in quake) in the m3tests m3makefile. Can you do that? If not, I may have a look myself at the weekend... Olaf PS: We really should get all our code base into a state that you can run all your applications without any problems on current. After all there aren't that many active M3 users... PPS: I need to return to my offline testing of LDAP protocol handlers for online charging servers now :-/ Quoting Mika Nystrom : > Hi again m3devel, > > Does anyone understand the code in ThreadPosix.m3? > > After looking through it again I think my attached solution (sent out > yesterday afternoon) is right, but I'm still just a little bit fuzzy on > how the code works (is supposed to work) but I have some hypotheses. > > I realized it's been a long, long, loooong time since I coded anything > with signals and signal handlers and I've forgotten much of the semantics. > After all since Concurrent Pascal came out in '76 signals has always been > the "old" way of doing things, right? (And I quit programming C in favor > of M3 in 1999...) > > In ThreadPosixC.c, there's a function called setup_sigvtalrm that > registers the Modula-3 procedure switch_thread as a signal handler. > > My understanding is that in C, signals are masked atomically on the > call into the signal handler and unmasked atomically when the handler > returns (unless the signal mask is changed, etc., while in the handler). > > However, switch_thread... doesn't return? Or? Or it does a longjmp > somewhere? Which means signals may not be unmasked(?) Which is the reason > for the calling "allow_sigvtalrm" as the first statement of the signal > handler? > > Now I can't just turn on ALL signals in the signal handler because then > I may get very deeply nested signal handlers, so I only call allow_allsignals > (or whatever we call it) from the first-level signal handler. Hmm. > Maybe that is not quite right? Do we eventually get back there? Yes > I think we must because certain things depend on inCritical being zero... > and when inCritical is zero SIGCHLD is again enabled. > > No I still don't fully understand how the code is supposed to work, > especially in the presence of nested signal handling. Does anyone > understand better? > > (I am still going to try to commit my patch later today because I don't > think it could possibly make things worse and it does make a significant > real-world improvement to the compiler performance under user threads.) > > Mika > > Mika Nystrom writes: >> Hi Tony, >> >> I found a problem with the change. I would appreciate it if you and >> Jay could think about what I found and my solution (and anybody else >> is welcome to as well, of course). >> >> When fiddling with cm3 to get it to run in parallel, I pushed it up to >> fork up to 100 processes at once, just to see what happens. When I did >> that (with my SIGCHLD patch) I got a few errors such as illegal instruction >> and a bus error or two. I believe what was happening was that I was >> taking too many signals at once, because of the following code in >> ThreadPosix.m3: >> >> PROCEDURE switch_thread (sig: int) RAISES {Alerted} = >> BEGIN >> allow_sigvtalrm (); (* here *) >> >> INC(inCritical); >> (* mark signal as being delivered *) >> IF sig >= 0 AND sig < MaxSigs THEN >> gotSigs := gotSigs + SET OF Sig { sig } >> END; >> DEC(inCritical); >> >> IF inCritical = 0 AND heapState.inCritical = 0 THEN >> InternalYield () >> END; >> END switch_thread; >> >> I found it a bit odd that the first action of the signal handler is to >> enable signals. With VTALRM this is innocuous, but in my case, imagine >> what happens if you have a lot of exited child processes queued up. >> What happens is that your signal stack gets wound deeper and deeper, and >> each time you catch one SIGCHLD you ask for another. Eventually you >> run out of stack space. >> >> My change, which appears to fix this problem, is to split up the "allow" >> function as follows: >> >> PROCEDURE switch_thread (sig: int) RAISES {Alerted} = >> BEGIN >> allow_sigvtalrm (); (* VTALRM only *) >> >> INC(inCritical); >> IF inCritical = 1 THEN allow_othersigs (); END; (* SIGCHLD *) >> >> (* mark signal as being delivered *) >> IF sig >= 0 AND sig < MaxSigs THEN >> gotSigs := gotSigs + SET OF Sig { sig } >> END; >> DEC(inCritical); >> >> IF inCritical = 0 AND heapState.inCritical = 0 THEN >> InternalYield () >> END; >> END switch_thread; >> >> This way, SIGCHLD is not taken in nested signal handlers. But is there a >> problem here? Well, the fallback is that it takes 0.1 seconds to wake >> up on Process.Wait . >> >> Is there a chance that SIGCHLD signals are left pending? My argument is >> no, since in the "base" environment (i.e., not executing in a >> signal handler) >> SIGCHLD is enabled as a postcondition of allow_othersigs. >> >> All *other* occurrences of allow_sigvtalrm are of course replaced with >> >> allow_sigvtalrm(); allow_othersigs(); >> >> The goal is that SIGCHLD be enabled precisely when SIGVTALRM is >> except inside >> the signal handler itself. >> >> Mika >> >> >> Tony Hosking writes: >>> Sorry, I missed you e-mail describing this change. Looks good. >>> >>> Antony Hosking | Associate Professor | Computer Science | Purdue = >>> University >>> 305 N. University Street | West Lafayette | IN 47907 | USA >>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>> >>> >>> >>> >>> On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: >>> >>>> Tony, >>>> =20 >>>> The old user threads has an old problem with Process.Wait. It has a=20= >>> >>>> Thread.Pause(0.1d0) in it. This causes the compiler to run three = >>> times >>>> slower than it should. The m3build I've been using for years has >>>> that Thread.Pause removed (so it uses 100% CPU instead, = >>> busy-waiting...) >>>> =20 >>>> But yeah user threads works the same as it used to (except for=20 >>>> now calling pthread_atfork, which I'll return to in another email...) >>>> =20 >>>> Mika > ... > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rcolebur at SCIRES.COM Tue Feb 15 23:05:37 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Tue, 15 Feb 2011 17:05:37 -0500 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , <20110213181717.4F0E91A2078@async.async.caltech.edu>, Message-ID: Jay, since I have a Windows2000 system, I can test the MAC address discovery (or anything else you need). Just let me know. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Monday, February 14, 2011 8:12 PM To: Coleburn, Randy; m3devel Subject: Re: [M3devel] More on threading Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Feb 16 01:56:49 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 16 Feb 2011 00:56:49 +0000 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , , , <20110213181717.4F0E91A2078@async.async.caltech.edu>, , , , Message-ID: It should work. http://dcvs.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/uid/WIN32/MachineID.c?rev=1.14;content-type=text%2Fplain Bing for "MSDN GetIfTable" => http://msdn.microsoft.com/en-us/library/aa365943(VS.85).aspx Mika, what made you think cm3 doesn't work on Windows 2000?? I don't think anything has been remotely suggested. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Tue, 15 Feb 2011 17:05:37 -0500 Subject: Re: [M3devel] More on threading Jay, since I have a Windows2000 system, I can test the MAC address discovery (or anything else you need). Just let me know. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Monday, February 14, 2011 8:12 PM To: Coleburn, Randy; m3devel Subject: Re: [M3devel] More on threading Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Wed Feb 16 05:17:06 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Tue, 15 Feb 2011 20:17:06 -0800 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , , , <20110213181717.4F0E91A2078@async.async.caltech.edu>, , , , Message-ID: <20110216041706.87A2D1A2078@async.async.caltech.edu> Jay K writes: >--_a2f17729-cb11-4f2b-8c70-c3b583cda751_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >It should work. >=20 >=20 >http://dcvs.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/uid/WIN3= >2/MachineID.c?rev=3D1.14=3Bcontent-type=3Dtext%2Fplain >=20 >=20 >Bing for "MSDN GetIfTable" >=3D> http://msdn.microsoft.com/en-us/library/aa365943(VS.85).aspx >=20 >=20 >Mika=2C what made you think cm3 doesn't work on Windows 2000?? >I don't think anything has been remotely suggested. Hi Jay, I searched my email and I do indeed misremember---sort of. I had an email exchange with you on Friday 28 August 2009 where I ran into a bunch of problems that I didn't know how to solve, and I gave up there. And you asked me if I "planned to upgrade to XP or newer anytime soon". I may have taken that as a sign that I shouldn't expect things to work on win2k but you didn't actually say that anywhere. Mika From wagner at elegosoft.com Thu Feb 17 23:47:30 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Thu, 17 Feb 2011 23:47:30 +0100 Subject: [M3devel] cm3-ide In-Reply-To: References: Message-ID: <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com> Quoting ??? ????????? : > Hi, > I am a student of Software Engineering. > I want to learn the language modula-3 and I'm trying to install IDE on my > computer. > I have windows 7 operating system and I can not find any suitable > installation file. > Do you have appropriate file or instructions how to do it on another system? > I need it for exercise in school and I'd love it if you could help me. > Thank you. > zeev blumentzvaig What exactly have you tried already? The archive for Windows listed on http://www.opencm3.net/download.html, http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should work for Windows 7, too, as far as I know. You simply need to unpack it and install a suitable linker, as described for example in http://www.opencm3.net/installation-windows.html. In order to help you, we need to know exactly what you have done and what failed. I'm not an expert on Windows; others on the list may be able to help better, so I CC this to m3devel. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Thu Feb 17 23:56:05 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Thu, 17 Feb 2011 14:56:05 -0800 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> References: <20110213220804.7A9001A2078@async.async.caltech.edu> <20110214211829.BA3761A2078@async.async.caltech.edu> <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> Message-ID: <20110217225605.644AC1A2078@async.async.caltech.edu> Hi Olaf, I have committed the SIGCHLD diff and run quite heavy jobs with it in the code, and so far I have not seen any problems. I am fairly sure there won't be any. Can you show me how to integrate the test program? Note that when it fails it quite often (usually) hangs rather than producing any specific error output. I have also definitely run into pthreads problems on I386_DARWIN now, not running the threadtest program. They show up when I enable the parallel back-end builds (I suspect they will do that on all architectures with pthreads---I've rebuilt all my CM3 installations with user threads). Mika Olaf Wagner writes: >Yes, please commit changes that improve matters, even if they are not >perfect. Perhaps Tony will review your latest suggestion again? > >Your thread test program should be integrated into the Hudson regression >test framework for the cm3 current line; I don't think it is already. >That way we would get an overview of the thread state for all platforms >at least once a day. It would be great if the tests would show up >in the cm3-current-test-m3tests-TARGET jobs; it shouldn't be too >difficult. Basically, only one script should need to be extended, and >some XML result generation added. IIRC it's in cm3/scripts/regression/defs.s= >h >in test_m3tests(), or (in quake) in the m3tests m3makefile. > >Can you do that? If not, I may have a look myself at the weekend... > >Olaf > >PS: We really should get all our code base into a state that you can >run all your applications without any problems on current. After all >there aren't that many active M3 users... > >PPS: I need to return to my offline testing of LDAP protocol handlers >for online charging servers now :-/ > >Quoting Mika Nystrom : > >> Hi again m3devel, >> >> Does anyone understand the code in ThreadPosix.m3? >> >> After looking through it again I think my attached solution (sent out >> yesterday afternoon) is right, but I'm still just a little bit fuzzy on >> how the code works (is supposed to work) but I have some hypotheses. >> >> I realized it's been a long, long, loooong time since I coded anything >> with signals and signal handlers and I've forgotten much of the semantics. >> After all since Concurrent Pascal came out in '76 signals has always been >> the "old" way of doing things, right? (And I quit programming C in favor >> of M3 in 1999...) >> >> In ThreadPosixC.c, there's a function called setup_sigvtalrm that >> registers the Modula-3 procedure switch_thread as a signal handler. >> >> My understanding is that in C, signals are masked atomically on the >> call into the signal handler and unmasked atomically when the handler >> returns (unless the signal mask is changed, etc., while in the handler). >> >> However, switch_thread... doesn't return? Or? Or it does a longjmp >> somewhere? Which means signals may not be unmasked(?) Which is the reaso= >n >> for the calling "allow_sigvtalrm" as the first statement of the signal >> handler? >> >> Now I can't just turn on ALL signals in the signal handler because then >> I may get very deeply nested signal handlers, so I only call allow_allsign= >als >> (or whatever we call it) from the first-level signal handler. Hmm. >> Maybe that is not quite right? Do we eventually get back there? Yes >> I think we must because certain things depend on inCritical being zero... >> and when inCritical is zero SIGCHLD is again enabled. >> >> No I still don't fully understand how the code is supposed to work, >> especially in the presence of nested signal handling. Does anyone >> understand better? >> >> (I am still going to try to commit my patch later today because I don't >> think it could possibly make things worse and it does make a significant >> real-world improvement to the compiler performance under user threads.) >> >> Mika >> >> Mika Nystrom writes: >>> Hi Tony, >>> >>> I found a problem with the change. I would appreciate it if you and >>> Jay could think about what I found and my solution (and anybody else >>> is welcome to as well, of course). >>> >>> When fiddling with cm3 to get it to run in parallel, I pushed it up to >>> fork up to 100 processes at once, just to see what happens. When I did >>> that (with my SIGCHLD patch) I got a few errors such as illegal instructi= >on >>> and a bus error or two. I believe what was happening was that I was >>> taking too many signals at once, because of the following code in >>> ThreadPosix.m3: >>> >>> PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>> BEGIN >>> allow_sigvtalrm (); (* here *) >>> >>> INC(inCritical); >>> (* mark signal as being delivered *) >>> IF sig >=3D 0 AND sig < MaxSigs THEN >>> gotSigs :=3D gotSigs + SET OF Sig { sig } >>> END; >>> DEC(inCritical); >>> >>> IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >>> InternalYield () >>> END; >>> END switch_thread; >>> >>> I found it a bit odd that the first action of the signal handler is to >>> enable signals. With VTALRM this is innocuous, but in my case, imagine >>> what happens if you have a lot of exited child processes queued up. >>> What happens is that your signal stack gets wound deeper and deeper, and >>> each time you catch one SIGCHLD you ask for another. Eventually you >>> run out of stack space. >>> >>> My change, which appears to fix this problem, is to split up the "allow" >>> function as follows: >>> >>> PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>> BEGIN >>> allow_sigvtalrm (); (* VTALRM only *) >>> >>> INC(inCritical); >>> IF inCritical =3D 1 THEN allow_othersigs (); END; (* SIGCHLD *) >>> >>> (* mark signal as being delivered *) >>> IF sig >=3D 0 AND sig < MaxSigs THEN >>> gotSigs :=3D gotSigs + SET OF Sig { sig } >>> END; >>> DEC(inCritical); >>> >>> IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >>> InternalYield () >>> END; >>> END switch_thread; >>> >>> This way, SIGCHLD is not taken in nested signal handlers. But is there a >>> problem here? Well, the fallback is that it takes 0.1 seconds to wake >>> up on Process.Wait . >>> >>> Is there a chance that SIGCHLD signals are left pending? My argument is >>> no, since in the "base" environment (i.e., not executing in a =20 >>> signal handler) >>> SIGCHLD is enabled as a postcondition of allow_othersigs. >>> >>> All *other* occurrences of allow_sigvtalrm are of course replaced with >>> >>> allow_sigvtalrm(); allow_othersigs(); >>> >>> The goal is that SIGCHLD be enabled precisely when SIGVTALRM is =20 >>> except inside >>> the signal handler itself. >>> >>> Mika >>> >>> >>> Tony Hosking writes: >>>> Sorry, I missed you e-mail describing this change. Looks good. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue =3D >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: >>>> >>>>> Tony, >>>>> =3D20 >>>>> The old user threads has an old problem with Process.Wait. It has a=3D= >20=3D >>>> >>>>> Thread.Pause(0.1d0) in it. This causes the compiler to run three =3D >>>> times >>>>> slower than it should. The m3build I've been using for years has >>>>> that Thread.Pause removed (so it uses 100% CPU instead, =3D >>>> busy-waiting...) >>>>> =3D20 >>>>> But yeah user threads works the same as it used to (except for=3D20 >>>>> now calling pthread_atfork, which I'll return to in another email...) >>>>> =3D20 >>>>> Mika >> ... >> > > > >--=20 >Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa= >ny >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz= >: Berlin >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Fri Feb 18 09:32:46 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 18 Feb 2011 09:32:46 +0100 Subject: [M3devel] Fwd: Re: cm3-ide Message-ID: <20110218093246.fzjch5luokkwgokk@mail.elegosoft.com> FYI ----- Forwarded message from zeev.blu at gmail.com ----- Date: Fri, 18 Feb 2011 09:03:38 +0200 From: ??? ????????? Reply-To: ??? ????????? Subject: Re: cm3-ide To: Olaf Wagner Cc: m3devel at elegosoft.com, m3-support at elego.de, moriah.moshe at gmail.com Hi, First, thanks for the quick reply. I have tried to use the file you gave me, and some other files from that site. I have some questions: 1. If I understand correctly, the method of working with this compiler is through the CMD in WINDOWS. Is not this language a IDE to work with? 2. When I tried to compile , I got a linking error: " link is not recognized as an internal or external command, operable program or batch file." And then another error: "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " And finally I received: " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" failed. " Thanks for your help zeev blumentzvaig 2011/2/18 Olaf Wagner > Quoting ??? ????????? : > > Hi, >> I am a student of Software Engineering. >> I want to learn the language modula-3 and I'm trying to install IDE on my >> computer. >> I have windows 7 operating system and I can not find any suitable >> installation file. >> Do you have appropriate file or instructions how to do it on another >> system? >> I need it for exercise in school and I'd love it if you could help me. >> Thank you. >> zeev blumentzvaig >> > > What exactly have you tried already? > > The archive for Windows listed on http://www.opencm3.net/download.html, > http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should > work for Windows 7, too, as far as I know. You simply need to > unpack it and install a suitable linker, as described for example in > http://www.opencm3.net/installation-windows.html. > > In order to help you, we need to know exactly what you have done and > what failed. I'm not an expert on Windows; others on the list may > be able to help better, so I CC this to m3devel. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 > 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: > DE163214194 > > ----- End forwarded message ----- -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 -------------- next part -------------- Hi, First, thanks for the quick reply. I have tried to use the file you gave me, and some other files from that site. I have some questions: 1. If I understand correctly, the method of working with this compiler is through the CMD in WINDOWS. Is not this language a IDE to work with? 2. When I tried to compile , I got a linking error: " link is not recognized as an internal or external command, operable program or batch file." And then another error: "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " And finally I received: " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" failed. " Thanks for your help zeev blumentzvaig 2011/2/18 Olaf Wagner > Quoting ??? ????????? : > > Hi, >> I am a student of Software Engineering. >> I want to learn the language modula-3 and I'm trying to install IDE on my >> computer. >> I have windows 7 operating system and I can not find any suitable >> installation file. >> Do you have appropriate file or instructions how to do it on another >> system? >> I need it for exercise in school and I'd love it if you could help me. >> Thank you. >> zeev blumentzvaig >> > > What exactly have you tried already? > > The archive for Windows listed on http://www.opencm3.net/download.html, > http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should > work for Windows 7, too, as far as I know. You simply need to > unpack it and install a suitable linker, as described for example in > http://www.opencm3.net/installation-windows.html. > > In order to help you, we need to know exactly what you have done and > what failed. I'm not an expert on Windows; others on the list may > be able to help better, so I CC this to m3devel. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 > 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: > DE163214194 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Fri Feb 18 09:40:14 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 18 Feb 2011 09:40:14 +0100 Subject: [M3devel] cm3-ide In-Reply-To: References: <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com> Message-ID: <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> Quoting ??? ????????? : > Hi, > First, thanks for the quick reply. > > I have tried to use the file you gave me, and some other files from that > site. > I have some questions: > 1. If I understand correctly, the method of working with this compiler is > through the CMD in WINDOWS. Is not this language a IDE to work with? A language and its implementation in a compiler does not necessarily require a full-fledged intergrated development environment like Eclipse. My faourite IDE when I was actively using M3 was always XEmacs :-) There _is_ a web-based cm3-ide (which should be contained in that archive) that allows you to build and manage your projects with a standard web browser, but that might be quite different from your expectations of an IDE, too ;-) > 2. When I tried to compile , I got a linking error: > > " link is not recognized as an internal or external command, operable > program or batch file." > > And then another error: > > "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " > > And finally I received: > > " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" > failed. " You either missed the installation of a suitable system linker as described in the "Windows installation notes" referred to below or you didn't set the appropriate variables in your environment or in the cm3 configuration files. CM3 does not include a system linker, and on Windows you have got to install one yourself, while on all other systems it usually is installed by default. You can use the cm3 option "-commands" to actually view the external commands the builder is calling. I hope this helps, Olaf > Thanks for your help > > zeev blumentzvaig > > > 2011/2/18 Olaf Wagner > >> Quoting ??? ????????? : >> >> Hi, >>> I am a student of Software Engineering. >>> I want to learn the language modula-3 and I'm trying to install IDE on my >>> computer. >>> I have windows 7 operating system and I can not find any suitable >>> installation file. >>> Do you have appropriate file or instructions how to do it on another >>> system? >>> I need it for exercise in school and I'd love it if you could help me. >>> Thank you. >>> zeev blumentzvaig >>> >> >> What exactly have you tried already? >> >> The archive for Windows listed on http://www.opencm3.net/download.html, >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should >> work for Windows 7, too, as far as I know. You simply need to >> unpack it and install a suitable linker, as described for example in >> http://www.opencm3.net/installation-windows.html. >> >> In order to help you, we need to know exactly what you have done and >> what failed. I'm not an expert on Windows; others on the list may >> be able to help better, so I CC this to m3devel. >> >> Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Fri Feb 18 10:14:14 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 18 Feb 2011 10:14:14 +0100 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110217225605.644AC1A2078@async.async.caltech.edu> References: <20110213220804.7A9001A2078@async.async.caltech.edu> <20110214211829.BA3761A2078@async.async.caltech.edu> <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> <20110217225605.644AC1A2078@async.async.caltech.edu> Message-ID: <20110218101414.yu4y8iyg0kkw40cg@mail.elegosoft.com> Quoting Mika Nystrom : > Hi Olaf, > > I have committed the SIGCHLD diff and run quite heavy jobs with it > in the code, and so far I have not seen any problems. I am fairly sure > there won't be any. Sounds good. > Can you show me how to integrate the test program? I already gave some hints below. I think what you definitely could do is writing the required XML output, either in M3 or as shell script. There's a simple schema describing what is required; some time ago we've put a copy on our web server: http://www.elegosoft.com/misc/JUnitXSchema.xsd It may be easier to just look at an example, e.g. http://hudson.modula3.com:8080/job/cm3-current-test-m3tests-AMD64_FREEBSD/ws/cm3/m3-sys/m3tests/m3tests-results.xml > Note that when it fails > it quite often (usually) hangs rather than producing any specific error > output. If we cannot avoid non-termination on our own, we must rely on the Hudson job timeout. This would definitely mean that we run the program in their own jobs or at the end of the other tests. Still, I think the whole test job will fail if it doesn't terminate. A runtime supervision for regression testing is something that is missing in our M3 world as has been noted before :-( If you provide the XML output, I can care for the integration into Hudson this or the next weekend. Or alternatively, you can do that on your own if you like and I help as good as I can. If you haven't got an account on Hudson, I can quickly create one. > I have also definitely run into pthreads problems on I386_DARWIN now, > not running the threadtest program. They show up when I enable the > parallel back-end builds (I suspect they will do that on all architectures > with pthreads---I've rebuilt all my CM3 installations with user threads). Hm, sounds like the complete switch to pthreads might have been a little too early, but we haven't had any reliable data to decide on IIRC. All the world uses pthreads, so I assumed they would be OK for us, too :-/ However, we need to fix the problems now; I hope Antony can spare some time. He's been the main architect for that. And it's good to know that user threads still work. Olaf > Mika > > Olaf Wagner writes: >> Yes, please commit changes that improve matters, even if they are not >> perfect. Perhaps Tony will review your latest suggestion again? >> >> Your thread test program should be integrated into the Hudson regression >> test framework for the cm3 current line; I don't think it is already. >> That way we would get an overview of the thread state for all platforms >> at least once a day. It would be great if the tests would show up >> in the cm3-current-test-m3tests-TARGET jobs; it shouldn't be too >> difficult. Basically, only one script should need to be extended, and >> some XML result generation added. IIRC it's in >> cm3/scripts/regression/defs.s= >> h >> in test_m3tests(), or (in quake) in the m3tests m3makefile. >> >> Can you do that? If not, I may have a look myself at the weekend... >> >> Olaf >> >> PS: We really should get all our code base into a state that you can >> run all your applications without any problems on current. After all >> there aren't that many active M3 users... >> >> PPS: I need to return to my offline testing of LDAP protocol handlers >> for online charging servers now :-/ >> >> Quoting Mika Nystrom : >> >>> Hi again m3devel, >>> >>> Does anyone understand the code in ThreadPosix.m3? >>> >>> After looking through it again I think my attached solution (sent out >>> yesterday afternoon) is right, but I'm still just a little bit fuzzy on >>> how the code works (is supposed to work) but I have some hypotheses. >>> >>> I realized it's been a long, long, loooong time since I coded anything >>> with signals and signal handlers and I've forgotten much of the semantics. >>> After all since Concurrent Pascal came out in '76 signals has always been >>> the "old" way of doing things, right? (And I quit programming C in favor >>> of M3 in 1999...) >>> >>> In ThreadPosixC.c, there's a function called setup_sigvtalrm that >>> registers the Modula-3 procedure switch_thread as a signal handler. >>> >>> My understanding is that in C, signals are masked atomically on the >>> call into the signal handler and unmasked atomically when the handler >>> returns (unless the signal mask is changed, etc., while in the handler). >>> >>> However, switch_thread... doesn't return? Or? Or it does a longjmp >>> somewhere? Which means signals may not be unmasked(?) Which is the reaso= >> n >>> for the calling "allow_sigvtalrm" as the first statement of the signal >>> handler? >>> >>> Now I can't just turn on ALL signals in the signal handler because then >>> I may get very deeply nested signal handlers, so I only call allow_allsign= >> als >>> (or whatever we call it) from the first-level signal handler. Hmm. >>> Maybe that is not quite right? Do we eventually get back there? Yes >>> I think we must because certain things depend on inCritical being zero... >>> and when inCritical is zero SIGCHLD is again enabled. >>> >>> No I still don't fully understand how the code is supposed to work, >>> especially in the presence of nested signal handling. Does anyone >>> understand better? >>> >>> (I am still going to try to commit my patch later today because I don't >>> think it could possibly make things worse and it does make a significant >>> real-world improvement to the compiler performance under user threads.) >>> >>> Mika >>> >>> Mika Nystrom writes: >>>> Hi Tony, >>>> >>>> I found a problem with the change. I would appreciate it if you and >>>> Jay could think about what I found and my solution (and anybody else >>>> is welcome to as well, of course). >>>> >>>> When fiddling with cm3 to get it to run in parallel, I pushed it up to >>>> fork up to 100 processes at once, just to see what happens. When I did >>>> that (with my SIGCHLD patch) I got a few errors such as illegal instructi= >> on >>>> and a bus error or two. I believe what was happening was that I was >>>> taking too many signals at once, because of the following code in >>>> ThreadPosix.m3: >>>> >>>> PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>>> BEGIN >>>> allow_sigvtalrm (); (* here *) >>>> >>>> INC(inCritical); >>>> (* mark signal as being delivered *) >>>> IF sig >=3D 0 AND sig < MaxSigs THEN >>>> gotSigs :=3D gotSigs + SET OF Sig { sig } >>>> END; >>>> DEC(inCritical); >>>> >>>> IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >>>> InternalYield () >>>> END; >>>> END switch_thread; >>>> >>>> I found it a bit odd that the first action of the signal handler is to >>>> enable signals. With VTALRM this is innocuous, but in my case, imagine >>>> what happens if you have a lot of exited child processes queued up. >>>> What happens is that your signal stack gets wound deeper and deeper, and >>>> each time you catch one SIGCHLD you ask for another. Eventually you >>>> run out of stack space. >>>> >>>> My change, which appears to fix this problem, is to split up the "allow" >>>> function as follows: >>>> >>>> PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>>> BEGIN >>>> allow_sigvtalrm (); (* VTALRM only *) >>>> >>>> INC(inCritical); >>>> IF inCritical =3D 1 THEN allow_othersigs (); END; (* SIGCHLD *) >>>> >>>> (* mark signal as being delivered *) >>>> IF sig >=3D 0 AND sig < MaxSigs THEN >>>> gotSigs :=3D gotSigs + SET OF Sig { sig } >>>> END; >>>> DEC(inCritical); >>>> >>>> IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >>>> InternalYield () >>>> END; >>>> END switch_thread; >>>> >>>> This way, SIGCHLD is not taken in nested signal handlers. But is there a >>>> problem here? Well, the fallback is that it takes 0.1 seconds to wake >>>> up on Process.Wait . >>>> >>>> Is there a chance that SIGCHLD signals are left pending? My argument is >>>> no, since in the "base" environment (i.e., not executing in a =20 >>>> signal handler) >>>> SIGCHLD is enabled as a postcondition of allow_othersigs. >>>> >>>> All *other* occurrences of allow_sigvtalrm are of course replaced with >>>> >>>> allow_sigvtalrm(); allow_othersigs(); >>>> >>>> The goal is that SIGCHLD be enabled precisely when SIGVTALRM is =20 >>>> except inside >>>> the signal handler itself. >>>> >>>> Mika >>>> >>>> >>>> Tony Hosking writes: >>>>> Sorry, I missed you e-mail describing this change. Looks good. >>>>> >>>>> Antony Hosking | Associate Professor | Computer Science | Purdue =3D >>>>> University >>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>> >>>>> >>>>> >>>>> >>>>> On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: >>>>> >>>>>> Tony, >>>>>> =3D20 >>>>>> The old user threads has an old problem with Process.Wait. It has a=3D= >> 20=3D >>>>> >>>>>> Thread.Pause(0.1d0) in it. This causes the compiler to run three =3D >>>>> times >>>>>> slower than it should. The m3build I've been using for years has >>>>>> that Thread.Pause removed (so it uses 100% CPU instead, =3D >>>>> busy-waiting...) >>>>>> =3D20 >>>>>> But yeah user threads works the same as it used to (except for=3D20 >>>>>> now calling pthread_atfork, which I'll return to in another email...) >>>>>> =3D20 >>>>>> Mika >>> ... >>> >> >> >> >> --=20 >> Olaf Wagner -- elego Software Solutions GmbH >> Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa= >> ny >> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 >> http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz= >> : Berlin >> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Fri Feb 18 13:55:32 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 18 Feb 2011 12:55:32 +0000 Subject: [M3devel] cm3-ide In-Reply-To: <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> References: , <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com>, , <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> Message-ID: > A language and its implementation in a compiler does not necessarily > require a full-fledged intergrated development environment like Eclipse. Programmers today are not used to using plain text editors. They are used to wizards and intellisense. It would take a lot of work I think to make Modula-3 at all usable to any programmer under 30 years old. Even older folks have largely become spoiled. - Jay > Date: Fri, 18 Feb 2011 09:40:14 +0100 > From: wagner at elegosoft.com > To: zeev.blu at gmail.com > CC: m3devel at elegosoft.com; m3-support at elego.de; moriah.moshe at gmail.com > Subject: Re: [M3devel] cm3-ide > > Quoting ??? ????????? : > > > Hi, > > First, thanks for the quick reply. > > > > I have tried to use the file you gave me, and some other files from that > > site. > > I have some questions: > > 1. If I understand correctly, the method of working with this compiler is > > through the CMD in WINDOWS. Is not this language a IDE to work with? > > A language and its implementation in a compiler does not necessarily > require a full-fledged intergrated development environment like Eclipse. > My faourite IDE when I was actively using M3 was always XEmacs :-) > There _is_ a web-based cm3-ide (which should be contained in that > archive) that allows you to build and manage your projects with a > standard web browser, but that might be quite different from your > expectations of an IDE, too ;-) > > > 2. When I tried to compile , I got a linking error: > > > > " link is not recognized as an internal or external command, operable > > program or batch file." > > > > And then another error: > > > > "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " > > > > And finally I received: > > > > " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" > > failed. " > > You either missed the installation of a suitable system linker as > described in the "Windows installation notes" referred to below or > you didn't set the appropriate variables in your environment or in > the cm3 configuration files. CM3 does not include a system linker, > and on Windows you have got to install one yourself, while on all > other systems it usually is installed by default. > > You can use the cm3 option "-commands" to actually view the external > commands the builder is calling. > > I hope this helps, > > Olaf > > > Thanks for your help > > > > zeev blumentzvaig > > > > > > 2011/2/18 Olaf Wagner > > > >> Quoting ??? ????????? : > >> > >> Hi, > >>> I am a student of Software Engineering. > >>> I want to learn the language modula-3 and I'm trying to install IDE on my > >>> computer. > >>> I have windows 7 operating system and I can not find any suitable > >>> installation file. > >>> Do you have appropriate file or instructions how to do it on another > >>> system? > >>> I need it for exercise in school and I'd love it if you could help me. > >>> Thank you. > >>> zeev blumentzvaig > >>> > >> > >> What exactly have you tried already? > >> > >> The archive for Windows listed on http://www.opencm3.net/download.html, > >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should > >> work for Windows 7, too, as far as I know. You simply need to > >> unpack it and install a suitable linker, as described for example in > >> http://www.opencm3.net/installation-windows.html. > >> > >> In order to help you, we need to know exactly what you have done and > >> what failed. I'm not an expert on Windows; others on the list may > >> be able to help better, so I CC this to m3devel. > >> > >> Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Fri Feb 18 14:12:21 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 18 Feb 2011 14:12:21 +0100 Subject: [M3devel] cm3-ide In-Reply-To: References: , <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com>, , <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> Message-ID: <20110218141221.075c2m16gwcw4s4g@mail.elegosoft.com> Quoting Jay K : > > A language and its implementation in a compiler does not necessarily > > require a full-fledged intergrated development environment like Eclipse. > > Programmers today are not used to using plain text editors. > They are used to wizards and intellisense. > It would take a lot of work I think to make Modula-3 > at all usable to any programmer under 30 years old. I don't think that an integration of M3 into Eclispe for example would be very difficult. As I'm not using it myself, someone else will have to provide it though. > Even older folks have largely become spoiled. I'm still fairly content with using the command line and text editors, even though that must seem old-fashioned ;-) Olaf > - Jay >> Date: Fri, 18 Feb 2011 09:40:14 +0100 >> From: wagner at elegosoft.com >> To: zeev.blu at gmail.com >> CC: m3devel at elegosoft.com; m3-support at elego.de; moriah.moshe at gmail.com >> Subject: Re: [M3devel] cm3-ide >> >> Quoting ??? ????????? : >> >> > Hi, >> > First, thanks for the quick reply. >> > >> > I have tried to use the file you gave me, and some other files from that >> > site. >> > I have some questions: >> > 1. If I understand correctly, the method of working with this compiler is >> > through the CMD in WINDOWS. Is not this language a IDE to work with? >> >> A language and its implementation in a compiler does not necessarily >> require a full-fledged intergrated development environment like Eclipse. >> My faourite IDE when I was actively using M3 was always XEmacs :-) >> There _is_ a web-based cm3-ide (which should be contained in that >> archive) that allows you to build and manage your projects with a >> standard web browser, but that might be quite different from your >> expectations of an IDE, too ;-) >> >> > 2. When I tried to compile , I got a linking error: >> > >> > " link is not recognized as an internal or external command, operable >> > program or batch file." >> > >> > And then another error: >> > >> > "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " >> > >> > And finally I received: >> > >> > " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" >> > failed. " >> >> You either missed the installation of a suitable system linker as >> described in the "Windows installation notes" referred to below or >> you didn't set the appropriate variables in your environment or in >> the cm3 configuration files. CM3 does not include a system linker, >> and on Windows you have got to install one yourself, while on all >> other systems it usually is installed by default. >> >> You can use the cm3 option "-commands" to actually view the external >> commands the builder is calling. >> >> I hope this helps, >> >> Olaf >> >> > Thanks for your help >> > >> > zeev blumentzvaig >> > >> > >> > 2011/2/18 Olaf Wagner >> > >> >> Quoting ??? ????????? : >> >> >> >> Hi, >> >>> I am a student of Software Engineering. >> >>> I want to learn the language modula-3 and I'm trying to install >> IDE on my >> >>> computer. >> >>> I have windows 7 operating system and I can not find any suitable >> >>> installation file. >> >>> Do you have appropriate file or instructions how to do it on another >> >>> system? >> >>> I need it for exercise in school and I'd love it if you could help me. >> >>> Thank you. >> >>> zeev blumentzvaig >> >>> >> >> >> >> What exactly have you tried already? >> >> >> >> The archive for Windows listed on http://www.opencm3.net/download.html, >> >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should >> >> work for Windows 7, too, as far as I know. You simply need to >> >> unpack it and install a suitable linker, as described for example in >> >> http://www.opencm3.net/installation-windows.html. >> >> >> >> In order to help you, we need to know exactly what you have done and >> >> what failed. I'm not an expert on Windows; others on the list may >> >> be able to help better, so I CC this to m3devel. >> >> >> >> Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hosking at cs.purdue.edu Fri Feb 18 15:42:58 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 18 Feb 2011 09:42:58 -0500 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110218101414.yu4y8iyg0kkw40cg@mail.elegosoft.com> References: <20110213220804.7A9001A2078@async.async.caltech.edu> <20110214211829.BA3761A2078@async.async.caltech.edu> <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> <20110217225605.644AC1A2078@async.async.caltech.edu> <20110218101414.yu4y8iyg0kkw40cg@mail.elegosoft.com> Message-ID: <947FF939-07BD-42F5-9E0B-DBCA9A51E68B@cs.purdue.edu> There has been a lot of untested churn in the pthreads implementation since I last looked closely at it. I think we have seen some bitrot that needs attention. I hope to have some time soon. On Feb 18, 2011, at 4:14 AM, Olaf Wagner wrote: > Hm, sounds like the complete switch to pthreads might have been a little > too early, but we haven't had any reliable data to decide on IIRC. All > the world uses pthreads, so I assumed they would be OK for us, too :-/ > However, we need to fix the problems now; I hope Antony can spare some > time. He's been the main architect for that. From dabenavidesd at yahoo.es Fri Feb 18 15:37:24 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Fri, 18 Feb 2011 14:37:24 +0000 (GMT) Subject: [M3devel] cm3-ide In-Reply-To: <20110218141221.075c2m16gwcw4s4g@mail.elegosoft.com> Message-ID: <31891.31208.qm@web29708.mail.ird.yahoo.com> Hi all: yes, it's agood idea, in fact there is one still in preparation M3clipse, the syntactic analyzer, parser and abstract tree builder and has not yet its semantic analyzer, all are written in Java, so it helps to this purposes, perhaps you can can have a hand of it, for now it just colorize its syntax and so see it: http://sourceforge.net/projects/m3clipse/files (it doesn't have any files right now, I believe because of recent crashed project repository server, I can try to put it on the web if any interest its license is EPL) You have this feature in another integrated development environment in another good package, cooledit, I think it features what you are suggesting, even it has an integrated keyword highlighting and code completion but still though it has integrated gdb window support, you might want to add m3gdb support to it (ideally I would say): http://freshmeat.net/projects/cooledit If you need cooledit Modula-3 syntax files you can tell me or be put on the repository for everybody if any interested just tell me, this is written by Pino Zollo. Yet another "cool" option is an "intelligent" or simply advanced integrated programming environment, it has a graphical debugger (in an internal debugging language) and so you can take a look at it in its book: http://www.cs.brown.edu/~spr/research/envfield.html And finally about cm3ide, I think it makes sense to have an distributed environment for reading its hyperlinked source code in almost any web browser by free, i.e a browser plugin which has an speech synthesizer for blind or low vision people. Of course any other custom plugin for any type of checking is a wanted feature, i.e. static analysis or runtime failures could be detected and prompted by it, which is sort of good if have any failure is good to be reported about it, and just not only throw a program debug image file don't you think? Hope it helps, thanks in advance. --- El vie, 18/2/11, Olaf Wagner escribi?: > De: Olaf Wagner > Asunto: Re: [M3devel] cm3-ide > Para: "Jay K" > CC: "m3devel" , m3-support at elego.de, moriah.moshe at gmail.com, zeev.blu at gmail.com > Fecha: viernes, 18 de febrero, 2011 08:12 > Quoting Jay K : > > > > A language and its implementation in a > compiler does not necessarily > > > require a full-fledged intergrated > development environment like Eclipse. > > > > Programmers today are not used to using plain text > editors. > > They are used to wizards and intellisense. > > It would take a lot of work I think to make Modula-3 > > at all usable to any programmer under 30 years old. > > I don't think that an integration of M3 into Eclispe for > example > would be very difficult. As I'm not using it myself, > someone else > will have to provide it though. > > > Even older folks have largely become spoiled. > > I'm still fairly content with using the command line and > text > editors, even though that must seem old-fashioned ;-) > > Olaf > > > - Jay > >> Date: Fri, 18 Feb 2011 09:40:14 +0100 > >> From: wagner at elegosoft.com > >> To: zeev.blu at gmail.com > >> CC: m3devel at elegosoft.com; > m3-support at elego.de; > moriah.moshe at gmail.com > >> Subject: Re: [M3devel] cm3-ide > >> > >> Quoting ??? ????????? : > >> > >> > Hi, > >> > First, thanks for the quick reply. > >> > > >> > I have tried to use the file you gave me, and > some other files from that > >> > site. > >> > I have some questions: > >> > 1. If I understand correctly, the method of > working with this compiler is > >> > through the CMD in WINDOWS. Is not this > language a IDE to work with? > >> > >> A language and its implementation in a compiler > does not necessarily > >> require a full-fledged intergrated development > environment like Eclipse. > >> My faourite IDE when I was actively using M3 was > always XEmacs :-) > >> There _is_ a web-based cm3-ide (which should be > contained in that > >> archive) that allows you to build and manage your > projects with a > >> standard web browser, but that might be quite > different from your > >> expectations of an IDE, too ;-) > >> > >> > 2. When I tried to compile , I got a linking > error: > >> > > >> > " link is not recognized as an internal or > external command, operable > >> > program or batch file." > >> > > >> > And then another error: > >> > > >> > "... \cm3 \ bin \ cm3.cfg ", line 455: quake > runtime error: link failed " > >> > > >> > And finally I received: > >> > > >> > " Fatal Error: procedure "m3_link" defined in > "... \ cm3 \ bin \ cm3.cfg" > >> > failed. " > >> > >> You either missed the installation of a suitable > system linker as > >> described in the "Windows installation notes" > referred to below or > >> you didn't set the appropriate variables in your > environment or in > >> the cm3 configuration files. CM3 does not include > a system linker, > >> and on Windows you have got to install one > yourself, while on all > >> other systems it usually is installed by default. > >> > >> You can use the cm3 option "-commands" to actually > view the external > >> commands the builder is calling. > >> > >> I hope this helps, > >> > >> Olaf > >> > >> > Thanks for your help > >> > > >> > zeev blumentzvaig > >> > > >> > > >> > 2011/2/18 Olaf Wagner > >> > > >> >> Quoting ??? ????????? : > >> >> > >> >> Hi, > >> >>> I am a student of Software > Engineering. > >> >>> I want to learn the language modula-3 > and I'm trying to install > >> IDE on my > >> >>> computer. > >> >>> I have windows 7 operating system and > I can not find any suitable > >> >>> installation file. > >> >>> Do you have appropriate file or > instructions how to do it on another > >> >>> system? > >> >>> I need it for exercise in school and > I'd love it if you could help me. > >> >>> Thank you. > >> >>> zeev blumentzvaig > >> >>> > >> >> > >> >> What exactly have you tried already? > >> >> > >> >> The archive for Windows listed on http://www.opencm3.net/download.html, > >> >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, > should > >> >> work for Windows 7, too, as far as I > know. You simply need to > >> >> unpack it and install a suitable linker, > as described for example in > >> >> http://www.opencm3.net/installation-windows.html. > >> >> > >> >> In order to help you, we need to know > exactly what you have done and > >> >> what failed. I'm not an expert on > Windows; others on the list may > >> >> be able to help better, so I CC this to > m3devel. > >> >> > >> >> Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 > 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | > Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | > USt-IdNr: DE163214194 > > From hosking at cs.purdue.edu Fri Feb 18 15:55:40 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 18 Feb 2011 09:55:40 -0500 Subject: [M3devel] cm3-ide In-Reply-To: References: , <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com>, , <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> Message-ID: Ergo: let's get M3 integrated with Eclipse. Unfortunately, adding new languages properly to Eclipse is a painful process. On Feb 18, 2011, at 7:55 AM, Jay K wrote: > > A language and its implementation in a compiler does not necessarily > > require a full-fledged intergrated development environment like Eclipse. > > > Programmers today are not used to using plain text editors. > They are used to wizards and intellisense. > It would take a lot of work I think to make Modula-3 > at all usable to any programmer under 30 years old. > Even older folks have largely become spoiled. > > - Jay > > Date: Fri, 18 Feb 2011 09:40:14 +0100 > > From: wagner at elegosoft.com > > To: zeev.blu at gmail.com > > CC: m3devel at elegosoft.com; m3-support at elego.de; moriah.moshe at gmail.com > > Subject: Re: [M3devel] cm3-ide > > > > Quoting ??? ????????? : > > > > > Hi, > > > First, thanks for the quick reply. > > > > > > I have tried to use the file you gave me, and some other files from that > > > site. > > > I have some questions: > > > 1. If I understand correctly, the method of working with this compiler is > > > through the CMD in WINDOWS. Is not this language a IDE to work with? > > > > A language and its implementation in a compiler does not necessarily > > require a full-fledged intergrated development environment like Eclipse. > > My faourite IDE when I was actively using M3 was always XEmacs :-) > > There _is_ a web-based cm3-ide (which should be contained in that > > archive) that allows you to build and manage your projects with a > > standard web browser, but that might be quite different from your > > expectations of an IDE, too ;-) > > > > > 2. When I tried to compile , I got a linking error: > > > > > > " link is not recognized as an internal or external command, operable > > > program or batch file." > > > > > > And then another error: > > > > > > "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " > > > > > > And finally I received: > > > > > > " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" > > > failed. " > > > > You either missed the installation of a suitable system linker as > > described in the "Windows installation notes" referred to below or > > you didn't set the appropriate variables in your environment or in > > the cm3 configuration files. CM3 does not include a system linker, > > and on Windows you have got to install one yourself, while on all > > other systems it usually is installed by default. > > > > You can use the cm3 option "-commands" to actually view the external > > commands the builder is calling. > > > > I hope this helps, > > > > Olaf > > > > > Thanks for your help > > > > > > zeev blumentzvaig > > > > > > > > > 2011/2/18 Olaf Wagner > > > > > >> Quoting ??? ????????? : > > >> > > >> Hi, > > >>> I am a student of Software Engineering. > > >>> I want to learn the language modula-3 and I'm trying to install IDE on my > > >>> computer. > > >>> I have windows 7 operating system and I can not find any suitable > > >>> installation file. > > >>> Do you have appropriate file or instructions how to do it on another > > >>> system? > > >>> I need it for exercise in school and I'd love it if you could help me. > > >>> Thank you. > > >>> zeev blumentzvaig > > >>> > > >> > > >> What exactly have you tried already? > > >> > > >> The archive for Windows listed on http://www.opencm3.net/download.html, > > >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should > > >> work for Windows 7, too, as far as I know. You simply need to > > >> unpack it and install a suitable linker, as described for example in > > >> http://www.opencm3.net/installation-windows.html. > > >> > > >> In order to help you, we need to know exactly what you have done and > > >> what failed. I'm not an expert on Windows; others on the list may > > >> be able to help better, so I CC this to m3devel. > > >> > > >> Olaf > > -- > > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 19 01:22:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 18 Feb 2011 16:22:50 -0800 Subject: [M3devel] Text.Hash in CM3 vs PM3 Message-ID: <20110219002250.9E9EE1A2078@async.async.caltech.edu> Hi m3devel, Am I reading the code correctly if I say that Text.Hash (e.g., String8.Hash) is approximately four times slower (on i386 and other 32-bit systems) in CM3 than it was in PM3 (eight times on Alpha and amd64) (i.e., UnsafeHash.Hash)? More, I suppose, on Alpha 21164 and Alpha 21064, which don't even have byte operations... Does anyone out there depend on the specific hash values of existing strings? (I'm not sure the SRC/PM3 algorithm can be adapted to CM3, maybe it can...) Mika From rcolebur at SCIRES.COM Sat Feb 19 10:34:38 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sat, 19 Feb 2011 04:34:38 -0500 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , , , <20110213181717.4F0E91A2078@async.async.caltech.edu>, , , , , Message-ID: Jay: For Windows 2000 SP4, using Visual C++ 2005 Express Edition: 1. I tried building the MachineID.c program in the first example link, but it runs into an unresolved symbol at link time. cl MachineID.c Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 Copyright (C) Microsoft Corporation. All rights reserved. MachineID.c Microsoft (R) Incremental Linker Version 8.00.50727.762 Copyright (C) Microsoft Corporation. All rights reserved. /out:MachineID.exe MachineID.obj LIBCMT.lib(crt0.obj) : error LNK2019: unresolved external symbol _main referenced in function ___tmainCRTStartup MachineID.exe : fatal error LNK1120: 1 unresolved externals 2. I was able to build the program in the 2nd link. Here is the output from running this program: Num Entries: 2 Index[0]: 16777219 InterfaceName[0]: osoft Visual Studio 8\Common7\Tools\ Description[0]: EL574ND4 Ethernet Adapter Type[0]: Ethernet Mtu[0]: 1500 Speed[0]: 100000000 Physical Addr: 00-00-86-5E-DF-A0 Admin Status[0]: 1 Oper Status[0]: Operational Index[1]: 1 InterfaceName[1]: Description[1]: MS TCP Loopback interface Type[1]: Software Lookback Mtu[1]: 1500 Speed[1]: 10000000 Physical Addr: Admin Status[1]: 1 Oper Status[1]: Operational Regards, Randy ________________________________ From: jayk123 at hotmail.com [jayk123 at hotmail.com] On Behalf Of Jay K [jay.krell at cornell.edu] Sent: Tuesday, February 15, 2011 7:56 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] More on threading It should work. http://dcvs.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/uid/WIN32/MachineID.c?rev=1.14;content-type=text%2Fplain Bing for "MSDN GetIfTable" => http://msdn.microsoft.com/en-us/library/aa365943(VS.85).aspx Mika, what made you think cm3 doesn't work on Windows 2000?? I don't think anything has been remotely suggested. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Tue, 15 Feb 2011 17:05:37 -0500 Subject: Re: [M3devel] More on threading Jay, since I have a Windows2000 system, I can test the MAC address discovery (or anything else you need). Just let me know. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Monday, February 14, 2011 8:12 PM To: Coleburn, Randy; m3devel Subject: Re: [M3devel] More on threading Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 19 11:03:54 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 19 Feb 2011 10:03:54 +0000 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, ,,, ,,<20110213181717.4F0E91A2078@async.async.caltech.edu>, , , , , , , , , , Message-ID: 1. That is the expected result. Please please look closer. 2. Yeah, the sample I derived from is a bit buggy but I think I fixed its bugs, I vaguely recall. I'm sure there's no problem here, anyway. At least on Windows 2000 and newer. Maybe on NT 3.x, NT 4.0, Win9x, maybe depending on the install of IE. But I doubt anyone cares. Heck, for a very long time, this code didn't really work for MacOSX, FreeBSD, NetBSD, OpenBSD, Solaris, only Windows, Linux, OSF/1, and nobody seemed to notice.. (it falls back to the local IPv4 address, which isn't very unique typically). I think it works everywhere now, through heavy #ifdef. (Given the level of #ifdef, I wonder if we should either combine Windows and Posix into one file, or split up Posix into multiple files, or just leave it alone..) Thanks, - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Sat, 19 Feb 2011 04:34:38 -0500 Subject: Re: [M3devel] More on threading Jay: For Windows 2000 SP4, using Visual C++ 2005 Express Edition: 1. I tried building the MachineID.c program in the first example link, but it runs into an unresolved symbol at link time. cl MachineID.c Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 Copyright (C) Microsoft Corporation. All rights reserved. MachineID.c Microsoft (R) Incremental Linker Version 8.00.50727.762 Copyright (C) Microsoft Corporation. All rights reserved. /out:MachineID.exe MachineID.obj LIBCMT.lib(crt0.obj) : error LNK2019: unresolved external symbol _main referenced in function ___tmainCRTStartup MachineID.exe : fatal error LNK1120: 1 unresolved externals 2. I was able to build the program in the 2nd link. Here is the output from running this program: Num Entries: 2 Index[0]: 16777219 InterfaceName[0]: osoft Visual Studio 8\Common7\Tools\ Description[0]: EL574ND4 Ethernet Adapter Type[0]: Ethernet Mtu[0]: 1500 Speed[0]: 100000000 Physical Addr: 00-00-86-5E-DF-A0 Admin Status[0]: 1 Oper Status[0]: Operational Index[1]: 1 InterfaceName[1]: Description[1]: MS TCP Loopback interface Type[1]: Software Lookback Mtu[1]: 1500 Speed[1]: 10000000 Physical Addr: Admin Status[1]: 1 Oper Status[1]: Operational Regards, Randy From: jayk123 at hotmail.com [jayk123 at hotmail.com] On Behalf Of Jay K [jay.krell at cornell.edu] Sent: Tuesday, February 15, 2011 7:56 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] More on threading It should work. http://dcvs.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/uid/WIN32/MachineID.c?rev=1.14;content-type=text%2Fplain Bing for "MSDN GetIfTable" => http://msdn.microsoft.com/en-us/library/aa365943(VS.85).aspx Mika, what made you think cm3 doesn't work on Windows 2000?? I don't think anything has been remotely suggested. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Tue, 15 Feb 2011 17:05:37 -0500 Subject: Re: [M3devel] More on threading Jay, since I have a Windows2000 system, I can test the MAC address discovery (or anything else you need). Just let me know. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Monday, February 14, 2011 8:12 PM To: Coleburn, Randy; m3devel Subject: Re: [M3devel] More on threading Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 19 19:29:30 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 10:29:30 -0800 Subject: [M3devel] SEGV mapping to RuntimeError Message-ID: <20110219182930.BDCAB1A2078@async.async.caltech.edu> Dear m3devel, For a while it has annoyed me that segmentation violations cause an unconditional program abort. I've changed that now so that (under user threads at least) we instead get a RuntimeError. Here's an example of the mechanism at work in an interactive Scheme environment. Consider the unhelpful interface and module Crash: INTERFACE Crash; PROCEDURE Me(); END Crash. MODULE Crash; PROCEDURE Me() = VAR ptr : REF INTEGER := NIL; BEGIN ptr^ := 0 END Me; BEGIN END Crash. Here's an example of what happens if you now call this from an interactive interpreter that catches the exception RuntimeError.E: M-Scheme Experimental LITHP ITH LITHENING. > (require-modules "m3") #t > (Crash.Me) EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > (+ 3 4) 7 > I just realized I may have broken pthreads, let me go back and double-check it. runtime/POSIX and thread/POSIX don't refer to the same thing do they... Mika From jay.krell at cornell.edu Sat Feb 19 20:53:49 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 19 Feb 2011 19:53:49 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110219182930.BDCAB1A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> Message-ID: Letting any code run after a SIGSEGV is dubious. Imagine the heap is corrupted. And then you run more code. And the code happens to call malloc. Or printf to log something. I suppose there might be an application that maps memory gradually, as pieces of a buffer are hit. Might. - Jay > To: m3devel at elegosoft.com > Date: Sat, 19 Feb 2011 10:29:30 -0800 > From: mika at async.caltech.edu > Subject: [M3devel] SEGV mapping to RuntimeError > > > Dear m3devel, > > For a while it has annoyed me that segmentation violations cause an > unconditional program abort. I've changed that now so that (under user > threads at least) we instead get a RuntimeError. Here's an example of > the mechanism at work in an interactive Scheme environment. Consider > the unhelpful interface and module Crash: > > INTERFACE Crash; PROCEDURE Me(); END Crash. > > MODULE Crash; > > PROCEDURE Me() = > VAR ptr : REF INTEGER := NIL; BEGIN > ptr^ := 0 > END Me; > > BEGIN END Crash. > > Here's an example of what happens if you now call this from an interactive > interpreter that catches the exception RuntimeError.E: > > M-Scheme Experimental > LITHP ITH LITHENING. > > (require-modules "m3") > #t > > (Crash.Me) > EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > > (+ 3 4) > 7 > > > > I just realized I may have broken pthreads, let me go back and double-check it. > runtime/POSIX and thread/POSIX don't refer to the same thing do they... > > Mika > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 19 21:38:21 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 12:38:21 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> Message-ID: <20110219203821.2016B1A2078@async.async.caltech.edu> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, you know! "Corrupting the heap" is something that can only happen as a result of an unchecked runtime error. Unchecked runtime errors cannot happen in modules not marked UNSAFE. SEGV is, however, used by the CM3 implementation (and its predecessors) to signal a certain kind of *checked* runtime error, namely, the dereferencing of a NIL reference. Correct me if I am wrong, but an attempt to dereference NIL is not going to leave the heap corrupted? And if you stick to safe code, the only SEGVs I think you get in the current CM3 are ones from NIL dereferences. Hence, as long as you stick with safe code, the only time the code I checked in earlier gets triggered is for NIL dereferences, which should never corrupt the heap. So SEGV is not sometimes, but in fact always recoverable. :-) Mika P.S. the bit above "if you stick to safe code": if you actually program in Modula-3 you almost never use UNSAFE. I went through my repository and I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, many of the UNSAFE modules are glue code to Fortran routines, which could relatively easily be verified to be safe in the Modula-3 sense. Almost all what remains is glue to some C library, which wouldn't be necessary if the rest of the world would wake up out of the dark ages, but I don't have the time to rewrite every single library from scratch myself. Jay K writes: >--_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Letting any code run after a SIGSEGV is dubious. >Imagine the heap is corrupted. >And then you run more code. >And the code happens to call malloc. >Or printf to log something. >=20 >I suppose there might be an application that maps memory >gradually=2C as pieces of a buffer are hit. Might. >=20 > - Jay >=20 >> To: m3devel at elegosoft.com >> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >> From: mika at async.caltech.edu >> Subject: [M3devel] SEGV mapping to RuntimeError >>=20 >>=20 >> Dear m3devel=2C >>=20 >> For a while it has annoyed me that segmentation violations cause an >> unconditional program abort. I've changed that now so that (under user >> threads at least) we instead get a RuntimeError. Here's an example of >> the mechanism at work in an interactive Scheme environment. Consider >> the unhelpful interface and module Crash: >>=20 >> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>=20 >> MODULE Crash=3B >>=20 >> PROCEDURE Me() =3D >> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >> ptr^ :=3D 0 >> END Me=3B >>=20 >> BEGIN END Crash. >>=20 >> Here's an example of what happens if you now call this from an interactiv= >e >> interpreter that catches the exception RuntimeError.E: >>=20 >> M-Scheme Experimental >> LITHP ITH LITHENING. >> > (require-modules "m3") >> #t >> > (Crash.Me) >> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >> > (+ 3 4)=20 >> 7 >> >=20 >>=20 >> I just realized I may have broken pthreads=2C let me go back and double-c= >heck it.=20 >> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>=20 >> Mika >>=20 > = > >--_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >Letting any code run after a SIGSEGV is dubious.
>Imagine the heap =3Bis corrupted.
>And then you run more code.
>And the code happens to call malloc.
>Or printf to log something.
> =3B
>I suppose there might be an application that maps memory
>gradually=2C as pieces of a buffer are hit. Might.
> =3B
> =3B- Jay
 =3B
>>=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >=2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >olations cause an
>=3B unconditional program abort. I've changed that = >now so that (under user
>=3B threads at least) we instead get a Runtim= >eError. Here's an example of
>=3B the mechanism at work in an interact= >ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>=3B Here's an example of what happens if you now call this from an inter= >active
>=3B interpreter that catches the exception RuntimeError.E:
= >>=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >gt=3B >=3B (require-modules "m3")
>=3B #t
>=3B >=3B (Crash.Me= >)
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > location.
>=3B >=3B (+ 3 4)
>=3B 7
>=3B >=3B
>= >=3B
>=3B I just realized I may have broken pthreads=2C let me go back= > and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >= > >--_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- From rodney_bates at lcwb.coop Sun Feb 20 00:17:55 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sat, 19 Feb 2011 17:17:55 -0600 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110219203821.2016B1A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> Message-ID: <4D604FA3.3060708@lcwb.coop> I know of one other place the compilers rely on hardware memory protection to detect a checked runtime error, and that is stack overflow. This won't corrupt anything, but is hard to distinguish from dereferencing NIL. This could probably be distinguished after the fact by some low-level, target-dependent code. I have found it by looking at assembly code at the point of failure--usually right after a stack pointer push. Detecting this via compiler-generated checks would probably be more extravagant than many other checks, as it is so frequent. I am not aware of any really good solution to this in any implementation of any language. On 02/19/2011 02:38 PM, Mika Nystrom wrote: > Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > you know! > > "Corrupting the heap" is something that can only happen as a result of > an unchecked runtime error. Unchecked runtime errors cannot happen in > modules not marked UNSAFE. > > SEGV is, however, used by the CM3 implementation (and its predecessors) > to signal a certain kind of *checked* runtime error, namely, the > dereferencing of a NIL reference. Correct me if I am wrong, but an > attempt to dereference NIL is not going to leave the heap corrupted? > > And if you stick to safe code, the only SEGVs I think you get in the > current CM3 are ones from NIL dereferences. > > Hence, as long as you stick with safe code, the only time the code I > checked in earlier gets triggered is for NIL dereferences, which should > never corrupt the heap. So SEGV is not sometimes, but in fact always > recoverable. > > :-) > > Mika > > P.S. the bit above "if you stick to safe code": if you actually program in > Modula-3 you almost never use UNSAFE. I went through my repository and > I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > many of the UNSAFE modules are glue code to Fortran routines, which > could relatively easily be verified to be safe in the Modula-3 sense. > Almost all what remains is glue to some C library, which wouldn't be > necessary if the rest of the world would wake up out of the dark ages, but > I don't have the time to rewrite every single library from scratch myself. > > > Jay K writes: >> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >> Content-Type: text/plain; charset="iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> >> Letting any code run after a SIGSEGV is dubious. >> Imagine the heap is corrupted. >> And then you run more code. >> And the code happens to call malloc. >> Or printf to log something. >> =20 >> I suppose there might be an application that maps memory >> gradually=2C as pieces of a buffer are hit. Might. >> =20 >> - Jay >> =20 >>> To: m3devel at elegosoft.com >>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>> From: mika at async.caltech.edu >>> Subject: [M3devel] SEGV mapping to RuntimeError >>> =20 >>> =20 >>> Dear m3devel=2C >>> =20 >>> For a while it has annoyed me that segmentation violations cause an >>> unconditional program abort. I've changed that now so that (under user >>> threads at least) we instead get a RuntimeError. Here's an example of >>> the mechanism at work in an interactive Scheme environment. Consider >>> the unhelpful interface and module Crash: >>> =20 >>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>> =20 >>> MODULE Crash=3B >>> =20 >>> PROCEDURE Me() =3D >>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>> ptr^ :=3D 0 >>> END Me=3B >>> =20 >>> BEGIN END Crash. >>> =20 >>> Here's an example of what happens if you now call this from an interactiv= >> e >>> interpreter that catches the exception RuntimeError.E: >>> =20 >>> M-Scheme Experimental >>> LITHP ITH LITHENING. >>>> (require-modules "m3") >>> #t >>>> (Crash.Me) >>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>> (+ 3 4)=20 >>> 7 >>>> =20 >>> =20 >>> I just realized I may have broken pthreads=2C let me go back and double-c= >> heck it.=20 >>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>> =20 >>> Mika >>> =20 >> = >> >> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >> Content-Type: text/html; charset="iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> >> >> >> >> >> Letting any code run after a SIGSEGV is dubious.
>> Imagine the heap =3Bis corrupted.
>> And then you run more code.
>> And the code happens to call malloc.
>> Or printf to log something.
>>  =3B
>> I suppose there might be an application that maps memory
>> gradually=2C as pieces of a buffer are hit. Might.
>>  =3B
>>  =3B- Jay
 =3B
>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >> olations cause an
>=3B unconditional program abort. I've changed that = >> now so that (under user
>=3B threads at least) we instead get a Runtim= >> eError. Here's an example of
>=3B the mechanism at work in an interact= >> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >> >=3B Here's an example of what happens if you now call this from an inter= >> active
>=3B interpreter that catches the exception RuntimeError.E:
= >> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >> >> = >> >> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > From mika at async.caltech.edu Sun Feb 20 00:27:29 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 15:27:29 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <4D604FA3.3060708@lcwb.coop> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> Message-ID: <20110219232729.53A991A2078@async.async.caltech.edu> Ah, yes, stack protection. Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. Hmm, I get SIGILL on AMD64_FREEBSD as well: time ../AMD64_FREEBSD/stubexample M-Scheme Experimental LITHP ITH LITHENING. > (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) f > (f 0) Illegal instruction 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w What absolutely must not happen, of course, is that the runtime hangs while executing only safe code... Mika "Rodney M. Bates" writes: >I know of one other place the compilers rely on hardware memory protection >to detect a checked runtime error, and that is stack overflow. This won't >corrupt anything, but is hard to distinguish from dereferencing NIL. >This could probably be distinguished after the fact by some low-level, >target-dependent code. I have found it by looking at assembly code at >the point of failure--usually right after a stack pointer push. > >Detecting this via compiler-generated checks would probably be more >extravagant than many other checks, as it is so frequent. I am not >aware of any really good solution to this in any implementation of any >language. > >On 02/19/2011 02:38 PM, Mika Nystrom wrote: >> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >> you know! >> >> "Corrupting the heap" is something that can only happen as a result of >> an unchecked runtime error. Unchecked runtime errors cannot happen in >> modules not marked UNSAFE. >> >> SEGV is, however, used by the CM3 implementation (and its predecessors) >> to signal a certain kind of *checked* runtime error, namely, the >> dereferencing of a NIL reference. Correct me if I am wrong, but an >> attempt to dereference NIL is not going to leave the heap corrupted? >> >> And if you stick to safe code, the only SEGVs I think you get in the >> current CM3 are ones from NIL dereferences. >> >> Hence, as long as you stick with safe code, the only time the code I >> checked in earlier gets triggered is for NIL dereferences, which should >> never corrupt the heap. So SEGV is not sometimes, but in fact always >> recoverable. >> >> :-) >> >> Mika >> >> P.S. the bit above "if you stick to safe code": if you actually program in >> Modula-3 you almost never use UNSAFE. I went through my repository and >> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >> many of the UNSAFE modules are glue code to Fortran routines, which >> could relatively easily be verified to be safe in the Modula-3 sense. >> Almost all what remains is glue to some C library, which wouldn't be >> necessary if the rest of the world would wake up out of the dark ages, but >> I don't have the time to rewrite every single library from scratch myself. >> >> >> Jay K writes: >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>> Content-Type: text/plain; charset="iso-8859-1" >>> Content-Transfer-Encoding: quoted-printable >>> >>> >>> Letting any code run after a SIGSEGV is dubious. >>> Imagine the heap is corrupted. >>> And then you run more code. >>> And the code happens to call malloc. >>> Or printf to log something. >>> =20 >>> I suppose there might be an application that maps memory >>> gradually=2C as pieces of a buffer are hit. Might. >>> =20 >>> - Jay >>> =20 >>>> To: m3devel at elegosoft.com >>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>> From: mika at async.caltech.edu >>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>> =20 >>>> =20 >>>> Dear m3devel=2C >>>> =20 >>>> For a while it has annoyed me that segmentation violations cause an >>>> unconditional program abort. I've changed that now so that (under user >>>> threads at least) we instead get a RuntimeError. Here's an example of >>>> the mechanism at work in an interactive Scheme environment. Consider >>>> the unhelpful interface and module Crash: >>>> =20 >>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>> =20 >>>> MODULE Crash=3B >>>> =20 >>>> PROCEDURE Me() =3D >>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>> ptr^ :=3D 0 >>>> END Me=3B >>>> =20 >>>> BEGIN END Crash. >>>> =20 >>>> Here's an example of what happens if you now call this from an interactiv= >>> e >>>> interpreter that catches the exception RuntimeError.E: >>>> =20 >>>> M-Scheme Experimental >>>> LITHP ITH LITHENING. >>>>> (require-modules "m3") >>>> #t >>>>> (Crash.Me) >>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>> (+ 3 4)=20 >>>> 7 >>>>> =20 >>>> =20 >>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>> heck it.=20 >>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>> =20 >>>> Mika >>>> =20 >>> = >>> >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>> Content-Type: text/html; charset="iso-8859-1" >>> Content-Transfer-Encoding: quoted-printable >>> >>> >>> >>> >>> >>> >>> Letting any code run after a SIGSEGV is dubious.
>>> Imagine the heap =3Bis corrupted.
>>> And then you run more code.
>>> And the code happens to call malloc.
>>> Or printf to log something.
>>>  =3B
>>> I suppose there might be an application that maps memory
>>> gradually=2C as pieces of a buffer are hit. Might.
>>>  =3B
>>>  =3B- Jay
 =3B
>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>> olations cause an
>=3B unconditional program abort. I've changed that = >>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>> >=3B Here's an example of what happens if you now call this from an inter= >>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>> >>> = >>> >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >> From jay.krell at cornell.edu Sun Feb 20 00:36:25 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 19 Feb 2011 23:36:25 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110219232729.53A991A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu> Message-ID: Modula-3 doesn't live in a vacuum. The vast majority of surrounding code is C and C++. There are bugs everywhere. - Jay > To: rodney_bates at lcwb.coop > Date: Sat, 19 Feb 2011 15:27:29 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > Ah, yes, stack protection. > > Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > > Hmm, I get SIGILL on AMD64_FREEBSD as well: > > time ../AMD64_FREEBSD/stubexample > M-Scheme Experimental > LITHP ITH LITHENING. > > (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > f > > (f 0) > Illegal instruction > 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > > What absolutely must not happen, of course, is that the runtime hangs > while executing only safe code... > > Mika > > "Rodney M. Bates" writes: > >I know of one other place the compilers rely on hardware memory protection > >to detect a checked runtime error, and that is stack overflow. This won't > >corrupt anything, but is hard to distinguish from dereferencing NIL. > >This could probably be distinguished after the fact by some low-level, > >target-dependent code. I have found it by looking at assembly code at > >the point of failure--usually right after a stack pointer push. > > > >Detecting this via compiler-generated checks would probably be more > >extravagant than many other checks, as it is so frequent. I am not > >aware of any really good solution to this in any implementation of any > >language. > > > >On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >> you know! > >> > >> "Corrupting the heap" is something that can only happen as a result of > >> an unchecked runtime error. Unchecked runtime errors cannot happen in > >> modules not marked UNSAFE. > >> > >> SEGV is, however, used by the CM3 implementation (and its predecessors) > >> to signal a certain kind of *checked* runtime error, namely, the > >> dereferencing of a NIL reference. Correct me if I am wrong, but an > >> attempt to dereference NIL is not going to leave the heap corrupted? > >> > >> And if you stick to safe code, the only SEGVs I think you get in the > >> current CM3 are ones from NIL dereferences. > >> > >> Hence, as long as you stick with safe code, the only time the code I > >> checked in earlier gets triggered is for NIL dereferences, which should > >> never corrupt the heap. So SEGV is not sometimes, but in fact always > >> recoverable. > >> > >> :-) > >> > >> Mika > >> > >> P.S. the bit above "if you stick to safe code": if you actually program in > >> Modula-3 you almost never use UNSAFE. I went through my repository and > >> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >> many of the UNSAFE modules are glue code to Fortran routines, which > >> could relatively easily be verified to be safe in the Modula-3 sense. > >> Almost all what remains is glue to some C library, which wouldn't be > >> necessary if the rest of the world would wake up out of the dark ages, but > >> I don't have the time to rewrite every single library from scratch myself. > >> > >> > >> Jay K writes: > >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>> Content-Type: text/plain; charset="iso-8859-1" > >>> Content-Transfer-Encoding: quoted-printable > >>> > >>> > >>> Letting any code run after a SIGSEGV is dubious. > >>> Imagine the heap is corrupted. > >>> And then you run more code. > >>> And the code happens to call malloc. > >>> Or printf to log something. > >>> =20 > >>> I suppose there might be an application that maps memory > >>> gradually=2C as pieces of a buffer are hit. Might. > >>> =20 > >>> - Jay > >>> =20 > >>>> To: m3devel at elegosoft.com > >>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>> From: mika at async.caltech.edu > >>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>> =20 > >>>> =20 > >>>> Dear m3devel=2C > >>>> =20 > >>>> For a while it has annoyed me that segmentation violations cause an > >>>> unconditional program abort. I've changed that now so that (under user > >>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>> the mechanism at work in an interactive Scheme environment. Consider > >>>> the unhelpful interface and module Crash: > >>>> =20 > >>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>> =20 > >>>> MODULE Crash=3B > >>>> =20 > >>>> PROCEDURE Me() =3D > >>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>> ptr^ :=3D 0 > >>>> END Me=3B > >>>> =20 > >>>> BEGIN END Crash. > >>>> =20 > >>>> Here's an example of what happens if you now call this from an interactiv= > >>> e > >>>> interpreter that catches the exception RuntimeError.E: > >>>> =20 > >>>> M-Scheme Experimental > >>>> LITHP ITH LITHENING. > >>>>> (require-modules "m3") > >>>> #t > >>>>> (Crash.Me) > >>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>> (+ 3 4)=20 > >>>> 7 > >>>>> =20 > >>>> =20 > >>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>> heck it.=20 > >>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>> =20 > >>>> Mika > >>>> =20 > >>> = > >>> > >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>> Content-Type: text/html; charset="iso-8859-1" > >>> Content-Transfer-Encoding: quoted-printable > >>> > >>> > >>> > >>> > >>> > >>> > >>> Letting any code run after a SIGSEGV is dubious.
> >>> Imagine the heap =3Bis corrupted.
> >>> And then you run more code.
> >>> And the code happens to call malloc.
> >>> Or printf to log something.
> >>>  =3B
> >>> I suppose there might be an application that maps memory
> >>> gradually=2C as pieces of a buffer are hit. Might.
> >>>  =3B
> >>>  =3B- Jay
 =3B
> >>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>> olations cause an
>=3B unconditional program abort. I've changed that = > >>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>> >=3B Here's an example of what happens if you now call this from an inter= > >>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>> > >>> = > >>> > >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 20 01:13:14 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 16:13:14 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu> Message-ID: <20110220001314.7F2951A2078@async.async.caltech.edu> Jay K writes: >--_1802b75d-f353-43c9-bd6c-02a34e2b7ae8_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Modula-3 doesn't live in a vacuum. >The vast majority of surrounding code is C and C++. >There are bugs everywhere. > - Jay It's rather up to the programmer using it, isn't it? If you write your code in pure Modula-3, you should be able to rely on the guarantees of the Green Book (and therefore that SIGSEGV means something relatively specific, not that "the world just ended"). If you insist on linking with broken C and C++... well you get what you deserve, probably best not to try to handle RuntimeError.E in that case. In any case, to an interactive user of my Scheme interpreter it is very useful that a NIL dereference drops him at the read-eval-print loop rather than killing the world. E.g., if as I had yesterday, a 1-gigabyte datastructure parsed in memory, and I lost the world because I happened to try to set-car! on a null list somewhere. Mika From jay.krell at cornell.edu Sun Feb 20 01:26:30 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 20 Feb 2011 00:26:30 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220001314.7F2951A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , <4D604FA3.3060708@lcwb.coop>, , <20110219232729.53A991A2078@async.async.caltech.edu>, , <20110220001314.7F2951A2078@async.async.caltech.edu> Message-ID: NIL might be a special case. But by and large, continuing to do anything after a SIGSEGV is a bad idea. You think (you say) you are continuing on and preserving state, but the state of the world is now arbitrarily altered and unknown and all future computation based on it is suspect. Best to throw it away. And avoid incorrect results when there is a such a strong hint that things have gone awry. In the Windows C programming world, you can catch access violatios (SIGSEGV) very easily with __except. But prevailing wisdom now and for a long time is exactly what I'm saying -- that catching and continuing isn't "robust", it is letting the damage of bugs continue further. I'm undecided as to the level of guarantees of the safety guarantees, given the surrounding unsafe world. I believe safety depends on a lack of bugs in certain places, certain fairly large places, and therefore the safety guarantees should be considered significant and useful, but not absolute. There really are bugs everywhere. In the compiler (Modula-3, C, etc.), in the C runtime, in the kernels, in the CPUs. Recent AMD processors have a bug in integer division even..one of the most surprising things I've seen in a long time.. (really, recent processors, integer division, I'm not talking about the ancient Pentium FDIV bug). There is a published errata for it and it has been fixed. But I managed to hit it in my code.. - Jay > To: jay.krell at cornell.edu > Date: Sat, 19 Feb 2011 16:13:14 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > Jay K writes: > >--_1802b75d-f353-43c9-bd6c-02a34e2b7ae8_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >Modula-3 doesn't live in a vacuum. > >The vast majority of surrounding code is C and C++. > >There are bugs everywhere. > > - Jay > > It's rather up to the programmer using it, isn't it? If you write your > code in pure Modula-3, you should be able to rely on the guarantees of > the Green Book (and therefore that SIGSEGV means something relatively > specific, not that "the world just ended"). If you insist on linking > with broken C and C++... well you get what you deserve, probably best > not to try to handle RuntimeError.E in that case. > > In any case, to an interactive user of my Scheme interpreter it is very > useful that a NIL dereference drops him at the read-eval-print loop > rather than killing the world. E.g., if as I had yesterday, a 1-gigabyte > datastructure parsed in memory, and I lost the world because I happened > to try to set-car! on a null list somewhere. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 20 01:59:04 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 16:59:04 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , <4D604FA3.3060708@lcwb.coop>, , <20110219232729.53A991A2078@async.async.caltech.edu>, , <20110220001314.7F2951A2078@async.async.caltech.edu> Message-ID: <20110220005904.A15F21A2078@async.async.caltech.edu> Jay K writes: >--_e4a53a7c-4058-44e7-bcce-18b6ae35efbe_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >NIL might be a special case. >But by and large=2C continuing to do anything after a SIGSEGV is a bad idea= >. >You think (you say) you are continuing on and preserving state=2C but the s= >tate >of the world is now arbitrarily altered and unknown and all future computat= >ion >based on it is suspect. Best to throw it away. And avoid incorrect results >when there is a such a strong hint that things have gone awry. Let's try this again. The Green Book unequivocally says that non-UNSAFE code can't get the world into a "state arbitrarily altered and unknown." That doesn't depend on the special case NIL, that's inherent in the design of the language and runtime. Yet non-UNSAFE code can (for efficiency reasons of the implementation) cause a SIGSEGV. If the state of the world is somehow arbitrarily altered and unknown I would suggest tracking down the bugs (in UNSAFE code, compiler, C code, wherever) and fixing them rather than repeating the C programmer's mantra that SIGSEGV is or ought to be fatal to the point that you need to start from a clean slate. This sort of thing is precisely what Modula-3 was invented for!!!! Why do you think (crazy) people still use the programming language, fifteen years after it's been declared dead? By the way, every year that I teach a class, I always ask the students what programming languages they use/learn/want to learn. Over the years I've seen a steady progression of C++, Java, Python... until last year. The most popular answer was "C". I must say it felt a bit odd to be, technologically, 20 years ahead of people 20 years younger. Now, I always used to feel a bit bad about forcing students to study a "dead" programming language that would not help them in employment, research, etc. (Modular-3? What's that?) But better than a language that should be dead. Mika From jay.krell at cornell.edu Sun Feb 20 02:30:01 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 20 Feb 2011 01:30:01 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220005904.A15F21A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, ,,, ,,<20110219203821.2016B1A2078@async.async.caltech.edu>, ,,<4D604FA3.3060708@lcwb.coop>, , , <20110219232729.53A991A2078@async.async.caltech.edu>, , , , <20110220001314.7F2951A2078@async.async.caltech.edu>, , <20110220005904.A15F21A2078@async.async.caltech.edu> Message-ID: I just don't think because the Green Book says something, or because it is inherent in the design, that is necessarily true or inherent in the implementation. There are bugs *everywhere*, and they gradually chip away at all aspects of correctness. Much of this, there is nothing you can do about. The integer divide bug is an example that there is precious little you can do about. You can check that the errata is fixed, and not depend on the correctness until you have first check for it, but inevitably tons and tons of code will run before your correctness check (e.g. booting the OS) and so the state is already not guaranteed. I've also had my code fail to page in. That is another thing you can't do anything about, in usermode...except maybe keep your code to under a page in size. :) Simple progressing from one line of code to the next, or pieces within the line, can fail. But SIGSEGV I think is in another realm. It is a sign of problems that should generally be ignored. There is a term "fail fast" -- at a sign of problem, fail. Don't press on. There are arguments either way. The world, really, the larger human enterprise and setting, is not fail fast. We have lots of problems, yet almost everything almost always goes on. - Jay > To: jay.krell at cornell.edu > Date: Sat, 19 Feb 2011 16:59:04 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > Jay K writes: > >--_e4a53a7c-4058-44e7-bcce-18b6ae35efbe_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >NIL might be a special case. > >But by and large=2C continuing to do anything after a SIGSEGV is a bad idea= > >. > >You think (you say) you are continuing on and preserving state=2C but the s= > >tate > >of the world is now arbitrarily altered and unknown and all future computat= > >ion > >based on it is suspect. Best to throw it away. And avoid incorrect results > >when there is a such a strong hint that things have gone awry. > > Let's try this again. > > The Green Book unequivocally says that non-UNSAFE code can't get the > world into a "state arbitrarily altered and unknown." That doesn't > depend on the special case NIL, that's inherent in the design of the > language and runtime. > > Yet non-UNSAFE code can (for efficiency reasons of the implementation) > cause a SIGSEGV. > > If the state of the world is somehow arbitrarily altered and unknown I > would suggest tracking down the bugs (in UNSAFE code, compiler, C code, > wherever) and fixing them rather than repeating the C programmer's mantra > that SIGSEGV is or ought to be fatal to the point that you need to start > from a clean slate. > > This sort of thing is precisely what Modula-3 was invented for!!!! > Why do you think (crazy) people still use the programming language, > fifteen years after it's been declared dead? > > By the way, every year that I teach a class, I always ask the students > what programming languages they use/learn/want to learn. Over the years > I've seen a steady progression of C++, Java, Python... until last year. > The most popular answer was "C". I must say it felt a bit odd to be, > technologically, 20 years ahead of people 20 years younger. Now, > I always used to feel a bit bad about forcing students to study a > "dead" programming language that would not help them in employment, > research, etc. (Modular-3? What's that?) But better than a language > that should be dead. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 20 02:51:07 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 17:51:07 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , , <4D604FA3.3060708@lcwb.coop>, , , <20110219232729.53A991A2078@async.async.caltech.edu>, , , , <20110220001314.7F2951A2078@async.async.caltech.edu>, , <20110220005904.A15F21A2078@async.async.caltech.edu> Message-ID: <20110220015107.110461A2078@async.async.caltech.edu> >But SIGSEGV I think is in another realm. It is a sign of problems that shou= >ld generally >be ignored. >There is a term "fail fast" -- at a sign of problem=2C fail. Don't press on= >. >There are arguments either way. >The world=2C really=2C the larger human enterprise and setting=2C is not fa= >il fast. >We have lots of problems=2C yet almost everything almost always goes on. Well, see, when you're using a Lisp interpreter, SIGSEGV in called code is not in another realm (this is my problem). It is also not in another realm if you are implementing an operating system without hardware memory protection (e.g., SPIN) or if you're running untrusted outside code in a "sandbox" (which you CAN build with Modula-3, again without memory protection). How would you like it if you had to reboot your computer every time you dereferenced a null pointer? Sounds a bit like the 1950s doesn't it? Why should I have to restart my Lisp interpreter if I slip on the keyboard? These are things you would never dream of doing in C but that Modula-3 is eminently suited for doing. For precisely these sorts of purposes, there's a big distinction designed into the language between failures in "safe" code (which mean just that the code failed) and failures in "unsafe" code (which mean that the world ended). If you're not doing any of the special things listed above you're free not to catch RuntimeError.E, in which case your program will crash on an uncaught exception if it receives a SIGSEGV. >I just don't think because the Green Book says something=2C or because it i= >s inherent in the >design=2C that is necessarily true or inherent in the implementation. >There are bugs *everywhere*=2C and they gradually chip away at all aspects = >of correctness. Dijkstra teaches that one should code to the specification. Coding to the implementation is a sure path to losing one's mind. Mika From jay.krell at cornell.edu Sun Feb 20 07:36:12 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 20 Feb 2011 06:36:12 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220015107.110461A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , ,,, , ,,<20110219203821.2016B1A2078@async.async.caltech.edu>, , ,,<4D604FA3.3060708@lcwb.coop>, , ,,<20110219232729.53A991A2078@async.async.caltech.edu>, ,,, , , <20110220001314.7F2951A2078@async.async.caltech.edu>, , , , <20110220005904.A15F21A2078@async.async.caltech.edu>, , <20110220015107.110461A2078@async.async.caltech.edu> Message-ID: - You can't get SIGSEGV w/o memory protection. SIGSEGV is generated by hardware. - Untrusted code should not be run in-process, at least unsafe untrusted code. - I don't have to reboot my computer for SIGSEGV, but individual processes are killed. The damage is contained as soon as it is detected. The nice thing about safe languages is the error is caught just before any damage is caused, i.e. array bounds checking. - Jay > To: jay.krell at cornell.edu > Date: Sat, 19 Feb 2011 17:51:07 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > >But SIGSEGV I think is in another realm. It is a sign of problems that shou= > >ld generally > >be ignored. > >There is a term "fail fast" -- at a sign of problem=2C fail. Don't press on= > >. > >There are arguments either way. > >The world=2C really=2C the larger human enterprise and setting=2C is not fa= > >il fast. > >We have lots of problems=2C yet almost everything almost always goes on. > > Well, see, when you're using a Lisp interpreter, SIGSEGV in called > code is not in another realm (this is my problem). It is also not > in another realm if you are implementing an operating system without > hardware memory protection (e.g., SPIN) or if you're running untrusted > outside code in a "sandbox" (which you CAN build with Modula-3, again > without memory protection). How would you like it if you had to reboot > your computer every time you dereferenced a null pointer? Sounds a bit > like the 1950s doesn't it? Why should I have to restart my Lisp > interpreter if I slip on the keyboard? > > These are things you would never dream of doing in C but that Modula-3 > is eminently suited for doing. For precisely these sorts of purposes, > there's a big distinction designed into the language between failures > in "safe" code (which mean just that the code failed) and failures in > "unsafe" code (which mean that the world ended). > > If you're not doing any of the special things listed above you're free > not to catch RuntimeError.E, in which case your program will crash on > an uncaught exception if it receives a SIGSEGV. > > > >I just don't think because the Green Book says something=2C or because it i= > >s inherent in the > >design=2C that is necessarily true or inherent in the implementation. > >There are bugs *everywhere*=2C and they gradually chip away at all aspects = > >of correctness. > > Dijkstra teaches that one should code to the specification. Coding to > the implementation is a sure path to losing one's mind. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Sun Feb 20 13:28:52 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 20 Feb 2011 13:28:52 +0100 Subject: [M3devel] platform-independent kill process Message-ID: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> I've been thinking a bit about the hanging tests problem in our regression test framework. I'd like to provide a simple utility to start a program with time supervision. But as far as I can see, Modula-3 does not provide a platform-independent way of terminating a process. Of course, on Unix I can send signals; but what would I do on Windows? A quick scan of the win32 interfaces in m3core hasn't brought up anything, but I might have looked for the wrong texts. Could we easily extend the Process interface e.g. by PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException ? Any opinions about that? Different interface suggestions? Rather create ProcessExtras than extending a `standard' interface? We currently only have to consider Unix and Windows, or are there other active flavours of OSs to care for? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Sun Feb 20 17:54:00 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 08:54:00 -0800 Subject: [M3devel] platform-independent kill process In-Reply-To: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> References: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> Message-ID: <20110220165400.384291A2078@async.async.caltech.edu> While poking around the compiler sources I found something called M3Process.Interrupt in m3middle---note the comment about Win32: PROCEDURE Interrupt (t: Process.T) RAISES {OSError.E}; (* Send an interrupt signal to process 't'. *) PROCEDURE Abort (t: Process.T) RAISES {OSError.E}; (* Make a best effort attempt to stop process 't'. Warning: aborting a process may leave it in a broken state or fail to release its OS resources (ie. Win32 is broken!) *) POSIX implementation: PROCEDURE Interrupt (t: Process.T) RAISES {OSError.E} = VAR pid := Process.GetID (t); BEGIN IF Usignal.kill (pid, Usignal.SIGINT) # 0 THEN OSErrorPosix.Raise (); END; END Interrupt; PROCEDURE Abort (t: Process.T) RAISES {OSError.E} = VAR pid := Process.GetID (t); BEGIN IF Usignal.kill (pid, Usignal.SIGKILL) # 0 THEN OSErrorPosix.Raise (); END; END Abort; WIN32: PROCEDURE Interrupt (t: Process.T) RAISES {OSError.E} = BEGIN Abort (t); (* can't do any better yet... *) END Interrupt; PROCEDURE Abort (t: Process.T) RAISES {OSError.E} = VAR hProcess := LOOPHOLE (Process.GetID (t), WinNT.HANDLE); BEGIN IF WinBase.TerminateProcess (hProcess, 1) = 0 THEN OSErrorWin32.Raise (); END; END Abort; I don't know enough about Windows to deciper the warning about releasing resources... Mika Olaf Wagner writes: >I've been thinking a bit about the hanging tests problem in our regression >test framework. I'd like to provide a simple utility to start a program >with time supervision. But as far as I can see, Modula-3 does not provide >a platform-independent way of terminating a process. Of course, on Unix >I can send signals; but what would I do on Windows? A quick scan of the >win32 interfaces in m3core hasn't brought up anything, but I might >have looked for the wrong texts. > >Could we easily extend the Process interface e.g. by > > PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException > >? >Any opinions about that? Different interface suggestions? >Rather create ProcessExtras than extending a `standard' interface? > >We currently only have to consider Unix and Windows, or are there >other active flavours of OSs to care for? > >Olaf >--=20 >Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb=E4ude 12, 13355 Berlin, Germany >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch=E4ftsf=FChrer: Olaf Wagner | Sitz: Berl= >in >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Sun Feb 20 18:12:19 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 09:12:19 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , , , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , , , <4D604FA3.3060708@lcwb.coop>, , , , <20110219232729.53A991A2078@async.async.caltech.edu>, , , , , , <20110220001314.7F2951A2078@async.async.caltech.edu>, , , , <20110220005904.A15F21A2078@async.async.caltech.edu>, , <20110220015107.110461A2078@async.async.caltech.edu> Message-ID: <20110220171219.9CE011A2078@async.async.caltech.edu> Jay K writes: > - You can't get SIGSEGV w/o memory protection. SIGSEGV is generated by har= >dware. >=20 I'm not interested in SIGSEGV per se. The point is to map whatever bad memory reference can be generated by non-UNSAFE code to a RuntimeError.E. If you have the hardware to do that, you use it. A Modula-3 implementation on a system without memory protection would have to generate RuntimeError.E using a different mechanism. > - Untrusted code should not be run in-process=2C at least unsafe untrusted= > code. Why not? If you can prove it can't cause damage, then it can't. Why should Java be able to run untrusted code and not we? >=20 > - I don't have to reboot my computer for SIGSEGV=2C but individual process= >es are killed. It's the same argument. Why do you trust your kernel? It's buggy too! If a program crashes, better to reboot the machine to be sure you didn't infect something else. Heck, unplug it for three minutes just to make sure... > The damage is contained as soon as it is detected. The nice thing about = >safe languages > is the error is caught just before any damage is caused=2C i.e. array b= >ounds checking. My point exactly. No reason to start from a clean slate, if you have some obvious earlier place to jump back to. All the examples I've given of situations where it's useful to catch the RuntimeError.E corresponding to SIGSEGV are systems that can be thought of as pseudo-OSes. You have a "job" or "process" doing something within the context of a larger system, or at least there are levels of trust. You're running "an untrusted module" or "a user command". I won't call them "processes" because the terminology would get confusing, but it's the same idea. If one of these modules crashes, there is really no need to abort the whole world, and it would be extremely annoying to user(s) to have to do so. Mika From rodney_bates at lcwb.coop Sun Feb 20 18:58:27 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 20 Feb 2011 11:58:27 -0600 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110219232729.53A991A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> <20110219232729.53A991A2078@async.async.caltech.edu> Message-ID: <4D615643.7060107@lcwb.coop> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. Probably a fully protected guard page at the end of the stack. This technique always worries me a bit because a procedure with a really big activation record could jump right past it. Probably it would almost always access the first page of the big area before storing anything into later pages. On 02/19/2011 05:27 PM, Mika Nystrom wrote: > Ah, yes, stack protection. > > Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > > Hmm, I get SIGILL on AMD64_FREEBSD as well: > > time ../AMD64_FREEBSD/stubexample > M-Scheme Experimental > LITHP ITH LITHENING. >> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > f >> (f 0) > Illegal instruction > 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > > What absolutely must not happen, of course, is that the runtime hangs > while executing only safe code... > > Mika > > "Rodney M. Bates" writes: >> I know of one other place the compilers rely on hardware memory protection >> to detect a checked runtime error, and that is stack overflow. This won't >> corrupt anything, but is hard to distinguish from dereferencing NIL. >> This could probably be distinguished after the fact by some low-level, >> target-dependent code. I have found it by looking at assembly code at >> the point of failure--usually right after a stack pointer push. >> >> Detecting this via compiler-generated checks would probably be more >> extravagant than many other checks, as it is so frequent. I am not >> aware of any really good solution to this in any implementation of any >> language. >> >> On 02/19/2011 02:38 PM, Mika Nystrom wrote: >>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >>> you know! >>> >>> "Corrupting the heap" is something that can only happen as a result of >>> an unchecked runtime error. Unchecked runtime errors cannot happen in >>> modules not marked UNSAFE. >>> >>> SEGV is, however, used by the CM3 implementation (and its predecessors) >>> to signal a certain kind of *checked* runtime error, namely, the >>> dereferencing of a NIL reference. Correct me if I am wrong, but an >>> attempt to dereference NIL is not going to leave the heap corrupted? >>> >>> And if you stick to safe code, the only SEGVs I think you get in the >>> current CM3 are ones from NIL dereferences. >>> >>> Hence, as long as you stick with safe code, the only time the code I >>> checked in earlier gets triggered is for NIL dereferences, which should >>> never corrupt the heap. So SEGV is not sometimes, but in fact always >>> recoverable. >>> >>> :-) >>> >>> Mika >>> >>> P.S. the bit above "if you stick to safe code": if you actually program in >>> Modula-3 you almost never use UNSAFE. I went through my repository and >>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >>> many of the UNSAFE modules are glue code to Fortran routines, which >>> could relatively easily be verified to be safe in the Modula-3 sense. >>> Almost all what remains is glue to some C library, which wouldn't be >>> necessary if the rest of the world would wake up out of the dark ages, but >>> I don't have the time to rewrite every single library from scratch myself. >>> >>> >>> Jay K writes: >>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>> Content-Type: text/plain; charset="iso-8859-1" >>>> Content-Transfer-Encoding: quoted-printable >>>> >>>> >>>> Letting any code run after a SIGSEGV is dubious. >>>> Imagine the heap is corrupted. >>>> And then you run more code. >>>> And the code happens to call malloc. >>>> Or printf to log something. >>>> =20 >>>> I suppose there might be an application that maps memory >>>> gradually=2C as pieces of a buffer are hit. Might. >>>> =20 >>>> - Jay >>>> =20 >>>>> To: m3devel at elegosoft.com >>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>>> From: mika at async.caltech.edu >>>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>>> =20 >>>>> =20 >>>>> Dear m3devel=2C >>>>> =20 >>>>> For a while it has annoyed me that segmentation violations cause an >>>>> unconditional program abort. I've changed that now so that (under user >>>>> threads at least) we instead get a RuntimeError. Here's an example of >>>>> the mechanism at work in an interactive Scheme environment. Consider >>>>> the unhelpful interface and module Crash: >>>>> =20 >>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>>> =20 >>>>> MODULE Crash=3B >>>>> =20 >>>>> PROCEDURE Me() =3D >>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>>> ptr^ :=3D 0 >>>>> END Me=3B >>>>> =20 >>>>> BEGIN END Crash. >>>>> =20 >>>>> Here's an example of what happens if you now call this from an interactiv= >>>> e >>>>> interpreter that catches the exception RuntimeError.E: >>>>> =20 >>>>> M-Scheme Experimental >>>>> LITHP ITH LITHENING. >>>>>> (require-modules "m3") >>>>> #t >>>>>> (Crash.Me) >>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>>> (+ 3 4)=20 >>>>> 7 >>>>>> =20 >>>>> =20 >>>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>>> heck it.=20 >>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>>> =20 >>>>> Mika >>>>> =20 >>>> = >>>> >>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>> Content-Type: text/html; charset="iso-8859-1" >>>> Content-Transfer-Encoding: quoted-printable >>>> >>>> >>>> >>>> >>>> >>>> >>>> Letting any code run after a SIGSEGV is dubious.
>>>> Imagine the heap =3Bis corrupted.
>>>> And then you run more code.
>>>> And the code happens to call malloc.
>>>> Or printf to log something.
>>>>  =3B
>>>> I suppose there might be an application that maps memory
>>>> gradually=2C as pieces of a buffer are hit. Might.
>>>>  =3B
>>>>  =3B- Jay
 =3B
>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>>> olations cause an
>=3B unconditional program abort. I've changed that = >>>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>>> >=3B Here's an example of what happens if you now call this from an inter= >>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>>> >>>> = >>>> >>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >>> > From mika at async.caltech.edu Sun Feb 20 19:37:46 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 10:37:46 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <4D615643.7060107@lcwb.coop> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> <20110219232729.53A991A2078@async.async.caltech.edu> <4D615643.7060107@lcwb.coop> Message-ID: <20110220183746.195C11A2078@async.async.caltech.edu> On a 64-bit machine, at least, there ought to be enough virtual memory that you could just have a gap between thread stacks big enough to allow for a protection area larger than the largest possible (implementation-defined) activation record, no? I know I've run into trouble with very large activation records in the past (and not because I was running out of stack space, either). Or at least a procedure with a very large activation record (or a procedure calling it) could be required to call some sort of check routine "EnoughStackSpaceRemaining()" before starting to scribble on the activation record? Also the end of the activation record must be written to at least once, or else the memory protection won't be triggered. In any case if this is done properly the same mechanism I proposed for SIGSEGV ought to be able to catch stack overflow, no? Well, as long as signals are delivered on a separate stack. If signals are delivered on the same stack, the signal handler would get nastier, it would have to make space through some manipulations (maybe temporarily unporotecting the redzone page?) for its own purposes... but I don't see why it couldn't be done. Not sure why I'm getting SIGILL... maybe I am getting my signal handler activated inside the redzone page because of a difference in signal handling..? I remember reading something about sigaltstack... I would of course love to be able to recover from stack overflow, too. In some sense, since it's a generally unknown limit, it's even less of a fatal error than a NIL dereference (hence makes even more sense to catch it). Mika "Rodney M. Bates" writes: >I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. >Probably a fully protected guard page at the end of the stack. This technique >always worries me a bit because a procedure with a really big activation record >could jump right past it. Probably it would almost always access the first page >of the big area before storing anything into later pages. > >On 02/19/2011 05:27 PM, Mika Nystrom wrote: >> Ah, yes, stack protection. >> >> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. >> >> Hmm, I get SIGILL on AMD64_FREEBSD as well: >> >> time ../AMD64_FREEBSD/stubexample >> M-Scheme Experimental >> LITHP ITH LITHENING. >>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) >> f >>> (f 0) >> Illegal instruction >> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w >> >> What absolutely must not happen, of course, is that the runtime hangs >> while executing only safe code... >> >> Mika >> >> "Rodney M. Bates" writes: >>> I know of one other place the compilers rely on hardware memory protection >>> to detect a checked runtime error, and that is stack overflow. This won't >>> corrupt anything, but is hard to distinguish from dereferencing NIL. >>> This could probably be distinguished after the fact by some low-level, >>> target-dependent code. I have found it by looking at assembly code at >>> the point of failure--usually right after a stack pointer push. >>> >>> Detecting this via compiler-generated checks would probably be more >>> extravagant than many other checks, as it is so frequent. I am not >>> aware of any really good solution to this in any implementation of any >>> language. >>> >>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: >>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >>>> you know! >>>> >>>> "Corrupting the heap" is something that can only happen as a result of >>>> an unchecked runtime error. Unchecked runtime errors cannot happen in >>>> modules not marked UNSAFE. >>>> >>>> SEGV is, however, used by the CM3 implementation (and its predecessors) >>>> to signal a certain kind of *checked* runtime error, namely, the >>>> dereferencing of a NIL reference. Correct me if I am wrong, but an >>>> attempt to dereference NIL is not going to leave the heap corrupted? >>>> >>>> And if you stick to safe code, the only SEGVs I think you get in the >>>> current CM3 are ones from NIL dereferences. >>>> >>>> Hence, as long as you stick with safe code, the only time the code I >>>> checked in earlier gets triggered is for NIL dereferences, which should >>>> never corrupt the heap. So SEGV is not sometimes, but in fact always >>>> recoverable. >>>> >>>> :-) >>>> >>>> Mika >>>> >>>> P.S. the bit above "if you stick to safe code": if you actually program in >>>> Modula-3 you almost never use UNSAFE. I went through my repository and >>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >>>> many of the UNSAFE modules are glue code to Fortran routines, which >>>> could relatively easily be verified to be safe in the Modula-3 sense. >>>> Almost all what remains is glue to some C library, which wouldn't be >>>> necessary if the rest of the world would wake up out of the dark ages, but >>>> I don't have the time to rewrite every single library from scratch myself. >>>> >>>> >>>> Jay K writes: >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>> Content-Type: text/plain; charset="iso-8859-1" >>>>> Content-Transfer-Encoding: quoted-printable >>>>> >>>>> >>>>> Letting any code run after a SIGSEGV is dubious. >>>>> Imagine the heap is corrupted. >>>>> And then you run more code. >>>>> And the code happens to call malloc. >>>>> Or printf to log something. >>>>> =20 >>>>> I suppose there might be an application that maps memory >>>>> gradually=2C as pieces of a buffer are hit. Might. >>>>> =20 >>>>> - Jay >>>>> =20 >>>>>> To: m3devel at elegosoft.com >>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>>>> From: mika at async.caltech.edu >>>>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>>>> =20 >>>>>> =20 >>>>>> Dear m3devel=2C >>>>>> =20 >>>>>> For a while it has annoyed me that segmentation violations cause an >>>>>> unconditional program abort. I've changed that now so that (under user >>>>>> threads at least) we instead get a RuntimeError. Here's an example of >>>>>> the mechanism at work in an interactive Scheme environment. Consider >>>>>> the unhelpful interface and module Crash: >>>>>> =20 >>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>>>> =20 >>>>>> MODULE Crash=3B >>>>>> =20 >>>>>> PROCEDURE Me() =3D >>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>>>> ptr^ :=3D 0 >>>>>> END Me=3B >>>>>> =20 >>>>>> BEGIN END Crash. >>>>>> =20 >>>>>> Here's an example of what happens if you now call this from an interactiv= >>>>> e >>>>>> interpreter that catches the exception RuntimeError.E: >>>>>> =20 >>>>>> M-Scheme Experimental >>>>>> LITHP ITH LITHENING. >>>>>>> (require-modules "m3") >>>>>> #t >>>>>>> (Crash.Me) >>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>>>> (+ 3 4)=20 >>>>>> 7 >>>>>>> =20 >>>>>> =20 >>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>>>> heck it.=20 >>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>>>> =20 >>>>>> Mika >>>>>> =20 >>>>> = >>>>> >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>> Content-Type: text/html; charset="iso-8859-1" >>>>> Content-Transfer-Encoding: quoted-printable >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Letting any code run after a SIGSEGV is dubious.
>>>>> Imagine the heap =3Bis corrupted.
>>>>> And then you run more code.
>>>>> And the code happens to call malloc.
>>>>> Or printf to log something.
>>>>>  =3B
>>>>> I suppose there might be an application that maps memory
>>>>> gradually=2C as pieces of a buffer are hit. Might.
>>>>>  =3B
>>>>>  =3B- Jay
 =3B
>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>>>> olations cause an
>=3B unconditional program abort. I've changed that = >>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>>>> >=3B Here's an example of what happens if you now call this from an inter= >>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>>>> >>>>> = >>>>> >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >>>> >> From jay.krell at cornell.edu Sun Feb 20 23:38:19 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 20 Feb 2011 22:38:19 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220183746.195C11A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu>, <4D615643.7060107@lcwb.coop>, <20110220183746.195C11A2078@async.async.caltech.edu> Message-ID: Probably for this reason: There is a requirement on NT that stack pages be touched in order. Functions with locals totally more than 4K call _chkstk (aka _alloca) to allocate their stack, instead of the usual register subtraction. It contains a loop that touches a byte every 4K. Otherwise, if you have lots of functions with small frames, the stack is touched by virtue of the call function pushing the return address. Modula-3 has long failed to uphold this contract, and still does. I've never seen it clearly documented, but you can see it is what the C compiler does. I had thought there were other reasons for this behavior. I thought you actually get an exception of the stack is touched out of order ("the first time"). But I think I did an experiment long ago with Modula-3 and there was no exception. I don't know about other platforms. I've also seen evidence in gcc and/or Target.i3 that compiler writers know well about such mechanisms -- there being a constant to set as to what size locals trigger special behavior. But m3back doesn't do anything here. Probably we could use _chkstk unconditonally as well -- need to see what it does for numbers smaller than 4K. But that'd be a deoptimization in the common case, and deoptimizing only as necessary should be easy enough. - Jay > To: rodney_bates at lcwb.coop > Date: Sun, 20 Feb 2011 10:37:46 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > On a 64-bit machine, at least, there ought to be enough virtual > memory that you could just have a gap between thread stacks big > enough to allow for a protection area larger than the largest possible > (implementation-defined) activation record, no? I know I've run into > trouble with very large activation records in the past (and not because > I was running out of stack space, either). > > Or at least a procedure with a very large activation record (or > a procedure calling it) could be required to call some sort of check > routine "EnoughStackSpaceRemaining()" before starting to scribble > on the activation record? > > Also the end of the activation record must be written to at least once, > or else the memory protection won't be triggered. > > In any case if this is done properly the same mechanism I proposed for > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > signals are delivered on a separate stack. If signals are delivered on > the same stack, the signal handler would get nastier, it would have to > make space through some manipulations (maybe temporarily unporotecting > the redzone page?) for its own purposes... but I don't see why it > couldn't be done. > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > activated inside the redzone page because of a difference in signal > handling..? I remember reading something about sigaltstack... > > I would of course love to be able to recover from stack overflow, too. > In some sense, since it's a generally unknown limit, it's even less of > a fatal error than a NIL dereference (hence makes even more sense to > catch it). > > Mika > > "Rodney M. Bates" writes: > >I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >Probably a fully protected guard page at the end of the stack. This technique > >always worries me a bit because a procedure with a really big activation record > >could jump right past it. Probably it would almost always access the first page > >of the big area before storing anything into later pages. > > > >On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >> Ah, yes, stack protection. > >> > >> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >> > >> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >> > >> time ../AMD64_FREEBSD/stubexample > >> M-Scheme Experimental > >> LITHP ITH LITHENING. > >>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >> f > >>> (f 0) > >> Illegal instruction > >> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >> > >> What absolutely must not happen, of course, is that the runtime hangs > >> while executing only safe code... > >> > >> Mika > >> > >> "Rodney M. Bates" writes: > >>> I know of one other place the compilers rely on hardware memory protection > >>> to detect a checked runtime error, and that is stack overflow. This won't > >>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>> This could probably be distinguished after the fact by some low-level, > >>> target-dependent code. I have found it by looking at assembly code at > >>> the point of failure--usually right after a stack pointer push. > >>> > >>> Detecting this via compiler-generated checks would probably be more > >>> extravagant than many other checks, as it is so frequent. I am not > >>> aware of any really good solution to this in any implementation of any > >>> language. > >>> > >>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>> you know! > >>>> > >>>> "Corrupting the heap" is something that can only happen as a result of > >>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>> modules not marked UNSAFE. > >>>> > >>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>> to signal a certain kind of *checked* runtime error, namely, the > >>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>> > >>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>> current CM3 are ones from NIL dereferences. > >>>> > >>>> Hence, as long as you stick with safe code, the only time the code I > >>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>> recoverable. > >>>> > >>>> :-) > >>>> > >>>> Mika > >>>> > >>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>> Almost all what remains is glue to some C library, which wouldn't be > >>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>> I don't have the time to rewrite every single library from scratch myself. > >>>> > >>>> > >>>> Jay K writes: > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious. > >>>>> Imagine the heap is corrupted. > >>>>> And then you run more code. > >>>>> And the code happens to call malloc. > >>>>> Or printf to log something. > >>>>> =20 > >>>>> I suppose there might be an application that maps memory > >>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>> =20 > >>>>> - Jay > >>>>> =20 > >>>>>> To: m3devel at elegosoft.com > >>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>> From: mika at async.caltech.edu > >>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>> =20 > >>>>>> =20 > >>>>>> Dear m3devel=2C > >>>>>> =20 > >>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>> the unhelpful interface and module Crash: > >>>>>> =20 > >>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>> =20 > >>>>>> MODULE Crash=3B > >>>>>> =20 > >>>>>> PROCEDURE Me() =3D > >>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>> ptr^ :=3D 0 > >>>>>> END Me=3B > >>>>>> =20 > >>>>>> BEGIN END Crash. > >>>>>> =20 > >>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>> e > >>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>> =20 > >>>>>> M-Scheme Experimental > >>>>>> LITHP ITH LITHENING. > >>>>>>> (require-modules "m3") > >>>>>> #t > >>>>>>> (Crash.Me) > >>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>> (+ 3 4)=20 > >>>>>> 7 > >>>>>>> =20 > >>>>>> =20 > >>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>> heck it.=20 > >>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>> =20 > >>>>>> Mika > >>>>>> =20 > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>> Imagine the heap =3Bis corrupted.
> >>>>> And then you run more code.
> >>>>> And the code happens to call malloc.
> >>>>> Or printf to log something.
> >>>>>  =3B
> >>>>> I suppose there might be an application that maps memory
> >>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>  =3B
> >>>>>  =3B- Jay
 =3B
> >>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>> > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>> > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 01:41:28 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 00:41:28 +0000 Subject: [M3devel] libm3/Process Message-ID: Process.Wait frees up resources. Is that the right way to go? Shouldn't it register a cleanup function for the garbage collector to call, like we do for mutexes? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Mon Feb 21 02:15:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 17:15:50 -0800 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20110221003248.11C9C2474005@birch.elegosoft.com>, Message-ID: <20110221011550.0D2791A2078@async.async.caltech.edu> Jay I didn't quite follow your previous messages. What is the correct solution, and what did you mean when you were talking about the garbage collector? It does seem like---if a Process handle goes out of scope and gets collected then yes the runtime should do the equivalent of "EVAL Process.Wait" at some future time. But that seems unrelated/orthogonal to the issues with the Windows process killing... The process killing in m3middle/M3Process I think is only called in a few places in the compiler and then under what already are dubious circumstances. For instance I see it done where Process.Create has failed. But if Process.Create has failed, then there's no process to kill, so it all seems rather pointless. It is also called as one of the last actions before exiting the parent process, so there's another reason for its being pointless. In other words I think it's safe to say that if you have an idea for how to implement process-killing correctly on Windows you can do it however you like without fear of breaking any existing code... Mika But I can Jay K writes: >--_d36c70d3-e71b-433d-abdc-06703a929f89_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >The code here confuses hProcess and dwProcessId. >Can we introduce a public function here in libm3 instead of m3middle? >ie: to avoid answering the question as to how to expose hProcess? >And to avoid the loophole as well? > >Thanks=2C > - Jay > >From: jay.krell at cornell.edu >To: jkrell at elego.de=3B m3commit at elegosoft.com >Subject: RE: [M3commit] CVS Update: cm3 >Date: Mon=2C 21 Feb 2011 00:35:38 +0000 > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= >rect here. > > - Jay > >> Date: Mon=2C 21 Feb 2011 01:32:47 +0000 >> To: m3commit at elegosoft.com >> From: jkrell at elego.de >> Subject: [M3commit] CVS Update: cm3 >>=20 >> CVSROOT: /usr/cvs >> Changes by: jkrell at birch. 11/02/21 01:32:47 >>=20 >> Modified files: >> cm3/m3-sys/m3middle/src/: M3Process.i3=20 >>=20 >> Log message: >> remove incorrect comments=3B Win32 is not broken >>=20 > = > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >The code here confuses hProcess and dwProcessId.
Can we introduce a publ= >ic function here in libm3 instead of m3middle?
ie: to avoid answering th= >e question as to how to expose hProcess?
And to avoid the loophole as we= >ll?

Thanks=2C
 =3B- Jay

From: = >jay.krell at cornell.edu
To: jkrell at elego.de=3B m3commit at elegosoft.com
S= >ubject: RE: [M3commit] CVS Update: cm3
Date: Mon=2C 21 Feb 2011 00:35:38= > +0000

> >"> > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= >rect here.

 =3B- Jay

>=3B Date: Mon=2C 21 Feb 2011 01:3= >2:47 +0000
>=3B To: m3commit at elegosoft.com
>=3B From: jkrell at eleg= >o.de
>=3B Subject: [M3commit] CVS Update: cm3
>=3B
>=3B CVS= >ROOT: /usr/cvs
>=3B Changes by: jkrell at birch. 11/02/21 01:32:47
>= >=3B
>=3B Modified files:
>=3B cm3/m3-sys/m3middle/src/: M3Proce= >ss.i3
>=3B
>=3B Log message:
>=3B remove incorrect commen= >ts=3B Win32 is not broken
>=3B
>= > >--_d36c70d3-e71b-433d-abdc-06703a929f89_-- From rodney_bates at lcwb.coop Mon Feb 21 03:21:27 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 20 Feb 2011 20:21:27 -0600 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220183746.195C11A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> <20110219232729.53A991A2078@async.async.caltech.edu> <4D615643.7060107@lcwb.coop> <20110220183746.195C11A2078@async.async.caltech.edu> Message-ID: <4D61CC27.7070704@lcwb.coop> On 02/20/2011 12:37 PM, Mika Nystrom wrote: > On a 64-bit machine, at least, there ought to be enough virtual > memory that you could just have a gap between thread stacks big > enough to allow for a protection area larger than the largest possible > (implementation-defined) activation record, no? I know I've run into > trouble with very large activation records in the past (and not because > I was running out of stack space, either). > > Or at least a procedure with a very large activation record (or > a procedure calling it) could be required to call some sort of check > routine "EnoughStackSpaceRemaining()" before starting to scribble > on the activation record? Hmm, I like this idea. It would introduce normal-case runtime overhead only for such procedures, and these are likely rare. Also, assuming the procedure actually uses very much of its large AR, it should also have enough computation time to wash out the stack check overhead. > > Also the end of the activation record must be written to at least once, > or else the memory protection won't be triggered. > I was thinking (as an alternative mechanism) of having the compiler intentionally add enough artificial write(s) as necessary to ensure storing within the red zone, and not just beyond it. This seems trickier to get right and harder to distinguish after the fact from a NIL dereference. > In any case if this is done properly the same mechanism I proposed for > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > signals are delivered on a separate stack. If signals are delivered on > the same stack, the signal handler would get nastier, it would have to > make space through some manipulations (maybe temporarily unporotecting > the redzone page?) for its own purposes... but I don't see why it > couldn't be done. > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > activated inside the redzone page because of a difference in signal > handling..? I remember reading something about sigaltstack... > > I would of course love to be able to recover from stack overflow, too. > In some sense, since it's a generally unknown limit, it's even less of > a fatal error than a NIL dereference (hence makes even more sense to > catch it). I think this would be a nice mechanism to have available. It would have to be used with some care. In any case, it would be really nice and more frequently so, to at least have runtime error messages that distinguished stack overflow from NIL deref. > > Mika > > "Rodney M. Bates" writes: >> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. >> Probably a fully protected guard page at the end of the stack. This technique >> always worries me a bit because a procedure with a really big activation record >> could jump right past it. Probably it would almost always access the first page >> of the big area before storing anything into later pages. >> >> On 02/19/2011 05:27 PM, Mika Nystrom wrote: >>> Ah, yes, stack protection. >>> >>> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. >>> >>> Hmm, I get SIGILL on AMD64_FREEBSD as well: >>> >>> time ../AMD64_FREEBSD/stubexample >>> M-Scheme Experimental >>> LITHP ITH LITHENING. >>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) >>> f >>>> (f 0) >>> Illegal instruction >>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w >>> >>> What absolutely must not happen, of course, is that the runtime hangs >>> while executing only safe code... >>> >>> Mika >>> >>> "Rodney M. Bates" writes: >>>> I know of one other place the compilers rely on hardware memory protection >>>> to detect a checked runtime error, and that is stack overflow. This won't >>>> corrupt anything, but is hard to distinguish from dereferencing NIL. >>>> This could probably be distinguished after the fact by some low-level, >>>> target-dependent code. I have found it by looking at assembly code at >>>> the point of failure--usually right after a stack pointer push. >>>> >>>> Detecting this via compiler-generated checks would probably be more >>>> extravagant than many other checks, as it is so frequent. I am not >>>> aware of any really good solution to this in any implementation of any >>>> language. >>>> >>>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: >>>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >>>>> you know! >>>>> >>>>> "Corrupting the heap" is something that can only happen as a result of >>>>> an unchecked runtime error. Unchecked runtime errors cannot happen in >>>>> modules not marked UNSAFE. >>>>> >>>>> SEGV is, however, used by the CM3 implementation (and its predecessors) >>>>> to signal a certain kind of *checked* runtime error, namely, the >>>>> dereferencing of a NIL reference. Correct me if I am wrong, but an >>>>> attempt to dereference NIL is not going to leave the heap corrupted? >>>>> >>>>> And if you stick to safe code, the only SEGVs I think you get in the >>>>> current CM3 are ones from NIL dereferences. >>>>> >>>>> Hence, as long as you stick with safe code, the only time the code I >>>>> checked in earlier gets triggered is for NIL dereferences, which should >>>>> never corrupt the heap. So SEGV is not sometimes, but in fact always >>>>> recoverable. >>>>> >>>>> :-) >>>>> >>>>> Mika >>>>> >>>>> P.S. the bit above "if you stick to safe code": if you actually program in >>>>> Modula-3 you almost never use UNSAFE. I went through my repository and >>>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >>>>> many of the UNSAFE modules are glue code to Fortran routines, which >>>>> could relatively easily be verified to be safe in the Modula-3 sense. >>>>> Almost all what remains is glue to some C library, which wouldn't be >>>>> necessary if the rest of the world would wake up out of the dark ages, but >>>>> I don't have the time to rewrite every single library from scratch myself. >>>>> >>>>> >>>>> Jay K writes: >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>>> Content-Type: text/plain; charset="iso-8859-1" >>>>>> Content-Transfer-Encoding: quoted-printable >>>>>> >>>>>> >>>>>> Letting any code run after a SIGSEGV is dubious. >>>>>> Imagine the heap is corrupted. >>>>>> And then you run more code. >>>>>> And the code happens to call malloc. >>>>>> Or printf to log something. >>>>>> =20 >>>>>> I suppose there might be an application that maps memory >>>>>> gradually=2C as pieces of a buffer are hit. Might. >>>>>> =20 >>>>>> - Jay >>>>>> =20 >>>>>>> To: m3devel at elegosoft.com >>>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>>>>> From: mika at async.caltech.edu >>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>>>>> =20 >>>>>>> =20 >>>>>>> Dear m3devel=2C >>>>>>> =20 >>>>>>> For a while it has annoyed me that segmentation violations cause an >>>>>>> unconditional program abort. I've changed that now so that (under user >>>>>>> threads at least) we instead get a RuntimeError. Here's an example of >>>>>>> the mechanism at work in an interactive Scheme environment. Consider >>>>>>> the unhelpful interface and module Crash: >>>>>>> =20 >>>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>>>>> =20 >>>>>>> MODULE Crash=3B >>>>>>> =20 >>>>>>> PROCEDURE Me() =3D >>>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>>>>> ptr^ :=3D 0 >>>>>>> END Me=3B >>>>>>> =20 >>>>>>> BEGIN END Crash. >>>>>>> =20 >>>>>>> Here's an example of what happens if you now call this from an interactiv= >>>>>> e >>>>>>> interpreter that catches the exception RuntimeError.E: >>>>>>> =20 >>>>>>> M-Scheme Experimental >>>>>>> LITHP ITH LITHENING. >>>>>>>> (require-modules "m3") >>>>>>> #t >>>>>>>> (Crash.Me) >>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>>>>> (+ 3 4)=20 >>>>>>> 7 >>>>>>>> =20 >>>>>>> =20 >>>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>>>>> heck it.=20 >>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>>>>> =20 >>>>>>> Mika >>>>>>> =20 >>>>>> = >>>>>> >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>>> Content-Type: text/html; charset="iso-8859-1" >>>>>> Content-Transfer-Encoding: quoted-printable >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Letting any code run after a SIGSEGV is dubious.
>>>>>> Imagine the heap =3Bis corrupted.
>>>>>> And then you run more code.
>>>>>> And the code happens to call malloc.
>>>>>> Or printf to log something.
>>>>>>  =3B
>>>>>> I suppose there might be an application that maps memory
>>>>>> gradually=2C as pieces of a buffer are hit. Might.
>>>>>>  =3B
>>>>>>  =3B- Jay
 =3B
>>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>>>>> olations cause an
>=3B unconditional program abort. I've changed that = >>>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>>>>> >=3B Here's an example of what happens if you now call this from an inter= >>>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>>>>> >>>>>> = >>>>>> >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >>>>> >>> > From jay.krell at cornell.edu Mon Feb 21 04:24:53 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 03:24:53 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <4D61CC27.7070704@lcwb.coop> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu>, <4D615643.7060107@lcwb.coop>, <20110220183746.195C11A2078@async.async.caltech.edu>, <4D61CC27.7070704@lcwb.coop> Message-ID: We should look at what C does on the various targets. NT already has a specific easy simple mechanism/policy here, that Modula-3 should follow but doesn't. Specifically, C code in NT can overflow its stack, but it is caught "right away" and not via random corruption of neighboring memory. Perhaps, hopefully, C code in other targets is this good. If not, well, the NT mechanism makes a lot of sense for all targets: if function has less than a page of locals, do nothing special. if function has more than a page of locals, touch each page, in order, at the start of the function. "page" is hardware specific, however I believe all of our targets have 4K or 8K pages, and if you just hardcode 4K, that works, with a slight slight pessimization on targets with 8K pages (e.g. I think Sparc and IA64, but I'd have to dig around). Actually NT reserves like two pages at the end of the stack. One to trigger stack overflow exception, one to give an exception handler a little bit of room to deal with it, such as by capturing a dump of some sort, and exiting, and then if the exception handler uses more than its page, the process is I think terminated. FURTHERMORE, this is a detail that SHOULD be handled by the existing gcc backend. I don't know if it does, but it ought to. But the NT backend does not, and should. It's possible this problem has regressed with the 1K jmpbuf I put in. (I'm very very inclined to shrink that down either to precise levels or approximate but usually smaller levels. m3core did have an assert at startup that the size was large enough. Still, we can't get the 128 bit alignment some targets prefer (powerpc) or require (hppa64). Still, the alloca solution is better, if done right.) - Jay > Date: Sun, 20 Feb 2011 20:21:27 -0600 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > > On 02/20/2011 12:37 PM, Mika Nystrom wrote: > > On a 64-bit machine, at least, there ought to be enough virtual > > memory that you could just have a gap between thread stacks big > > enough to allow for a protection area larger than the largest possible > > (implementation-defined) activation record, no? I know I've run into > > trouble with very large activation records in the past (and not because > > I was running out of stack space, either). > > > > Or at least a procedure with a very large activation record (or > > a procedure calling it) could be required to call some sort of check > > routine "EnoughStackSpaceRemaining()" before starting to scribble > > on the activation record? > > Hmm, I like this idea. It would introduce normal-case runtime overhead > only for such procedures, and these are likely rare. Also, assuming the procedure > actually uses very much of its large AR, it should also have enough computation > time to wash out the stack check overhead. > > > > > Also the end of the activation record must be written to at least once, > > or else the memory protection won't be triggered. > > > > I was thinking (as an alternative mechanism) of having the compiler intentionally > add enough artificial write(s) as necessary to ensure storing within the > red zone, and not just beyond it. This seems trickier to get right and > harder to distinguish after the fact from a NIL dereference. > > > In any case if this is done properly the same mechanism I proposed for > > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > > signals are delivered on a separate stack. If signals are delivered on > > the same stack, the signal handler would get nastier, it would have to > > make space through some manipulations (maybe temporarily unporotecting > > the redzone page?) for its own purposes... but I don't see why it > > couldn't be done. > > > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > > activated inside the redzone page because of a difference in signal > > handling..? I remember reading something about sigaltstack... > > > > I would of course love to be able to recover from stack overflow, too. > > In some sense, since it's a generally unknown limit, it's even less of > > a fatal error than a NIL dereference (hence makes even more sense to > > catch it). > > I think this would be a nice mechanism to have available. It would have to > be used with some care. In any case, it would be really nice and more > frequently so, to at least have runtime error messages that distinguished > stack overflow from NIL deref. > > > > > Mika > > > > "Rodney M. Bates" writes: > >> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >> Probably a fully protected guard page at the end of the stack. This technique > >> always worries me a bit because a procedure with a really big activation record > >> could jump right past it. Probably it would almost always access the first page > >> of the big area before storing anything into later pages. > >> > >> On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >>> Ah, yes, stack protection. > >>> > >>> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >>> > >>> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >>> > >>> time ../AMD64_FREEBSD/stubexample > >>> M-Scheme Experimental > >>> LITHP ITH LITHENING. > >>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >>> f > >>>> (f 0) > >>> Illegal instruction > >>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >>> > >>> What absolutely must not happen, of course, is that the runtime hangs > >>> while executing only safe code... > >>> > >>> Mika > >>> > >>> "Rodney M. Bates" writes: > >>>> I know of one other place the compilers rely on hardware memory protection > >>>> to detect a checked runtime error, and that is stack overflow. This won't > >>>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>>> This could probably be distinguished after the fact by some low-level, > >>>> target-dependent code. I have found it by looking at assembly code at > >>>> the point of failure--usually right after a stack pointer push. > >>>> > >>>> Detecting this via compiler-generated checks would probably be more > >>>> extravagant than many other checks, as it is so frequent. I am not > >>>> aware of any really good solution to this in any implementation of any > >>>> language. > >>>> > >>>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>>> you know! > >>>>> > >>>>> "Corrupting the heap" is something that can only happen as a result of > >>>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>>> modules not marked UNSAFE. > >>>>> > >>>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>>> to signal a certain kind of *checked* runtime error, namely, the > >>>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>>> > >>>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>>> current CM3 are ones from NIL dereferences. > >>>>> > >>>>> Hence, as long as you stick with safe code, the only time the code I > >>>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>>> recoverable. > >>>>> > >>>>> :-) > >>>>> > >>>>> Mika > >>>>> > >>>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>>> Almost all what remains is glue to some C library, which wouldn't be > >>>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>>> I don't have the time to rewrite every single library from scratch myself. > >>>>> > >>>>> > >>>>> Jay K writes: > >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>>> Content-Transfer-Encoding: quoted-printable > >>>>>> > >>>>>> > >>>>>> Letting any code run after a SIGSEGV is dubious. > >>>>>> Imagine the heap is corrupted. > >>>>>> And then you run more code. > >>>>>> And the code happens to call malloc. > >>>>>> Or printf to log something. > >>>>>> =20 > >>>>>> I suppose there might be an application that maps memory > >>>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>>> =20 > >>>>>> - Jay > >>>>>> =20 > >>>>>>> To: m3devel at elegosoft.com > >>>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>>> From: mika at async.caltech.edu > >>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>>> =20 > >>>>>>> =20 > >>>>>>> Dear m3devel=2C > >>>>>>> =20 > >>>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>>> the unhelpful interface and module Crash: > >>>>>>> =20 > >>>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>>> =20 > >>>>>>> MODULE Crash=3B > >>>>>>> =20 > >>>>>>> PROCEDURE Me() =3D > >>>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>>> ptr^ :=3D 0 > >>>>>>> END Me=3B > >>>>>>> =20 > >>>>>>> BEGIN END Crash. > >>>>>>> =20 > >>>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>>> e > >>>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>>> =20 > >>>>>>> M-Scheme Experimental > >>>>>>> LITHP ITH LITHENING. > >>>>>>>> (require-modules "m3") > >>>>>>> #t > >>>>>>>> (Crash.Me) > >>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>>> (+ 3 4)=20 > >>>>>>> 7 > >>>>>>>> =20 > >>>>>>> =20 > >>>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>>> heck it.=20 > >>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>>> =20 > >>>>>>> Mika > >>>>>>> =20 > >>>>>> = > >>>>>> > >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>>> Content-Transfer-Encoding: quoted-printable > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>>> Imagine the heap =3Bis corrupted.
> >>>>>> And then you run more code.
> >>>>>> And the code happens to call malloc.
> >>>>>> Or printf to log something.
> >>>>>>  =3B
> >>>>>> I suppose there might be an application that maps memory
> >>>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>>  =3B
> >>>>>>  =3B- Jay
 =3B
> >>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>>> > >>>>>> = > >>>>>> > >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>>> > >>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 04:28:10 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 03:28:10 +0000 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: <20110221011550.0D2791A2078@async.async.caltech.edu> References: <20110221003248.11C9C2474005@birch.elegosoft.com>, , , , <20110221011550.0D2791A2078@async.async.caltech.edu> Message-ID: Wait? No, that is probably just bogus posix thinking. You can just close your handle to the process. You don't have to wait for it to exit. It is somewhat orthogonal. The thing is, the request for Process.Kill, got me looking at when does libm3 close the Process handle and when does it use the handle correctly (i.e. not m3middle's Process.Abort!) It does the CloseHandle in Process.Wait, so if that is never called, there is a leak. Anyway, this should be easy to make correct. (So easy, though, it boggles the mind as to why it was done this way in the first place. I do have to check somethig -- it is SLIGHTLY possible I broke it. I know I fiddled around with the definition of process ids, made them match the underlying platform. I'll check the history. m3middle uses a LOOPHOLE on some "id" to get the "handle". Anyway, I'll check and make it correct.) - Jay > To: jay.krell at cornell.edu > Date: Sun, 20 Feb 2011 17:15:50 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] [M3commit] CVS Update: cm3 > > > Jay I didn't quite follow your previous messages. What is the correct > solution, and what did you mean when you were talking about the garbage > collector? It does seem like---if a Process handle goes out of scope > and gets collected then yes the runtime should do the equivalent of "EVAL > Process.Wait" at some future time. But that seems unrelated/orthogonal > to the issues with the Windows process killing... > > The process killing in m3middle/M3Process I think is only called in > a few places in the compiler and then under what already are dubious > circumstances. For instance I see it done where Process.Create has > failed. But if Process.Create has failed, then there's no process to > kill, so it all seems rather pointless. It is also called as one of > the last actions before exiting the parent process, so there's another > reason for its being pointless. > > In other words I think it's safe to say that if you have an idea for > how to implement process-killing correctly on Windows you can do it > however you like without fear of breaking any existing code... > > Mika > > But I can > > Jay K writes: > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >The code here confuses hProcess and dwProcessId. > >Can we introduce a public function here in libm3 instead of m3middle? > >ie: to avoid answering the question as to how to expose hProcess? > >And to avoid the loophole as well? > > > >Thanks=2C > > - Jay > > > >From: jay.krell at cornell.edu > >To: jkrell at elego.de=3B m3commit at elegosoft.com > >Subject: RE: [M3commit] CVS Update: cm3 > >Date: Mon=2C 21 Feb 2011 00:35:38 +0000 > > > > > > > > > > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= > >rect here. > > > > - Jay > > > >> Date: Mon=2C 21 Feb 2011 01:32:47 +0000 > >> To: m3commit at elegosoft.com > >> From: jkrell at elego.de > >> Subject: [M3commit] CVS Update: cm3 > >>=20 > >> CVSROOT: /usr/cvs > >> Changes by: jkrell at birch. 11/02/21 01:32:47 > >>=20 > >> Modified files: > >> cm3/m3-sys/m3middle/src/: M3Process.i3=20 > >>=20 > >> Log message: > >> remove incorrect comments=3B Win32 is not broken > >>=20 > > = > > > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ > >Content-Type: text/html; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > > > > > > > >The code here confuses hProcess and dwProcessId.
Can we introduce a publ= > >ic function here in libm3 instead of m3middle?
ie: to avoid answering th= > >e question as to how to expose hProcess?
And to avoid the loophole as we= > >ll?

Thanks=2C
 =3B- Jay

From: = > >jay.krell at cornell.edu
To: jkrell at elego.de=3B m3commit at elegosoft.com
S= > >ubject: RE: [M3commit] CVS Update: cm3
Date: Mon=2C 21 Feb 2011 00:35:38= > > +0000

> > > > >"> > > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= > >rect here.

 =3B- Jay

>=3B Date: Mon=2C 21 Feb 2011 01:3= > >2:47 +0000
>=3B To: m3commit at elegosoft.com
>=3B From: jkrell at eleg= > >o.de
>=3B Subject: [M3commit] CVS Update: cm3
>=3B
>=3B CVS= > >ROOT: /usr/cvs
>=3B Changes by: jkrell at birch. 11/02/21 01:32:47
>= > >=3B
>=3B Modified files:
>=3B cm3/m3-sys/m3middle/src/: M3Proce= > >ss.i3
>=3B
>=3B Log message:
>=3B remove incorrect commen= > >ts=3B Win32 is not broken
>=3B
> >= > > > >--_d36c70d3-e71b-433d-abdc-06703a929f89_-- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 04:46:04 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 03:46:04 +0000 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20110221003248.11C9C2474005@birch.elegosoft.com>, , , , , , , <20110221011550.0D2791A2078@async.async.caltech.edu>, Message-ID: Ok, it looks like I did break it. http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/os/WIN32/ProcessWin32.m3.diff?r1=1.6;r2=1.7 GetID used to return the handle. I changed it to return the id. m3middle/Process.Abort LOOPHOLEs GetID to a handle. I think the GetID change should probably stay. But certainly one or the other must change. The handle is not unique across the system, just the process, whereas the id is unique across the system. If Process.i3 only mandates within the process, then maybe the old way. Maybe. - Jay From: jay.krell at cornell.edu To: mika at async.caltech.edu Date: Mon, 21 Feb 2011 03:28:10 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] [M3commit] CVS Update: cm3 Wait? No, that is probably just bogus posix thinking. You can just close your handle to the process. You don't have to wait for it to exit. It is somewhat orthogonal. The thing is, the request for Process.Kill, got me looking at when does libm3 close the Process handle and when does it use the handle correctly (i.e. not m3middle's Process.Abort!) It does the CloseHandle in Process.Wait, so if that is never called, there is a leak. Anyway, this should be easy to make correct. (So easy, though, it boggles the mind as to why it was done this way in the first place. I do have to check somethig -- it is SLIGHTLY possible I broke it. I know I fiddled around with the definition of process ids, made them match the underlying platform. I'll check the history. m3middle uses a LOOPHOLE on some "id" to get the "handle". Anyway, I'll check and make it correct.) - Jay > To: jay.krell at cornell.edu > Date: Sun, 20 Feb 2011 17:15:50 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] [M3commit] CVS Update: cm3 > > > Jay I didn't quite follow your previous messages. What is the correct > solution, and what did you mean when you were talking about the garbage > collector? It does seem like---if a Process handle goes out of scope > and gets collected then yes the runtime should do the equivalent of "EVAL > Process.Wait" at some future time. But that seems unrelated/orthogonal > to the issues with the Windows process killing... > > The process killing in m3middle/M3Process I think is only called in > a few places in the compiler and then under what already are dubious > circumstances. For instance I see it done where Process.Create has > failed. But if Process.Create has failed, then there's no process to > kill, so it all seems rather pointless. It is also called as one of > the last actions before exiting the parent process, so there's another > reason for its being pointless. > > In other words I think it's safe to say that if you have an idea for > how to implement process-killing correctly on Windows you can do it > however you like without fear of breaking any existing code... > > Mika > > But I can > > Jay K writes: > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >The code here confuses hProcess and dwProcessId. > >Can we introduce a public function here in libm3 instead of m3middle? > >ie: to avoid answering the question as to how to expose hProcess? > >And to avoid the loophole as well? > > > >Thanks=2C > > - Jay > > > >From: jay.krell at cornell.edu > >To: jkrell at elego.de=3B m3commit at elegosoft.com > >Subject: RE: [M3commit] CVS Update: cm3 > >Date: Mon=2C 21 Feb 2011 00:35:38 +0000 > > > > > > > > > > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= > >rect here. > > > > - Jay > > > >> Date: Mon=2C 21 Feb 2011 01:32:47 +0000 > >> To: m3commit at elegosoft.com > >> From: jkrell at elego.de > >> Subject: [M3commit] CVS Update: cm3 > >>=20 > >> CVSROOT: /usr/cvs > >> Changes by: jkrell at birch. 11/02/21 01:32:47 > >>=20 > >> Modified files: > >> cm3/m3-sys/m3middle/src/: M3Process.i3=20 > >>=20 > >> Log message: > >> remove incorrect comments=3B Win32 is not broken > >>=20 > > = > > > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ > >Content-Type: text/html; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > > > > > > > >The code here confuses hProcess and dwProcessId.
Can we introduce a publ= > >ic function here in libm3 instead of m3middle?
ie: to avoid answering th= > >e question as to how to expose hProcess?
And to avoid the loophole as we= > >ll?

Thanks=2C
 =3B- Jay

From: = > >jay.krell at cornell.edu
To: jkrell at elego.de=3B m3commit at elegosoft.com
S= > >ubject: RE: [M3commit] CVS Update: cm3
Date: Mon=2C 21 Feb 2011 00:35:38= > > +0000

> > > > >"> > > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= > >rect here.

 =3B- Jay

>=3B Date: Mon=2C 21 Feb 2011 01:3= > >2:47 +0000
>=3B To: m3commit at elegosoft.com
>=3B From: jkrell at eleg= > >o.de
>=3B Subject: [M3commit] CVS Update: cm3
>=3B
>=3B CVS= > >ROOT: /usr/cvs
>=3B Changes by: jkrell at birch. 11/02/21 01:32:47
>= > >=3B
>=3B Modified files:
>=3B cm3/m3-sys/m3middle/src/: M3Proce= > >ss.i3
>=3B
>=3B Log message:
>=3B remove incorrect commen= > >ts=3B Win32 is not broken
>=3B
> >= > > > >--_d36c70d3-e71b-433d-abdc-06703a929f89_-- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 04:49:34 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 03:49:34 +0000 Subject: [M3devel] platform-independent kill process In-Reply-To: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> References: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> Message-ID: Yes, please. I'm not sure Terminate should return anything though. In Win32 you would use TerminateProcess(). If you must have the exit code, that would be GetExitCodeProcess. Though the process might still be running and not have an orderly exit code. Oh, and TerminateProcess does have to set the exit code. In m3middle they set it to 1. > We currently only have to consider Unix and Windows, or are there > other active flavours of OSs to care for? I don't think so. There was OS/2, BeOS, Amiga, MacOS < 10.... - Jay > Date: Sun, 20 Feb 2011 13:28:52 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: [M3devel] platform-independent kill process > > I've been thinking a bit about the hanging tests problem in our regression > test framework. I'd like to provide a simple utility to start a program > with time supervision. But as far as I can see, Modula-3 does not provide > a platform-independent way of terminating a process. Of course, on Unix > I can send signals; but what would I do on Windows? A quick scan of the > win32 interfaces in m3core hasn't brought up anything, but I might > have looked for the wrong texts. > > Could we easily extend the Process interface e.g. by > > PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException > > ? > Any opinions about that? Different interface suggestions? > Rather create ProcessExtras than extending a `standard' interface? > > We currently only have to consider Unix and Windows, or are there > other active flavours of OSs to care for? > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Mon Feb 21 05:13:54 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sun, 20 Feb 2011 23:13:54 -0500 Subject: [M3devel] platform-independent kill process In-Reply-To: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> References: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> Message-ID: Olaf: If the regression tests are written in Modula-3, why not use Network Objects to provide an interface that can be called to tell the program to stop? This would be a "graceful" shutdown mechanism, rather than just killing the process at the OS level. I've used this technique before with other types of programs. Regards, Randy -----Original Message----- From: Olaf Wagner [mailto:wagner at elegosoft.com] Sent: Sunday, February 20, 2011 7:29 AM To: m3devel at elegosoft.com Subject: [M3devel] platform-independent kill process I've been thinking a bit about the hanging tests problem in our regression test framework. I'd like to provide a simple utility to start a program with time supervision. But as far as I can see, Modula-3 does not provide a platform-independent way of terminating a process. Of course, on Unix I can send signals; but what would I do on Windows? A quick scan of the win32 interfaces in m3core hasn't brought up anything, but I might have looked for the wrong texts. Could we easily extend the Process interface e.g. by PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException ? Any opinions about that? Different interface suggestions? Rather create ProcessExtras than extending a `standard' interface? We currently only have to consider Unix and Windows, or are there other active flavours of OSs to care for? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Mon Feb 21 05:50:31 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 20:50:31 -0800 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20110221003248.11C9C2474005@birch.elegosoft.com>, , , , <20110221011550.0D2791A2078@async.async.caltech.edu> Message-ID: <20110221045031.1D5D81A2078@async.async.caltech.edu> Jay K writes: >--_0b0e48b4-5c54-4fbb-8adf-2eb48374df09_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Wait? No=2C that is probably just bogus posix thinking. >You can just close your handle to the process. >You don't have to wait for it to exit. Well on Unix I don't think you can do that, can you? You can't tell the system that you're not going to be calling "wait4" on a particular PID... or can you? If not, the portable thing to do is to do what I said... register a GC cleanup such that when a Process.T goes out of scope the runtime does a Process.Wait in the background. It'd work on both Unix and Windows and use existing code rather than introducing a new abstraction. One issue is that with my SIGCHLD modification you will get zombies if you don't call wait4 on Unix. Process.i3 does mention this possibility so I don't think I "broke" anything. (Coding to the specification cuts both ways...) Mika From wagner at elegosoft.com Mon Feb 21 07:49:08 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 21 Feb 2011 07:49:08 +0100 Subject: [M3devel] platform-independent kill process In-Reply-To: References: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> Message-ID: <20110221074908.jndwjzx2bqcoo880@mail.elegosoft.com> Quoting "Coleburn, Randy" : > Olaf: > > If the regression tests are written in Modula-3, why not use Network > Objects to provide an interface that can be called to tell the > program to stop? Baucse that involves too many other sub-systems and components. I'd like to keep the requirements for tests limited. When we test such basic things like threading, we shouldn't have to assume that the network subsystem and RPC are working well, too. > This would be a "graceful" shutdown mechanism, rather than just > killing the process at the OS level. I'm not sure that it would work in the presence of threading problems, but I may be mistaken there. > I've used this technique before with other types of programs. Nothing speaking against that, if your application needs all the functionality anyway. However, I think M3 should support a basic mechanism on process level. Olaf > Regards, > Randy > > -----Original Message----- > From: Olaf Wagner [mailto:wagner at elegosoft.com] > Sent: Sunday, February 20, 2011 7:29 AM > To: m3devel at elegosoft.com > Subject: [M3devel] platform-independent kill process > > I've been thinking a bit about the hanging tests problem in our regression > test framework. I'd like to provide a simple utility to start a program > with time supervision. But as far as I can see, Modula-3 does not provide > a platform-independent way of terminating a process. Of course, on Unix > I can send signals; but what would I do on Windows? A quick scan of the > win32 interfaces in m3core hasn't brought up anything, but I might > have looked for the wrong texts. > > Could we easily extend the Process interface e.g. by > > PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException > > ? > Any opinions about that? Different interface suggestions? > Rather create ProcessExtras than extending a `standard' interface? > > We currently only have to consider Unix and Windows, or are there > other active flavours of OSs to care for? > > Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Mon Feb 21 12:12:10 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 11:12:10 +0000 Subject: [M3devel] libm3/Process.ExitCode? Message-ID: TYPE ExitCode = [0 .. 16_7FFFFFFF]; (* An exit code (or status) of zero normally means successful termination, and a non-zero value normally indicates an error, but the exact conventions vary between systems and programs. *) Is there a good reason this isn't a full INTEGER? Or at least a full Cstdint.int32_t? In the Win32 case, we mask the actual exit code with this, which will lead to the non-zero 0x80000000 becoming zero. I'm inclined to change it like: before: RETURN Word.And(status, LAST(ExitCode)) after: (* for some reason, cut off the high bit, but * if that turns non-zero into zero, then still return non-zero *) maskedStatus := Word.And(status, LAST(ExitCode)); IF status <> 0 and maskedStatus = 0 THEN maskedStatus := 1; END; RETURN maskedStatus Or change ExitCode to plain lossless INTEGER. But maybe negative values were avoided for some reason? I understand that on Posix, I think, you just get an 8 bit unsigned value. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 13:00:14 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 12:00:14 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , <4D604FA3.3060708@lcwb.coop>, , <20110219232729.53A991A2078@async.async.caltech.edu>, , <4D615643.7060107@lcwb.coop>, , <20110220183746.195C11A2078@async.async.caltech.edu>, Message-ID: Ok, I'm disappointed to report that I did some quick checking on Darwin/x86, Linux/x86, Solaris/sparc32/x86 and none of them seem to be smart about this. Er, less they were being really smart. int F1(void) { volatile char a[4000]; return a[3999]; } int F2(void) { volatile char a[6000]; return a[5999]; } int F4(void) { volatile char a[60000]; return a[59999]; } I guess I should redo all the tests but accessing [0] also/instead. So, yes, we should definitely consider doing something. However...however, the frontend doesn't know which locals wiil be optimized away. The backend is setup to do this...but it is only used for NT systems, darn. jbook2:gcc jay$ grep CHECK_STACK_LIMIT */*/*/* gcc/config/i386/cygming.h:#define CHECK_STACK_LIMIT 4000 gcc/config/i386/cygwin.asm: than CHECK_STACK_LIMIT bytes in one go. Touching the stack at 4K gcc/config/i386/i386-interix.h:#define CHECK_STACK_LIMIT 0x1000 gcc/config/i386/i386.c:#ifndef CHECK_STACK_LIMIT gcc/config/i386/i386.c:#define CHECK_STACK_LIMIT (-1) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT)) gcc/config/i386/i386.c: else if (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT))) gcc/config/i386/i386.md:#ifndef CHECK_STACK_LIMIT gcc/config/i386/i386.md:#define CHECK_STACK_LIMIT 0 gcc/config/i386/i386.md: if (CHECK_STACK_LIMIT && CONST_INT_P (operands[1]) gcc/config/i386/i386.md: && INTVAL (operands[1]) < CHECK_STACK_LIMIT) jbook2:gcc jay$ grep CHECK_STACK_LIMIT */*/*/*/* jbook2:gcc jay$ grep TARGET_STACK_PROBE * jbook2:gcc jay$ grep TARGET_STACK_PROBE */* gcc/ChangeLog: * config/i386/i386.c (ix86_expand_prologue) [TARGET_STACK_PROBE]: gcc/ChangeLog-2005: (TARGET_TLS_DIRECT_SEG_REFS, TARGET_STACK_PROBE) jbook2:gcc jay$ grep TARGET_STACK_PROBE */*/* jbook2:gcc jay$ grep TARGET_STACK_PROBE */*/*/* gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT)) gcc/config/i386/i386.c: else if (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT))) gcc/config/i386/i386.md: "!TARGET_64BIT && TARGET_STACK_PROBE" gcc/config/i386/i386.md: "TARGET_64BIT && TARGET_STACK_PROBE" gcc/config/i386/i386.md: "TARGET_STACK_PROBE" - Jay From: jay.krell at cornell.edu To: mika at async.caltech.edu; rodney_bates at lcwb.coop Date: Sun, 20 Feb 2011 22:38:19 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] SEGV mapping to RuntimeError Probably for this reason: There is a requirement on NT that stack pages be touched in order. Functions with locals totally more than 4K call _chkstk (aka _alloca) to allocate their stack, instead of the usual register subtraction. It contains a loop that touches a byte every 4K. Otherwise, if you have lots of functions with small frames, the stack is touched by virtue of the call function pushing the return address. Modula-3 has long failed to uphold this contract, and still does. I've never seen it clearly documented, but you can see it is what the C compiler does. I had thought there were other reasons for this behavior. I thought you actually get an exception of the stack is touched out of order ("the first time"). But I think I did an experiment long ago with Modula-3 and there was no exception. I don't know about other platforms. I've also seen evidence in gcc and/or Target.i3 that compiler writers know well about such mechanisms -- there being a constant to set as to what size locals trigger special behavior. But m3back doesn't do anything here. Probably we could use _chkstk unconditonally as well -- need to see what it does for numbers smaller than 4K. But that'd be a deoptimization in the common case, and deoptimizing only as necessary should be easy enough. - Jay > To: rodney_bates at lcwb.coop > Date: Sun, 20 Feb 2011 10:37:46 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > On a 64-bit machine, at least, there ought to be enough virtual > memory that you could just have a gap between thread stacks big > enough to allow for a protection area larger than the largest possible > (implementation-defined) activation record, no? I know I've run into > trouble with very large activation records in the past (and not because > I was running out of stack space, either). > > Or at least a procedure with a very large activation record (or > a procedure calling it) could be required to call some sort of check > routine "EnoughStackSpaceRemaining()" before starting to scribble > on the activation record? > > Also the end of the activation record must be written to at least once, > or else the memory protection won't be triggered. > > In any case if this is done properly the same mechanism I proposed for > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > signals are delivered on a separate stack. If signals are delivered on > the same stack, the signal handler would get nastier, it would have to > make space through some manipulations (maybe temporarily unporotecting > the redzone page?) for its own purposes... but I don't see why it > couldn't be done. > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > activated inside the redzone page because of a difference in signal > handling..? I remember reading something about sigaltstack... > > I would of course love to be able to recover from stack overflow, too. > In some sense, since it's a generally unknown limit, it's even less of > a fatal error than a NIL dereference (hence makes even more sense to > catch it). > > Mika > > "Rodney M. Bates" writes: > >I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >Probably a fully protected guard page at the end of the stack. This technique > >always worries me a bit because a procedure with a really big activation record > >could jump right past it. Probably it would almost always access the first page > >of the big area before storing anything into later pages. > > > >On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >> Ah, yes, stack protection. > >> > >> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >> > >> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >> > >> time ../AMD64_FREEBSD/stubexample > >> M-Scheme Experimental > >> LITHP ITH LITHENING. > >>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >> f > >>> (f 0) > >> Illegal instruction > >> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >> > >> What absolutely must not happen, of course, is that the runtime hangs > >> while executing only safe code... > >> > >> Mika > >> > >> "Rodney M. Bates" writes: > >>> I know of one other place the compilers rely on hardware memory protection > >>> to detect a checked runtime error, and that is stack overflow. This won't > >>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>> This could probably be distinguished after the fact by some low-level, > >>> target-dependent code. I have found it by looking at assembly code at > >>> the point of failure--usually right after a stack pointer push. > >>> > >>> Detecting this via compiler-generated checks would probably be more > >>> extravagant than many other checks, as it is so frequent. I am not > >>> aware of any really good solution to this in any implementation of any > >>> language. > >>> > >>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>> you know! > >>>> > >>>> "Corrupting the heap" is something that can only happen as a result of > >>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>> modules not marked UNSAFE. > >>>> > >>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>> to signal a certain kind of *checked* runtime error, namely, the > >>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>> > >>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>> current CM3 are ones from NIL dereferences. > >>>> > >>>> Hence, as long as you stick with safe code, the only time the code I > >>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>> recoverable. > >>>> > >>>> :-) > >>>> > >>>> Mika > >>>> > >>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>> Almost all what remains is glue to some C library, which wouldn't be > >>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>> I don't have the time to rewrite every single library from scratch myself. > >>>> > >>>> > >>>> Jay K writes: > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious. > >>>>> Imagine the heap is corrupted. > >>>>> And then you run more code. > >>>>> And the code happens to call malloc. > >>>>> Or printf to log something. > >>>>> =20 > >>>>> I suppose there might be an application that maps memory > >>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>> =20 > >>>>> - Jay > >>>>> =20 > >>>>>> To: m3devel at elegosoft.com > >>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>> From: mika at async.caltech.edu > >>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>> =20 > >>>>>> =20 > >>>>>> Dear m3devel=2C > >>>>>> =20 > >>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>> the unhelpful interface and module Crash: > >>>>>> =20 > >>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>> =20 > >>>>>> MODULE Crash=3B > >>>>>> =20 > >>>>>> PROCEDURE Me() =3D > >>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>> ptr^ :=3D 0 > >>>>>> END Me=3B > >>>>>> =20 > >>>>>> BEGIN END Crash. > >>>>>> =20 > >>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>> e > >>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>> =20 > >>>>>> M-Scheme Experimental > >>>>>> LITHP ITH LITHENING. > >>>>>>> (require-modules "m3") > >>>>>> #t > >>>>>>> (Crash.Me) > >>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>> (+ 3 4)=20 > >>>>>> 7 > >>>>>>> =20 > >>>>>> =20 > >>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>> heck it.=20 > >>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>> =20 > >>>>>> Mika > >>>>>> =20 > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>> Imagine the heap =3Bis corrupted.
> >>>>> And then you run more code.
> >>>>> And the code happens to call malloc.
> >>>>> Or printf to log something.
> >>>>>  =3B
> >>>>> I suppose there might be an application that maps memory
> >>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>  =3B
> >>>>>  =3B- Jay
 =3B
> >>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>> > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>> > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 13:05:33 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 12:05:33 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, ,,, ,,<20110219203821.2016B1A2078@async.async.caltech.edu>, ,,<4D604FA3.3060708@lcwb.coop>, ,,<20110219232729.53A991A2078@async.async.caltech.edu>, ,,<4D615643.7060107@lcwb.coop>, , , <20110220183746.195C11A2078@async.async.caltech.edu>, , , Message-ID: Hm. Maybe we just need to pass -fstack-check to the backend. Needs experimentation... - Jay From: jay.krell at cornell.edu To: mika at async.caltech.edu; rodney_bates at lcwb.coop Date: Mon, 21 Feb 2011 12:00:14 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] SEGV mapping to RuntimeError Ok, I'm disappointed to report that I did some quick checking on Darwin/x86, Linux/x86, Solaris/sparc32/x86 and none of them seem to be smart about this. Er, less they were being really smart. int F1(void) { volatile char a[4000]; return a[3999]; } int F2(void) { volatile char a[6000]; return a[5999]; } int F4(void) { volatile char a[60000]; return a[59999]; } I guess I should redo all the tests but accessing [0] also/instead. So, yes, we should definitely consider doing something. However...however, the frontend doesn't know which locals wiil be optimized away. The backend is setup to do this...but it is only used for NT systems, darn. jbook2:gcc jay$ grep CHECK_STACK_LIMIT */*/*/* gcc/config/i386/cygming.h:#define CHECK_STACK_LIMIT 4000 gcc/config/i386/cygwin.asm: than CHECK_STACK_LIMIT bytes in one go. Touching the stack at 4K gcc/config/i386/i386-interix.h:#define CHECK_STACK_LIMIT 0x1000 gcc/config/i386/i386.c:#ifndef CHECK_STACK_LIMIT gcc/config/i386/i386.c:#define CHECK_STACK_LIMIT (-1) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT)) gcc/config/i386/i386.c: else if (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT))) gcc/config/i386/i386.md:#ifndef CHECK_STACK_LIMIT gcc/config/i386/i386.md:#define CHECK_STACK_LIMIT 0 gcc/config/i386/i386.md: if (CHECK_STACK_LIMIT && CONST_INT_P (operands[1]) gcc/config/i386/i386.md: && INTVAL (operands[1]) < CHECK_STACK_LIMIT) jbook2:gcc jay$ grep CHECK_STACK_LIMIT */*/*/*/* jbook2:gcc jay$ grep TARGET_STACK_PROBE * jbook2:gcc jay$ grep TARGET_STACK_PROBE */* gcc/ChangeLog: * config/i386/i386.c (ix86_expand_prologue) [TARGET_STACK_PROBE]: gcc/ChangeLog-2005: (TARGET_TLS_DIRECT_SEG_REFS, TARGET_STACK_PROBE) jbook2:gcc jay$ grep TARGET_STACK_PROBE */*/* jbook2:gcc jay$ grep TARGET_STACK_PROBE */*/*/* gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT)) gcc/config/i386/i386.c: else if (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT))) gcc/config/i386/i386.md: "!TARGET_64BIT && TARGET_STACK_PROBE" gcc/config/i386/i386.md: "TARGET_64BIT && TARGET_STACK_PROBE" gcc/config/i386/i386.md: "TARGET_STACK_PROBE" - Jay From: jay.krell at cornell.edu To: mika at async.caltech.edu; rodney_bates at lcwb.coop Date: Sun, 20 Feb 2011 22:38:19 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] SEGV mapping to RuntimeError Probably for this reason: There is a requirement on NT that stack pages be touched in order. Functions with locals totally more than 4K call _chkstk (aka _alloca) to allocate their stack, instead of the usual register subtraction. It contains a loop that touches a byte every 4K. Otherwise, if you have lots of functions with small frames, the stack is touched by virtue of the call function pushing the return address. Modula-3 has long failed to uphold this contract, and still does. I've never seen it clearly documented, but you can see it is what the C compiler does. I had thought there were other reasons for this behavior. I thought you actually get an exception of the stack is touched out of order ("the first time"). But I think I did an experiment long ago with Modula-3 and there was no exception. I don't know about other platforms. I've also seen evidence in gcc and/or Target.i3 that compiler writers know well about such mechanisms -- there being a constant to set as to what size locals trigger special behavior. But m3back doesn't do anything here. Probably we could use _chkstk unconditonally as well -- need to see what it does for numbers smaller than 4K. But that'd be a deoptimization in the common case, and deoptimizing only as necessary should be easy enough. - Jay > To: rodney_bates at lcwb.coop > Date: Sun, 20 Feb 2011 10:37:46 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > On a 64-bit machine, at least, there ought to be enough virtual > memory that you could just have a gap between thread stacks big > enough to allow for a protection area larger than the largest possible > (implementation-defined) activation record, no? I know I've run into > trouble with very large activation records in the past (and not because > I was running out of stack space, either). > > Or at least a procedure with a very large activation record (or > a procedure calling it) could be required to call some sort of check > routine "EnoughStackSpaceRemaining()" before starting to scribble > on the activation record? > > Also the end of the activation record must be written to at least once, > or else the memory protection won't be triggered. > > In any case if this is done properly the same mechanism I proposed for > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > signals are delivered on a separate stack. If signals are delivered on > the same stack, the signal handler would get nastier, it would have to > make space through some manipulations (maybe temporarily unporotecting > the redzone page?) for its own purposes... but I don't see why it > couldn't be done. > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > activated inside the redzone page because of a difference in signal > handling..? I remember reading something about sigaltstack... > > I would of course love to be able to recover from stack overflow, too. > In some sense, since it's a generally unknown limit, it's even less of > a fatal error than a NIL dereference (hence makes even more sense to > catch it). > > Mika > > "Rodney M. Bates" writes: > >I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >Probably a fully protected guard page at the end of the stack. This technique > >always worries me a bit because a procedure with a really big activation record > >could jump right past it. Probably it would almost always access the first page > >of the big area before storing anything into later pages. > > > >On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >> Ah, yes, stack protection. > >> > >> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >> > >> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >> > >> time ../AMD64_FREEBSD/stubexample > >> M-Scheme Experimental > >> LITHP ITH LITHENING. > >>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >> f > >>> (f 0) > >> Illegal instruction > >> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >> > >> What absolutely must not happen, of course, is that the runtime hangs > >> while executing only safe code... > >> > >> Mika > >> > >> "Rodney M. Bates" writes: > >>> I know of one other place the compilers rely on hardware memory protection > >>> to detect a checked runtime error, and that is stack overflow. This won't > >>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>> This could probably be distinguished after the fact by some low-level, > >>> target-dependent code. I have found it by looking at assembly code at > >>> the point of failure--usually right after a stack pointer push. > >>> > >>> Detecting this via compiler-generated checks would probably be more > >>> extravagant than many other checks, as it is so frequent. I am not > >>> aware of any really good solution to this in any implementation of any > >>> language. > >>> > >>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>> you know! > >>>> > >>>> "Corrupting the heap" is something that can only happen as a result of > >>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>> modules not marked UNSAFE. > >>>> > >>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>> to signal a certain kind of *checked* runtime error, namely, the > >>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>> > >>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>> current CM3 are ones from NIL dereferences. > >>>> > >>>> Hence, as long as you stick with safe code, the only time the code I > >>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>> recoverable. > >>>> > >>>> :-) > >>>> > >>>> Mika > >>>> > >>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>> Almost all what remains is glue to some C library, which wouldn't be > >>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>> I don't have the time to rewrite every single library from scratch myself. > >>>> > >>>> > >>>> Jay K writes: > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious. > >>>>> Imagine the heap is corrupted. > >>>>> And then you run more code. > >>>>> And the code happens to call malloc. > >>>>> Or printf to log something. > >>>>> =20 > >>>>> I suppose there might be an application that maps memory > >>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>> =20 > >>>>> - Jay > >>>>> =20 > >>>>>> To: m3devel at elegosoft.com > >>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>> From: mika at async.caltech.edu > >>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>> =20 > >>>>>> =20 > >>>>>> Dear m3devel=2C > >>>>>> =20 > >>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>> the unhelpful interface and module Crash: > >>>>>> =20 > >>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>> =20 > >>>>>> MODULE Crash=3B > >>>>>> =20 > >>>>>> PROCEDURE Me() =3D > >>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>> ptr^ :=3D 0 > >>>>>> END Me=3B > >>>>>> =20 > >>>>>> BEGIN END Crash. > >>>>>> =20 > >>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>> e > >>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>> =20 > >>>>>> M-Scheme Experimental > >>>>>> LITHP ITH LITHENING. > >>>>>>> (require-modules "m3") > >>>>>> #t > >>>>>>> (Crash.Me) > >>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>> (+ 3 4)=20 > >>>>>> 7 > >>>>>>> =20 > >>>>>> =20 > >>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>> heck it.=20 > >>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>> =20 > >>>>>> Mika > >>>>>> =20 > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>> Imagine the heap =3Bis corrupted.
> >>>>> And then you run more code.
> >>>>> And the code happens to call malloc.
> >>>>> Or printf to log something.
> >>>>>  =3B
> >>>>> I suppose there might be an application that maps memory
> >>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>  =3B
> >>>>>  =3B- Jay
 =3B
> >>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>> > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>> > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Tue Feb 22 01:01:12 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Mon, 21 Feb 2011 16:01:12 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> Message-ID: <20110222000112.5A7131A2078@async.async.caltech.edu> There is in fact a small problem with my change to the way SIGSEGV is handled. It now appears that bad memory references are all happening in ThreadPosix.m3 (i.e., we've lost some of the debugging help of the error message). I'm not quite sure how to fix this although I'm sure all the bits are around. My guess is that instead of performing RAISE RuntimeError.E in the thread switcher, the code ought to call RTHooks.Raise with an argument derived from the faulting address as reported to the signal handler. This is probably quite easy, but does anyone know the right routines to call? One would need to track down the routine that faulted as well as the RT0 record corresponding to the exception RuntimeError.E. Mika Jay K writes: >Letting any code run after a SIGSEGV is dubious. >Imagine the heap is corrupted. >And then you run more code. >And the code happens to call malloc. >Or printf to log something. >=20 >I suppose there might be an application that maps memory >gradually=2C as pieces of a buffer are hit. Might. From dabenavidesd at yahoo.es Tue Feb 22 02:07:45 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Tue, 22 Feb 2011 01:07:45 +0000 (GMT) Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222000112.5A7131A2078@async.async.caltech.edu> Message-ID: <533385.35225.qm@web29704.mail.ird.yahoo.com> Hi all: although this is not of my full understanding, CM3 JVM was supposed to handle in such a way the exceptions of jvm ?-op fault script programs, given that: there was a runtime above the CM3 runtime and besides having its own spaces for stack, thread stacks and runtime machinery, loader, etc. Please just to advert this is and I guess was the case for CM3 JVM, if so, let the system handle it in that way, why do you need to fully manage to handle that as in Modula-3 runtime per se, my believe is that you could do that for the language environment routines separately, that's why pragmas are really useful. I don't believe it allowed to signal like in Modula-3 terms threads but you could for instance lock a thread until you have time in the scheduler from Modula-3, still this is more flexible nowadays I believe with multiprocessors, etc. Similarly SPIN used or performed protection mechanism, like domain level, and at language level the heap freezes when handling untrusted code (even extra checked array bounds violation faults in non-Modula-3 code), so you have same runtime I believe for doing that too in Modula-3, again, is just another time of problem arises, perhaps this can be tried for thread stacks, etc and in kernel threads well as something alike, I really think could be done. Thanks in advance --- El lun, 21/2/11, Mika Nystrom escribi?: > De: Mika Nystrom > Asunto: Re: [M3devel] SEGV mapping to RuntimeError > Para: "Jay K" , m3devel at elegosoft.com > Fecha: lunes, 21 de febrero, 2011 19:01 > > There is in fact a small problem with my change to the way > SIGSEGV is > handled. It now appears that bad memory references > are all happening > in ThreadPosix.m3 (i.e., we've lost some of the debugging > help of the > error message). > > I'm not quite sure how to fix this although I'm sure all > the bits are > around. My guess is that instead of performing RAISE > RuntimeError.E > in the thread switcher, the code ought to call > RTHooks.Raise with an > argument derived from the faulting address as reported to > the signal > handler. This is probably quite easy, but does anyone > know the > right routines to call? One would need to track down > the routine that > faulted as well as the RT0 record corresponding to the > exception > RuntimeError.E. > > Mika > > Jay K writes: > >Letting any code run after a SIGSEGV is dubious. > >Imagine the heap is corrupted. > >And then you run more code. > >And the code happens to call malloc. > >Or printf to log something. > >=20 > >I suppose there might be an application that maps > memory > >gradually=2C as pieces of a buffer are hit. Might. > From mika at async.caltech.edu Tue Feb 22 03:12:00 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Mon, 21 Feb 2011 18:12:00 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <533385.35225.qm@web29704.mail.ird.yahoo.com> References: <533385.35225.qm@web29704.mail.ird.yahoo.com> Message-ID: <20110222021200.9F0EE1A2078@async.async.caltech.edu> There are several issues here and yes this the RuntimeError.E exception is taken either from SPIN or the CM3 JVM, I don't know which. I only understand the user threads code so I have made that raise the RuntimeError.E instead of dumping core on a segfault. I think this is what the CM JVM was supposed to do as well---but honestly I can't tell if that work was ever completed. They do support mapping various other runtime errors (subrange mismatch, e.g.) into exceptions, and that works well. However the segmentation fault mapping was not functional: there was no other way of handling it than dumping core on the whole M3 environment and runtime before my changes last week. My last email was about a bug in my changes---not in the actual functionality but the fact that if you *don't* catch the RuntimeError.E exception that results from a bad memory reference, the runtime quits with a slightly confusing error message, apparently implicating the threading system when the bug is actually in normal, non-UNSAFE code. Mika "Daniel Alejandro Benavides D." writes: >Hi all: >although this is not of my full understanding, CM3 JVM was supposed to hand= >le in such a way the exceptions of jvm =B5-op fault script programs, given = >that: there was a runtime above the CM3 runtime and besides having its own = >spaces for stack, thread stacks and runtime machinery, loader, etc. >Please just to advert this is and I guess was the case for CM3 JVM, if so, = >let the system handle it in that way, why do you need to fully manage to ha= >ndle that as in Modula-3 runtime per se, my believe is that you could do th= >at for the language environment routines separately, that's why pragmas are= > really useful. I don't believe it allowed to signal like in Modula-3 terms= > threads but you could for instance lock a thread until you have time in th= >e scheduler from Modula-3, still this is more flexible nowadays I believe w= >ith multiprocessors, etc. >Similarly SPIN used or performed protection mechanism, like domain level, a= >nd at language level the heap freezes when handling untrusted code (even ex= >tra checked array bounds violation faults in non-Modula-3 code), so you hav= >e same runtime I believe for doing that too in Modula-3, again, is just ano= >ther time of problem arises, perhaps this can be tried for thread stacks, e= >tc and in kernel threads well as something alike, I really think could be d= >one. >Thanks in advance > >--- El lun, 21/2/11, Mika Nystrom escribi=F3: > >> De: Mika Nystrom >> Asunto: Re: [M3devel] SEGV mapping to RuntimeError >> Para: "Jay K" , m3devel at elegosoft.com >> Fecha: lunes, 21 de febrero, 2011 19:01 >>=20 >> There is in fact a small problem with my change to the way >> SIGSEGV is >> handled. It now appears that bad memory references >> are all happening >> in ThreadPosix.m3 (i.e., we've lost some of the debugging >> help of the >> error message). >>=20 >> I'm not quite sure how to fix this although I'm sure all >> the bits are >> around. My guess is that instead of performing RAISE >> RuntimeError.E >> in the thread switcher, the code ought to call >> RTHooks.Raise with an >> argument derived from the faulting address as reported to >> the signal >> handler. This is probably quite easy, but does anyone >> know the=20 >> right routines to call? One would need to track down >> the routine that >> faulted as well as the RT0 record corresponding to the >> exception >> RuntimeError.E. >>=20 >> Mika >>=20 >> Jay K writes: >> >Letting any code run after a SIGSEGV is dubious. >> >Imagine the heap is corrupted. >> >And then you run more code. >> >And the code happens to call malloc. >> >Or printf to log something. >> >=3D20 >> >I suppose there might be an application that maps >> memory >> >gradually=3D2C as pieces of a buffer are hit. Might. >> =0A=0A=0A From dabenavidesd at yahoo.es Tue Feb 22 05:31:58 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Tue, 22 Feb 2011 04:31:58 +0000 (GMT) Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222021200.9F0EE1A2078@async.async.caltech.edu> Message-ID: <806751.73807.qm@web29712.mail.ird.yahoo.com> Hi all: yes I see, things are harder than I thought here but still seems the problem is not directly in the runtime environment system, perhaps this is more just another time of System native restrictions: according to this guys: http://grsecurity.net/confighelp.php/ (see for Automatically emulate sigreturn trampolines) that's why perhaps the SPIN could do that without requiring the process to terminate which is otherwise is very odd I believe for such restrictions, still need to check more details: Interesting this emails go into the details of more advanced programs to analyze ine ach platform the runtime behavior many years ago, look for Hp-UX implementation "bug". Again grsecurity OS doers talk again even of (see CVSup): http://pax.grsecurity.net/docs/randexec.txt I hope it helps a bit, thanks in advance --- El lun, 21/2/11, Mika Nystrom escribi?: > De: Mika Nystrom > Asunto: Re: [M3devel] SEGV mapping to RuntimeError > Para: "Daniel Alejandro Benavides D." > CC: m3devel at elegosoft.com > Fecha: lunes, 21 de febrero, 2011 21:12 > > There are several issues here and yes this the > RuntimeError.E exception > is taken either from SPIN or the CM3 JVM, I don't know > which. > > I only understand the user threads code so I have made that > raise the > RuntimeError.E instead of dumping core on a segfault. > I think this is what > the CM JVM was supposed to do as well---but honestly I > can't tell if that > work was ever completed. They do support mapping > various other runtime > errors (subrange mismatch, e.g.) into exceptions, and that > works well. > However the segmentation fault mapping was not functional: > there was no > other way of handling it than dumping core on the whole M3 > environment > and runtime before my changes last week. > > My last email was about a bug in my changes---not in the > actual functionality > but the fact that if you *don't* catch the RuntimeError.E > exception that > results from a bad memory reference, the runtime quits with > a slightly > confusing error message, apparently implicating the > threading system when > the bug is actually in normal, non-UNSAFE code. > > Mika > > "Daniel Alejandro Benavides D." writes: > >Hi all: > >although this is not of my full understanding, CM3 JVM > was supposed to hand= > >le in such a way the exceptions of jvm =B5-op fault > script programs, given = > >that: there was a runtime above the CM3 runtime and > besides having its own = > >spaces for stack, thread stacks and runtime machinery, > loader, etc. > >Please just to advert this is and I guess was the case > for CM3 JVM, if so, = > >let the system handle it in that way, why do you need > to fully manage to ha= > >ndle that as in Modula-3 runtime per se, my believe is > that you could do th= > >at for the language environment routines separately, > that's why pragmas are= > > really useful. I don't believe it allowed to signal > like in Modula-3 terms= > > threads but you could for instance lock a thread until > you have time in th= > >e scheduler from Modula-3, still this is more flexible > nowadays I believe w= > >ith multiprocessors, etc. > >Similarly SPIN used or performed protection mechanism, > like domain level, a= > >nd at language level the heap freezes when handling > untrusted code (even ex= > >tra checked array bounds violation faults in > non-Modula-3 code), so you hav= > >e same runtime I believe for doing that too in > Modula-3, again, is just ano= > >ther time of problem arises, perhaps this can be tried > for thread stacks, e= > >tc and in kernel threads well as something alike, I > really think could be d= > >one. > >Thanks in advance > > > >--- El lun, 21/2/11, Mika Nystrom > escribi=F3: > > > >> De: Mika Nystrom > >> Asunto: Re: [M3devel] SEGV mapping to > RuntimeError > >> Para: "Jay K" , > m3devel at elegosoft.com > >> Fecha: lunes, 21 de febrero, 2011 19:01 > >>=20 > >> There is in fact a small problem with my change to > the way > >> SIGSEGV is > >> handled. It now appears that bad memory > references > >> are all happening > >> in ThreadPosix.m3 (i.e., we've lost some of the > debugging > >> help of the > >> error message). > >>=20 > >> I'm not quite sure how to fix this although I'm > sure all > >> the bits are > >> around. My guess is that instead of > performing RAISE > >> RuntimeError.E > >> in the thread switcher, the code ought to call > >> RTHooks.Raise with an > >> argument derived from the faulting address as > reported to > >> the signal > >> handler. This is probably quite easy, but > does anyone > >> know the=20 > >> right routines to call? One would need to > track down > >> the routine that > >> faulted as well as the RT0 record corresponding to > the > >> exception > >> RuntimeError.E. > >>=20 > >> Mika > >>=20 > >> Jay K writes: > >> >Letting any code run after a SIGSEGV is > dubious. > >> >Imagine the heap is corrupted. > >> >And then you run more code. > >> >And the code happens to call malloc. > >> >Or printf to log something. > >> >=3D20 > >> >I suppose there might be an application that > maps > >> memory > >> >gradually=3D2C as pieces of a buffer are hit. > Might. > >> =0A=0A=0A > From dabenavidesd at yahoo.es Tue Feb 22 05:40:34 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Tue, 22 Feb 2011 04:40:34 +0000 (GMT) Subject: [M3devel] SEGV mapping to RuntimeError Message-ID: <748970.47663.qm@web29719.mail.ird.yahoo.com> Hi all: sorry missed the second link in the comments about HP--UX and Nil dereferences: http://www.usenix.org/publications/library/proceedings/sd96/full_papers/small.txt Still they seemed to be some option in the compiler to check where platform needed to catch the RuntimeError either from compiler inserted checks for NIL and from trapped OS call or just ignore it in some circunstances like HP-UX case, see for that it. Thanks in advance --- El lun, 21/2/11, Daniel Alejandro Benavides D. escribi?: > De: Daniel Alejandro Benavides D. > Asunto: Re: [M3devel] SEGV mapping to RuntimeError > Para: "Mika Nystrom" > CC: m3devel at elegosoft.com > Fecha: lunes, 21 de febrero, 2011 23:31 > Hi all: > yes I see, things are harder than I thought here but still > seems the problem is not directly in the runtime environment > system, perhaps this is more just another time of System > native restrictions: > according to this guys: > http://grsecurity.net/confighelp.php/ > (see for Automatically emulate sigreturn trampolines) > that's why perhaps the SPIN could do that without requiring > the process to terminate which is otherwise is very odd I > believe for such restrictions, still need to check more > details: > Interesting this emails go into the details of more > advanced programs to analyze ine ach platform the runtime > behavior many years ago, look for Hp-UX implementation > "bug". > Again grsecurity OS doers talk again even of (see CVSup): > http://pax.grsecurity.net/docs/randexec.txt > I hope it helps a bit, thanks in advance > > --- El lun, 21/2/11, Mika Nystrom > escribi?: > > > De: Mika Nystrom > > Asunto: Re: [M3devel] SEGV mapping to RuntimeError > > Para: "Daniel Alejandro Benavides D." > > CC: m3devel at elegosoft.com > > Fecha: lunes, 21 de febrero, 2011 21:12 > > > > There are several issues here and yes this the > > RuntimeError.E exception > > is taken either from SPIN or the CM3 JVM, I don't > know > > which. > > > > I only understand the user threads code so I have made > that > > raise the > > RuntimeError.E instead of dumping core on a segfault. > > > I think this is what > > the CM JVM was supposed to do as well---but honestly > I > > can't tell if that > > work was ever completed. They do support > mapping > > various other runtime > > errors (subrange mismatch, e.g.) into exceptions, and > that > > works well. > > However the segmentation fault mapping was not > functional: > > there was no > > other way of handling it than dumping core on the > whole M3 > > environment > > and runtime before my changes last week. > > > > My last email was about a bug in my changes---not in > the > > actual functionality > > but the fact that if you *don't* catch the > RuntimeError.E > > exception that > > results from a bad memory reference, the runtime quits > with > > a slightly > > confusing error message, apparently implicating the > > threading system when > > the bug is actually in normal, non-UNSAFE code. > > > > Mika > > > > "Daniel Alejandro Benavides D." writes: > > >Hi all: > > >although this is not of my full understanding, CM3 > JVM > > was supposed to hand= > > >le in such a way the exceptions of jvm =B5-op > fault > > script programs, given = > > >that: there was a runtime above the CM3 runtime > and > > besides having its own = > > >spaces for stack, thread stacks and runtime > machinery, > > loader, etc. > > >Please just to advert this is and I guess was the > case > > for CM3 JVM, if so, = > > >let the system handle it in that way, why do you > need > > to fully manage to ha= > > >ndle that as in Modula-3 runtime per se, my > believe is > > that you could do th= > > >at for the language environment routines > separately, > > that's why pragmas are= > > > really useful. I don't believe it allowed to > signal > > like in Modula-3 terms= > > > threads but you could for instance lock a thread > until > > you have time in th= > > >e scheduler from Modula-3, still this is more > flexible > > nowadays I believe w= > > >ith multiprocessors, etc. > > >Similarly SPIN used or performed protection > mechanism, > > like domain level, a= > > >nd at language level the heap freezes when > handling > > untrusted code (even ex= > > >tra checked array bounds violation faults in > > non-Modula-3 code), so you hav= > > >e same runtime I believe for doing that too in > > Modula-3, again, is just ano= > > >ther time of problem arises, perhaps this can be > tried > > for thread stacks, e= > > >tc and in kernel threads well as something alike, > I > > really think could be d= > > >one. > > >Thanks in advance > > > > > >--- El lun, 21/2/11, Mika Nystrom > > escribi=F3: > > > > > >> De: Mika Nystrom > > >> Asunto: Re: [M3devel] SEGV mapping to > > RuntimeError > > >> Para: "Jay K" , > > m3devel at elegosoft.com > > >> Fecha: lunes, 21 de febrero, 2011 19:01 > > >>=20 > > >> There is in fact a small problem with my > change to > > the way > > >> SIGSEGV is > > >> handled. It now appears that bad > memory > > references > > >> are all happening > > >> in ThreadPosix.m3 (i.e., we've lost some of > the > > debugging > > >> help of the > > >> error message). > > >>=20 > > >> I'm not quite sure how to fix this although > I'm > > sure all > > >> the bits are > > >> around. My guess is that instead of > > performing RAISE > > >> RuntimeError.E > > >> in the thread switcher, the code ought to > call > > >> RTHooks.Raise with an > > >> argument derived from the faulting address > as > > reported to > > >> the signal > > >> handler. This is probably quite easy, > but > > does anyone > > >> know the=20 > > >> right routines to call? One would need > to > > track down > > >> the routine that > > >> faulted as well as the RT0 record > corresponding to > > the > > >> exception > > >> RuntimeError.E. > > >>=20 > > >> Mika > > >>=20 > > >> Jay K writes: > > >> >Letting any code run after a SIGSEGV is > > dubious. > > >> >Imagine the heap is corrupted. > > >> >And then you run more code. > > >> >And the code happens to call malloc. > > >> >Or printf to log something. > > >> >=3D20 > > >> >I suppose there might be an application > that > > maps > > >> memory > > >> >gradually=3D2C as pieces of a buffer are > hit. > > Might. > > >> =0A=0A=0A > > > > > > From mika at async.caltech.edu Tue Feb 22 18:44:05 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Tue, 22 Feb 2011 09:44:05 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <4D61CC27.7070704@lcwb.coop> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> <20110219232729.53A991A2078@async.async.caltech.edu> <4D615643.7060107@lcwb.coop> <20110220183746.195C11A2078@async.async.caltech.edu> <4D61CC27.7070704@lcwb.coop> Message-ID: <20110222174405.E15481A2078@async.async.caltech.edu> Ok so I was thinking about this. Why on earth is stack overflow even a problem? Consider the following procedure call (in my code, stack grows upwards): (* sp at x, pc at y *) y: P(args) z: next_statement decompose as follows: (* sp at x, pc at y *) y: Push(args etc. and ret. address z) Jump(P) z: next_statement Now, we say: y: ok := check_stack(size of frame) IF NOT ok THEN abort() END; Push(args etc. and ret. address z) Jump(P) z: next_statement (note check_stack and the following IF can be implemented by hardware, need not actually be an instruction) Let me change the code a tad: y: ok := check_stack(size of frame) y':IF NOT ok THEN WITH new_stack_bottom = malloc(stack_size) huge_amount = new_stack_bottom - sp DO create_redzone_at(new_stack_bottom+stack_size-redzone_size) EVAL alloca(huge_amount) END END; Push(args etc. and ret. address z) Jump(P) z: IF NOT ok THEN destroy_redzone(...); free(new_stack_bottom) END Note 1. cleanup of redzone could be postponed to return of caller....when alloca in any case has to be cleaned up. Note 2. the test IF NOT ok at z is more expensive to implement than the one at y because you can't really use hardware for it. A hardware callback can be arranged though: VAR ptr := sp; y: ok := check_stack(size of frame) y':IF NOT ok THEN ptr := 0; (* illegal address *) fault_address := z; WITH new_stack_bottom = malloc(stack_size) huge_amount = new_stack_bottom - sp DO create_redzone_at(new_stack_bottom+stack_size-redzone_size) EVAL alloca(huge_amount) END END; Push(args etc. and ret. address z) Jump(P) z: EVAL ptr^ (* [ NOT ok -> hardware callback to SEGV: ] *) SEGV(signalpc): IF NOT ok AND signalpc = fault_address THEN destroy_redzone(...); free(new_stack_bottom) END Mika "Rodney M. Bates" writes: > > >On 02/20/2011 12:37 PM, Mika Nystrom wrote: >> On a 64-bit machine, at least, there ought to be enough virtual >> memory that you could just have a gap between thread stacks big >> enough to allow for a protection area larger than the largest possible >> (implementation-defined) activation record, no? I know I've run into >> trouble with very large activation records in the past (and not because >> I was running out of stack space, either). >> >> Or at least a procedure with a very large activation record (or >> a procedure calling it) could be required to call some sort of check >> routine "EnoughStackSpaceRemaining()" before starting to scribble >> on the activation record? > >Hmm, I like this idea. It would introduce normal-case runtime overhead >only for such procedures, and these are likely rare. Also, assuming the procedure >actually uses very much of its large AR, it should also have enough computation >time to wash out the stack check overhead. > >> >> Also the end of the activation record must be written to at least once, >> or else the memory protection won't be triggered. >> > >I was thinking (as an alternative mechanism) of having the compiler intentionally >add enough artificial write(s) as necessary to ensure storing within the >red zone, and not just beyond it. This seems trickier to get right and >harder to distinguish after the fact from a NIL dereference. > >> In any case if this is done properly the same mechanism I proposed for >> SIGSEGV ought to be able to catch stack overflow, no? Well, as long as >> signals are delivered on a separate stack. If signals are delivered on >> the same stack, the signal handler would get nastier, it would have to >> make space through some manipulations (maybe temporarily unporotecting >> the redzone page?) for its own purposes... but I don't see why it >> couldn't be done. >> >> Not sure why I'm getting SIGILL... maybe I am getting my signal handler >> activated inside the redzone page because of a difference in signal >> handling..? I remember reading something about sigaltstack... >> >> I would of course love to be able to recover from stack overflow, too. >> In some sense, since it's a generally unknown limit, it's even less of >> a fatal error than a NIL dereference (hence makes even more sense to >> catch it). > >I think this would be a nice mechanism to have available. It would have to >be used with some care. In any case, it would be really nice and more >frequently so, to at least have runtime error messages that distinguished >stack overflow from NIL deref. > >> >> Mika >> >> "Rodney M. Bates" writes: >>> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. >>> Probably a fully protected guard page at the end of the stack. This technique >>> always worries me a bit because a procedure with a really big activation record >>> could jump right past it. Probably it would almost always access the first page >>> of the big area before storing anything into later pages. >>> >>> On 02/19/2011 05:27 PM, Mika Nystrom wrote: >>>> Ah, yes, stack protection. >>>> >>>> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. >>>> >>>> Hmm, I get SIGILL on AMD64_FREEBSD as well: >>>> >>>> time ../AMD64_FREEBSD/stubexample >>>> M-Scheme Experimental >>>> LITHP ITH LITHENING. >>>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) >>>> f >>>>> (f 0) >>>> Illegal instruction >>>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w >>>> >>>> What absolutely must not happen, of course, is that the runtime hangs >>>> while executing only safe code... >>>> >>>> Mika >>>> >>>> "Rodney M. Bates" writes: >>>>> I know of one other place the compilers rely on hardware memory protection >>>>> to detect a checked runtime error, and that is stack overflow. This won't >>>>> corrupt anything, but is hard to distinguish from dereferencing NIL. >>>>> This could probably be distinguished after the fact by some low-level, >>>>> target-dependent code. I have found it by looking at assembly code at >>>>> the point of failure--usually right after a stack pointer push. >>>>> >>>>> Detecting this via compiler-generated checks would probably be more >>>>> extravagant than many other checks, as it is so frequent. I am not >>>>> aware of any really good solution to this in any implementation of any >>>>> language. >>>>> >>>>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: >>>>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >>>>>> you know! >>>>>> >>>>>> "Corrupting the heap" is something that can only happen as a result of >>>>>> an unchecked runtime error. Unchecked runtime errors cannot happen in >>>>>> modules not marked UNSAFE. >>>>>> >>>>>> SEGV is, however, used by the CM3 implementation (and its predecessors) >>>>>> to signal a certain kind of *checked* runtime error, namely, the >>>>>> dereferencing of a NIL reference. Correct me if I am wrong, but an >>>>>> attempt to dereference NIL is not going to leave the heap corrupted? >>>>>> >>>>>> And if you stick to safe code, the only SEGVs I think you get in the >>>>>> current CM3 are ones from NIL dereferences. >>>>>> >>>>>> Hence, as long as you stick with safe code, the only time the code I >>>>>> checked in earlier gets triggered is for NIL dereferences, which should >>>>>> never corrupt the heap. So SEGV is not sometimes, but in fact always >>>>>> recoverable. >>>>>> >>>>>> :-) >>>>>> >>>>>> Mika >>>>>> >>>>>> P.S. the bit above "if you stick to safe code": if you actually program in >>>>>> Modula-3 you almost never use UNSAFE. I went through my repository and >>>>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >>>>>> many of the UNSAFE modules are glue code to Fortran routines, which >>>>>> could relatively easily be verified to be safe in the Modula-3 sense. >>>>>> Almost all what remains is glue to some C library, which wouldn't be >>>>>> necessary if the rest of the world would wake up out of the dark ages, but >>>>>> I don't have the time to rewrite every single library from scratch myself. >>>>>> >>>>>> >>>>>> Jay K writes: >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>>>> Content-Type: text/plain; charset="iso-8859-1" >>>>>>> Content-Transfer-Encoding: quoted-printable >>>>>>> >>>>>>> >>>>>>> Letting any code run after a SIGSEGV is dubious. >>>>>>> Imagine the heap is corrupted. >>>>>>> And then you run more code. >>>>>>> And the code happens to call malloc. >>>>>>> Or printf to log something. >>>>>>> =20 >>>>>>> I suppose there might be an application that maps memory >>>>>>> gradually=2C as pieces of a buffer are hit. Might. >>>>>>> =20 >>>>>>> - Jay >>>>>>> =20 >>>>>>>> To: m3devel at elegosoft.com >>>>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>>>>>> From: mika at async.caltech.edu >>>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>>>>>> =20 >>>>>>>> =20 >>>>>>>> Dear m3devel=2C >>>>>>>> =20 >>>>>>>> For a while it has annoyed me that segmentation violations cause an >>>>>>>> unconditional program abort. I've changed that now so that (under user >>>>>>>> threads at least) we instead get a RuntimeError. Here's an example of >>>>>>>> the mechanism at work in an interactive Scheme environment. Consider >>>>>>>> the unhelpful interface and module Crash: >>>>>>>> =20 >>>>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>>>>>> =20 >>>>>>>> MODULE Crash=3B >>>>>>>> =20 >>>>>>>> PROCEDURE Me() =3D >>>>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>>>>>> ptr^ :=3D 0 >>>>>>>> END Me=3B >>>>>>>> =20 >>>>>>>> BEGIN END Crash. >>>>>>>> =20 >>>>>>>> Here's an example of what happens if you now call this from an interactiv= >>>>>>> e >>>>>>>> interpreter that catches the exception RuntimeError.E: >>>>>>>> =20 >>>>>>>> M-Scheme Experimental >>>>>>>> LITHP ITH LITHENING. >>>>>>>>> (require-modules "m3") >>>>>>>> #t >>>>>>>>> (Crash.Me) >>>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>>>>>> (+ 3 4)=20 >>>>>>>> 7 >>>>>>>>> =20 >>>>>>>> =20 >>>>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>>>>>> heck it.=20 >>>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>>>>>> =20 >>>>>>>> Mika >>>>>>>> =20 >>>>>>> = >>>>>>> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>>>> Content-Type: text/html; charset="iso-8859-1" >>>>>>> Content-Transfer-Encoding: quoted-printable >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Letting any code run after a SIGSEGV is dubious.
>>>>>>> Imagine the heap =3Bis corrupted.
>>>>>>> And then you run more code.
>>>>>>> And the code happens to call malloc.
>>>>>>> Or printf to log something.
>>>>>>>  =3B
>>>>>>> I suppose there might be an application that maps memory
>>>>>>> gradually=2C as pieces of a buffer are hit. Might.
>>>>>>>  =3B
>>>>>>>  =3B- Jay
 =3B
>>>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>>>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>>>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>>>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>>>>>> olations cause an
>=3B unconditional program abort. I've changed that = >>>>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>>>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>>>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>>>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>>>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>>>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>>>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>>>>>> >=3B Here's an example of what happens if you now call this from an inter= >>>>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>>>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>>>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>>>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>>>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>>>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>>>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>>>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>>>>>> >>>>>>> = >>>>>>> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >>>>>> >>>> >> From rodney_bates at lcwb.coop Tue Feb 22 18:59:08 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Tue, 22 Feb 2011 11:59:08 -0600 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222021200.9F0EE1A2078@async.async.caltech.edu> References: <533385.35225.qm@web29704.mail.ird.yahoo.com> <20110222021200.9F0EE1A2078@async.async.caltech.edu> Message-ID: <4D63F96C.7020500@lcwb.coop> I don't know how relevant this is, but FWIW, I played with some patches in an interactive program I have that catches its own assertion failures and recovers from some of them. I was already using RTProcess.RegisterExitor with my own procedure. I made the exitor just call RTException.Raise. Then I deliberately did a NIL deref inside a try block catching RTException.E and ignoring it. It did prevent termination of my program. But there were two other problems: 1) The SIGSEGV messages came out on the command line before the exitor was invoked. 2) After the exception was caught, the program was catatonic, without using CPU. This is with pthreads. I haven't figured out why the wait. m3gdb seems to be useless in looking at other threads than the one it happens to get into, when using pthreads. On 02/21/2011 08:12 PM, Mika Nystrom wrote: > There are several issues here and yes this the RuntimeError.E exception > is taken either from SPIN or the CM3 JVM, I don't know which. > > I only understand the user threads code so I have made that raise the > RuntimeError.E instead of dumping core on a segfault. I think this is what > the CM JVM was supposed to do as well---but honestly I can't tell if that > work was ever completed. They do support mapping various other runtime > errors (subrange mismatch, e.g.) into exceptions, and that works well. > However the segmentation fault mapping was not functional: there was no > other way of handling it than dumping core on the whole M3 environment > and runtime before my changes last week. > > My last email was about a bug in my changes---not in the actual functionality > but the fact that if you *don't* catch the RuntimeError.E exception that > results from a bad memory reference, the runtime quits with a slightly > confusing error message, apparently implicating the threading system when > the bug is actually in normal, non-UNSAFE code. > > Mika > From jay.krell at cornell.edu Tue Feb 22 19:06:42 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 22 Feb 2011 18:06:42 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222174405.E15481A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu>, <4D615643.7060107@lcwb.coop>, <20110220183746.195C11A2078@async.async.caltech.edu>, <4D61CC27.7070704@lcwb.coop>, <20110222174405.E15481A2078@async.async.caltech.edu> Message-ID: (aside, and possible agreement: right -- an interpreter should consider NOT recursing on the machine stack whenever code it is interpreting recurses, but definitely some do) - Jay > To: rodney_bates at lcwb.coop > Date: Tue, 22 Feb 2011 09:44:05 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > Ok so I was thinking about this. > > Why on earth is stack overflow even a problem? > > Consider the following procedure call (in my code, stack grows upwards): > > (* sp at x, pc at y *) > y: P(args) > z: next_statement > > decompose as follows: > > (* sp at x, pc at y *) > y: Push(args etc. and ret. address z) > Jump(P) > z: next_statement > > Now, we say: > > y: ok := check_stack(size of frame) > IF NOT ok THEN abort() END; > Push(args etc. and ret. address z) > Jump(P) > z: next_statement > > (note check_stack and the following IF can be implemented by hardware, > need not actually be an instruction) > > Let me change the code a tad: > > y: ok := check_stack(size of frame) > y':IF NOT ok THEN > WITH new_stack_bottom = malloc(stack_size) > huge_amount = new_stack_bottom - sp DO > create_redzone_at(new_stack_bottom+stack_size-redzone_size) > EVAL alloca(huge_amount) > END > END; > Push(args etc. and ret. address z) > Jump(P) > z: IF NOT ok THEN destroy_redzone(...); free(new_stack_bottom) END > > Note 1. cleanup of redzone could be postponed to return of caller....when > alloca in any case has to be cleaned up. > > Note 2. the test IF NOT ok at z is more expensive to implement than the > one at y because you can't really use hardware for it. A hardware callback > can be arranged though: > > VAR ptr := sp; > y: ok := check_stack(size of frame) > y':IF NOT ok THEN > ptr := 0; (* illegal address *) > fault_address := z; > WITH new_stack_bottom = malloc(stack_size) > huge_amount = new_stack_bottom - sp DO > create_redzone_at(new_stack_bottom+stack_size-redzone_size) > EVAL alloca(huge_amount) > END > END; > Push(args etc. and ret. address z) > Jump(P) > z: EVAL ptr^ (* [ NOT ok -> hardware callback to SEGV: ] *) > > SEGV(signalpc): IF NOT ok AND signalpc = fault_address THEN destroy_redzone(...); free(new_stack_bottom) END > > Mika > > > > > > "Rodney M. Bates" writes: > > > > > >On 02/20/2011 12:37 PM, Mika Nystrom wrote: > >> On a 64-bit machine, at least, there ought to be enough virtual > >> memory that you could just have a gap between thread stacks big > >> enough to allow for a protection area larger than the largest possible > >> (implementation-defined) activation record, no? I know I've run into > >> trouble with very large activation records in the past (and not because > >> I was running out of stack space, either). > >> > >> Or at least a procedure with a very large activation record (or > >> a procedure calling it) could be required to call some sort of check > >> routine "EnoughStackSpaceRemaining()" before starting to scribble > >> on the activation record? > > > >Hmm, I like this idea. It would introduce normal-case runtime overhead > >only for such procedures, and these are likely rare. Also, assuming the procedure > >actually uses very much of its large AR, it should also have enough computation > >time to wash out the stack check overhead. > > > >> > >> Also the end of the activation record must be written to at least once, > >> or else the memory protection won't be triggered. > >> > > > >I was thinking (as an alternative mechanism) of having the compiler intentionally > >add enough artificial write(s) as necessary to ensure storing within the > >red zone, and not just beyond it. This seems trickier to get right and > >harder to distinguish after the fact from a NIL dereference. > > > >> In any case if this is done properly the same mechanism I proposed for > >> SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > >> signals are delivered on a separate stack. If signals are delivered on > >> the same stack, the signal handler would get nastier, it would have to > >> make space through some manipulations (maybe temporarily unporotecting > >> the redzone page?) for its own purposes... but I don't see why it > >> couldn't be done. > >> > >> Not sure why I'm getting SIGILL... maybe I am getting my signal handler > >> activated inside the redzone page because of a difference in signal > >> handling..? I remember reading something about sigaltstack... > >> > >> I would of course love to be able to recover from stack overflow, too. > >> In some sense, since it's a generally unknown limit, it's even less of > >> a fatal error than a NIL dereference (hence makes even more sense to > >> catch it). > > > >I think this would be a nice mechanism to have available. It would have to > >be used with some care. In any case, it would be really nice and more > >frequently so, to at least have runtime error messages that distinguished > >stack overflow from NIL deref. > > > >> > >> Mika > >> > >> "Rodney M. Bates" writes: > >>> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >>> Probably a fully protected guard page at the end of the stack. This technique > >>> always worries me a bit because a procedure with a really big activation record > >>> could jump right past it. Probably it would almost always access the first page > >>> of the big area before storing anything into later pages. > >>> > >>> On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >>>> Ah, yes, stack protection. > >>>> > >>>> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >>>> > >>>> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >>>> > >>>> time ../AMD64_FREEBSD/stubexample > >>>> M-Scheme Experimental > >>>> LITHP ITH LITHENING. > >>>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >>>> f > >>>>> (f 0) > >>>> Illegal instruction > >>>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >>>> > >>>> What absolutely must not happen, of course, is that the runtime hangs > >>>> while executing only safe code... > >>>> > >>>> Mika > >>>> > >>>> "Rodney M. Bates" writes: > >>>>> I know of one other place the compilers rely on hardware memory protection > >>>>> to detect a checked runtime error, and that is stack overflow. This won't > >>>>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>>>> This could probably be distinguished after the fact by some low-level, > >>>>> target-dependent code. I have found it by looking at assembly code at > >>>>> the point of failure--usually right after a stack pointer push. > >>>>> > >>>>> Detecting this via compiler-generated checks would probably be more > >>>>> extravagant than many other checks, as it is so frequent. I am not > >>>>> aware of any really good solution to this in any implementation of any > >>>>> language. > >>>>> > >>>>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>>>> you know! > >>>>>> > >>>>>> "Corrupting the heap" is something that can only happen as a result of > >>>>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>>>> modules not marked UNSAFE. > >>>>>> > >>>>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>>>> to signal a certain kind of *checked* runtime error, namely, the > >>>>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>>>> > >>>>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>>>> current CM3 are ones from NIL dereferences. > >>>>>> > >>>>>> Hence, as long as you stick with safe code, the only time the code I > >>>>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>>>> recoverable. > >>>>>> > >>>>>> :-) > >>>>>> > >>>>>> Mika > >>>>>> > >>>>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>>>> Almost all what remains is glue to some C library, which wouldn't be > >>>>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>>>> I don't have the time to rewrite every single library from scratch myself. > >>>>>> > >>>>>> > >>>>>> Jay K writes: > >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>>>> Content-Transfer-Encoding: quoted-printable > >>>>>>> > >>>>>>> > >>>>>>> Letting any code run after a SIGSEGV is dubious. > >>>>>>> Imagine the heap is corrupted. > >>>>>>> And then you run more code. > >>>>>>> And the code happens to call malloc. > >>>>>>> Or printf to log something. > >>>>>>> =20 > >>>>>>> I suppose there might be an application that maps memory > >>>>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>>>> =20 > >>>>>>> - Jay > >>>>>>> =20 > >>>>>>>> To: m3devel at elegosoft.com > >>>>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>>>> From: mika at async.caltech.edu > >>>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>>>> =20 > >>>>>>>> =20 > >>>>>>>> Dear m3devel=2C > >>>>>>>> =20 > >>>>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>>>> the unhelpful interface and module Crash: > >>>>>>>> =20 > >>>>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>>>> =20 > >>>>>>>> MODULE Crash=3B > >>>>>>>> =20 > >>>>>>>> PROCEDURE Me() =3D > >>>>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>>>> ptr^ :=3D 0 > >>>>>>>> END Me=3B > >>>>>>>> =20 > >>>>>>>> BEGIN END Crash. > >>>>>>>> =20 > >>>>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>>>> e > >>>>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>>>> =20 > >>>>>>>> M-Scheme Experimental > >>>>>>>> LITHP ITH LITHENING. > >>>>>>>>> (require-modules "m3") > >>>>>>>> #t > >>>>>>>>> (Crash.Me) > >>>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>>>> (+ 3 4)=20 > >>>>>>>> 7 > >>>>>>>>> =20 > >>>>>>>> =20 > >>>>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>>>> heck it.=20 > >>>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>>>> =20 > >>>>>>>> Mika > >>>>>>>> =20 > >>>>>>> = > >>>>>>> > >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>>>> Content-Transfer-Encoding: quoted-printable > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>>>> Imagine the heap =3Bis corrupted.
> >>>>>>> And then you run more code.
> >>>>>>> And the code happens to call malloc.
> >>>>>>> Or printf to log something.
> >>>>>>>  =3B
> >>>>>>> I suppose there might be an application that maps memory
> >>>>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>>>  =3B
> >>>>>>>  =3B- Jay
 =3B
> >>>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>>>> > >>>>>>> = > >>>>>>> > >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>>>> > >>>> > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Tue Feb 22 19:45:02 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Tue, 22 Feb 2011 10:45:02 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu>, <4D615643.7060107@lcwb.coop>, <20110220183746.195C11A2078@async.async.caltech.edu>, <4D61CC27.7070704@lcwb.coop>, <20110222174405.E15481A2078@async.async.caltech.edu> Message-ID: <20110222184502.88B921A2078@async.async.caltech.edu> Jay K writes: >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >(aside=2C and possible agreement: right -- an interpreter should consider N= >OT recursing on the machine >stack whenever code it is interpreting recurses=2C but definitely some do) >=20 > - Jay Jay, yes I agree---but: 1. of course it's easier to code the interpreter to recurse on the machine stack 2. code called out from the interpreter will in any case recurse on the (a) machine stack 3. again if one can "fix" the stack mechanism so that running out of stack space is non-fatal, there's no reason not to use the machine stack I've found that Modula-3's garbage collector is brutally slow and that converting code to using stack-allocated memory whenever possible is a huge performance win under both PM3 and CM3. Perhaps this is a problem with the garbage collector. I also have a partial design for a Scheme compiler (emitting Modula-3) that would use the machine stack whenever possible rather than heap allocation and do a "tricky thing" with local procedures to copy the stack whenever a reference to the stack escapes. (Most of the time you can implement Scheme on a stack but sometimes you need to keep part of it live after you return; hence implementations (including mine) heap-allocate local variables...) I realize my code is perhaps difficult to read. The gist of it is this: when you run out of stack space, allocate a new stack anywhere else in memory. Make it appear that you have alloca'd all the space between the old and new stacks (into a variable that you of course never touch), then continue. An issue is what to do when you return, you have to make sure that you restore your old redzone.. perhaps you don't need to free the stack eagerly: you could instead remember you have that page allocated and re-use it next time you hit the redzone, or put it in a pool to allow it to be used by another thread. The arbitrary choice of stack sizes has always bothered me greatly. Some of my programs have thousands of threads and they use stack space irregularly... Mika >> To: rodney_bates at lcwb.coop >> Date: Tue=2C 22 Feb 2011 09:44:05 -0800 >> From: mika at async.caltech.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] SEGV mapping to RuntimeError >>=20 >>=20 >> Ok so I was thinking about this. >>=20 >> Why on earth is stack overflow even a problem? >>=20 >> Consider the following procedure call (in my code=2C stack grows upwards)= >: >>=20 >> (* sp at x=2C pc at y *) >> y: P(args) >> z: next_statement >>=20 >> decompose as follows: >>=20 >> (* sp at x=2C pc at y *) >> y: Push(args etc. and ret. address z) >> Jump(P) >> z: next_statement >>=20 >> Now=2C we say: >>=20 >> y: ok :=3D check_stack(size of frame) >> IF NOT ok THEN abort() END=3B >> Push(args etc. and ret. address z) >> Jump(P) >> z: next_statement >>=20 >> (note check_stack and the following IF can be implemented by hardware=2C >> need not actually be an instruction) >>=20 >> Let me change the code a tad: >>=20 >> y: ok :=3D check_stack(size of frame) >> y':IF NOT ok THEN=20 >> WITH new_stack_bottom =3D malloc(stack_size) >> huge_amount =3D new_stack_bottom - sp DO >> create_redzone_at(new_stack_bottom+stack_size-redzone_size) >> EVAL alloca(huge_amount)=20 >> END >> END=3B >> Push(args etc. and ret. address z) >> Jump(P) >> z: IF NOT ok THEN destroy_redzone(...)=3B free(new_stack_bottom) END >>=20 >> Note 1. cleanup of redzone could be postponed to return of caller....when >> alloca in any case has to be cleaned up. >>=20 >> Note 2. the test IF NOT ok at z is more expensive to implement than the >> one at y because you can't really use hardware for it. A hardware callbac= >k >> can be arranged though: >>=20 >> VAR ptr :=3D sp=3B >> y: ok :=3D check_stack(size of frame) >> y':IF NOT ok THEN=20 >> ptr :=3D 0=3B (* illegal address *) >> fault_address :=3D z=3B >> WITH new_stack_bottom =3D malloc(stack_size) >> huge_amount =3D new_stack_bottom - sp DO >> create_redzone_at(new_stack_bottom+stack_size-redzone_size) >> EVAL alloca(huge_amount)=20 >> END >> END=3B >> Push(args etc. and ret. address z) >> Jump(P) >> z: EVAL ptr^ (* [ NOT ok -> hardware callback to SEGV: ] *) >>=20 >> SEGV(signalpc): IF NOT ok AND signalpc =3D fault_address THEN destroy_red= >zone(...)=3B free(new_stack_bottom) END >>=20 >> Mika >>=20 >>=20 >>=20 >>=20 >>=20 >> "Rodney M. Bates" writes: >> > >> > >> >On 02/20/2011 12:37 PM=2C Mika Nystrom wrote: >> >> On a 64-bit machine=2C at least=2C there ought to be enough virtual >> >> memory that you could just have a gap between thread stacks big >> >> enough to allow for a protection area larger than the largest possible >> >> (implementation-defined) activation record=2C no? I know I've run into >> >> trouble with very large activation records in the past (and not becaus= >e >> >> I was running out of stack space=2C either). >> >> >> >> Or at least a procedure with a very large activation record (or >> >> a procedure calling it) could be required to call some sort of check >> >> routine "EnoughStackSpaceRemaining()" before starting to scribble >> >> on the activation record? >> > >> >Hmm=2C I like this idea. It would introduce normal-case runtime overhead >> >only for such procedures=2C and these are likely rare. Also=2C assuming = >the procedure >> >actually uses very much of its large AR=2C it should also have enough co= >mputation >> >time to wash out the stack check overhead. >> > >> >> >> >> Also the end of the activation record must be written to at least once= >=2C >> >> or else the memory protection won't be triggered. >> >> >> > >> >I was thinking (as an alternative mechanism) of having the compiler inte= >ntionally >> >add enough artificial write(s) as necessary to ensure storing within the >> >red zone=2C and not just beyond it. This seems trickier to get right and >> >harder to distinguish after the fact from a NIL dereference. >> > >> >> In any case if this is done properly the same mechanism I proposed for >> >> SIGSEGV ought to be able to catch stack overflow=2C no? Well=2C as lon= >g as >> >> signals are delivered on a separate stack. If signals are delivered on >> >> the same stack=2C the signal handler would get nastier=2C it would hav= >e to >> >> make space through some manipulations (maybe temporarily unporotecting >> >> the redzone page?) for its own purposes... but I don't see why it >> >> couldn't be done. >> >> >> >> Not sure why I'm getting SIGILL... maybe I am getting my signal handle= >r >> >> activated inside the redzone page because of a difference in signal >> >> handling..? I remember reading something about sigaltstack... >> >> >> >> I would of course love to be able to recover from stack overflow=2C to= >o. >> >> In some sense=2C since it's a generally unknown limit=2C it's even les= >s of >> >> a fatal error than a NIL dereference (hence makes even more sense to >> >> catch it). >> > >> >I think this would be a nice mechanism to have available. It would have = >to >> >be used with some care. In any case=2C it would be really nice and more >> >frequently so=2C to at least have runtime error messages that distinguis= >hed >> >stack overflow from NIL deref. >> > >> >> >> >> Mika >> >> >> >> "Rodney M. Bates" writes: >> >>> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AM= >D64_LINUX. >> >>> Probably a fully protected guard page at the end of the stack. This t= >echnique >> >>> always worries me a bit because a procedure with a really big activat= >ion record >> >>> could jump right past it. Probably it would almost always access the = >first page >> >>> of the big area before storing anything into later pages. >> >>> >> >>> On 02/19/2011 05:27 PM=2C Mika Nystrom wrote: >> >>>> Ah=2C yes=2C stack protection. >> >>>> >> >>>> Do you know if it's a SIGSEGV=2C not a SIGBUS? I know I have seen SI= >GILL on Macs. >> >>>> >> >>>> Hmm=2C I get SIGILL on AMD64_FREEBSD as well: >> >>>> >> >>>> time ../AMD64_FREEBSD/stubexample >> >>>> M-Scheme Experimental >> >>>> LITHP ITH LITHENING. >> >>>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) >> >>>> f >> >>>>> (f 0) >> >>>> Illegal instruction >> >>>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w >> >>>> >> >>>> What absolutely must not happen=2C of course=2C is that the runtime = >hangs >> >>>> while executing only safe code... >> >>>> >> >>>> Mika >> >>>> >> >>>> "Rodney M. Bates" writes: >> >>>>> I know of one other place the compilers rely on hardware memory pro= >tection >> >>>>> to detect a checked runtime error=2C and that is stack overflow. Th= >is won't >> >>>>> corrupt anything=2C but is hard to distinguish from dereferencing N= >IL. >> >>>>> This could probably be distinguished after the fact by some low-lev= >el=2C >> >>>>> target-dependent code. I have found it by looking at assembly code = >at >> >>>>> the point of failure--usually right after a stack pointer push. >> >>>>> >> >>>>> Detecting this via compiler-generated checks would probably be more >> >>>>> extravagant than many other checks=2C as it is so frequent. I am no= >t >> >>>>> aware of any really good solution to this in any implementation of = >any >> >>>>> language. >> >>>>> >> >>>>> On 02/19/2011 02:38 PM=2C Mika Nystrom wrote: >> >>>>>> Jay=2C sometimes I wonder about you: this is a Modula-3 mailing li= >st=2C >> >>>>>> you know! >> >>>>>> >> >>>>>> "Corrupting the heap" is something that can only happen as a resul= >t of >> >>>>>> an unchecked runtime error. Unchecked runtime errors cannot happen= > in >> >>>>>> modules not marked UNSAFE. >> >>>>>> >> >>>>>> SEGV is=2C however=2C used by the CM3 implementation (and its pred= >ecessors) >> >>>>>> to signal a certain kind of *checked* runtime error=2C namely=2C t= >he >> >>>>>> dereferencing of a NIL reference. Correct me if I am wrong=2C but = >an >> >>>>>> attempt to dereference NIL is not going to leave the heap corrupte= >d? >> >>>>>> >> >>>>>> And if you stick to safe code=2C the only SEGVs I think you get in= > the >> >>>>>> current CM3 are ones from NIL dereferences. >> >>>>>> >> >>>>>> Hence=2C as long as you stick with safe code=2C the only time the = >code I >> >>>>>> checked in earlier gets triggered is for NIL dereferences=2C which= > should >> >>>>>> never corrupt the heap. So SEGV is not sometimes=2C but in fact al= >ways >> >>>>>> recoverable. >> >>>>>> >> >>>>>> :-) >> >>>>>> >> >>>>>> Mika >> >>>>>> >> >>>>>> P.S. the bit above "if you stick to safe code": if you actually pr= >ogram in >> >>>>>> Modula-3 you almost never use UNSAFE. I went through my repository= > and >> >>>>>> I have 40 modules using UNSAFE out of a total of 4=2C559. Furtherm= >ore=2C >> >>>>>> many of the UNSAFE modules are glue code to Fortran routines=2C wh= >ich >> >>>>>> could relatively easily be verified to be safe in the Modula-3 sen= >se. >> >>>>>> Almost all what remains is glue to some C library=2C which wouldn'= >t be >> >>>>>> necessary if the rest of the world would wake up out of the dark a= >ges=2C but >> >>>>>> I don't have the time to rewrite every single library from scratch= > myself. >> >>>>>> >> >>>>>> >> >>>>>> Jay K writes: >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >> >>>>>>> Content-Type: text/plain=3B charset=3D"iso-8859-1" >> >>>>>>> Content-Transfer-Encoding: quoted-printable >> >>>>>>> >> >>>>>>> >> >>>>>>> Letting any code run after a SIGSEGV is dubious. >> >>>>>>> Imagine the heap is corrupted. >> >>>>>>> And then you run more code. >> >>>>>>> And the code happens to call malloc. >> >>>>>>> Or printf to log something. >> >>>>>>> =3D20 >> >>>>>>> I suppose there might be an application that maps memory >> >>>>>>> gradually=3D2C as pieces of a buffer are hit. Might. >> >>>>>>> =3D20 >> >>>>>>> - Jay >> >>>>>>> =3D20 >> >>>>>>>> To: m3devel at elegosoft.com >> >>>>>>>> Date: Sat=3D2C 19 Feb 2011 10:29:30 -0800 >> >>>>>>>> From: mika at async.caltech.edu >> >>>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError >> >>>>>>>> =3D20 >> >>>>>>>> =3D20 >> >>>>>>>> Dear m3devel=3D2C >> >>>>>>>> =3D20 >> >>>>>>>> For a while it has annoyed me that segmentation violations cause= > an >> >>>>>>>> unconditional program abort. I've changed that now so that (unde= >r user >> >>>>>>>> threads at least) we instead get a RuntimeError. Here's an examp= >le of >> >>>>>>>> the mechanism at work in an interactive Scheme environment. Cons= >ider >> >>>>>>>> the unhelpful interface and module Crash: >> >>>>>>>> =3D20 >> >>>>>>>> INTERFACE Crash=3D3B PROCEDURE Me()=3D3B END Crash. >> >>>>>>>> =3D20 >> >>>>>>>> MODULE Crash=3D3B >> >>>>>>>> =3D20 >> >>>>>>>> PROCEDURE Me() =3D3D >> >>>>>>>> VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN >> >>>>>>>> ptr^ :=3D3D 0 >> >>>>>>>> END Me=3D3B >> >>>>>>>> =3D20 >> >>>>>>>> BEGIN END Crash. >> >>>>>>>> =3D20 >> >>>>>>>> Here's an example of what happens if you now call this from an i= >nteractiv=3D >> >>>>>>> e >> >>>>>>>> interpreter that catches the exception RuntimeError.E: >> >>>>>>>> =3D20 >> >>>>>>>> M-Scheme Experimental >> >>>>>>>> LITHP ITH LITHENING. >> >>>>>>>>> (require-modules "m3") >> >>>>>>>> #t >> >>>>>>>>> (Crash.Me) >> >>>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory = >location. >> >>>>>>>>> (+ 3 4)=3D20 >> >>>>>>>> 7 >> >>>>>>>>> =3D20 >> >>>>>>>> =3D20 >> >>>>>>>> I just realized I may have broken pthreads=3D2C let me go back a= >nd double-c=3D >> >>>>>>> heck it.=3D20 >> >>>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do = >they... >> >>>>>>>> =3D20 >> >>>>>>>> Mika >> >>>>>>>> =3D20 >> >>>>>>> =3D >> >>>>>>> >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >> >>>>>>> Content-Type: text/html=3B charset=3D"iso-8859-1" >> >>>>>>> Content-Transfer-Encoding: quoted-printable >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> Letting any code run after a SIGSEGV is dubious.
>> >>>>>>> Imagine the heap =3D3Bis corrupted.
>> >>>>>>> And then you run more code.
>> >>>>>>> And the code happens to call malloc.
>> >>>>>>> Or printf to log something.
>> >>>>>>>  =3D3B
>> >>>>>>> I suppose there might be an application that maps memory
>> >>>>>>> gradually=3D2C as pieces of a buffer are hit. Might.
>> >>>>>>>  =3D3B
>> >>>>>>>  =3D3B- Jay
 =3D3B
>> >>>>>>> >=3D3B To: m3devel at elegosoft.com
>=3D3B Date: Sat=3D2C 19 = >Feb 2011 10:29:3=3D >> >>>>>>> 0 -0800
>=3D3B From: mika at async.caltech.edu
>=3D3B Subj= >ect: [M3devel]=3D >> >>>>>>> SEGV mapping to RuntimeError
>=3D3B
>=3D3B
>=3D3B = >Dear m3devel=3D >> >>>>>>> =3D2C
>=3D3B
>=3D3B For a while it has annoyed me that = >segmentation vi=3D >> >>>>>>> olations cause an
>=3D3B unconditional program abort. I've c= >hanged that =3D >> >>>>>>> now so that (under user
>=3D3B threads at least) we instead = >get a Runtim=3D >> >>>>>>> eError. Here's an example of
>=3D3B the mechanism at work in= > an interact=3D >> >>>>>>> ive Scheme environment. Consider
>=3D3B the unhelpful interf= >ace and modu=3D >> >>>>>>> le Crash:
>=3D3B
>=3D3B INTERFACE Crash=3D3B PROCEDURE = >Me()=3D3B END Cra=3D >> >>>>>>> sh.
>=3D3B
>=3D3B MODULE Crash=3D3B
>=3D3B
>= >=3D3B PROCEDURE Me(=3D >> >>>>>>> ) =3D3D
>=3D3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGINR>>=3D3B ptr^ :=3D3D=3D >> >>>>>>> 0
>=3D3B END Me=3D3B
>=3D3B
>=3D3B BEGIN END Crash= >.
>=3D3B
=3D >> >>>>>>> >=3D3B Here's an example of what happens if you now call this f= >rom an inter=3D >> >>>>>>> active
>=3D3B interpreter that catches the exception Runtime= >Error.E:
=3D >> >>>>>>> >=3D3B
>=3D3B M-Scheme Experimental
>=3D3B LITHP ITH = >LITHENING.
&=3D >> >>>>>>> gt=3D3B>=3D3B (require-modules "m3")
>=3D3B #t
>=3D3B= >>=3D3B (Crash.Me=3D >> >>>>>>> )
>=3D3B EXCEPTION! RuntimeError! Attempt to reference an il= >legal memory=3D >> >>>>>>> location.
>=3D3B>=3D3B (+ 3 4)
>=3D3B 7
>=3D3B&g= >t=3D3B
>=3D >> >>>>>>> =3D3B
>=3D3B I just realized I may have broken pthreads=3D2C= > let me go back=3D >> >>>>>>> and double-check it.
>=3D3B runtime/POSIX and thread/POSIX d= >on't refer=3D >> >>>>>>> to the same thing do they...
>=3D3B
>=3D3B Mika
>= >=3D3B
=3D >> >>>>>>> >> >>>>>>> =3D >> >>>>>>> >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >> >>>>>> >> >>>> >> >> > = > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >(aside=2C and possible agreement: right -- an interpreter should consider N= >OT recursing on the machine
>stack whenever code it is interpreting recurses=2C but definitely some do)<= >BR> > =3B
> =3B- Jay
 =3B
>>=3B To: rodney_bates at lcwb.coop
>=3B Date: Tue=2C 22 Feb 2011 09:44:= >05 -0800
>=3B From: mika at async.caltech.edu
>=3B CC: m3devel at elego= >soft.com
>=3B Subject: Re: [M3devel] SEGV mapping to RuntimeError
&= >gt=3B
>=3B
>=3B Ok so I was thinking about this.
>=3B
= >>=3B Why on earth is stack overflow even a problem?
>=3B
>=3B = >Consider the following procedure call (in my code=2C stack grows upwards):<= >BR>>=3B
>=3B (* sp at x=2C pc at y *)
>=3B y: P(args)
>= >=3B z: next_statement
>=3B
>=3B decompose as follows:
>=3B = >
>=3B (* sp at x=2C pc at y *)
>=3B y: Push(args etc. and ret. ad= >dress z)
>=3B Jump(P)
>=3B z: next_statement
>=3B
>=3B= > Now=2C we say:
>=3B
>=3B y: ok :=3D check_stack(size of frame)<= >BR>>=3B IF NOT ok THEN abort() END=3B
>=3B Push(args etc. and ret. a= >ddress z)
>=3B Jump(P)
>=3B z: next_statement
>=3B
>= >=3B (note check_stack and the following IF can be implemented by hardware= >=2C
>=3B need not actually be an instruction)
>=3B
>=3B Let= > me change the code a tad:
>=3B
>=3B y: ok :=3D check_stack(size= > of frame)
>=3B y':IF NOT ok THEN
>=3B WITH new_stack_bottom =3D= > malloc(stack_size)
>=3B huge_amount =3D new_stack_bottom - sp DO
&= >gt=3B create_redzone_at(new_stack_bottom+stack_size-redzone_size)
>=3B= > EVAL alloca(huge_amount)
>=3B END
>=3B END=3B
>=3B Push(ar= >gs etc. and ret. address z)
>=3B Jump(P)
>=3B z: IF NOT ok THEN d= >estroy_redzone(...)=3B free(new_stack_bottom) END
>=3B
>=3B Note= > 1. cleanup of redzone could be postponed to return of caller....when
&g= >t=3B alloca in any case has to be cleaned up.
>=3B
>=3B Note 2. = >the test IF NOT ok at z is more expensive to implement than the
>=3B o= >ne at y because you can't really use hardware for it. A hardware callbackR>>=3B can be arranged though:
>=3B
>=3B VAR ptr :=3D sp=3B>>=3B y: ok :=3D check_stack(size of frame)
>=3B y':IF NOT ok THEN <= >BR>>=3B ptr :=3D 0=3B (* illegal address *)
>=3B fault_address :=3D = >z=3B
>=3B WITH new_stack_bottom =3D malloc(stack_size)
>=3B huge_= >amount =3D new_stack_bottom - sp DO
>=3B create_redzone_at(new_stack_b= >ottom+stack_size-redzone_size)
>=3B EVAL alloca(huge_amount)
>= >=3B END
>=3B END=3B
>=3B Push(args etc. and ret. address z)
&g= >t=3B Jump(P)
>=3B z: EVAL ptr^ (* [ NOT ok ->=3B hardware callback t= >o SEGV: ] *)
>=3B
>=3B SEGV(signalpc): IF NOT ok AND signalpc = >=3D fault_address THEN destroy_redzone(...)=3B free(new_stack_bottom) ENDR>>=3B
>=3B Mika
>=3B
>=3B
>=3B
>=3B
>= >=3B
>=3B "Rodney M. Bates" writes:
>=3B >=3B
>=3B >=3B<= >BR>>=3B >=3BOn 02/20/2011 12:37 PM=2C Mika Nystrom wrote:
>=3B >= >=3B>=3B On a 64-bit machine=2C at least=2C there ought to be enough virtu= >al
>=3B >=3B>=3B memory that you could just have a gap between thr= >ead stacks big
>=3B >=3B>=3B enough to allow for a protection area= > larger than the largest possible
>=3B >=3B>=3B (implementation-de= >fined) activation record=2C no? I know I've run into
>=3B >=3B>=3B= > trouble with very large activation records in the past (and not because>>=3B >=3B>=3B I was running out of stack space=2C either).
>=3B= > >=3B>=3B
>=3B >=3B>=3B Or at least a procedure with a very la= >rge activation record (or
>=3B >=3B>=3B a procedure calling it) co= >uld be required to call some sort of check
>=3B >=3B>=3B routine "= >EnoughStackSpaceRemaining()" before starting to scribble
>=3B >=3B&g= >t=3B on the activation record?
>=3B >=3B
>=3B >=3BHmm=2C I li= >ke this idea. It would introduce normal-case runtime overhead
>=3B >= >=3Bonly for such procedures=2C and these are likely rare. Also=2C assuming = >the procedure
>=3B >=3Bactually uses very much of its large AR=2C it= > should also have enough computation
>=3B >=3Btime to wash out the s= >tack check overhead.
>=3B >=3B
>=3B >=3B>=3B
>=3B >= >=3B>=3B Also the end of the activation record must be written to at least= > once=2C
>=3B >=3B>=3B or else the memory protection won't be trig= >gered.
>=3B >=3B>=3B
>=3B >=3B
>=3B >=3BI was thinki= >ng (as an alternative mechanism) of having the compiler intentionally
&g= >t=3B >=3Badd enough artificial write(s) as necessary to ensure storing wi= >thin the
>=3B >=3Bred zone=2C and not just beyond it. This seems tri= >ckier to get right and
>=3B >=3Bharder to distinguish after the fact= > from a NIL dereference.
>=3B >=3B
>=3B >=3B>=3B In any cas= >e if this is done properly the same mechanism I proposed for
>=3B >= >=3B>=3B SIGSEGV ought to be able to catch stack overflow=2C no? Well=2C a= >s long as
>=3B >=3B>=3B signals are delivered on a separate stack.= > If signals are delivered on
>=3B >=3B>=3B the same stack=2C the s= >ignal handler would get nastier=2C it would have to
>=3B >=3B>=3B = >make space through some manipulations (maybe temporarily unporotecting
&= >gt=3B >=3B>=3B the redzone page?) for its own purposes... but I don't s= >ee why it
>=3B >=3B>=3B couldn't be done.
>=3B >=3B>=3BR>>=3B >=3B>=3B Not sure why I'm getting SIGILL... maybe I am getting= > my signal handler
>=3B >=3B>=3B activated inside the redzone page= > because of a difference in signal
>=3B >=3B>=3B handling..? I rem= >ember reading something about sigaltstack...
>=3B >=3B>=3B
>= >=3B >=3B>=3B I would of course love to be able to recover from stack ov= >erflow=2C too.
>=3B >=3B>=3B In some sense=2C since it's a general= >ly unknown limit=2C it's even less of
>=3B >=3B>=3B a fatal error = >than a NIL dereference (hence makes even more sense to
>=3B >=3B>= >=3B catch it).
>=3B >=3B
>=3B >=3BI think this would be a nic= >e mechanism to have available. It would have to
>=3B >=3Bbe used wit= >h some care. In any case=2C it would be really nice and more
>=3B >= >=3Bfrequently so=2C to at least have runtime error messages that distinguis= >hed
>=3B >=3Bstack overflow from NIL deref.
>=3B >=3B
>= >=3B >=3B>=3B
>=3B >=3B>=3B Mika
>=3B >=3B>=3B
>= >=3B >=3B>=3B "Rodney M. Bates" writes:
>=3B >=3B>=3B>=3B I a= >m pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX= >.
>=3B >=3B>=3B>=3B Probably a fully protected guard page at the= > end of the stack. This technique
>=3B >=3B>=3B>=3B always worri= >es me a bit because a procedure with a really big activation record
>= >=3B >=3B>=3B>=3B could jump right past it. Probably it would almost a= >lways access the first page
>=3B >=3B>=3B>=3B of the big area be= >fore storing anything into later pages.
>=3B >=3B>=3B>=3B
>= >=3B >=3B>=3B>=3B On 02/19/2011 05:27 PM=2C Mika Nystrom wrote:
>= >=3B >=3B>=3B>=3B>=3B Ah=2C yes=2C stack protection.
>=3B >= >=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B Do you know if it'= >s a SIGSEGV=2C not a SIGBUS? I know I have seen SIGILL on Macs.
>=3B &= >gt=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B Hmm=2C I get SIG= >ILL on AMD64_FREEBSD as well:
>=3B >=3B>=3B>=3B>=3B
>=3B = >>=3B>=3B>=3B>=3B time ../AMD64_FREEBSD/stubexample
>=3B >=3B= >>=3B>=3B>=3B M-Scheme Experimental
>=3B >=3B>=3B>=3B>=3B= > LITHP ITH LITHENING.
>=3B >=3B>=3B>=3B>=3B>=3B (define (f a= >) (+ (f (+ a 1)) (f (+ a 2))))
>=3B >=3B>=3B>=3B>=3B f
>= >=3B >=3B>=3B>=3B>=3B>=3B (f 0)
>=3B >=3B>=3B>=3B>=3B= > Illegal instruction
>=3B >=3B>=3B>=3B>=3B 3.847u 0.368s 0:13.= >32 31.5% 2160+284478k 0+0io 0pf+0w
>=3B >=3B>=3B>=3B>=3B
&g= >t=3B >=3B>=3B>=3B>=3B What absolutely must not happen=2C of course= >=2C is that the runtime hangs
>=3B >=3B>=3B>=3B>=3B while exec= >uting only safe code...
>=3B >=3B>=3B>=3B>=3B
>=3B >=3B= >>=3B>=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B
>=3B >=3B= >>=3B>=3B>=3B "Rodney M. Bates" writes:
>=3B >=3B>=3B>=3B&g= >t=3B>=3B I know of one other place the compilers rely on hardware memory = >protection
>=3B >=3B>=3B>=3B>=3B>=3B to detect a checked run= >time error=2C and that is stack overflow. This won't
>=3B >=3B>=3B= >>=3B>=3B>=3B corrupt anything=2C but is hard to distinguish from dere= >ferencing NIL.
>=3B >=3B>=3B>=3B>=3B>=3B This could probably= > be distinguished after the fact by some low-level=2C
>=3B >=3B>= >=3B>=3B>=3B>=3B target-dependent code. I have found it by looking at = >assembly code at
>=3B >=3B>=3B>=3B>=3B>=3B the point of fail= >ure--usually right after a stack pointer push.
>=3B >=3B>=3B>=3B= >>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B Detecting this via co= >mpiler-generated checks would probably be more
>=3B >=3B>=3B>=3B= >>=3B>=3B extravagant than many other checks=2C as it is so frequent. I = >am not
>=3B >=3B>=3B>=3B>=3B>=3B aware of any really good so= >lution to this in any implementation of any
>=3B >=3B>=3B>=3B>= >=3B>=3B language.
>=3B >=3B>=3B>=3B>=3B>=3B
>=3B >= >=3B>=3B>=3B>=3B>=3B On 02/19/2011 02:38 PM=2C Mika Nystrom wrote:R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B Jay=2C sometimes I wonder abo= >ut you: this is a Modula-3 mailing list=2C
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B you know!
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>>=3B >=3B>=3B>=3B>=3B>=3B>=3B "Corrupting the heap" is somet= hing that can only happen as a result of
>=3B >=3B>=3B>=3B>=3B= >>=3B>=3B an unchecked runtime error. Unchecked runtime errors cannot ha= >ppen in
>=3B >=3B>=3B>=3B>=3B>=3B>=3B modules not marked U= >NSAFE.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B SEGV is=2C however=2C used by the CM3 implement= >ation (and its predecessors)
>=3B >=3B>=3B>=3B>=3B>=3B>=3B= > to signal a certain kind of *checked* runtime error=2C namely=2C the
&g= >t=3B >=3B>=3B>=3B>=3B>=3B>=3B dereferencing of a NIL reference.= > Correct me if I am wrong=2C but an
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B attempt to dereference NIL is not going to leave the heap corrupt= >ed?
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B&g= >t=3B>=3B>=3B>=3B And if you stick to safe code=2C the only SEGVs I th= >ink you get in the
>=3B >=3B>=3B>=3B>=3B>=3B>=3B current C= >M3 are ones from NIL dereferences.
>=3B >=3B>=3B>=3B>=3B>=3B= >>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B Hence=2C as long as y= >ou stick with safe code=2C the only time the code I
>=3B >=3B>=3B&= >gt=3B>=3B>=3B>=3B checked in earlier gets triggered is for NIL derefe= >rences=2C which should
>=3B >=3B>=3B>=3B>=3B>=3B>=3B never= > corrupt the heap. So SEGV is not sometimes=2C but in fact always
>=3B= > >=3B>=3B>=3B>=3B>=3B>=3B recoverable.
>=3B >=3B>=3B&g= >t=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B :-)R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B&= >gt=3B>=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
&g= >t=3B >=3B>=3B>=3B>=3B>=3B>=3B P.S. the bit above "if you stick = >to safe code": if you actually program in
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B Modula-3 you almost never use UNSAFE. I went through my rep= >ository and
>=3B >=3B>=3B>=3B>=3B>=3B>=3B I have 40 module= >s using UNSAFE out of a total of 4=2C559. Furthermore=2C
>=3B >=3B&g= >t=3B>=3B>=3B>=3B>=3B many of the UNSAFE modules are glue code to Fo= >rtran routines=2C which
>=3B >=3B>=3B>=3B>=3B>=3B>=3B coul= >d relatively easily be verified to be safe in the Modula-3 sense.
>=3B= > >=3B>=3B>=3B>=3B>=3B>=3B Almost all what remains is glue to so= >me C library=2C which wouldn't be
>=3B >=3B>=3B>=3B>=3B>=3B&= >gt=3B necessary if the rest of the world would wake up out of the dark ages= >=2C but
>=3B >=3B>=3B>=3B>=3B>=3B>=3B I don't have the tim= >e to rewrite every single library from scratch myself.
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B<= >BR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B Jay K writes:
>=3B >= >=3B>=3B>=3B>=3B>=3B>=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912= >854f_
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Content-Type: te= >xt/plain=3B charset=3D"iso-8859-1"
>=3B >=3B>=3B>=3B>=3B>=3B= >>=3B>=3B Content-Transfer-Encoding: quoted-printable
>=3B >=3B&g= >t=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B= >>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Letting a= >ny code run after a SIGSEGV is dubious.
>=3B >=3B>=3B>=3B>=3B&= >gt=3B>=3B>=3B Imagine the heap is corrupted.
>=3B >=3B>=3B>= >=3B>=3B>=3B>=3B>=3B And then you run more code.
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B And the code happens to call malloc.
&= >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Or printf to log something= >.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>=3B >= >=3B>=3B>=3B>=3B>=3B>=3B>=3B I suppose there might be an applica= >tion that maps memory
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B = >gradually=3D2C as pieces of a buffer are hit. Might.
>=3B >=3B>=3B= >>=3B>=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B - Jay
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B = >=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B To: m3deve= >l at elegosoft.com
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = >Date: Sat=3D2C 19 Feb 2011 10:29:30 -0800
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B>=3B>=3B From: mika at async.caltech.edu
>=3B >=3B&g= >t=3B>=3B>=3B>=3B>=3B>=3B>=3B Subject: [M3devel] SEGV mapping to= > RuntimeError
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = >=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
&= >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B Dear m3devel=3D2C>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>=3B &g= >t=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B For a while it has annoyed m= >e that segmentation violations cause an
>=3B >=3B>=3B>=3B>=3B&= >gt=3B>=3B>=3B>=3B unconditional program abort. I've changed that now = >so that (under user
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>= >=3B threads at least) we instead get a RuntimeError. Here's an example ofR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B the mechanism at = >work in an interactive Scheme environment. Consider
>=3B >=3B>=3B&= >gt=3B>=3B>=3B>=3B>=3B>=3B the unhelpful interface and module Cras= >h:
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>= >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B INTERFACE Crash=3D3B P= >ROCEDURE Me()=3D3B END Crash.
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B MODULE Crash=3D3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= >=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = >PROCEDURE Me() =3D3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&g= >t=3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN
>=3B >=3B>=3B>= >=3B>=3B>=3B>=3B>=3B>=3B ptr^ :=3D3D 0
>=3B >=3B>=3B>= >=3B>=3B>=3B>=3B>=3B>=3B END Me=3D3B
>=3B >=3B>=3B>=3B&= >gt=3B>=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B>=3B BEGIN END Crash.
>=3B >=3B>=3B>=3B>=3B&g= >t=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B= >>=3B>=3B Here's an example of what happens if you now call this from an= > interactiv=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B e
&g= >t=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B interpreter that catc= >hes the exception RuntimeError.E:
>=3B >=3B>=3B>=3B>=3B>=3B&= >gt=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= >=3B>=3B M-Scheme Experimental
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B>=3B LITHP ITH LITHENING.
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B>=3B>=3B (require-modules "m3")
>=3B >=3B>=3B&g= >t=3B>=3B>=3B>=3B>=3B>=3B #t
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B>=3B>=3B (Crash.Me)
>=3B >=3B>=3B>=3B>=3B&g= >t=3B>=3B>=3B>=3B EXCEPTION! RuntimeError! Attempt to reference an ill= >egal memory location.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B>=3B (+ 3 4)=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= >=3B>=3B 7
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B>= >=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20<= >BR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B I just realized = >I may have broken pthreads=3D2C let me go back and double-c=3D
>=3B &g= >t=3B>=3B>=3B>=3B>=3B>=3B>=3B heck it.=3D20
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B>=3B runtime/POSIX and thread/POSIX don'= >t refer to the same thing do they...
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B= > =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D
>=3B &= >gt=3B>=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_
>=3B &g= >t=3B>=3B>=3B>=3B>=3B>=3B>=3B Content-Type: text/html=3B charset= >=3D"iso-8859-1"
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Conten= >t-Transfer-Encoding: quoted-printable
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3B= >html>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3Bhead&= >gt=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3Bstyle>= >=3B<=3B!--
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B .hmmessag= >e P
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B {
>=3B >=3B= >>=3B>=3B>=3B>=3B>=3B>=3B margin:0px=3D3B
>=3B >=3B>=3B= >>=3B>=3B>=3B>=3B>=3B padding:0px
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B>=3B }
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= >=3B body.hmmessage
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B {R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B font-size: 10pt=3D3B>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B font-family:Tahoma
&g= >t=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B }
>=3B >=3B>=3B>= >=3B>=3B>=3B>=3B>=3B -->=3B<=3B/style>=3B
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B <=3B/head>=3B
>=3B >=3B>=3B&= >gt=3B>=3B>=3B>=3B>=3B <=3Bbody class=3D3D'hmmessage'>=3B
>= >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Letting any code run after a= > SIGSEGV is dubious.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B= >>=3B>=3B Imagine the heap&=3Bnbsp=3D3Bis corrupted.<=3BBR>=3B>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B And then you run more co= >de.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B And = >the code happens to call malloc.<=3BBR>=3B
>=3B >=3B>=3B>=3B= >>=3B>=3B>=3B>=3B Or printf to log something.<=3BBR>=3B
>= >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B &=3Bnbsp=3D3B<=3BBR>= >=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B I suppose there mi= >ght be an application that maps memory<=3BBR>=3B
>=3B >=3B>=3B= >>=3B>=3B>=3B>=3B>=3B gradually=3D2C as pieces of a buffer are hit= >. Might.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B= > &=3Bnbsp=3D3B<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B &=3Bnbsp=3D3B- Jay<=3BBR>=3B&=3Bnbsp=3D3B<=3BBR>=3B= >
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B &=3Bgt=3D3B To: m3= >devel at elegosoft.com<=3BBR>=3B&=3Bgt=3D3B Date: Sat=3D2C 19 Feb 2011 = >10:29:3=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B 0 -0800<= >=3BBR>=3B&=3Bgt=3D3B From: mika at async.caltech.edu<=3BBR>=3B&=3B= >gt=3D3B Subject: [M3devel]=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B SEGV mapping to RuntimeError<=3BBR>=3B&=3Bgt=3D3B<=3BBR&= >gt=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B Dear m3devel=3D
>=3B &= >gt=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D2C<=3BBR>=3B&=3Bgt=3D3B= ><=3BBR>=3B&=3Bgt=3D3B For a while it has annoyed me that segmentatio= >n vi=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B olations cause= > an<=3BBR>=3B&=3Bgt=3D3B unconditional program abort. I've changed t= >hat =3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B now so that (u= >nder user<=3BBR>=3B&=3Bgt=3D3B threads at least) we instead get a Ru= >ntim=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B eError. Here's= > an example of<=3BBR>=3B&=3Bgt=3D3B the mechanism at work in an inte= >ract=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B ive Scheme env= >ironment. Consider<=3BBR>=3B&=3Bgt=3D3B the unhelpful interface and = >modu=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B le Crash:<= >=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B INTERFACE Crash=3D3B = >PROCEDURE Me()=3D3B END Cra=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B sh.<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B MODUL= >E Crash=3D3B<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B PROCED= >URE Me(=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B ) =3D3D<= >=3BBR>=3B&=3Bgt=3D3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN<=3B= >BR>=3B&=3Bgt=3D3B ptr^ :=3D3D=3D
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B 0<=3BBR>=3B&=3Bgt=3D3B END Me=3D3B<=3BBR>=3B&= >=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B BEGIN END Crash.<=3BBR>=3B&= >=3Bgt=3D3B<=3BBR>=3B=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B &=3Bgt=3D3B Here's an example of what happens if you now call this= > from an inter=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B acti= >ve<=3BBR>=3B&=3Bgt=3D3B interpreter that catches the exception Runti= >meError.E:<=3BBR>=3B=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B &=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B M-Scheme Experimental<= >=3BBR>=3B&=3Bgt=3D3B LITHP ITH LITHENING.<=3BBR>=3B&=3B=3D
&= >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B gt=3D3B&=3Bgt=3D3B (req= >uire-modules "m3")<=3BBR>=3B&=3Bgt=3D3B #t<=3BBR>=3B&=3Bgt=3D= >3B&=3Bgt=3D3B (Crash.Me=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B )<=3BBR>=3B&=3Bgt=3D3B EXCEPTION! RuntimeError! Attempt to= > reference an illegal memory=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B location.<=3BBR>=3B&=3Bgt=3D3B&=3Bgt=3D3B (+ 3 4)<=3B= >BR>=3B&=3Bgt=3D3B 7<=3BBR>=3B&=3Bgt=3D3B&=3Bgt=3D3B<=3BBR&= >gt=3B&=3Bgt=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D3= >B<=3BBR>=3B&=3Bgt=3D3B I just realized I may have broken pthreads=3D= >2C let me go back=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B a= >nd double-check it.<=3BBR>=3B&=3Bgt=3D3B runtime/POSIX and thread/PO= >SIX don't refer=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B to = >the same thing do they...<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bg= >t=3D3B Mika<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B =3D
>=3B >=3B&= >gt=3B>=3B>=3B>=3B>=3B>=3B <=3B/body>=3B
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B <=3B/html>=3B=3D
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B&= >gt=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_--
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B
>=3B &g= >t=3B>=3B
>= > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_-- From jay.krell at cornell.edu Tue Feb 22 23:08:58 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 22 Feb 2011 22:08:58 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222184502.88B921A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , <4D604FA3.3060708@lcwb.coop>, , <20110219232729.53A991A2078@async.async.caltech.edu>, , <4D615643.7060107@lcwb.coop>, , <20110220183746.195C11A2078@async.async.caltech.edu>, , <4D61CC27.7070704@lcwb.coop>, , <20110222174405.E15481A2078@async.async.caltech.edu>, , <20110222184502.88B921A2078@async.async.caltech.edu> Message-ID: Heap vs. stack is indeed an unsolvable dilemna. Stack is much faster, but some arbitrary size must be chosen, that bothers me too, and detection and recovery from out of stack is tricky and not portable. Heap is slow, but portable, and generally limited to address space (but subject to fragmentation), easy to detect exhaustion. GC heap is usually fast to allocate, competitive with stack allocation, but the more you allocate, the more work for the GC to do. There is no free lunch -- not even as cheap as it might seem. :) Some systems..which I have never worked on..statically allocate everything. They need to know they will never run out of memory. > I realize my code is perhaps difficult to read. Not necessarily. I didn't even try yet. :) For many programmers, it is difficult to get them to read any code. :) - Jay > To: jay.krell at cornell.edu > Date: Tue, 22 Feb 2011 10:45:02 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > Jay K writes: > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >(aside=2C and possible agreement: right -- an interpreter should consider N= > >OT recursing on the machine > >stack whenever code it is interpreting recurses=2C but definitely some do) > >=20 > > - Jay > > > Jay, yes I agree---but: > > 1. of course it's easier to code the interpreter to recurse on the machine > stack > > 2. code called out from the interpreter will in any case recurse on the (a) > machine stack > > 3. again if one can "fix" the stack mechanism so that running out of > stack space is non-fatal, there's no reason not to use the machine stack > > I've found that Modula-3's garbage collector is brutally slow and that > converting code to using stack-allocated memory whenever possible is a > huge performance win under both PM3 and CM3. Perhaps this is a problem > with the garbage collector. > > I also have a partial design for a Scheme compiler (emitting Modula-3) > that would use the machine stack whenever possible rather than heap > allocation and do a "tricky thing" with local procedures to copy the > stack whenever a reference to the stack escapes. (Most of the time you > can implement Scheme on a stack but sometimes you need to keep part > of it live after you return; hence implementations (including mine) > heap-allocate local variables...) > > I realize my code is perhaps difficult to read. The gist of it is this: > when you run out of stack space, allocate a new stack anywhere else > in memory. Make it appear that you have alloca'd all the space between > the old and new stacks (into a variable that you of course never touch), > then continue. An issue is what to do when you return, you have to make > sure that you restore your old redzone.. perhaps you don't need to free > the stack eagerly: you could instead remember you have that page allocated > and re-use it next time you hit the redzone, or put it in a pool to allow > it to be used by another thread. > > The arbitrary choice of stack sizes has always bothered me greatly. > Some of my programs have thousands of threads and they use stack space > irregularly... > > Mika > > > >> To: rodney_bates at lcwb.coop > >> Date: Tue=2C 22 Feb 2011 09:44:05 -0800 > >> From: mika at async.caltech.edu > >> CC: m3devel at elegosoft.com > >> Subject: Re: [M3devel] SEGV mapping to RuntimeError > >>=20 > >>=20 > >> Ok so I was thinking about this. > >>=20 > >> Why on earth is stack overflow even a problem? > >>=20 > >> Consider the following procedure call (in my code=2C stack grows upwards)= > >: > >>=20 > >> (* sp at x=2C pc at y *) > >> y: P(args) > >> z: next_statement > >>=20 > >> decompose as follows: > >>=20 > >> (* sp at x=2C pc at y *) > >> y: Push(args etc. and ret. address z) > >> Jump(P) > >> z: next_statement > >>=20 > >> Now=2C we say: > >>=20 > >> y: ok :=3D check_stack(size of frame) > >> IF NOT ok THEN abort() END=3B > >> Push(args etc. and ret. address z) > >> Jump(P) > >> z: next_statement > >>=20 > >> (note check_stack and the following IF can be implemented by hardware=2C > >> need not actually be an instruction) > >>=20 > >> Let me change the code a tad: > >>=20 > >> y: ok :=3D check_stack(size of frame) > >> y':IF NOT ok THEN=20 > >> WITH new_stack_bottom =3D malloc(stack_size) > >> huge_amount =3D new_stack_bottom - sp DO > >> create_redzone_at(new_stack_bottom+stack_size-redzone_size) > >> EVAL alloca(huge_amount)=20 > >> END > >> END=3B > >> Push(args etc. and ret. address z) > >> Jump(P) > >> z: IF NOT ok THEN destroy_redzone(...)=3B free(new_stack_bottom) END > >>=20 > >> Note 1. cleanup of redzone could be postponed to return of caller....when > >> alloca in any case has to be cleaned up. > >>=20 > >> Note 2. the test IF NOT ok at z is more expensive to implement than the > >> one at y because you can't really use hardware for it. A hardware callbac= > >k > >> can be arranged though: > >>=20 > >> VAR ptr :=3D sp=3B > >> y: ok :=3D check_stack(size of frame) > >> y':IF NOT ok THEN=20 > >> ptr :=3D 0=3B (* illegal address *) > >> fault_address :=3D z=3B > >> WITH new_stack_bottom =3D malloc(stack_size) > >> huge_amount =3D new_stack_bottom - sp DO > >> create_redzone_at(new_stack_bottom+stack_size-redzone_size) > >> EVAL alloca(huge_amount)=20 > >> END > >> END=3B > >> Push(args etc. and ret. address z) > >> Jump(P) > >> z: EVAL ptr^ (* [ NOT ok -> hardware callback to SEGV: ] *) > >>=20 > >> SEGV(signalpc): IF NOT ok AND signalpc =3D fault_address THEN destroy_red= > >zone(...)=3B free(new_stack_bottom) END > >>=20 > >> Mika > >>=20 > >>=20 > >>=20 > >>=20 > >>=20 > >> "Rodney M. Bates" writes: > >> > > >> > > >> >On 02/20/2011 12:37 PM=2C Mika Nystrom wrote: > >> >> On a 64-bit machine=2C at least=2C there ought to be enough virtual > >> >> memory that you could just have a gap between thread stacks big > >> >> enough to allow for a protection area larger than the largest possible > >> >> (implementation-defined) activation record=2C no? I know I've run into > >> >> trouble with very large activation records in the past (and not becaus= > >e > >> >> I was running out of stack space=2C either). > >> >> > >> >> Or at least a procedure with a very large activation record (or > >> >> a procedure calling it) could be required to call some sort of check > >> >> routine "EnoughStackSpaceRemaining()" before starting to scribble > >> >> on the activation record? > >> > > >> >Hmm=2C I like this idea. It would introduce normal-case runtime overhead > >> >only for such procedures=2C and these are likely rare. Also=2C assuming = > >the procedure > >> >actually uses very much of its large AR=2C it should also have enough co= > >mputation > >> >time to wash out the stack check overhead. > >> > > >> >> > >> >> Also the end of the activation record must be written to at least once= > >=2C > >> >> or else the memory protection won't be triggered. > >> >> > >> > > >> >I was thinking (as an alternative mechanism) of having the compiler inte= > >ntionally > >> >add enough artificial write(s) as necessary to ensure storing within the > >> >red zone=2C and not just beyond it. This seems trickier to get right and > >> >harder to distinguish after the fact from a NIL dereference. > >> > > >> >> In any case if this is done properly the same mechanism I proposed for > >> >> SIGSEGV ought to be able to catch stack overflow=2C no? Well=2C as lon= > >g as > >> >> signals are delivered on a separate stack. If signals are delivered on > >> >> the same stack=2C the signal handler would get nastier=2C it would hav= > >e to > >> >> make space through some manipulations (maybe temporarily unporotecting > >> >> the redzone page?) for its own purposes... but I don't see why it > >> >> couldn't be done. > >> >> > >> >> Not sure why I'm getting SIGILL... maybe I am getting my signal handle= > >r > >> >> activated inside the redzone page because of a difference in signal > >> >> handling..? I remember reading something about sigaltstack... > >> >> > >> >> I would of course love to be able to recover from stack overflow=2C to= > >o. > >> >> In some sense=2C since it's a generally unknown limit=2C it's even les= > >s of > >> >> a fatal error than a NIL dereference (hence makes even more sense to > >> >> catch it). > >> > > >> >I think this would be a nice mechanism to have available. It would have = > >to > >> >be used with some care. In any case=2C it would be really nice and more > >> >frequently so=2C to at least have runtime error messages that distinguis= > >hed > >> >stack overflow from NIL deref. > >> > > >> >> > >> >> Mika > >> >> > >> >> "Rodney M. Bates" writes: > >> >>> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AM= > >D64_LINUX. > >> >>> Probably a fully protected guard page at the end of the stack. This t= > >echnique > >> >>> always worries me a bit because a procedure with a really big activat= > >ion record > >> >>> could jump right past it. Probably it would almost always access the = > >first page > >> >>> of the big area before storing anything into later pages. > >> >>> > >> >>> On 02/19/2011 05:27 PM=2C Mika Nystrom wrote: > >> >>>> Ah=2C yes=2C stack protection. > >> >>>> > >> >>>> Do you know if it's a SIGSEGV=2C not a SIGBUS? I know I have seen SI= > >GILL on Macs. > >> >>>> > >> >>>> Hmm=2C I get SIGILL on AMD64_FREEBSD as well: > >> >>>> > >> >>>> time ../AMD64_FREEBSD/stubexample > >> >>>> M-Scheme Experimental > >> >>>> LITHP ITH LITHENING. > >> >>>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >> >>>> f > >> >>>>> (f 0) > >> >>>> Illegal instruction > >> >>>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >> >>>> > >> >>>> What absolutely must not happen=2C of course=2C is that the runtime = > >hangs > >> >>>> while executing only safe code... > >> >>>> > >> >>>> Mika > >> >>>> > >> >>>> "Rodney M. Bates" writes: > >> >>>>> I know of one other place the compilers rely on hardware memory pro= > >tection > >> >>>>> to detect a checked runtime error=2C and that is stack overflow. Th= > >is won't > >> >>>>> corrupt anything=2C but is hard to distinguish from dereferencing N= > >IL. > >> >>>>> This could probably be distinguished after the fact by some low-lev= > >el=2C > >> >>>>> target-dependent code. I have found it by looking at assembly code = > >at > >> >>>>> the point of failure--usually right after a stack pointer push. > >> >>>>> > >> >>>>> Detecting this via compiler-generated checks would probably be more > >> >>>>> extravagant than many other checks=2C as it is so frequent. I am no= > >t > >> >>>>> aware of any really good solution to this in any implementation of = > >any > >> >>>>> language. > >> >>>>> > >> >>>>> On 02/19/2011 02:38 PM=2C Mika Nystrom wrote: > >> >>>>>> Jay=2C sometimes I wonder about you: this is a Modula-3 mailing li= > >st=2C > >> >>>>>> you know! > >> >>>>>> > >> >>>>>> "Corrupting the heap" is something that can only happen as a resul= > >t of > >> >>>>>> an unchecked runtime error. Unchecked runtime errors cannot happen= > > in > >> >>>>>> modules not marked UNSAFE. > >> >>>>>> > >> >>>>>> SEGV is=2C however=2C used by the CM3 implementation (and its pred= > >ecessors) > >> >>>>>> to signal a certain kind of *checked* runtime error=2C namely=2C t= > >he > >> >>>>>> dereferencing of a NIL reference. Correct me if I am wrong=2C but = > >an > >> >>>>>> attempt to dereference NIL is not going to leave the heap corrupte= > >d? > >> >>>>>> > >> >>>>>> And if you stick to safe code=2C the only SEGVs I think you get in= > > the > >> >>>>>> current CM3 are ones from NIL dereferences. > >> >>>>>> > >> >>>>>> Hence=2C as long as you stick with safe code=2C the only time the = > >code I > >> >>>>>> checked in earlier gets triggered is for NIL dereferences=2C which= > > should > >> >>>>>> never corrupt the heap. So SEGV is not sometimes=2C but in fact al= > >ways > >> >>>>>> recoverable. > >> >>>>>> > >> >>>>>> :-) > >> >>>>>> > >> >>>>>> Mika > >> >>>>>> > >> >>>>>> P.S. the bit above "if you stick to safe code": if you actually pr= > >ogram in > >> >>>>>> Modula-3 you almost never use UNSAFE. I went through my repository= > > and > >> >>>>>> I have 40 modules using UNSAFE out of a total of 4=2C559. Furtherm= > >ore=2C > >> >>>>>> many of the UNSAFE modules are glue code to Fortran routines=2C wh= > >ich > >> >>>>>> could relatively easily be verified to be safe in the Modula-3 sen= > >se. > >> >>>>>> Almost all what remains is glue to some C library=2C which wouldn'= > >t be > >> >>>>>> necessary if the rest of the world would wake up out of the dark a= > >ges=2C but > >> >>>>>> I don't have the time to rewrite every single library from scratch= > > myself. > >> >>>>>> > >> >>>>>> > >> >>>>>> Jay K writes: > >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >> >>>>>>> Content-Type: text/plain=3B charset=3D"iso-8859-1" > >> >>>>>>> Content-Transfer-Encoding: quoted-printable > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> Letting any code run after a SIGSEGV is dubious. > >> >>>>>>> Imagine the heap is corrupted. > >> >>>>>>> And then you run more code. > >> >>>>>>> And the code happens to call malloc. > >> >>>>>>> Or printf to log something. > >> >>>>>>> =3D20 > >> >>>>>>> I suppose there might be an application that maps memory > >> >>>>>>> gradually=3D2C as pieces of a buffer are hit. Might. > >> >>>>>>> =3D20 > >> >>>>>>> - Jay > >> >>>>>>> =3D20 > >> >>>>>>>> To: m3devel at elegosoft.com > >> >>>>>>>> Date: Sat=3D2C 19 Feb 2011 10:29:30 -0800 > >> >>>>>>>> From: mika at async.caltech.edu > >> >>>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >> >>>>>>>> =3D20 > >> >>>>>>>> =3D20 > >> >>>>>>>> Dear m3devel=3D2C > >> >>>>>>>> =3D20 > >> >>>>>>>> For a while it has annoyed me that segmentation violations cause= > > an > >> >>>>>>>> unconditional program abort. I've changed that now so that (unde= > >r user > >> >>>>>>>> threads at least) we instead get a RuntimeError. Here's an examp= > >le of > >> >>>>>>>> the mechanism at work in an interactive Scheme environment. Cons= > >ider > >> >>>>>>>> the unhelpful interface and module Crash: > >> >>>>>>>> =3D20 > >> >>>>>>>> INTERFACE Crash=3D3B PROCEDURE Me()=3D3B END Crash. > >> >>>>>>>> =3D20 > >> >>>>>>>> MODULE Crash=3D3B > >> >>>>>>>> =3D20 > >> >>>>>>>> PROCEDURE Me() =3D3D > >> >>>>>>>> VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN > >> >>>>>>>> ptr^ :=3D3D 0 > >> >>>>>>>> END Me=3D3B > >> >>>>>>>> =3D20 > >> >>>>>>>> BEGIN END Crash. > >> >>>>>>>> =3D20 > >> >>>>>>>> Here's an example of what happens if you now call this from an i= > >nteractiv=3D > >> >>>>>>> e > >> >>>>>>>> interpreter that catches the exception RuntimeError.E: > >> >>>>>>>> =3D20 > >> >>>>>>>> M-Scheme Experimental > >> >>>>>>>> LITHP ITH LITHENING. > >> >>>>>>>>> (require-modules "m3") > >> >>>>>>>> #t > >> >>>>>>>>> (Crash.Me) > >> >>>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory = > >location. > >> >>>>>>>>> (+ 3 4)=3D20 > >> >>>>>>>> 7 > >> >>>>>>>>> =3D20 > >> >>>>>>>> =3D20 > >> >>>>>>>> I just realized I may have broken pthreads=3D2C let me go back a= > >nd double-c=3D > >> >>>>>>> heck it.=3D20 > >> >>>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do = > >they... > >> >>>>>>>> =3D20 > >> >>>>>>>> Mika > >> >>>>>>>> =3D20 > >> >>>>>>> =3D > >> >>>>>>> > >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >> >>>>>>> Content-Type: text/html=3B charset=3D"iso-8859-1" > >> >>>>>>> Content-Transfer-Encoding: quoted-printable > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> Letting any code run after a SIGSEGV is dubious.
> >> >>>>>>> Imagine the heap =3D3Bis corrupted.
> >> >>>>>>> And then you run more code.
> >> >>>>>>> And the code happens to call malloc.
> >> >>>>>>> Or printf to log something.
> >> >>>>>>>  =3D3B
> >> >>>>>>> I suppose there might be an application that maps memory
> >> >>>>>>> gradually=3D2C as pieces of a buffer are hit. Might.
> >> >>>>>>>  =3D3B
> >> >>>>>>>  =3D3B- Jay
 =3D3B
> >> >>>>>>> >=3D3B To: m3devel at elegosoft.com
>=3D3B Date: Sat=3D2C 19 = > >Feb 2011 10:29:3=3D > >> >>>>>>> 0 -0800
>=3D3B From: mika at async.caltech.edu
>=3D3B Subj= > >ect: [M3devel]=3D > >> >>>>>>> SEGV mapping to RuntimeError
>=3D3B
>=3D3B
>=3D3B = > >Dear m3devel=3D > >> >>>>>>> =3D2C
>=3D3B
>=3D3B For a while it has annoyed me that = > >segmentation vi=3D > >> >>>>>>> olations cause an
>=3D3B unconditional program abort. I've c= > >hanged that =3D > >> >>>>>>> now so that (under user
>=3D3B threads at least) we instead = > >get a Runtim=3D > >> >>>>>>> eError. Here's an example of
>=3D3B the mechanism at work in= > > an interact=3D > >> >>>>>>> ive Scheme environment. Consider
>=3D3B the unhelpful interf= > >ace and modu=3D > >> >>>>>>> le Crash:
>=3D3B
>=3D3B INTERFACE Crash=3D3B PROCEDURE = > >Me()=3D3B END Cra=3D > >> >>>>>>> sh.
>=3D3B
>=3D3B MODULE Crash=3D3B
>=3D3B
>= > >=3D3B PROCEDURE Me(=3D > >> >>>>>>> ) =3D3D
>=3D3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN >R>>=3D3B ptr^ :=3D3D=3D > >> >>>>>>> 0
>=3D3B END Me=3D3B
>=3D3B
>=3D3B BEGIN END Crash= > >.
>=3D3B
=3D > >> >>>>>>> >=3D3B Here's an example of what happens if you now call this f= > >rom an inter=3D > >> >>>>>>> active
>=3D3B interpreter that catches the exception Runtime= > >Error.E:
=3D > >> >>>>>>> >=3D3B
>=3D3B M-Scheme Experimental
>=3D3B LITHP ITH = > >LITHENING.
&=3D > >> >>>>>>> gt=3D3B>=3D3B (require-modules "m3")
>=3D3B #t
>=3D3B= > >>=3D3B (Crash.Me=3D > >> >>>>>>> )
>=3D3B EXCEPTION! RuntimeError! Attempt to reference an il= > >legal memory=3D > >> >>>>>>> location.
>=3D3B>=3D3B (+ 3 4)
>=3D3B 7
>=3D3B&g= > >t=3D3B
>=3D > >> >>>>>>> =3D3B
>=3D3B I just realized I may have broken pthreads=3D2C= > > let me go back=3D > >> >>>>>>> and double-check it.
>=3D3B runtime/POSIX and thread/POSIX d= > >on't refer=3D > >> >>>>>>> to the same thing do they...
>=3D3B
>=3D3B Mika
>= > >=3D3B
=3D > >> >>>>>>> > >> >>>>>>> =3D > >> >>>>>>> > >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >> >>>>>> > >> >>>> > >> >> > > = > > > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_ > >Content-Type: text/html; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > > > > > > > >(aside=2C and possible agreement: right -- an interpreter should consider N= > >OT recursing on the machine
> >stack whenever code it is interpreting recurses=2C but definitely some do)<= > >BR> > > =3B
> > =3B- Jay
 =3B
> >>=3B To: rodney_bates at lcwb.coop
>=3B Date: Tue=2C 22 Feb 2011 09:44:= > >05 -0800
>=3B From: mika at async.caltech.edu
>=3B CC: m3devel at elego= > >soft.com
>=3B Subject: Re: [M3devel] SEGV mapping to RuntimeError
&= > >gt=3B
>=3B
>=3B Ok so I was thinking about this.
>=3B
= > >>=3B Why on earth is stack overflow even a problem?
>=3B
>=3B = > >Consider the following procedure call (in my code=2C stack grows upwards):<= > >BR>>=3B
>=3B (* sp at x=2C pc at y *)
>=3B y: P(args)
>= > >=3B z: next_statement
>=3B
>=3B decompose as follows:
>=3B = > >
>=3B (* sp at x=2C pc at y *)
>=3B y: Push(args etc. and ret. ad= > >dress z)
>=3B Jump(P)
>=3B z: next_statement
>=3B
>=3B= > > Now=2C we say:
>=3B
>=3B y: ok :=3D check_stack(size of frame)<= > >BR>>=3B IF NOT ok THEN abort() END=3B
>=3B Push(args etc. and ret. a= > >ddress z)
>=3B Jump(P)
>=3B z: next_statement
>=3B
>= > >=3B (note check_stack and the following IF can be implemented by hardware= > >=2C
>=3B need not actually be an instruction)
>=3B
>=3B Let= > > me change the code a tad:
>=3B
>=3B y: ok :=3D check_stack(size= > > of frame)
>=3B y':IF NOT ok THEN
>=3B WITH new_stack_bottom =3D= > > malloc(stack_size)
>=3B huge_amount =3D new_stack_bottom - sp DO
&= > >gt=3B create_redzone_at(new_stack_bottom+stack_size-redzone_size)
>=3B= > > EVAL alloca(huge_amount)
>=3B END
>=3B END=3B
>=3B Push(ar= > >gs etc. and ret. address z)
>=3B Jump(P)
>=3B z: IF NOT ok THEN d= > >estroy_redzone(...)=3B free(new_stack_bottom) END
>=3B
>=3B Note= > > 1. cleanup of redzone could be postponed to return of caller....when
&g= > >t=3B alloca in any case has to be cleaned up.
>=3B
>=3B Note 2. = > >the test IF NOT ok at z is more expensive to implement than the
>=3B o= > >ne at y because you can't really use hardware for it. A hardware callback >R>>=3B can be arranged though:
>=3B
>=3B VAR ptr :=3D sp=3B >>>=3B y: ok :=3D check_stack(size of frame)
>=3B y':IF NOT ok THEN <= > >BR>>=3B ptr :=3D 0=3B (* illegal address *)
>=3B fault_address :=3D = > >z=3B
>=3B WITH new_stack_bottom =3D malloc(stack_size)
>=3B huge_= > >amount =3D new_stack_bottom - sp DO
>=3B create_redzone_at(new_stack_b= > >ottom+stack_size-redzone_size)
>=3B EVAL alloca(huge_amount)
>= > >=3B END
>=3B END=3B
>=3B Push(args etc. and ret. address z)
&g= > >t=3B Jump(P)
>=3B z: EVAL ptr^ (* [ NOT ok ->=3B hardware callback t= > >o SEGV: ] *)
>=3B
>=3B SEGV(signalpc): IF NOT ok AND signalpc = > >=3D fault_address THEN destroy_redzone(...)=3B free(new_stack_bottom) END >R>>=3B
>=3B Mika
>=3B
>=3B
>=3B
>=3B
>= > >=3B
>=3B "Rodney M. Bates" writes:
>=3B >=3B
>=3B >=3B<= > >BR>>=3B >=3BOn 02/20/2011 12:37 PM=2C Mika Nystrom wrote:
>=3B >= > >=3B>=3B On a 64-bit machine=2C at least=2C there ought to be enough virtu= > >al
>=3B >=3B>=3B memory that you could just have a gap between thr= > >ead stacks big
>=3B >=3B>=3B enough to allow for a protection area= > > larger than the largest possible
>=3B >=3B>=3B (implementation-de= > >fined) activation record=2C no? I know I've run into
>=3B >=3B>=3B= > > trouble with very large activation records in the past (and not because >>>=3B >=3B>=3B I was running out of stack space=2C either).
>=3B= > > >=3B>=3B
>=3B >=3B>=3B Or at least a procedure with a very la= > >rge activation record (or
>=3B >=3B>=3B a procedure calling it) co= > >uld be required to call some sort of check
>=3B >=3B>=3B routine "= > >EnoughStackSpaceRemaining()" before starting to scribble
>=3B >=3B&g= > >t=3B on the activation record?
>=3B >=3B
>=3B >=3BHmm=2C I li= > >ke this idea. It would introduce normal-case runtime overhead
>=3B >= > >=3Bonly for such procedures=2C and these are likely rare. Also=2C assuming = > >the procedure
>=3B >=3Bactually uses very much of its large AR=2C it= > > should also have enough computation
>=3B >=3Btime to wash out the s= > >tack check overhead.
>=3B >=3B
>=3B >=3B>=3B
>=3B >= > >=3B>=3B Also the end of the activation record must be written to at least= > > once=2C
>=3B >=3B>=3B or else the memory protection won't be trig= > >gered.
>=3B >=3B>=3B
>=3B >=3B
>=3B >=3BI was thinki= > >ng (as an alternative mechanism) of having the compiler intentionally
&g= > >t=3B >=3Badd enough artificial write(s) as necessary to ensure storing wi= > >thin the
>=3B >=3Bred zone=2C and not just beyond it. This seems tri= > >ckier to get right and
>=3B >=3Bharder to distinguish after the fact= > > from a NIL dereference.
>=3B >=3B
>=3B >=3B>=3B In any cas= > >e if this is done properly the same mechanism I proposed for
>=3B >= > >=3B>=3B SIGSEGV ought to be able to catch stack overflow=2C no? Well=2C a= > >s long as
>=3B >=3B>=3B signals are delivered on a separate stack.= > > If signals are delivered on
>=3B >=3B>=3B the same stack=2C the s= > >ignal handler would get nastier=2C it would have to
>=3B >=3B>=3B = > >make space through some manipulations (maybe temporarily unporotecting
&= > >gt=3B >=3B>=3B the redzone page?) for its own purposes... but I don't s= > >ee why it
>=3B >=3B>=3B couldn't be done.
>=3B >=3B>=3B >R>>=3B >=3B>=3B Not sure why I'm getting SIGILL... maybe I am getting= > > my signal handler
>=3B >=3B>=3B activated inside the redzone page= > > because of a difference in signal
>=3B >=3B>=3B handling..? I rem= > >ember reading something about sigaltstack...
>=3B >=3B>=3B
>= > >=3B >=3B>=3B I would of course love to be able to recover from stack ov= > >erflow=2C too.
>=3B >=3B>=3B In some sense=2C since it's a general= > >ly unknown limit=2C it's even less of
>=3B >=3B>=3B a fatal error = > >than a NIL dereference (hence makes even more sense to
>=3B >=3B>= > >=3B catch it).
>=3B >=3B
>=3B >=3BI think this would be a nic= > >e mechanism to have available. It would have to
>=3B >=3Bbe used wit= > >h some care. In any case=2C it would be really nice and more
>=3B >= > >=3Bfrequently so=2C to at least have runtime error messages that distinguis= > >hed
>=3B >=3Bstack overflow from NIL deref.
>=3B >=3B
>= > >=3B >=3B>=3B
>=3B >=3B>=3B Mika
>=3B >=3B>=3B
>= > >=3B >=3B>=3B "Rodney M. Bates" writes:
>=3B >=3B>=3B>=3B I a= > >m pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX= > >.
>=3B >=3B>=3B>=3B Probably a fully protected guard page at the= > > end of the stack. This technique
>=3B >=3B>=3B>=3B always worri= > >es me a bit because a procedure with a really big activation record
>= > >=3B >=3B>=3B>=3B could jump right past it. Probably it would almost a= > >lways access the first page
>=3B >=3B>=3B>=3B of the big area be= > >fore storing anything into later pages.
>=3B >=3B>=3B>=3B
>= > >=3B >=3B>=3B>=3B On 02/19/2011 05:27 PM=2C Mika Nystrom wrote:
>= > >=3B >=3B>=3B>=3B>=3B Ah=2C yes=2C stack protection.
>=3B >= > >=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B Do you know if it'= > >s a SIGSEGV=2C not a SIGBUS? I know I have seen SIGILL on Macs.
>=3B &= > >gt=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B Hmm=2C I get SIG= > >ILL on AMD64_FREEBSD as well:
>=3B >=3B>=3B>=3B>=3B
>=3B = > >>=3B>=3B>=3B>=3B time ../AMD64_FREEBSD/stubexample
>=3B >=3B= > >>=3B>=3B>=3B M-Scheme Experimental
>=3B >=3B>=3B>=3B>=3B= > > LITHP ITH LITHENING.
>=3B >=3B>=3B>=3B>=3B>=3B (define (f a= > >) (+ (f (+ a 1)) (f (+ a 2))))
>=3B >=3B>=3B>=3B>=3B f
>= > >=3B >=3B>=3B>=3B>=3B>=3B (f 0)
>=3B >=3B>=3B>=3B>=3B= > > Illegal instruction
>=3B >=3B>=3B>=3B>=3B 3.847u 0.368s 0:13.= > >32 31.5% 2160+284478k 0+0io 0pf+0w
>=3B >=3B>=3B>=3B>=3B
&g= > >t=3B >=3B>=3B>=3B>=3B What absolutely must not happen=2C of course= > >=2C is that the runtime hangs
>=3B >=3B>=3B>=3B>=3B while exec= > >uting only safe code...
>=3B >=3B>=3B>=3B>=3B
>=3B >=3B= > >>=3B>=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B
>=3B >=3B= > >>=3B>=3B>=3B "Rodney M. Bates" writes:
>=3B >=3B>=3B>=3B&g= > >t=3B>=3B I know of one other place the compilers rely on hardware memory = > >protection
>=3B >=3B>=3B>=3B>=3B>=3B to detect a checked run= > >time error=2C and that is stack overflow. This won't
>=3B >=3B>=3B= > >>=3B>=3B>=3B corrupt anything=2C but is hard to distinguish from dere= > >ferencing NIL.
>=3B >=3B>=3B>=3B>=3B>=3B This could probably= > > be distinguished after the fact by some low-level=2C
>=3B >=3B>= > >=3B>=3B>=3B>=3B target-dependent code. I have found it by looking at = > >assembly code at
>=3B >=3B>=3B>=3B>=3B>=3B the point of fail= > >ure--usually right after a stack pointer push.
>=3B >=3B>=3B>=3B= > >>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B Detecting this via co= > >mpiler-generated checks would probably be more
>=3B >=3B>=3B>=3B= > >>=3B>=3B extravagant than many other checks=2C as it is so frequent. I = > >am not
>=3B >=3B>=3B>=3B>=3B>=3B aware of any really good so= > >lution to this in any implementation of any
>=3B >=3B>=3B>=3B>= > >=3B>=3B language.
>=3B >=3B>=3B>=3B>=3B>=3B
>=3B >= > >=3B>=3B>=3B>=3B>=3B On 02/19/2011 02:38 PM=2C Mika Nystrom wrote: >R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B Jay=2C sometimes I wonder abo= > >ut you: this is a Modula-3 mailing list=2C
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B you know!
>=3B >=3B>=3B>=3B>=3B>=3B>=3B >>>=3B >=3B>=3B>=3B>=3B>=3B>=3B "Corrupting the heap" is somet= > hing that can only happen as a result of
>=3B >=3B>=3B>=3B>=3B= > >>=3B>=3B an unchecked runtime error. Unchecked runtime errors cannot ha= > >ppen in
>=3B >=3B>=3B>=3B>=3B>=3B>=3B modules not marked U= > >NSAFE.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B SEGV is=2C however=2C used by the CM3 implement= > >ation (and its predecessors)
>=3B >=3B>=3B>=3B>=3B>=3B>=3B= > > to signal a certain kind of *checked* runtime error=2C namely=2C the
&g= > >t=3B >=3B>=3B>=3B>=3B>=3B>=3B dereferencing of a NIL reference.= > > Correct me if I am wrong=2C but an
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B attempt to dereference NIL is not going to leave the heap corrupt= > >ed?
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B&g= > >t=3B>=3B>=3B>=3B And if you stick to safe code=2C the only SEGVs I th= > >ink you get in the
>=3B >=3B>=3B>=3B>=3B>=3B>=3B current C= > >M3 are ones from NIL dereferences.
>=3B >=3B>=3B>=3B>=3B>=3B= > >>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B Hence=2C as long as y= > >ou stick with safe code=2C the only time the code I
>=3B >=3B>=3B&= > >gt=3B>=3B>=3B>=3B checked in earlier gets triggered is for NIL derefe= > >rences=2C which should
>=3B >=3B>=3B>=3B>=3B>=3B>=3B never= > > corrupt the heap. So SEGV is not sometimes=2C but in fact always
>=3B= > > >=3B>=3B>=3B>=3B>=3B>=3B recoverable.
>=3B >=3B>=3B&g= > >t=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B :-) >R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B&= > >gt=3B>=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
&g= > >t=3B >=3B>=3B>=3B>=3B>=3B>=3B P.S. the bit above "if you stick = > >to safe code": if you actually program in
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B Modula-3 you almost never use UNSAFE. I went through my rep= > >ository and
>=3B >=3B>=3B>=3B>=3B>=3B>=3B I have 40 module= > >s using UNSAFE out of a total of 4=2C559. Furthermore=2C
>=3B >=3B&g= > >t=3B>=3B>=3B>=3B>=3B many of the UNSAFE modules are glue code to Fo= > >rtran routines=2C which
>=3B >=3B>=3B>=3B>=3B>=3B>=3B coul= > >d relatively easily be verified to be safe in the Modula-3 sense.
>=3B= > > >=3B>=3B>=3B>=3B>=3B>=3B Almost all what remains is glue to so= > >me C library=2C which wouldn't be
>=3B >=3B>=3B>=3B>=3B>=3B&= > >gt=3B necessary if the rest of the world would wake up out of the dark ages= > >=2C but
>=3B >=3B>=3B>=3B>=3B>=3B>=3B I don't have the tim= > >e to rewrite every single library from scratch myself.
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B<= > >BR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B Jay K writes:
>=3B >= > >=3B>=3B>=3B>=3B>=3B>=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912= > >854f_
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Content-Type: te= > >xt/plain=3B charset=3D"iso-8859-1"
>=3B >=3B>=3B>=3B>=3B>=3B= > >>=3B>=3B Content-Transfer-Encoding: quoted-printable
>=3B >=3B&g= > >t=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B= > >>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Letting a= > >ny code run after a SIGSEGV is dubious.
>=3B >=3B>=3B>=3B>=3B&= > >gt=3B>=3B>=3B Imagine the heap is corrupted.
>=3B >=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B And then you run more code.
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B And the code happens to call malloc.
&= > >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Or printf to log something= > >.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>=3B >= > >=3B>=3B>=3B>=3B>=3B>=3B>=3B I suppose there might be an applica= > >tion that maps memory
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >gradually=3D2C as pieces of a buffer are hit. Might.
>=3B >=3B>=3B= > >>=3B>=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B - Jay
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B To: m3deve= > >l at elegosoft.com
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >Date: Sat=3D2C 19 Feb 2011 10:29:30 -0800
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B From: mika at async.caltech.edu
>=3B >=3B&g= > >t=3B>=3B>=3B>=3B>=3B>=3B>=3B Subject: [M3devel] SEGV mapping to= > > RuntimeError
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
&= > >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B Dear m3devel=3D2C >>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>=3B &g= > >t=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B For a while it has annoyed m= > >e that segmentation violations cause an
>=3B >=3B>=3B>=3B>=3B&= > >gt=3B>=3B>=3B>=3B unconditional program abort. I've changed that now = > >so that (under user
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B threads at least) we instead get a RuntimeError. Here's an example of >R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B the mechanism at = > >work in an interactive Scheme environment. Consider
>=3B >=3B>=3B&= > >gt=3B>=3B>=3B>=3B>=3B>=3B the unhelpful interface and module Cras= > >h:
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>= > >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B INTERFACE Crash=3D3B P= > >ROCEDURE Me()=3D3B END Crash.
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B MODULE Crash=3D3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >PROCEDURE Me() =3D3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&g= > >t=3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN
>=3B >=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B ptr^ :=3D3D 0
>=3B >=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B END Me=3D3B
>=3B >=3B>=3B>=3B&= > >gt=3B>=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B BEGIN END Crash.
>=3B >=3B>=3B>=3B>=3B&g= > >t=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B= > >>=3B>=3B Here's an example of what happens if you now call this from an= > > interactiv=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B e
&g= > >t=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B interpreter that catc= > >hes the exception RuntimeError.E:
>=3B >=3B>=3B>=3B>=3B>=3B&= > >gt=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B M-Scheme Experimental
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B LITHP ITH LITHENING.
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B (require-modules "m3")
>=3B >=3B>=3B&g= > >t=3B>=3B>=3B>=3B>=3B>=3B #t
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B (Crash.Me)
>=3B >=3B>=3B>=3B>=3B&g= > >t=3B>=3B>=3B>=3B EXCEPTION! RuntimeError! Attempt to reference an ill= > >egal memory location.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B>=3B (+ 3 4)=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B 7
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20<= > >BR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B I just realized = > >I may have broken pthreads=3D2C let me go back and double-c=3D
>=3B &g= > >t=3B>=3B>=3B>=3B>=3B>=3B>=3B heck it.=3D20
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B>=3B runtime/POSIX and thread/POSIX don'= > >t refer to the same thing do they...
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B= > > =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D
>=3B &= > >gt=3B>=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_
>=3B &g= > >t=3B>=3B>=3B>=3B>=3B>=3B>=3B Content-Type: text/html=3B charset= > >=3D"iso-8859-1"
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Conten= > >t-Transfer-Encoding: quoted-printable
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3B= > >html>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3Bhead&= > >gt=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3Bstyle>= > >=3B<=3B!--
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B .hmmessag= > >e P
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B {
>=3B >=3B= > >>=3B>=3B>=3B>=3B>=3B>=3B margin:0px=3D3B
>=3B >=3B>=3B= > >>=3B>=3B>=3B>=3B>=3B padding:0px
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B }
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B body.hmmessage
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B { >R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B font-size: 10pt=3D3B >>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B font-family:Tahoma
&g= > >t=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B }
>=3B >=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B -->=3B<=3B/style>=3B
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B <=3B/head>=3B
>=3B >=3B>=3B&= > >gt=3B>=3B>=3B>=3B>=3B <=3Bbody class=3D3D'hmmessage'>=3B
>= > >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Letting any code run after a= > > SIGSEGV is dubious.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B= > >>=3B>=3B Imagine the heap&=3Bnbsp=3D3Bis corrupted.<=3BBR>=3B >>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B And then you run more co= > >de.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B And = > >the code happens to call malloc.<=3BBR>=3B
>=3B >=3B>=3B>=3B= > >>=3B>=3B>=3B>=3B Or printf to log something.<=3BBR>=3B
>= > >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B &=3Bnbsp=3D3B<=3BBR>= > >=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B I suppose there mi= > >ght be an application that maps memory<=3BBR>=3B
>=3B >=3B>=3B= > >>=3B>=3B>=3B>=3B>=3B gradually=3D2C as pieces of a buffer are hit= > >. Might.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B= > > &=3Bnbsp=3D3B<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B &=3Bnbsp=3D3B- Jay<=3BBR>=3B&=3Bnbsp=3D3B<=3BBR>=3B= > >
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B &=3Bgt=3D3B To: m3= > >devel at elegosoft.com<=3BBR>=3B&=3Bgt=3D3B Date: Sat=3D2C 19 Feb 2011 = > >10:29:3=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B 0 -0800<= > >=3BBR>=3B&=3Bgt=3D3B From: mika at async.caltech.edu<=3BBR>=3B&=3B= > >gt=3D3B Subject: [M3devel]=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B SEGV mapping to RuntimeError<=3BBR>=3B&=3Bgt=3D3B<=3BBR&= > >gt=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B Dear m3devel=3D
>=3B &= > >gt=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D2C<=3BBR>=3B&=3Bgt=3D3B= > ><=3BBR>=3B&=3Bgt=3D3B For a while it has annoyed me that segmentatio= > >n vi=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B olations cause= > > an<=3BBR>=3B&=3Bgt=3D3B unconditional program abort. I've changed t= > >hat =3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B now so that (u= > >nder user<=3BBR>=3B&=3Bgt=3D3B threads at least) we instead get a Ru= > >ntim=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B eError. Here's= > > an example of<=3BBR>=3B&=3Bgt=3D3B the mechanism at work in an inte= > >ract=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B ive Scheme env= > >ironment. Consider<=3BBR>=3B&=3Bgt=3D3B the unhelpful interface and = > >modu=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B le Crash:<= > >=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B INTERFACE Crash=3D3B = > >PROCEDURE Me()=3D3B END Cra=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B sh.<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B MODUL= > >E Crash=3D3B<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B PROCED= > >URE Me(=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B ) =3D3D<= > >=3BBR>=3B&=3Bgt=3D3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN<=3B= > >BR>=3B&=3Bgt=3D3B ptr^ :=3D3D=3D
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B 0<=3BBR>=3B&=3Bgt=3D3B END Me=3D3B<=3BBR>=3B&= > >=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B BEGIN END Crash.<=3BBR>=3B&= > >=3Bgt=3D3B<=3BBR>=3B=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B &=3Bgt=3D3B Here's an example of what happens if you now call this= > > from an inter=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B acti= > >ve<=3BBR>=3B&=3Bgt=3D3B interpreter that catches the exception Runti= > >meError.E:<=3BBR>=3B=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B &=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B M-Scheme Experimental<= > >=3BBR>=3B&=3Bgt=3D3B LITHP ITH LITHENING.<=3BBR>=3B&=3B=3D
&= > >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B gt=3D3B&=3Bgt=3D3B (req= > >uire-modules "m3")<=3BBR>=3B&=3Bgt=3D3B #t<=3BBR>=3B&=3Bgt=3D= > >3B&=3Bgt=3D3B (Crash.Me=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B )<=3BBR>=3B&=3Bgt=3D3B EXCEPTION! RuntimeError! Attempt to= > > reference an illegal memory=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B location.<=3BBR>=3B&=3Bgt=3D3B&=3Bgt=3D3B (+ 3 4)<=3B= > >BR>=3B&=3Bgt=3D3B 7<=3BBR>=3B&=3Bgt=3D3B&=3Bgt=3D3B<=3BBR&= > >gt=3B&=3Bgt=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D3= > >B<=3BBR>=3B&=3Bgt=3D3B I just realized I may have broken pthreads=3D= > >2C let me go back=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B a= > >nd double-check it.<=3BBR>=3B&=3Bgt=3D3B runtime/POSIX and thread/PO= > >SIX don't refer=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B to = > >the same thing do they...<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bg= > >t=3D3B Mika<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B =3D
>=3B >=3B&= > >gt=3B>=3B>=3B>=3B>=3B>=3B <=3B/body>=3B
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B <=3B/html>=3B=3D
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B&= > >gt=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_--
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B
>=3B &g= > >t=3B>=3B
> >= > > > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_-- -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Wed Feb 23 17:42:06 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Wed, 23 Feb 2011 08:42:06 -0800 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" Message-ID: <20110223164206.BE6931A2078@async.async.caltech.edu> Notes: 1. the Modula-3 garbage collector gets confused (shoulda thought of that), so I have to turn it off 2. however if the GC were fixed, there's no need to "fix up" the stack after the return---you can just allocate the new stacks as GC-traced: even though the collector is compacting they are "pinned" because they are referenced from the thread stack! 3. alloca is built into gcc nowadays, so I had to track down an assembly version: I suspect that the compiler front-end could be modified to emit a call to alloca and thus get gcc to generate the same code, obviating the need for any assembly code at all. (* Unsafe interface ***************************************************) INTERFACE Unsafe; FROM Ctypes IMPORT int; PROCEDURE DoIt(); <*EXTERNAL alloca*> PROCEDURE alloca(size : int) : INTEGER; END Unsafe. (* Unsafe implementation **********************************************) UNSAFE MODULE Unsafe; IMPORT Thread, IO, Fmt; IMPORT RTCollector, Random; EXCEPTION Quit; VAR rand := NEW(Random.Default).init(); PROCEDURE P(cl : Closure; arg : ARRAY[0..100] OF INTEGER) RAISES { Quit } = CONST Fudge = 1024; VAR stack : REF ARRAY OF INTEGER := NIL; BEGIN IO.Put(Fmt.F("Thread %s GetSP()=0x%s\n", Fmt.Int(cl.id), Fmt.Int(GetSP(), base := 16)) ); (*Thread.Pause(0.1d0);*) IF GetSP() < cl.lo + Fudge THEN IF cl.stacks >= rand.integer(3,10) THEN IO.Put(Fmt.F("Thread %s quitting after %s stacks\n", Fmt.Int(cl.id), Fmt.Int(cl.stacks))); RAISE Quit END; stack := NEW(REF ARRAY OF INTEGER, 8*1024); WITH lo = LOOPHOLE(ADR(stack[0]),INTEGER), hi = LOOPHOLE(ADR(stack[LAST(stack^)]),INTEGER), sp = GetSP() DO IO.Put(Fmt.F("Thread %s sp %s new stack lo %s hi %s\n", Fmt.Int(cl.id), Fmt.Int(sp, base := 16), Fmt.Int(lo, base := 16), Fmt.Int(hi, base:= 16))); (* new stack starts sp - hi away *) EVAL alloca(sp-hi); IO.Put(Fmt.F("After alloca, sp %s\n", Fmt.Int(GetSP(),base := 16))); cl.lo := lo; INC(cl.stacks) END END; P(cl, arg) END P; TYPE Closure = Thread.Closure OBJECT id : CARDINAL; lo : INTEGER; stacks : CARDINAL; OVERRIDES apply := Apply END; PROCEDURE Apply(cl : Closure) : REFANY = BEGIN Thread.Pause(1.0d0); LOOP TRY cl.lo := GetSP() - 1024*8; (* pretend we have 8k left *) cl.stacks := 0; P(cl, ARRAY [0..100] OF INTEGER { 0, .. }); EXCEPT Quit => (* start over *) END END END Apply; PROCEDURE DoIt() = BEGIN RTCollector.Disable(); FOR i := 1 TO 2 DO EVAL Thread.Fork(NEW(Closure, id := i)) END; LOOP Thread.Pause(1.0d0) END END DoIt; PROCEDURE GetSP() : INTEGER = BEGIN RETURN alloca(1) END GetSP; BEGIN END Unsafe. (* Main can't be UNSAFE in Modula-3 ***********************************) MODULE Main; IMPORT Unsafe; BEGIN Unsafe.DoIt() END Main. (* alloca.s from FreeBSD (old code from BSD/386) **********************) /*- * Copyright (c) 1990 The Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ .asciz "@(#)alloca.s 5.2 (Berkeley) 5/14/90" #include /* like alloc, but automatic automatic free in return */ .globl alloca .type alloca, @function alloca: popl %edx /* pop return addr */ popl %eax /* pop amount to allocate */ movl %esp,%ecx addl $3,%eax /* round up to next word */ andl $0xfffffffc,%eax subl %eax,%esp movl %esp,%eax /* base of newly allocated space */ pushl 8(%ecx) /* copy possible saved registers */ pushl 4(%ecx) pushl 0(%ecx) pushl %eax /* dummy to pop at callsite */ jmp *%edx /* "return" */ From jay.krell at cornell.edu Wed Feb 23 20:53:42 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 23 Feb 2011 19:53:42 +0000 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: <20110223164206.BE6931A2078@async.async.caltech.edu> References: <20110223164206.BE6931A2078@async.async.caltech.edu> Message-ID: Stack must be contiguous on NT. Some platforms -- NT -- actually know how big the current stack is -- and won't let you alloca past it. Best way to let the target handle making stacks is with Posix makecontext, for the many-but-not-all platforms that have it, or CreateFiber on NT. (OpenBSD and I think Darwin/amd64 lack makecontext. It is in Posix, but deprecated; heck Darwin doesn't even have Posix semaphores seemingly..) To avoid the assembly you could use a hack like: void* Unsafe__alloca(size_t n) { return alloca(n); } Beware. Dangerous platform/compiler-specific stuff here. - Jay > To: m3devel at elegosoft.com > Date: Wed, 23 Feb 2011 08:42:06 -0800 > From: mika at async.caltech.edu > Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" > > Notes: > > 1. the Modula-3 garbage collector gets confused (shoulda thought of that), > so I have to turn it off > > 2. however if the GC were fixed, there's no need to "fix up" the stack > after the return---you can just allocate the new stacks as GC-traced: > even though the collector is compacting they are "pinned" because > they are referenced from the thread stack! > > 3. alloca is built into gcc nowadays, so I had to track down an assembly > version: I suspect that the compiler front-end could be modified to > emit a call to alloca and thus get gcc to generate the same code, > obviating the need for any assembly code at all. > > > (* Unsafe interface ***************************************************) > > INTERFACE Unsafe; > FROM Ctypes IMPORT int; > > PROCEDURE DoIt(); > > <*EXTERNAL alloca*> > PROCEDURE alloca(size : int) : INTEGER; > > END Unsafe. > > (* Unsafe implementation **********************************************) > > UNSAFE MODULE Unsafe; > > IMPORT Thread, IO, Fmt; > IMPORT RTCollector, Random; > > EXCEPTION Quit; > > VAR rand := NEW(Random.Default).init(); > > PROCEDURE P(cl : Closure; arg : ARRAY[0..100] OF INTEGER) RAISES { Quit } = > CONST Fudge = 1024; > VAR stack : REF ARRAY OF INTEGER := NIL; > BEGIN > IO.Put(Fmt.F("Thread %s GetSP()=0x%s\n", > Fmt.Int(cl.id), > Fmt.Int(GetSP(), base := 16)) > ); > (*Thread.Pause(0.1d0);*) > > IF GetSP() < cl.lo + Fudge THEN > IF cl.stacks >= rand.integer(3,10) THEN > IO.Put(Fmt.F("Thread %s quitting after %s stacks\n", > Fmt.Int(cl.id), Fmt.Int(cl.stacks))); > RAISE Quit > END; > > stack := NEW(REF ARRAY OF INTEGER, 8*1024); > WITH lo = LOOPHOLE(ADR(stack[0]),INTEGER), > hi = LOOPHOLE(ADR(stack[LAST(stack^)]),INTEGER), > sp = GetSP() DO > IO.Put(Fmt.F("Thread %s sp %s new stack lo %s hi %s\n", > Fmt.Int(cl.id), > Fmt.Int(sp, base := 16), > Fmt.Int(lo, base := 16), > Fmt.Int(hi, base:= 16))); > > (* new stack starts sp - hi away *) > EVAL alloca(sp-hi); > IO.Put(Fmt.F("After alloca, sp %s\n", > Fmt.Int(GetSP(),base := 16))); > cl.lo := lo; > INC(cl.stacks) > END > END; > > P(cl, arg) > END P; > > TYPE > Closure = Thread.Closure OBJECT > id : CARDINAL; > lo : INTEGER; > stacks : CARDINAL; > OVERRIDES > apply := Apply > END; > > PROCEDURE Apply(cl : Closure) : REFANY = > BEGIN > Thread.Pause(1.0d0); > > LOOP > TRY > cl.lo := GetSP() - 1024*8; (* pretend we have 8k left *) > cl.stacks := 0; > P(cl, ARRAY [0..100] OF INTEGER { 0, .. }); > EXCEPT > Quit => (* start over *) > END > END > END Apply; > > PROCEDURE DoIt() = > BEGIN > RTCollector.Disable(); > > FOR i := 1 TO 2 DO > EVAL Thread.Fork(NEW(Closure, id := i)) > END; > > LOOP Thread.Pause(1.0d0) END > END DoIt; > > PROCEDURE GetSP() : INTEGER = BEGIN RETURN alloca(1) END GetSP; > > BEGIN END Unsafe. > > (* Main can't be UNSAFE in Modula-3 ***********************************) > > MODULE Main; > IMPORT Unsafe; > > BEGIN Unsafe.DoIt() END Main. > > (* alloca.s from FreeBSD (old code from BSD/386) **********************) > > /*- > * Copyright (c) 1990 The Regents of the University of California. > * All rights reserved. > * > * This code is derived from software contributed to Berkeley by > * William Jolitz. > * > * Redistribution and use in source and binary forms, with or without > * modification, are permitted provided that the following conditions > * are met: > * 1. Redistributions of source code must retain the above copyright > * notice, this list of conditions and the following disclaimer. > * 2. Redistributions in binary form must reproduce the above copyright > * notice, this list of conditions and the following disclaimer in the > * documentation and/or other materials provided with the distribution. > * 3. All advertising materials mentioning features or use of this software > * must display the following acknowledgement: > * This product includes software developed by the University of > * California, Berkeley and its contributors. > * 4. Neither the name of the University nor the names of its contributors > * may be used to endorse or promote products derived from this software > * without specific prior written permission. > * > * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND > * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE > * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE > * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL > * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS > * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) > * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT > * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY > * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > * SUCH DAMAGE. > */ > > .asciz "@(#)alloca.s 5.2 (Berkeley) 5/14/90" > #include > > > /* like alloc, but automatic automatic free in return */ > > .globl alloca > .type alloca, @function > alloca: > popl %edx /* pop return addr */ > popl %eax /* pop amount to allocate */ > movl %esp,%ecx > addl $3,%eax /* round up to next word */ > andl $0xfffffffc,%eax > subl %eax,%esp > movl %esp,%eax /* base of newly allocated space */ > pushl 8(%ecx) /* copy possible saved registers */ > pushl 4(%ecx) > pushl 0(%ecx) > pushl %eax /* dummy to pop at callsite */ > jmp *%edx /* "return" */ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Wed Feb 23 21:39:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Wed, 23 Feb 2011 12:39:50 -0800 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: References: <20110223164206.BE6931A2078@async.async.caltech.edu> Message-ID: <20110223203950.30ABF1A2078@async.async.caltech.edu> Jay K writes: >--_60ddb4ae-bef3-432c-a5fd-a46c878ce951_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Stack must be contiguous on NT. >Some platforms -- NT -- actually know how big the current stack is -- and w= >on't let you >alloca past it. Best way to let the target handle making stacks is with Pos= >ix makecontext=2C >for the many-but-not-all platforms that have it=2C or CreateFiber on NT. >(OpenBSD and I think Darwin/amd64 lack makecontext. It is in Posix=2C but d= >eprecated=3B >heck Darwin doesn't even have Posix semaphores seemingly..) The problem with making stacks with makecontext, etc., to me seems to be: what do you do when you want to return to the original stack. Can it be made to "just work"? Note that in my design, the stacks are all contiguous---but overlapping! There are no "new" stacks and no switching of stacks. You just pretend you have a huge data structure sitting where the other guys' stacks (and data, etc.) are. Cleanup is also easy, it's handled by the ordinary GC. > >To avoid the assembly you could use a hack like: >void* Unsafe__alloca(size_t n) { return alloca(n)=3B } Won't work. alloca releases memory at the }. What would avoid it is generating the call to alloca in the front end. > >Beware. >Dangerous platform/compiler-specific stuff here. Not really that specific. It relies on precisely one special property of the underlying system: that alloca doesn't do any error checking. The only changes that need to be made to CM3 are coming up with a mechanism for detecting the need for expanding the stack (could be done by bumping the redzone before winding the stack and handling this in a SEGV handler) and modifying the garbage collector to understand the scheme. I suggest adding a back pointer at the beginning of the newly allocated area that would let the GC go back to the previous one while scanning (or however it works). Users/platforms that don't need or can't support "leapfrog stacks" would not change except that their garbage collectors would have some code that's never executed... I've discussed the above with people who've been doing concurrent programming since before the glaciers came thru (implementations of Concurrent Pascal in the 1970s) and they said that they used linked (noncontiguous) stacks in those days. The state of the art of 2011 seems rather unimpressive... My feeling is that the stack overflow issue is one of the things hindering the acceptance of truly multi-threaded programming (thousands, tens of thousands, millions of threads). Mika From jay.krell at cornell.edu Wed Feb 23 21:51:35 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 23 Feb 2011 20:51:35 +0000 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: <20110223203950.30ABF1A2078@async.async.caltech.edu> References: <20110223164206.BE6931A2078@async.async.caltech.edu>, , <20110223203950.30ABF1A2078@async.async.caltech.edu> Message-ID: > that alloca doesn't do any error checking No. There is also native exception handling to consider. NT essentially does error checking in both. > thousands, tens of thousands, millions of threads - 64bit address spaces - multiple machines ("cloud") > Won't work. alloca releases memory at the }. Only sort of. You can still return that point and still use the memory. You do risk, e.g. if you make any function calls, then the alloca memory and the function parameters might overlap. You could..platform-dependent..probably skip the first "few" bytes of the alloca memory. > Note that in my design, the stacks are all contiguous---but overlapping! Are your stacks also traversing heap and what-not, or are you just chopping up the current stack into lots of smaller stacks? i.e. alloca stack size times n and then create n threads? That's less bad. I had the impression you were skipping over other memory. You know, like, malloc your stack, and then alloca to get the stack pointer to hit the malloced memory. > What would avoid it is generating the call to alloca in the front end. We could w/o much difficulty provide alloca in the frontend. Probably limit it to unsafe modules. At least to support your experimenting and somewhat platform-dependent code. > in the 1970s.. I think the right place to experiment with this stuff is at a lower level. Or with more complete control of the system. ie. in the kernel, or on only one specific kernel, that you build/modify/write-from-scratch yourself. Gotta go, - Jay > To: jay.krell at cornell.edu > Date: Wed, 23 Feb 2011 12:39:50 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" > > Jay K writes: > >--_60ddb4ae-bef3-432c-a5fd-a46c878ce951_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >Stack must be contiguous on NT. > >Some platforms -- NT -- actually know how big the current stack is -- and w= > >on't let you > >alloca past it. Best way to let the target handle making stacks is with Pos= > >ix makecontext=2C > >for the many-but-not-all platforms that have it=2C or CreateFiber on NT. > >(OpenBSD and I think Darwin/amd64 lack makecontext. It is in Posix=2C but d= > >eprecated=3B > >heck Darwin doesn't even have Posix semaphores seemingly..) > > The problem with making stacks with makecontext, etc., to me seems > to be: what do you do when you want to return to the original stack. > Can it be made to "just work"? > > Note that in my design, the stacks are all contiguous---but overlapping! > There are no "new" stacks and no switching of stacks. You just pretend > you have a huge data structure sitting where the other guys' stacks > (and data, etc.) are. > > Cleanup is also easy, it's handled by the ordinary GC. > > > > >To avoid the assembly you could use a hack like: > >void* Unsafe__alloca(size_t n) { return alloca(n)=3B } > > Won't work. alloca releases memory at the }. > > What would avoid it is generating the call to alloca in the front end. > > > > >Beware. > >Dangerous platform/compiler-specific stuff here. > > Not really that specific. It relies on precisely one special property > of the underlying system: that alloca doesn't do any error checking. > > The only changes that need to be made to CM3 are coming up with a > mechanism for detecting the need for expanding the stack (could be done by > bumping the redzone before winding the stack and handling this in a SEGV > handler) and modifying the garbage collector to understand the scheme. > I suggest adding a back pointer at the beginning of the newly allocated > area that would let the GC go back to the previous one while scanning > (or however it works). > > Users/platforms that don't need or can't support "leapfrog stacks" > would not change except that their garbage collectors would have some > code that's never executed... > > I've discussed the above with people who've been doing concurrent > programming since before the glaciers came thru (implementations of > Concurrent Pascal in the 1970s) and they said that they used linked > (noncontiguous) stacks in those days. The state of the art of 2011 > seems rather unimpressive... > > My feeling is that the stack overflow issue is one of the things hindering > the acceptance of truly multi-threaded programming (thousands, tens of > thousands, millions of threads). > > Mika > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Thu Feb 24 03:20:29 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Wed, 23 Feb 2011 18:20:29 -0800 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: References: <20110223164206.BE6931A2078@async.async.caltech.edu>, , <20110223203950.30ABF1A2078@async.async.caltech.edu> Message-ID: <20110224022029.B016E1A2078@async.async.caltech.edu> Jay K writes: >--_6d651072-194c-4975-9e55-045a636fc1ed_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > that alloca doesn't do any error checking > > >No. There is also native exception handling to consider. >NT essentially does error checking in both. Native exception handling? What do you mean? > > >> thousands=2C tens of thousands=2C millions of threads > > - 64bit address spaces > - multiple machines ("cloud") >=20 > > > Won't work. alloca releases memory at the }. >=20 >=20 > Only sort of. > You can still return that point and still use the memory. > You do risk=2C e.g. if you make any function calls=2C then the > alloca memory and the function parameters might overlap. > You could..platform-dependent..probably skip the first > "few" bytes of the alloca memory. Well that's the whole point of the alloca, that it moves the stack pointer beyond the danger region...(and keeps it there till you return) if you let it come back ... it defeats the purpose. > > > > Note that in my design=2C the stacks are all contiguous---but overlappin= >g! >=20 > Are your stacks also traversing heap and what-not=2C > or are you just chopping up the current stack into lots > of smaller stacks? Yes traverses heap, protected pages, I/O buffers, text segments,... whatever junk is in between the old stack and the newly allocated area. >=20 > i.e. alloca stack size times n and then create n threads? > That's less bad. >=20 > I had the impression you were skipping over other memory. >=20 > You know=2C like=2C malloc your stack=2C and then alloca to get > the stack pointer to hit the malloced memory. That's exactly right. malloc a new stack and use alloca to fiddle the stack pointer so it lands at the start of the new stack. Now, I don't know a darn thing about Windows, but I've been looking at Unix code and I see the following: * all alloca does is it modifies the stack pointer. It does this from user code without calling any sort of operating system or other runtime routines. * nothing special is done to "stacks" in Modula-3's "POSIX" user threading library---threads are allocated with a NEW(UNTRACED REF...) using the normal Modula-3 memory management routines. * the concept of "stack" (another one of those EWD inventions, incidentally) itself is a bit of a misnomer. There is only the stack pointer + conventions on how you use it. The "stack" is simply the area between where the thread began execution and the current value of the stack pointer. It's a matter of definition, simply. > > > What would avoid it is generating the call to alloca in the front end. >=20 >We could w/o much difficulty provide alloca in the frontend. >Probably limit it to unsafe modules. >At least to support your experimenting and somewhat platform-dependent code= >. Sure that would be fun :) > > > in the 1970s.. >=20 >=20 >I think the right place to experiment with this stuff is at a lower level. >Or with more complete control of the system. >ie. in the kernel=2C or on only one specific kernel=2C that you >build/modify/write-from-scratch yourself. Let me put it like this: if Modula-3's "user threads" work on architecture/OS combination X, I think you can just about (perhaps you can actually) prove that "leapfrog stacks" would work on the same architecture/OS combination, just based on an analysis of how the user threads allocate and use their stacks. But if user threads don't work (say Windows?) then it probably won't work. We do know, however, that user threads work just about everywhere... Mika > > >Gotta go=2C > - Jay > > > > > > From jay.krell at cornell.edu Thu Feb 24 11:09:33 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 24 Feb 2011 10:09:33 +0000 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: <20110224022029.B016E1A2078@async.async.caltech.edu> References: <20110223164206.BE6931A2078@async.async.caltech.edu>, , , , <20110223203950.30ABF1A2078@async.async.caltech.edu>, , <20110224022029.B016E1A2078@async.async.caltech.edu> Message-ID: > But if user threads don't work (say Windows?) then it probably won't > work. We do know, however, that user threads work just about everywhere. They don't. One could use fibers if one really wanted to do one's own scheduling, and locking. (user threads implies writing your own locks). On NT, alloca touches every page between the current and the result. (not every byte, that'd be wasteful, just every page) Touching the end of the stack generates stack overflow exceptions. So you can't go past it. That is, the very problem people were worried about, is solved on NT. "Native exception handling" is what e.g. C++ uses. I believe it validates the stack to be in range. In particular, on NT, the stack range is available somewhere, and occasionally used. - Jay > To: jay.krell at cornell.edu > Date: Wed, 23 Feb 2011 18:20:29 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" > > Jay K writes: > >--_6d651072-194c-4975-9e55-045a636fc1ed_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > that alloca doesn't do any error checking > > > > > >No. There is also native exception handling to consider. > >NT essentially does error checking in both. > > Native exception handling? What do you mean? > > > > > > >> thousands=2C tens of thousands=2C millions of threads > > > > - 64bit address spaces > > - multiple machines ("cloud") > >=20 > > > > > Won't work. alloca releases memory at the }. > >=20 > >=20 > > Only sort of. > > You can still return that point and still use the memory. > > You do risk=2C e.g. if you make any function calls=2C then the > > alloca memory and the function parameters might overlap. > > You could..platform-dependent..probably skip the first > > "few" bytes of the alloca memory. > > Well that's the whole point of the alloca, that it moves the stack > pointer beyond the danger region...(and keeps it there till you return) > if you let it come back ... it defeats the purpose. > > > > > > > > Note that in my design=2C the stacks are all contiguous---but overlappin= > >g! > >=20 > > Are your stacks also traversing heap and what-not=2C > > or are you just chopping up the current stack into lots > > of smaller stacks? > > > Yes traverses heap, protected pages, I/O buffers, text segments,... whatever > junk is in between the old stack and the newly allocated area. > > > >=20 > > i.e. alloca stack size times n and then create n threads? > > That's less bad. > >=20 > > I had the impression you were skipping over other memory. > >=20 > > You know=2C like=2C malloc your stack=2C and then alloca to get > > the stack pointer to hit the malloced memory. > > That's exactly right. malloc a new stack and use alloca to fiddle > the stack pointer so it lands at the start of the new stack. > > Now, I don't know a darn thing about Windows, but I've been looking at > Unix code and I see the following: > > * all alloca does is it modifies the stack pointer. It does this from > user code without calling any sort of operating system or other runtime > routines. > > * nothing special is done to "stacks" in Modula-3's "POSIX" user threading > library---threads are allocated with a NEW(UNTRACED REF...) using the > normal Modula-3 memory management routines. > > * the concept of "stack" (another one of those EWD inventions, > incidentally) itself is a bit of a misnomer. There is only the > stack pointer + conventions on how you use it. The "stack" is simply > the area between where the thread began execution and the current > value of the stack pointer. It's a matter of definition, simply. > > > > > > What would avoid it is generating the call to alloca in the front end. > >=20 > >We could w/o much difficulty provide alloca in the frontend. > >Probably limit it to unsafe modules. > >At least to support your experimenting and somewhat platform-dependent code= > >. > > Sure that would be fun :) > > > > > > in the 1970s.. > >=20 > >=20 > >I think the right place to experiment with this stuff is at a lower level. > >Or with more complete control of the system. > >ie. in the kernel=2C or on only one specific kernel=2C that you > >build/modify/write-from-scratch yourself. > > Let me put it like this: if Modula-3's "user threads" work on > architecture/OS combination X, I think you can just about (perhaps > you can actually) prove that "leapfrog stacks" would work on the same > architecture/OS combination, just based on an analysis of how the user > threads allocate and use their stacks. But if user threads don't work > (say Windows?) then it probably won't work. We do know, however, that > user threads work just about everywhere... > > Mika > > > > > > >Gotta go=2C > > - Jay > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Feb 24 20:16:58 2011 From: dragisha at m3w.org (=?utf-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 24 Feb 2011 20:16:58 +0100 Subject: [M3devel] new target? :) Message-ID: http://code.google.com/chrome/nativeclient/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Thu Feb 24 22:46:17 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Thu, 24 Feb 2011 21:46:17 +0000 (GMT) Subject: [M3devel] new target? :) In-Reply-To: Message-ID: <758561.47161.qm@web29710.mail.ird.yahoo.com> Hi all: is interesting anyway but to write the web apps in c and good friends, hum ..., let me think if I can write a web server in less than 100 loc of Obliq and indeed one is written as an os extension, why would need any web page that renders c code, I think I would gain much more time writing myself rather than porting C (of course with its flavors) in Modula-3 or better if so in Obliq which is a mobile computing environment indeed, it makes me think, though the idea is good it makes little sense to make everything a client of a web browser rather than an instance of a structured platform, if they need only C why it has to be an instance of a web browser, it must be the browser makes better work for quick dynamic pages with C, or maybe another way of keeping it alive artificially, I mean what is the gain with that, if better runtime but, hum ...? but? it needs javascript, hum ... another loss of time (much time in inefficient checking), Nokia had the same idea with Qt and JS in WRT, (very little RT information), but still Obliq scripts can be type inferenced in much more smoothly way. I think this battle of JavaScript and Obliq and Modula-3 vs others could be fight yet one time again but needs more time for tools to make that a reality, still the others think they can do it, it's very interesting. Thanks in advance --- El jue, 24/2/11, Dragi?a Duri? escribi?: De: Dragi?a Duri? Asunto: [M3devel] new target? :) Para: "m3devel" Fecha: jueves, 24 de febrero, 2011 14:16 http://code.google.com/chrome/nativeclient/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 24 22:58:17 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 24 Feb 2011 21:58:17 +0000 Subject: [M3devel] new target? :) In-Reply-To: <758561.47161.qm@web29710.mail.ird.yahoo.com> References: , <758561.47161.qm@web29710.mail.ird.yahoo.com> Message-ID: I believe nativeclient depends on a fork of gcc. And I'm not sure if it always 32bit x86 or not. Part of what they do is limit the output such as to be easily disassembled and analyzed at load/runtime. If that is all, we could probably just take their patches. There is a paper about it.. I don't know what you link to though. Ultimately..you can tell what I'm going to say..a good solution is Modula-3-to-C, and then use whatever C compiler you want. Including the nativeclient gcc. An iPhone/iPad/iPod cross compiler. Older stuff like Irix/AIX/HPUX that we have no Hudson nodes for. NT/AMD64, NT/IA64. Android. Newer stuff that hasn't come along yet. Basically, generating C or C++ is a great portability mechanism, to old, existing, and new platforms. A bit better than depending on gcc. On Daniel's point, there's an ongoing race to provide the best new development environments that do or do not allow much code reuse. That are or are not "safe". not safe: iPhone, iPad, iPod, native for Android safe: Java on Android, Silverlight in web browser and Windows Phone 7, Go, JavaScript Those platforms have way more mindshare, support, tooling, libraries, than Modula-3 and I'm sure they always will. > let me think if I can write a web server in less than 100 loc It takes 5 lines in Python, as it is already builtin to the libraries -- confusing language and libraries, granted. - Jay Date: Thu, 24 Feb 2011 21:46:17 +0000 From: dabenavidesd at yahoo.es To: m3devel at elegosoft.com; dragisha at m3w.org Subject: Re: [M3devel] new target? :) Hi all: is interesting anyway but to write the web apps in c and good friends, hum ..., let me think if I can write a web server in less than 100 loc of Obliq and indeed one is written as an os extension, why would need any web page that renders c code, I think I would gain much more time writing myself rather than porting C (of course with its flavors) in Modula-3 or better if so in Obliq which is a mobile computing environment indeed, it makes me think, though the idea is good it makes little sense to make everything a client of a web browser rather than an instance of a structured platform, if they need only C why it has to be an instance of a web browser, it must be the browser makes better work for quick dynamic pages with C, or maybe another way of keeping it alive artificially, I mean what is the gain with that, if better runtime but, hum ...? but it needs javascript, hum ... another loss of time (much time in inefficient checking), Nokia had the same idea with Qt and JS in WRT, (very little RT information), but still Obliq scripts can be type inferenced in much more smoothly way. I think this battle of JavaScript and Obliq and Modula-3 vs others could be fight yet one time again but needs more time for tools to make that a reality, still the others think they can do it, it's very interesting. Thanks in advance --- El jue, 24/2/11, Dragi?a Duri? escribi?: De: Dragi?a Duri? Asunto: [M3devel] new target? :) Para: "m3devel" Fecha: jueves, 24 de febrero, 2011 14:16 http://code.google.com/chrome/nativeclient/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Thu Feb 24 23:15:51 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Thu, 24 Feb 2011 14:15:51 -0800 Subject: [M3devel] new target? :) In-Reply-To: References: , <758561.47161.qm@web29710.mail.ird.yahoo.com> Message-ID: <20110224221551.8B9271A2078@async.async.caltech.edu> Jay K writes: ... >Ultimately..you can tell what I'm going to say..a good solution is Modula-3= >-to-C=2C and then use whatever C compiler you want. >Including the nativeclient gcc. An iPhone/iPad/iPod cross compiler. Older s= I'm still curious why they abandoned this path. The first M3 compilers all generated C code. I read something about a Modula-2 compiler that was generating C code and therefore had problems with.. something. Also isn't the back-end m3cg actually quite an interesting program in its own right? You could imagine targeting other language front-ends to that back-end. I understand it is a good enough idea that RMS was very annoyed that m3cg even exists. Mika From jay.krell at cornell.edu Fri Feb 25 00:09:39 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 24 Feb 2011 23:09:39 +0000 Subject: [M3devel] new target? :) In-Reply-To: <20110224221551.8B9271A2078@async.async.caltech.edu> References: , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , <20110224221551.8B9271A2078@async.async.caltech.edu> Message-ID: I never used the system, but one can imagine and it has been said here: slower to compile less good debugging -- and also better debugging! However I still think the large gain in portability, and the actual *gain* in debuggability, would be very worthwhile. (There are pluses and minuses for debugging; you'd get to use gdb instead of m3gdb, and have C syntax). You'd gain portability and ports to various real, old, existing, new systems. e.g. NT/IA64, NT/AMD64, Android, NativeClient. You'd gain a "standard" distribution format of just source.tar.gz + possibly ./configure + make + make install. The that's a little more work. I think there's some issue with the metadata for garbage collection. I think we maybe store frame offsets for locals? We'd maybe have to put all locals in a struct or such? Impeding optimization? Similarly, nested functions would need some handling. You would also gain tremendously better exeption handling implementation. You'd generate C on NT/VMS/OSF1, and use __try/__except/__finally there. You'd generate C++ for all others and use try/catch/finally. You'd get stack walking on essentially all platforms. Except for the small number of platforms that use setjmp/longjmp for C++ exception handling. In some cases you'd get better codegen, as gcc is often not the best. RMS's problem is that normally the gcc frontends and backends are statically linked. frontends: C, C++, Java, Ada Therefore the frontends get infected by the GPL. Good in RMS opinion. m3cg is, technically, a gcc frontend, and therefore GPL. However its input is not human authored text. Its input is the output of a "real" front end -- m3front/cm3. By making the "call mechanism" be write to a file and run a process, the GPL doesn't cross the barrier, leaving cm3/m3front, and more importantly, m3core/libm3, not GPL. I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. - Jay > To: jay.krell at cornell.edu > Date: Thu, 24 Feb 2011 14:15:51 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] new target? :) > > Jay K writes: > ... > >Ultimately..you can tell what I'm going to say..a good solution is Modula-3= > >-to-C=2C and then use whatever C compiler you want. > >Including the nativeclient gcc. An iPhone/iPad/iPod cross compiler. Older s= > > I'm still curious why they abandoned this path. The first M3 compilers > all generated C code. I read something about a Modula-2 compiler that > was generating C code and therefore had problems with.. something. > > Also isn't the back-end m3cg actually quite an interesting program in > its own right? You could imagine targeting other language front-ends > to that back-end. I understand it is a good enough idea that RMS was > very annoyed that m3cg even exists. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 25 00:12:21 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Thu, 24 Feb 2011 23:12:21 +0000 (GMT) Subject: [M3devel] new target? :) In-Reply-To: <20110224221551.8B9271A2078@async.async.caltech.edu> Message-ID: <281816.90569.qm@web29709.mail.ird.yahoo.com> Hi all: yeah, but others had very distinct opinions in tis own right with two experimental projects: http://sandbox.parc.com/parctab/csl9501/node6.html and same house: http://www.cl.cam.ac.uk/research/origami/Origami1997c/ and prior another one which had experience with Trestle http://www.usenix.org/publications/library/proceedings/mobile93/kantarjiev.html So even with the space limitations the systems run in Modula-3 technically better than any other implementation for its computing environment. Perhaps that is the point he computing paradigm and specially performance is vastly superior than those days to run something like Modula-3, but maybe more easy for Obliq nowadays. Thanks in advance --- El jue, 24/2/11, Mika Nystrom escribi?: > De: Mika Nystrom > Asunto: Re: [M3devel] new target? :) > Para: "Jay K" > CC: m3devel at elegosoft.com > Fecha: jueves, 24 de febrero, 2011 17:15 > Jay K writes: > ... > >Ultimately..you can tell what I'm going to say..a good > solution is Modula-3= > >-to-C=2C and then use whatever C compiler you want. > >Including the nativeclient gcc. An iPhone/iPad/iPod > cross compiler. Older s= > > I'm still curious why they abandoned this path. The > first M3 compilers > all generated C code. I read something about a > Modula-2 compiler that > was generating C code and therefore had problems with.. > something. > > Also isn't the back-end m3cg actually quite an interesting > program in > its own right? You could imagine targeting other > language front-ends > to that back-end. I understand it is a good enough > idea that RMS was > very annoyed that m3cg even exists. > > Mika > From mika at async.caltech.edu Fri Feb 25 02:31:00 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Thu, 24 Feb 2011 17:31:00 -0800 Subject: [M3devel] new target? :) In-Reply-To: References: , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , <20110224221551.8B9271A2078@async.async.caltech.edu> Message-ID: <20110225013100.619CD1A2078@async.async.caltech.edu> Jay K writes: ... >m3cg is=2C technically=2C a gcc frontend=2C and therefore GPL. >However its input is not human authored text. >Its input is the output of a "real" front end -- m3front/cm3. >By making the "call mechanism" be write to a file and run a process=2C the = >GPL doesn't cross the barrier=2C >leaving cm3/m3front=2C and more importantly=2C m3core/libm3=2C not GPL. >=20 >=20 >I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. >=20 >=20 Good points all about C but I wonder about what is really lost. Also ...... Modula-3 is supposed to be portable to systems without C. You're supposed to be able to write an entire system in Modula-3. Ok generating C obviously doesn't break that feature but you might have to worry a bit about having too much C sneak into the runtime libraries. Well given that CM3 is released under what is more or less an MIT/BSD license, I think the problem is rather that Some Evil Software Company could write a completely non-free front-end and use m3cg to generate code... Mika From jay.krell at cornell.edu Fri Feb 25 02:55:52 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 25 Feb 2011 01:55:52 +0000 Subject: [M3devel] new target? :) In-Reply-To: <20110225013100.619CD1A2078@async.async.caltech.edu> References: , , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , <20110225013100.619CD1A2078@async.async.caltech.edu> Message-ID: When you write that system that doesn't have the support of any C compiler, sure, go ahead and write a backend just for it, and maybe a linker, and rewrite the small parts that depend on C and Posix in some sort of native way -- including writing all the underlying stuff. Getting Modula-3 working on that system will be a relatively small task, compared to whatever else is needed to build the system. Note for example, we can't e.g. open files in Modula-3, w/o underlying Win32 or Posix functionality. Everywhere that varies for Win32 vs. Posix would have to be revisited in this hypothetical system. This has little/no bearing on the vast vast vast vast majority of real world systems. The current Modula-3 system is NOT implemented entirely in Modula-3. It MUST depend on SOMETHING. So far, for the vast vast vast vast vast majority of systems, that something is written in C, and has a C runtime, compiler, and linker. Yes, I've heard of SPIN. You know..how are you actually going to network I/O or keyboard I/O or power management or video I/O or mouse I/O or sound I/O or preemptive multi-processor scheduling? You need device drivers. You need a processor-specific context switcher. i.e. you need a kernel and you need C AND/OR you need to do a lot of work. If you consider that our system is already forked along Win32 and Posix lines, you can consider these hypothetical systems a third (fourth, fifth, etc.) fork. A third (fourth) implementation of threading. A third implementation of file I/O. A third implementation of Trestle. etc. But this has little/no bearing how we should target existing real systems. You can say, not that "Modula-3 depends on C (runtime, compiler, linker)", but that "Modula-3 on Linux, NT, *BSD, Solaris, Darwin depends on C (runtime, compiler, linker)", and there's absolutely nothing wrong with that, as C and C++ are well supported on those systems and always will be. Increasing that dependency, on those systems, is fine. It would be nice to port the integrated backend to lots more systems, and to write our own linker. But not easy. And notice how the integrated backend also has the worst debugging story of the two/three options, currently. (I say three options because m3cg kind of has two modes, one is incomplete and currently disabled, the other requires m3gdb). One problem I see here, is that, you kind of sort of maybe forseeable want a backend that outputs Java or C#. C# particularly for Windows Phone 7/Nokia. Java kind of for Android, where Java is preferred over C/C++. i.e. there are systems that sort of lack C and C++. Not that they replace it with Modula-3 though. However this isn't super easy, like if you are to include the existing m3core/garbage collector. C# is optionally safe. I don't know if it would serve as a good target. (Optional safety imho makes C# among one of the closest languages to Modula-3. It is missing "compile to native", but that maybe is a good thing.) In addition, C# and Java do each compile down to one well documented "object" file format, so maybe skipping the text is probably sensible. Esp. since with C# there are in-proc libraries for codegen. More likely, if a Java/C# target were really needed, it'd include forking within m3core and replace the garbage collector. This is all highly specuative stuff. - Jay > To: jay.krell at cornell.edu > Date: Thu, 24 Feb 2011 17:31:00 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] new target? :) > > Jay K writes: > ... > >m3cg is=2C technically=2C a gcc frontend=2C and therefore GPL. > >However its input is not human authored text. > >Its input is the output of a "real" front end -- m3front/cm3. > >By making the "call mechanism" be write to a file and run a process=2C the = > >GPL doesn't cross the barrier=2C > >leaving cm3/m3front=2C and more importantly=2C m3core/libm3=2C not GPL. > >=20 > >=20 > >I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. > >=20 > >=20 > > Good points all about C but I wonder about what is really lost. Also ...... > Modula-3 is supposed to be portable to systems without C. You're supposed > to be able to write an entire system in Modula-3. Ok generating C obviously > doesn't break that feature but you might have to worry a bit about having > too much C sneak into the runtime libraries. > > Well given that CM3 is released under what is more or less an MIT/BSD license, > I think the problem is rather that Some Evil Software Company could write > a completely non-free front-end and use m3cg to generate code... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Fri Feb 25 03:35:28 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Thu, 24 Feb 2011 18:35:28 -0800 Subject: [M3devel] new target? :) In-Reply-To: References: , , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , <20110225013100.619CD1A2078@async.async.caltech.edu> Message-ID: <20110225023528.A615D1A2078@async.async.caltech.edu> In general I don't disagree with anything you say. On Linux, BSD, et al., we obviously use C for low-level system-specific stuff because the system call interfaces on those systems are documented for C and not for other languages. As for OSes in M3 I wasn't thinking as much about SPIN as about Topaz. Ok so it was Modula-2+, but it was the direct inspiration for the Modula-3 project: you're supposed to be able to write device drivers and such things in Modula-3. I agree though that targeting C doesn't directly weaken the case for M3 as a systems programming language. There are more similarities between C# and M3 than "unsafe". Look at the way methods are overridden in C#... (compared to Java). I also think almost everyone who had anything to do with M3 at DEC is now at Microsoft. Mika Jay K writes: >--_4e7f3be7-1b6e-44e9-92a2-cc8c4f6b0b8c_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >When you write that system that doesn't have the support of any C compiler= >=2C sure=2C go ahead and >write a backend just for it=2C and maybe a linker=2C and rewrite the small = >parts that depend on C and Posix in some >sort of native way -- including writing all the underlying stuff. >=20 >=20 >Getting Modula-3 working on that system will be a relatively small task=2C = >compared to whatever else > is needed to build the system. >=20 >Note for example=2C we can't e.g. open files in Modula-3=2C w/o underlying = >Win32 or Posix functionality. > Everywhere that varies for Win32 vs. Posix would have to be revisited in = >this hypothetical system. >=20 >=20 >This has little/no bearing on the vast vast vast vast majority of real worl= >d systems. >The current Modula-3 system is NOT implemented entirely in Modula-3. >It MUST depend on SOMETHING. >So far=2C for the vast vast vast vast vast majority of systems=2C that some= >thing is written in C=2C >and has a C runtime=2C compiler=2C and linker. >=20 >=20 >Yes=2C I've heard of SPIN. >=20 >=20 >You know..how are you actually going to network I/O or keyboard I/O or powe= >r management >or video I/O or mouse I/O or sound I/O or preemptive multi-processor schedu= >ling? >You need device drivers. You need a processor-specific context switcher. >i.e. you need a kernel and you need C AND/OR you need to do a lot of work. >=20 >=20 >If you consider that our system is already forked along Win32 and Posix lin= >es=2C you can consider >these hypothetical systems a third (fourth=2C fifth=2C etc.) fork. A third = >(fourth) implementation >of threading. A third implementation of file I/O. A third implementation of= > Trestle. etc. >=20 >=20 >But this has little/no bearing how we should target existing real systems. > You can say=2C not that "Modula-3 depends on C (runtime=2C compiler=2C lin= >ker)"=2C but that "Modula-3 on Linux=2C NT=2C *BSD=2C Solaris=2C Darwin >depends on C (runtime=2C compiler=2C linker)"=2C and there's absolutely not= >hing wrong with that=2C as C and C++ >are well supported on those systems and always will be. Increasing that dep= >endency=2C on those systems=2C is fine. >=20 >=20 >It would be nice to port the integrated backend to lots more systems=2C and= > to write our own linker. >But not easy. >And notice how the integrated backend also has the worst debugging story of= > the two/three options=2C currently. > (I say three options because m3cg kind of has two modes=2C one is incomple= >te and currently disabled=2C > the other requires m3gdb). >=20 >=20 >One problem I see here=2C is that=2C you kind of sort of maybe forseeable w= >ant a backend that outputs Java or C#. > C# particularly for Windows Phone 7/Nokia. > Java kind of for Android=2C where Java is preferred over C/C++. >=20 >=20 >i.e. there are systems that sort of lack C and C++. Not that they replace i= >t with Modula-3 though. >=20 >=20 >However this isn't super easy=2C like if you are to include the existing m3= >core/garbage collector. >C# is optionally safe. I don't know if it would serve as a good target. > (Optional safety imho makes C# among one of the closest languages to Modu= >la-3. It is > missing "compile to native"=2C but that maybe is a good thing.) >In addition=2C C# and Java do each compile down to one well documented "obj= >ect" file format=2C >so maybe skipping the text is probably sensible. Esp. since with C# there a= >re in-proc libraries for codegen. >=20 >=20 >More likely=2C if a Java/C# target were really needed=2C it'd include forki= >ng within m3core=20 >and replace the garbage collector. This is all highly specuative stuff. >=20 >=20 > - Jay > From jay.krell at cornell.edu Fri Feb 25 04:09:01 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 25 Feb 2011 03:09:01 +0000 Subject: [M3devel] new target? :) In-Reply-To: <20110225023528.A615D1A2078@async.async.caltech.edu> References: , , ,,<758561.47161.qm@web29710.mail.ird.yahoo.com>, ,,, , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , , , <20110225013100.619CD1A2078@async.async.caltech.edu>, , <20110225023528.A615D1A2078@async.async.caltech.edu> Message-ID: > I also think almost everyone who had anything to do with M3 at DEC is now at Microsoft Not just M3. M3 wasn't many people. Lots of people at Microsoft were at Digital. > Modula-3 project: you're supposed to be able to write device drivers > and such things in Modula-3. I agree though that targeting C doesn't Understood, agreed -- you can hypothetically write anything in Modula-3. The optional safety is fairly unique and important. Except for C# and Modula-3, all languages seem to be strictly safe or unsafe. I think, you can imagine, either an entire system in Modula-3, with m3core/libm3 coded "directly" in Modula-3 and simply statically or dynamically linked using "standard mechanism", or if there is still a user/kernel boundary, you either have to hack the compiler a bit, or resort to some assembly (possibly embedded in .c files). That is, calling the kernel is inevitably a special call sequence, and cm3 doesn't currently understand anything like that. About device drivers -- a critical piece is the runtime. Imagine writing an NT or Linux driver. Well, if you separate language from library...and limit language features to accomodate missing runtime...and retarget small parts of the runtime perhaps.. you know, there's a certain isomorphism? You can write almost anything in C or Modula-3 and most things will look very similar and map one-to-one. I think I read that Modula-3 can describe anything C can, except unions. You know, don't expect to have garbage collection in the kernel. Maybe not exception handling. Not necessarily the file I/O you are used to. But something. Not the threading you are used to. But something (at least on NT). But certainly you can interface Modula-3 to the NTkernel/Linux/*BSD runtimes, just like we interface to Win32/Posix. I'm not sure it'd buy much. I guess if you really like the syntax... It's another area where compiling to C would help. esp. for the exception handling on NT! __try/__finally are available in C on NT, usermode and kernel, and the implementation is much more efficient than what Modula-3 does currently. I really need to stop saying all this stuff though, either be quiet or do it... - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 25 15:20:48 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Fri, 25 Feb 2011 14:20:48 +0000 (GMT) Subject: [M3devel] new target? :) In-Reply-To: Message-ID: <669680.12232.qm@web29711.mail.ird.yahoo.com> Hi all: It was that there is some OS in Modula-2+ which run enterely in user space and you didn't want to get into the kernel but just for any option you would need to be unsafe (like machine starting, perhaps the OS managed software faults isolated from kernel space) that had the Runtime to time-out protect deadlocks, and it did it pretty much in a microkernel, just that company didn't feel sure o fsuch sophisticated system. Nowadays would be good if one could cooperate it in ARM VM for web platform developing, without the need for? GUI kernel, which indeed could crash the system if exploted, instead feels safer to run from "microkernel" traceable user space threads but making calls to a sort of video controller see below. Someone has the sources of it, just don't how complete, they did develop a UIMS and a window toolkit, plenty of tools to check: http://lists.cloud9.co.uk/pipermail/bbc-micro/2010-January/007795.html Apparently it had an Acorn C and Modula runtime: http://homepage.mac.com/dr2chase/resume.html Also about the machine and video chip: http://www.khantazi.org/Archives/MachineLst.html#A500 And its window toolkit: http://pages.sbcglobal.net/btlewis/resume-formatted.html Thanks in advance --- El jue, 24/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] new target? :) Para: "Mika Nystrom" CC: "m3devel" Fecha: jueves, 24 de febrero, 2011 22:09 ?> ?I also think almost everyone who had anything to do with M3 at DEC is now at Microsoft ? ? Not just M3. M3 wasn't many people. Lots of?people at Microsoft were?at Digital. ? ? > Modula-3 project: you're supposed to be able to write device drivers > and such things in Modula-3. I agree though that targeting C doesn't ? Understood, agreed -- you can hypothetically write anything in Modula-3. The optional safety is fairly unique and important. Except for C# and Modula-3, all languages seem to be strictly safe or unsafe. ? ? I think, you can imagine, either an entire system in Modula-3, with m3core/libm3 coded "directly" in Modula-3 and simply statically or dynamically linked using "standard mechanism", or if there is still a user/kernel boundary, you either have to hack the compiler a bit, or resort to some assembly (possibly embedded in .c files). That is, calling the kernel is inevitably a special call sequence, and cm3 doesn't currently understand anything like that. ? ? About device drivers -- a critical piece is the runtime. Imagine writing an NT or Linux driver. Well, if you separate language from library...and limit language features to accomodate?missing runtime...and retarget small parts of the runtime perhaps.. you know, there's a certain isomorphism? You can write almost anything in C or Modula-3 and most things will look very?similar and map one-to-one. I think I read that Modula-3 can describe anything C can, except unions. ? ? You know, don't expect to have garbage collection in the kernel. Maybe not exception handling. Not necessarily the file I/O you are used to. But something. Not the threading you are used to. But something (at least on NT). But certainly you can interface Modula-3 to the NTkernel/Linux/*BSD runtimes, just like we? interface to Win32/Posix. ? ? I'm not sure it'd buy much. I guess if you really like the syntax... ? ? It's another area where compiling to C would help. esp. for the exception handling on NT! __try/__finally are available in C on NT, usermode and kernel, and the implementation is much more efficient than what Modula-3 does currently. I really need to stop saying all this stuff though, either be quiet or do it... ? ? ?- Jay ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Feb 25 16:08:15 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 25 Feb 2011 10:08:15 -0500 Subject: [M3devel] new target? :) In-Reply-To: References: , , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , <20110225013100.619CD1A2078@async.async.caltech.edu> Message-ID: <04F51416-6EAC-4BBD-8C4B-623245437ECD@cs.purdue.edu> Just to follow up briefly as I have lots of other things on my plate. One of the reasons for targeting the gcc-based backend was to support easy generation of *source*-level debugging information. It is important that the debugger see types as Modula-3 sees them, rather than as C types generated form the Modula-3. Now, I agree that debugging the C-translated types is probably better than nothing. On Feb 24, 2011, at 8:55 PM, Jay K wrote: > When you write that system that doesn't have the support of any C compiler, sure, go ahead and > write a backend just for it, and maybe a linker, and rewrite the small parts that depend on C and Posix in some > sort of native way -- including writing all the underlying stuff. > > > Getting Modula-3 working on that system will be a relatively small task, compared to whatever else > is needed to build the system. > > Note for example, we can't e.g. open files in Modula-3, w/o underlying Win32 or Posix functionality. > Everywhere that varies for Win32 vs. Posix would have to be revisited in this hypothetical system. > > > This has little/no bearing on the vast vast vast vast majority of real world systems. > The current Modula-3 system is NOT implemented entirely in Modula-3. > It MUST depend on SOMETHING. > So far, for the vast vast vast vast vast majority of systems, that something is written in C, > and has a C runtime, compiler, and linker. > > > Yes, I've heard of SPIN. > > > You know..how are you actually going to network I/O or keyboard I/O or power management > or video I/O or mouse I/O or sound I/O or preemptive multi-processor scheduling? > You need device drivers. You need a processor-specific context switcher. > i.e. you need a kernel and you need C AND/OR you need to do a lot of work. > > > If you consider that our system is already forked along Win32 and Posix lines, you can consider > these hypothetical systems a third (fourth, fifth, etc.) fork. A third (fourth) implementation > of threading. A third implementation of file I/O. A third implementation of Trestle. etc. > > > But this has little/no bearing how we should target existing real systems. > You can say, not that "Modula-3 depends on C (runtime, compiler, linker)", but that "Modula-3 on Linux, NT, *BSD, Solaris, Darwin > depends on C (runtime, compiler, linker)", and there's absolutely nothing wrong with that, as C and C++ > are well supported on those systems and always will be. Increasing that dependency, on those systems, is fine. > > > It would be nice to port the integrated backend to lots more systems, and to write our own linker. > But not easy. > And notice how the integrated backend also has the worst debugging story of the two/three options, currently. > (I say three options because m3cg kind of has two modes, one is incomplete and currently disabled, > the other requires m3gdb). > > > One problem I see here, is that, you kind of sort of maybe forseeable want a backend that outputs Java or C#. > C# particularly for Windows Phone 7/Nokia. > Java kind of for Android, where Java is preferred over C/C++. > > > i.e. there are systems that sort of lack C and C++. Not that they replace it with Modula-3 though. > > > However this isn't super easy, like if you are to include the existing m3core/garbage collector. > C# is optionally safe. I don't know if it would serve as a good target. > (Optional safety imho makes C# among one of the closest languages to Modula-3. It is > missing "compile to native", but that maybe is a good thing.) > In addition, C# and Java do each compile down to one well documented "object" file format, > so maybe skipping the text is probably sensible. Esp. since with C# there are in-proc libraries for codegen. > > > More likely, if a Java/C# target were really needed, it'd include forking within m3core > and replace the garbage collector. This is all highly specuative stuff. > > > - Jay > > > > To: jay.krell at cornell.edu > > Date: Thu, 24 Feb 2011 17:31:00 -0800 > > From: mika at async.caltech.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] new target? :) > > > > Jay K writes: > > ... > > >m3cg is=2C technically=2C a gcc frontend=2C and therefore GPL. > > >However its input is not human authored text. > > >Its input is the output of a "real" front end -- m3front/cm3. > > >By making the "call mechanism" be write to a file and run a process=2C the = > > >GPL doesn't cross the barrier=2C > > >leaving cm3/m3front=2C and more importantly=2C m3core/libm3=2C not GPL. > > >=20 > > >=20 > > >I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. > > >=20 > > >=20 > > > > Good points all about C but I wonder about what is really lost. Also ...... > > Modula-3 is supposed to be portable to systems without C. You're supposed > > to be able to write an entire system in Modula-3. Ok generating C obviously > > doesn't break that feature but you might have to worry a bit about having > > too much C sneak into the runtime libraries. > > > > Well given that CM3 is released under what is more or less an MIT/BSD license, > > I think the problem is rather that Some Evil Software Company could write > > a completely non-free front-end and use m3cg to generate code... > > > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Feb 25 19:50:12 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 25 Feb 2011 18:50:12 +0000 Subject: [M3devel] new target? :) In-Reply-To: <04F51416-6EAC-4BBD-8C4B-623245437ECD@cs.purdue.edu> References: , , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , <20110225013100.619CD1A2078@async.async.caltech.edu> , <04F51416-6EAC-4BBD-8C4B-623245437ECD@cs.purdue.edu> Message-ID: #line directives should help source debugging. I agree the types would be a bit off. For example arrays, objects, texts, would have internals showing, and texts might take extra steps to see what you really want. - Jay Subject: Re: [M3devel] new target? :) From: hosking at cs.purdue.edu Date: Fri, 25 Feb 2011 10:08:15 -0500 CC: mika at async.caltech.edu; m3devel at elegosoft.com To: jay.krell at cornell.edu Just to follow up briefly as I have lots of other things on my plate.One of the reasons for targeting the gcc-based backend was to support easy generation of *source*-level debugging information. It is important that the debugger see types as Modula-3 sees them, rather than as C types generated form the Modula-3. Now, I agree that debugging the C-translated types is probably better than nothing. On Feb 24, 2011, at 8:55 PM, Jay K wrote:When you write that system that doesn't have the support of any C compiler, sure, go ahead and write a backend just for it, and maybe a linker, and rewrite the small parts that depend on C and Posix in some sort of native way -- including writing all the underlying stuff. Getting Modula-3 working on that system will be a relatively small task, compared to whatever else is needed to build the system. Note for example, we can't e.g. open files in Modula-3, w/o underlying Win32 or Posix functionality. Everywhere that varies for Win32 vs. Posix would have to be revisited in this hypothetical system. This has little/no bearing on the vast vast vast vast majority of real world systems. The current Modula-3 system is NOT implemented entirely in Modula-3. It MUST depend on SOMETHING. So far, for the vast vast vast vast vast majority of systems, that something is written in C, and has a C runtime, compiler, and linker. Yes, I've heard of SPIN. You know..how are you actually going to network I/O or keyboard I/O or power management or video I/O or mouse I/O or sound I/O or preemptive multi-processor scheduling? You need device drivers. You need a processor-specific context switcher. i.e. you need a kernel and you need C AND/OR you need to do a lot of work. If you consider that our system is already forked along Win32 and Posix lines, you can consider these hypothetical systems a third (fourth, fifth, etc.) fork. A third (fourth) implementation of threading. A third implementation of file I/O. A third implementation of Trestle. etc. But this has little/no bearing how we should target existing real systems. You can say, not that "Modula-3 depends on C (runtime, compiler, linker)", but that "Modula-3 on Linux, NT, *BSD, Solaris, Darwin depends on C (runtime, compiler, linker)", and there's absolutely nothing wrong with that, as C and C++ are well supported on those systems and always will be. Increasing that dependency, on those systems, is fine. It would be nice to port the integrated backend to lots more systems, and to write our own linker. But not easy. And notice how the integrated backend also has the worst debugging story of the two/three options, currently. (I say three options because m3cg kind of has two modes, one is incomplete and currently disabled, the other requires m3gdb). One problem I see here, is that, you kind of sort of maybe forseeable want a backend that outputs Java or C#. C# particularly for Windows Phone 7/Nokia. Java kind of for Android, where Java is preferred over C/C++. i.e. there are systems that sort of lack C and C++. Not that they replace it with Modula-3 though. However this isn't super easy, like if you are to include the existing m3core/garbage collector. C# is optionally safe. I don't know if it would serve as a good target. (Optional safety imho makes C# among one of the closest languages to Modula-3. It is missing "compile to native", but that maybe is a good thing.) In addition, C# and Java do each compile down to one well documented "object" file format, so maybe skipping the text is probably sensible. Esp. since with C# there are in-proc libraries for codegen. More likely, if a Java/C# target were really needed, it'd include forking within m3core and replace the garbage collector. This is all highly specuative stuff. - Jay > To: jay.krell at cornell.edu > Date: Thu, 24 Feb 2011 17:31:00 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] new target? :) > > Jay K writes: > ... > >m3cg is=2C technically=2C a gcc frontend=2C and therefore GPL. > >However its input is not human authored text. > >Its input is the output of a "real" front end -- m3front/cm3. > >By making the "call mechanism" be write to a file and run a process=2C the = > >GPL doesn't cross the barrier=2C > >leaving cm3/m3front=2C and more importantly=2C m3core/libm3=2C not GPL. > >=20 > >=20 > >I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. > >=20 > >=20 > > Good points all about C but I wonder about what is really lost. Also ...... > Modula-3 is supposed to be portable to systems without C. You're supposed > to be able to write an entire system in Modula-3. Ok generating C obviously > doesn't break that feature but you might have to worry a bit about having > too much C sneak into the runtime libraries. > > Well given that CM3 is released under what is more or less an MIT/BSD license, > I think the problem is rather that Some Evil Software Company could write > a completely non-free front-end and use m3cg to generate code... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Fri Feb 25 20:08:49 2011 From: dragisha at m3w.org (dragisha at m3w.org) Date: Fri, 25 Feb 2011 20:08:49 +0100 Subject: [M3devel] new target? :) In-Reply-To: References: Message-ID: <775012e5aaa51d171471059dac6ab8be@rstel.net> I am on the road so this is easiest way to amend this... As I see, some people steered this discussion to web dev... One of things I made some time ago is verry complete FastCGI support for Modula-3, and I will contribute it one of these days, I just need more time to go through everything and ensure it works on all platforms I have available now (OSX, Linux, Windows...)... FastCGI is server side, and there is no question about Modula-3 perfectness for anything server. This link I gave is a way for us to get our perfect platform client side. No oblets (IIRC), no nothing... Pure Modula-3. On Thu, 24 Feb 2011 20:16:58 +0100, Dragi?a Duri? wrote: > http://code.google.com/chrome/nativeclient/ [1] > > Links: > ------ > [1] http://code.google.com/chrome/nativeclient/ From rcolebur at SCIRES.COM Sat Feb 26 10:47:02 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sat, 26 Feb 2011 04:47:02 -0500 Subject: [M3devel] results of threadtest program on Windows7 Message-ID: Mika: I've finally managed to get cm3 rebuilt on Windows 7 again. So, I ran your threadtest program. Here are the results. Note the "..." is where I cut out a bunch of the repeating "ERROR FApply" messages. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe Writing file...done Creating read threads...done Creating fork threads...done Creating alloc threads...done Creating lock threads...done running...printing oldest/median age/newest ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. . . . *** *** runtime error: *** An enumeration or subrange value was out of range. *** file "..\src\Main.m3", line 340 *** laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. . . . laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. . . . ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. Stack trace: FP PC Procedure --------- --------- ------------------------------- 0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 Regards, Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 26 11:25:51 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 26 Feb 2011 10:25:51 +0000 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: Message-ID: The FApply problems are because the wants to run "sleep". I've checked in a sleep subset including a Win32 port (untested, but it is very simple). We should get that available to the test and have it run m3sleep instead of sleep. Or you can manually copy this m3sleep.exe to %windir%\system32\sleep.exe or such. Or put Cygwin sleep.exe in %PATH%, but that is overkill. Probably the thread tester could implement a -sleep flag and just call itself with that.. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Sat, 26 Feb 2011 04:47:02 -0500 Subject: [M3devel] results of threadtest program on Windows7 Mika: I?ve finally managed to get cm3 rebuilt on Windows 7 again. So, I ran your threadtest program. Here are the results. Note the ??? is where I cut out a bunch of the repeating ?ERROR FApply? messages. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exeWriting file...doneCreating read threads...doneCreating fork threads...doneCreating alloc threads...doneCreating lock threads...donerunning...printing oldest/median age/newestERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....****** runtime error:*** An enumeration or subrange value was out of range.*** file "..\src\Main.m3", line 340*** laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.Stack trace: FP PC Procedure--------- --------- ------------------------------- 0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 Regards,Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 26 11:42:47 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 26 Feb 2011 10:42:47 +0000 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: , Message-ID: On Darwin, we often get back EBUSY from pthread_mutex_delete. And often: *** *** runtime error: *** An array subscript was out of range. *** file "../src/runtime/common/RTCollector.m3", line 418 *** Abort trap I'll try to look into this more/sooner... - Jay From: jay.krell at cornell.edu To: rcolebur at scires.com; m3devel at elegosoft.com Date: Sat, 26 Feb 2011 10:25:51 +0000 Subject: Re: [M3devel] results of threadtest program on Windows7 The FApply problems are because the wants to run "sleep". I've checked in a sleep subset including a Win32 port (untested, but it is very simple). We should get that available to the test and have it run m3sleep instead of sleep. Or you can manually copy this m3sleep.exe to %windir%\system32\sleep.exe or such. Or put Cygwin sleep.exe in %PATH%, but that is overkill. Probably the thread tester could implement a -sleep flag and just call itself with that.. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Sat, 26 Feb 2011 04:47:02 -0500 Subject: [M3devel] results of threadtest program on Windows7 Mika: I?ve finally managed to get cm3 rebuilt on Windows 7 again. So, I ran your threadtest program. Here are the results. Note the ??? is where I cut out a bunch of the repeating ?ERROR FApply? messages. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exeWriting file...doneCreating read threads...doneCreating fork threads...doneCreating alloc threads...doneCreating lock threads...donerunning...printing oldest/median age/newestERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....****** runtime error:*** An enumeration or subrange value was out of range.*** file "..\src\Main.m3", line 340*** laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.Stack trace: FP PC Procedure--------- --------- ------------------------------- 0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 Regards,Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 26 18:55:11 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 26 Feb 2011 09:55:11 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: Message-ID: <20110226175511.3F7D21A2078@async.async.caltech.edu> Hi Randy, Hm yes it looks like my Windows programming skills leave something to be desired. You can run the thread tester while skipping a test as follows threadtest -tests ALL,-fork (for instance) if you just run threadtest -sadfassdaf it'll print the tests that are available. As it happens, I just had to upgrade my windows 2000 system to windows 7. Can you give me a very brief description of what you did to install Modula-3 on this system? Mika "Coleburn, Randy" writes: >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/plain; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > >Mika: > >I've finally managed to get cm3 rebuilt on Windows 7 again. > >So, I ran your threadtest program. > >Here are the results. Note the "..." is where I cut out a bunch of the rep= >eating "ERROR FApply" messages. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >*** >*** runtime error: >*** An enumeration or subrange value was out of range. >*** file "..\src\Main.m3", line 340 >*** > >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >Stack trace: > FP PC Procedure >--------- --------- ------------------------------- >0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 > >Regards, >Randy Coleburn > >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/html; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > From mika at async.caltech.edu Sat Feb 26 19:10:44 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 26 Feb 2011 10:10:44 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: Message-ID: <20110226181044.A44DD1A2078@async.async.caltech.edu> Hi Randy, I just implemented Jay's suggestion for removing the dependency on the external program "sleep". You can CVS update and try again now. Mika From jay.krell at cornell.edu Sat Feb 26 22:13:27 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 26 Feb 2011 21:13:27 +0000 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: <20110226181044.A44DD1A2078@async.async.caltech.edu> References: , <20110226181044.A44DD1A2078@async.async.caltech.edu> Message-ID: Thanks. jbook2:thread jay$ ./I386_DARWIN/threadtest Writing file...done Creating read threads...done Creating fork threads...done Creating alloc threads...done Creating lock threads...done running...printing oldest/median age/newest ..........laziest thread is 1/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 1/0/0 lock 1/1/1) ..........laziest thread is 1/0/0 (tests: read 1/0/0 fork 1/1/0 alloc 0/0/0 lock 0/0/0) those aren't errors, right? I can reproduce problems on Darwin -- assertion failures, errors in ThreadPThreadC.c like EBUSY and I think EINVAL. EBUSY is from pthread_mutex_destroy, which means we are freeing a mutex while it is locked, which means there is a problem. Such as the garbage collector collecting non-garbage. It ccould be something else though. I'm trying with user threads. User threads do print the above. But so far has a longer runtime than with pthreads, hasn't hit any SIGSEGV or assertion failure or anything yet. I do wonder .. oh, and then w/ user threads: All tests complete. Congratulations. I do wonder about scheduling order and when the preemptions occur. For example user threads does offer Disable/EnableScheduling. Maybe the timeslices vary a lot and userthreads preempt a lot less? Maybe we can try w/ tuning the user thread timeslice to match whatever pthread uses on whatever platform ( there likely no guarantee or documentation, but for experimental purposes we can likely figure it out experimentally). It will be interesting to see the results on Windows too. I'll probably try that right away. - Jay > To: rcolebur at SCIRES.COM > Date: Sat, 26 Feb 2011 10:10:44 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] results of threadtest program on Windows7 > > > Hi Randy, > > I just implemented Jay's suggestion for removing the dependency on the > external program "sleep". You can CVS update and try again now. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 27 00:10:55 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 26 Feb 2011 15:10:55 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: , <20110226181044.A44DD1A2078@async.async.caltech.edu> Message-ID: <20110226231055.932CA1A2078@async.async.caltech.edu> Jay K writes: >--_fe2b6a60-67dc-46f1-b844-2023fbc40258_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Thanks. > >jbook2:thread jay$ ./I386_DARWIN/threadtest=20 >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >..........laziest thread is 1/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 1/0/0= > lock 1/1/1) >..........laziest thread is 1/0/0 (tests: read 1/0/0 fork 1/1/0 alloc 0/0/0= > lock 0/0/0) > >those aren't errors=2C right? No those are just the noral diagnostics. The numbers are how many seconds the various threads are lagging. User threads do have a "more unfair" scheduler than you can expect with Darwin (see comments in the code)... You can change the number of threads etc but I find that the defaults seem to catch stuff. Lowering the number of threads per test from default 3 to "-n 2" simplifies debugging, perhaps. (More comments in the code.) Mika > >I can reproduce problems on Darwin -- assertion failures=2C errors in Threa= >dPThreadC.c like EBUSY and I think EINVAL. > EBUSY is from pthread_mutex_destroy=2C which means we are freeing a mutex= > while it is locked=2C which means > there is a problem. Such as the garbage collector collecting non-garbage= >. It ccould be something else though. > >I'm trying with user threads. User threads do print the above. But so far h= >as a longer runtime than with pthreads=2C > hasn't hit any SIGSEGV or assertion failure or anything yet. > >I do wonder .. oh=2C and then w/ user threads: > >All tests complete. Congratulations. > >I do wonder about scheduling order and when the preemptions occur. >For example user threads does offer Disable/EnableScheduling. >Maybe the timeslices vary a lot and userthreads preempt a lot less? >Maybe we can try w/ tuning the user thread timeslice to match whatever pthr= >ead uses on whatever platform ( >there likely no guarantee or documentation=2C but for experimental purposes= > we can likely figure it out >experimentally). > >It will be interesting to see the results on Windows too. I'll probably try= > that right away. > > - Jay From rcolebur at SCIRES.COM Sun Feb 27 20:09:08 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sun, 27 Feb 2011 14:09:08 -0500 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: <20110226175511.3F7D21A2078@async.async.caltech.edu> References: <20110226175511.3F7D21A2078@async.async.caltech.edu> Message-ID: Mika: Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. Here is what happens now when I run your threadtest program on Windows 7. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-fork Writing file...done Creating read threads...done Creating nxread threads...done Creating tryexcept threads...done Creating forktoomuch threads...done Creating alloc threads...done Creating creat threads...done Creating lock threads...done running...printing oldest/median age/newest *** *** runtime error: *** An array subscript was out of range. *** file "..\src\runtime\common\RTCollector.m3", line 418 *** *** *** runtime error: *** <*ASSERT*> failed. *** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 *** The last error repeats ad infinitum until I press CNTRL-C to abort. I'll send more info on the Windows install of Modula3 in a subsequent post. Regards, Randy Coleburn -----Original Message----- From: Mika Nystrom [mailto:mika at async.caltech.edu] Sent: Saturday, February 26, 2011 12:55 PM To: Coleburn, Randy Cc: m3devel at elegosoft.com Subject: Re: [M3devel] results of threadtest program on Windows7 Hi Randy, Hm yes it looks like my Windows programming skills leave something to be desired. You can run the thread tester while skipping a test as follows threadtest -tests ALL,-fork (for instance) if you just run threadtest -sadfassdaf it'll print the tests that are available. As it happens, I just had to upgrade my windows 2000 system to windows 7. Can you give me a very brief description of what you did to install Modula-3 on this system? Mika "Coleburn, Randy" writes: >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/plain; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > >Mika: > >I've finally managed to get cm3 rebuilt on Windows 7 again. > >So, I ran your threadtest program. > >Here are the results. Note the "..." is where I cut out a bunch of the rep= >eating "ERROR FApply" messages. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >*** >*** runtime error: >*** An enumeration or subrange value was out of range. >*** file "..\src\Main.m3", line 340 >*** > >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >Stack trace: > FP PC Procedure >--------- --------- ------------------------------- >0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 > >Regards, >Randy Coleburn > >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/html; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > From rcolebur at SCIRES.COM Sun Feb 27 20:45:29 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sun, 27 Feb 2011 14:45:29 -0500 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: <20110226175511.3F7D21A2078@async.async.caltech.edu> Message-ID: Mika et al: Thought I would try something else. I took the sources of your thread test program to an older XP machine that has CM3 circa August 2008. This is the machine and implementation I used when building a major project I did a couple years back. The thread test program does indeed build on this old system, but when I run it, I get different results than with the latest HEAD branch code. After it prints "running...printing oldest/median age/newest", on the next line I get two periods ".." and now the program seems hung. I'll let it "run" for a few more minutes to see if anything else happens before killing it. At least we don't get the subscript and assertion failures on this older CM3 platform. Regards, Randy Coleburn -----Original Message----- From: Coleburn, Randy Sent: Sunday, February 27, 2011 2:09 PM To: m3devel at elegosoft.com Subject: Re: [M3devel] results of threadtest program on Windows7 Mika: Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. Here is what happens now when I run your threadtest program on Windows 7. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-fork Writing file...done Creating read threads...done Creating nxread threads...done Creating tryexcept threads...done Creating forktoomuch threads...done Creating alloc threads...done Creating creat threads...done Creating lock threads...done running...printing oldest/median age/newest *** *** runtime error: *** An array subscript was out of range. *** file "..\src\runtime\common\RTCollector.m3", line 418 *** *** *** runtime error: *** <*ASSERT*> failed. *** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 *** The last error repeats ad infinitum until I press CNTRL-C to abort. I'll send more info on the Windows install of Modula3 in a subsequent post. Regards, Randy Coleburn -----Original Message----- From: Mika Nystrom [mailto:mika at async.caltech.edu] Sent: Saturday, February 26, 2011 12:55 PM To: Coleburn, Randy Cc: m3devel at elegosoft.com Subject: Re: [M3devel] results of threadtest program on Windows7 Hi Randy, Hm yes it looks like my Windows programming skills leave something to be desired. You can run the thread tester while skipping a test as follows threadtest -tests ALL,-fork (for instance) if you just run threadtest -sadfassdaf it'll print the tests that are available. As it happens, I just had to upgrade my windows 2000 system to windows 7. Can you give me a very brief description of what you did to install Modula-3 on this system? Mika "Coleburn, Randy" writes: >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/plain; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > >Mika: > >I've finally managed to get cm3 rebuilt on Windows 7 again. > >So, I ran your threadtest program. > >Here are the results. Note the "..." is where I cut out a bunch of the rep= >eating "ERROR FApply" messages. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >*** >*** runtime error: >*** An enumeration or subrange value was out of range. >*** file "..\src\Main.m3", line 340 >*** > >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >Stack trace: > FP PC Procedure >--------- --------- ------------------------------- >0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 > >Regards, >Randy Coleburn > >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/html; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > From mika at async.caltech.edu Sun Feb 27 21:29:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 27 Feb 2011 12:29:50 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: <20110226175511.3F7D21A2078@async.async.caltech.edu> Message-ID: <20110227202950.46F3B1A2078@async.async.caltech.edu> Hi Randy, You can try it with -tests POSIX as well. I find on user threads it runs very slowly (but it does run) because of how unfair the thread scheduler is. Next step might be to whittle down the tests and see if you can get a failure with a single test running and -n 2. That would likely be the simplest scenario to start further debugging from. Mika "Coleburn, Randy" writes: >Mika et al: > >Thought I would try something else. > >I took the sources of your thread test program to an older XP machine that = >has CM3 circa August 2008. This is the machine and implementation I used w= >hen building a major project I did a couple years back. > >The thread test program does indeed build on this old system, but when I ru= >n it, I get different results than with the latest HEAD branch code. =20 > >After it prints "running...printing oldest/median age/newest", on the next = >line I get two periods ".." and now the program seems hung. I'll let it "r= >un" for a few more minutes to see if anything else happens before killing i= >t. > >At least we don't get the subscript and assertion failures on this older CM= >3 platform. > >Regards, >Randy Coleburn > > >-----Original Message----- >From: Coleburn, Randy=20 >Sent: Sunday, February 27, 2011 2:09 PM >To: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7 > >Mika: > >Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. > >Here is what happens now when I run your threadtest program on Windows 7. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-for= >k >Writing file...done >Creating read threads...done >Creating nxread threads...done >Creating tryexcept threads...done >Creating forktoomuch threads...done >Creating alloc threads...done >Creating creat threads...done >Creating lock threads...done >running...printing oldest/median age/newest > > >*** >*** runtime error: >*** An array subscript was out of range. >*** file "..\src\runtime\common\RTCollector.m3", line 418 >*** > > > >*** >*** runtime error: >*** <*ASSERT*> failed. >*** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 >*** > >The last error repeats ad infinitum until I press CNTRL-C to abort. > >I'll send more info on the Windows install of Modula3 in a subsequent post. > >Regards, >Randy Coleburn > >-----Original Message----- >From: Mika Nystrom [mailto:mika at async.caltech.edu]=20 >Sent: Saturday, February 26, 2011 12:55 PM >To: Coleburn, Randy >Cc: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7=20 > >Hi Randy, > >Hm yes it looks like my Windows programming skills leave something >to be desired. > >You can run the thread tester while skipping a test as follows > > threadtest -tests ALL,-fork > >(for instance) > >if you just run=20 > > threadtest -sadfassdaf > >it'll print the tests that are available. > >As it happens, I just had to upgrade my windows 2000 system to windows 7. >Can you give me a very brief description of what you did to install Modula-= >3 >on this system? > > Mika > >"Coleburn, Randy" writes: >>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>Content-Type: text/plain; charset=3D"us-ascii" >>Content-Transfer-Encoding: quoted-printable >> >>Mika: >> >>I've finally managed to get cm3 rebuilt on Windows 7 again. >> >>So, I ran your threadtest program. >> >>Here are the results. Note the "..." is where I cut out a bunch of the re= >p=3D >>eating "ERROR FApply" messages. >> >>C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >>Writing file...done >>Creating read threads...done >>Creating fork threads...done >>Creating alloc threads...done >>Creating lock threads...done >>running...printing oldest/median age/newest >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>*** >>*** runtime error: >>*** An enumeration or subrange value was out of range. >>*** file "..\src\Main.m3", line 340 >>*** >> >>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D2: The syste= >m c=3D >>annot find the file specified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D2: The syste= >m c=3D >>annot find the file specified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>Stack trace: >> FP PC Procedure >>--------- --------- ------------------------------- >>0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >>0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 >> >>Regards, >>Randy Coleburn >> >>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>Content-Type: text/html; charset=3D"us-ascii" >>Content-Transfer-Encoding: quoted-printable >> From rcolebur at SCIRES.COM Sun Feb 27 22:03:13 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sun, 27 Feb 2011 16:03:13 -0500 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: <20110227202950.46F3B1A2078@async.async.caltech.edu> References: <20110226175511.3F7D21A2078@async.async.caltech.edu> <20110227202950.46F3B1A2078@async.async.caltech.edu> Message-ID: Mika: No change with "-tests POSIX". Interesting twist: On Windows 7, I thought I'd see what the command line options are, and I typed "threadtest -help" rather than reading the code. First time, it produced what appears to be a NIL deref crash. Then, I tried it again and it ran to completion. Something seems non-deterministic here. See below. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe -help Writing file...done Creating read threads...done Creating fork threads...done Creating alloc threads...done Creating lock threads...done running...printing oldest/median age/newest . *** *** runtime error: *** Attempt to reference an illegal memory location. *** pc = 0x77762262 *** Stack trace: FP PC Procedure --------- --------- ------------------------------- 0xcdf998 0x130351b SystemError + 0x64 in ..\src\runtime\NT386\RTSignal.m3 0xcdf9c0 0x77762262 0xcdf9d8 0x12e83b7 LockMutex + 0x4f in ..\src\thread\WIN32\ThreadWin32.m3 0xcdfa00 0x12c7b08 GetChar + 0x28 in ..\src\rw\Rd.m3 0xcdfb38 0x12c12e3 RApply + 0xd3 in ..\src\Main.m3 0xcdfb74 0x12e971f ThreadBase + 0x254 in ..\src\thread\WIN32\ThreadWin32.m3 0xcdfb80 0x76543677 0xcdfbc0 0x77779f02 ......... ......... ... more frames ... C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe -help Writing file...done Creating read threads...done Creating fork threads...done Creating alloc threads...done Creating lock threads...done running...printing oldest/median age/newest ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) All tests complete. Congratulations. Regards, Randy Coleburn -----Original Message----- From: Mika Nystrom [mailto:mika at async.caltech.edu] Sent: Sunday, February 27, 2011 3:30 PM To: Coleburn, Randy Cc: m3devel at elegosoft.com Subject: Re: [M3devel] results of threadtest program on Windows7 Hi Randy, You can try it with -tests POSIX as well. I find on user threads it runs very slowly (but it does run) because of how unfair the thread scheduler is. Next step might be to whittle down the tests and see if you can get a failure with a single test running and -n 2. That would likely be the simplest scenario to start further debugging from. Mika "Coleburn, Randy" writes: >Mika et al: > >Thought I would try something else. > >I took the sources of your thread test program to an older XP machine that = >has CM3 circa August 2008. This is the machine and implementation I used w= >hen building a major project I did a couple years back. > >The thread test program does indeed build on this old system, but when I ru= >n it, I get different results than with the latest HEAD branch code. =20 > >After it prints "running...printing oldest/median age/newest", on the next = >line I get two periods ".." and now the program seems hung. I'll let it "r= >un" for a few more minutes to see if anything else happens before killing i= >t. > >At least we don't get the subscript and assertion failures on this older CM= >3 platform. > >Regards, >Randy Coleburn > > >-----Original Message----- >From: Coleburn, Randy=20 >Sent: Sunday, February 27, 2011 2:09 PM >To: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7 > >Mika: > >Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. > >Here is what happens now when I run your threadtest program on Windows 7. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-for= >k >Writing file...done >Creating read threads...done >Creating nxread threads...done >Creating tryexcept threads...done >Creating forktoomuch threads...done >Creating alloc threads...done >Creating creat threads...done >Creating lock threads...done >running...printing oldest/median age/newest > > >*** >*** runtime error: >*** An array subscript was out of range. >*** file "..\src\runtime\common\RTCollector.m3", line 418 >*** > > > >*** >*** runtime error: >*** <*ASSERT*> failed. >*** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 >*** > >The last error repeats ad infinitum until I press CNTRL-C to abort. > >I'll send more info on the Windows install of Modula3 in a subsequent post. > >Regards, >Randy Coleburn > >-----Original Message----- >From: Mika Nystrom [mailto:mika at async.caltech.edu]=20 >Sent: Saturday, February 26, 2011 12:55 PM >To: Coleburn, Randy >Cc: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7=20 > >Hi Randy, > >Hm yes it looks like my Windows programming skills leave something >to be desired. > >You can run the thread tester while skipping a test as follows > > threadtest -tests ALL,-fork > >(for instance) > >if you just run=20 > > threadtest -sadfassdaf > >it'll print the tests that are available. > >As it happens, I just had to upgrade my windows 2000 system to windows 7. >Can you give me a very brief description of what you did to install Modula-= >3 >on this system? > > Mika > >"Coleburn, Randy" writes: >>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>Content-Type: text/plain; charset=3D"us-ascii" >>Content-Transfer-Encoding: quoted-printable >> >>Mika: >> >>I've finally managed to get cm3 rebuilt on Windows 7 again. >> >>So, I ran your threadtest program. >> >>Here are the results. Note the "..." is where I cut out a bunch of the re= >p=3D >>eating "ERROR FApply" messages. >> >>C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >>Writing file...done >>Creating read threads...done >>Creating fork threads...done >>Creating alloc threads...done >>Creating lock threads...done >>running...printing oldest/median age/newest >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>*** >>*** runtime error: >>*** An enumeration or subrange value was out of range. >>*** file "..\src\Main.m3", line 340 >>*** >> >>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D2: The syste= >m c=3D >>annot find the file specified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D2: The syste= >m c=3D >>annot find the file specified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>Stack trace: >> FP PC Procedure >>--------- --------- ------------------------------- >>0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >>0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 >> >>Regards, >>Randy Coleburn >> >>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>Content-Type: text/html; charset=3D"us-ascii" >>Content-Transfer-Encoding: quoted-printable >> From mika at async.caltech.edu Mon Feb 28 00:11:25 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 27 Feb 2011 15:11:25 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: <20110226175511.3F7D21A2078@async.async.caltech.edu> <20110227202950.46F3B1A2078@async.async.caltech.edu> Message-ID: <20110227231125.3DC611A2078@async.async.caltech.edu> Ah, it just doesn't check command-line arguments that carefully. I think what you did is equivalent to "-tests STD". Mika "Coleburn, Randy" writes: >Mika: > >No change with "-tests POSIX". > >Interesting twist: On Windows 7, I thought I'd see what the command line o= >ptions are, and I typed "threadtest -help" rather than reading the code. > >First time, it produced what appears to be a NIL deref crash. Then, I trie= >d it again and it ran to completion. Something seems non-deterministic her= >e. See below. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe -help >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >. > >*** >*** runtime error: >*** Attempt to reference an illegal memory location. >*** pc =3D 0x77762262 >*** > >Stack trace: > FP PC Procedure >--------- --------- ------------------------------- > 0xcdf998 0x130351b SystemError + 0x64 in ..\src\runtime\NT386\RTSignal.m= >3 > 0xcdf9c0 0x77762262 > 0xcdf9d8 0x12e83b7 LockMutex + 0x4f in ..\src\thread\WIN32\ThreadWin32.m= >3 > 0xcdfa00 0x12c7b08 GetChar + 0x28 in ..\src\rw\Rd.m3 > 0xcdfb38 0x12c12e3 RApply + 0xd3 in ..\src\Main.m3 > 0xcdfb74 0x12e971f ThreadBase + 0x254 in ..\src\thread\WIN32\ThreadWin32= >.m3 > 0xcdfb80 0x76543677 > 0xcdfbc0 0x77779f02 >......... ......... ... more frames ... > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe -help >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >All tests complete. Congratulations. > >Regards, >Randy Coleburn > >-----Original Message----- >From: Mika Nystrom [mailto:mika at async.caltech.edu]=20 >Sent: Sunday, February 27, 2011 3:30 PM >To: Coleburn, Randy >Cc: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7=20 > >Hi Randy, > >You can try it with -tests POSIX as well. > >I find on user threads it runs very slowly (but it does run) because of how= > unfair >the thread scheduler is. > >Next step might be to whittle down the tests and see if you can get a failu= >re with >a single test running and -n 2. That would likely be the simplest scenario= > to >start further debugging from. > > Mika > >"Coleburn, Randy" writes: >>Mika et al: >> >>Thought I would try something else. >> >>I took the sources of your thread test program to an older XP machine that= > =3D >>has CM3 circa August 2008. This is the machine and implementation I used = >w=3D >>hen building a major project I did a couple years back. >> >>The thread test program does indeed build on this old system, but when I r= >u=3D >>n it, I get different results than with the latest HEAD branch code. =3D20 >> >>After it prints "running...printing oldest/median age/newest", on the next= > =3D >>line I get two periods ".." and now the program seems hung. I'll let it "= >r=3D >>un" for a few more minutes to see if anything else happens before killing = >i=3D >>t. >> >>At least we don't get the subscript and assertion failures on this older C= >M=3D >>3 platform. >> >>Regards, >>Randy Coleburn >> >> >>-----Original Message----- >>From: Coleburn, Randy=3D20 >>Sent: Sunday, February 27, 2011 2:09 PM >>To: m3devel at elegosoft.com >>Subject: Re: [M3devel] results of threadtest program on Windows7 >> >>Mika: >> >>Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. >> >>Here is what happens now when I run your threadtest program on Windows 7. >> >>C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-fo= >r=3D >>k >>Writing file...done >>Creating read threads...done >>Creating nxread threads...done >>Creating tryexcept threads...done >>Creating forktoomuch threads...done >>Creating alloc threads...done >>Creating creat threads...done >>Creating lock threads...done >>running...printing oldest/median age/newest >> >> >>*** >>*** runtime error: >>*** An array subscript was out of range. >>*** file "..\src\runtime\common\RTCollector.m3", line 418 >>*** >> >> >> >>*** >>*** runtime error: >>*** <*ASSERT*> failed. >>*** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 >>*** >> >>The last error repeats ad infinitum until I press CNTRL-C to abort. >> >>I'll send more info on the Windows install of Modula3 in a subsequent post= >. >> >>Regards, >>Randy Coleburn >> >>-----Original Message----- >>From: Mika Nystrom [mailto:mika at async.caltech.edu]=3D20 >>Sent: Saturday, February 26, 2011 12:55 PM >>To: Coleburn, Randy >>Cc: m3devel at elegosoft.com >>Subject: Re: [M3devel] results of threadtest program on Windows7=3D20 >> >>Hi Randy, >> >>Hm yes it looks like my Windows programming skills leave something >>to be desired. >> >>You can run the thread tester while skipping a test as follows >> >> threadtest -tests ALL,-fork >> >>(for instance) >> >>if you just run=3D20 >> >> threadtest -sadfassdaf >> >>it'll print the tests that are available. >> >>As it happens, I just had to upgrade my windows 2000 system to windows 7. >>Can you give me a very brief description of what you did to install Modula= >-=3D >>3 >>on this system? >> >> Mika >> >>"Coleburn, Randy" writes: >>>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>>Content-Type: text/plain; charset=3D3D"us-ascii" >>>Content-Transfer-Encoding: quoted-printable >>> >>>Mika: >>> >>>I've finally managed to get cm3 rebuilt on Windows 7 again. >>> >>>So, I ran your threadtest program. >>> >>>Here are the results. Note the "..." is where I cut out a bunch of the r= >e=3D >>p=3D3D >>>eating "ERROR FApply" messages. >>> >>>C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >>>Writing file...done >>>Creating read threads...done >>>Creating fork threads...done >>>Creating alloc threads...done >>>Creating lock threads...done >>>running...printing oldest/median age/newest >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>. >>>. >>>. >>>*** >>>*** runtime error: >>>*** An enumeration or subrange value was out of range. >>>*** file "..\src\Main.m3", line 340 >>>*** >>> >>>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The sy= >ste=3D >>m c=3D3D >>>annot find the file specified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>. >>>. >>>. >>>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The sy= >ste=3D >>m c=3D3D >>>annot find the file specified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>. >>>. >>>. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>Stack trace: >>> FP PC Procedure >>>--------- --------- ------------------------------- >>>0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >>>0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 >>> >>>Regards, >>>Randy Coleburn >>> >>>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>>Content-Type: text/html; charset=3D3D"us-ascii" >>>Content-Transfer-Encoding: quoted-printable >>> From rcolebur at SCIRES.COM Tue Feb 1 01:48:21 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Message-ID: Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. Do you have a plan to fix? Anything I can do to help? --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 01:53:21 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 00:53:21 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: References: Message-ID: Oops, I was just wondering to self if I had fixed that. I can get to it fairly soon. We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling convention (which imho is a good reason to make it a separate m3cg call...) (The input parameter is in eax, and the output is in esp.) (Tony, you really don't want a separate m3cg call?) - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. Do you have a plan to fix? Anything I can do to help? --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 01:55:07 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 00:55:07 +0000 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: , , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, , Message-ID: ps: I'm happy to report that my available time here has become significantly reduced. Partly with that in mind, but even w/o it, I am willing to go back to the old way. m3front is full of mystery and I'm not sure I'm keen on understanding it. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 15:26:20 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I think we need to have a way to get the scope of the containing procedure. TryStmt.Parse is not the right place. If I get some time soon I can take a look at this. On Jan 31, 2011, at 2:19 PM, Jay K wrote: Right..I also tried Scope.Insert, didn't seem to help. But I think I might be doing it too late. I'll try doing it in Parse instead of Check. Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat more closely resembles the old pattern: IF Foo() THEN RETURN ELSE TRY FINALLY in the old way would use the stack, so we could too. But the allocation is slower/larger now (alloca possibly aligning up more than needed). Anyway, this point isn't the problem. The problem is my inability to work with m3front. I'll look at it a bit more -- i.e. as I said, moving the code to Parse. The other improvement I'd like to make: currently, if I had it working, what I'm aiming for is: struct EF1{ ... jmpbuf* jb } ef1; jmpbuf* jb = 0; jb = jb ? jb : alloca(sizeof(*jb)); ef1.jb = jb; setjmp(ef1.jb); but it'd be better to have: struct EF1{ ... jmpbuf* jb } ef1; ef1.jb = 0; ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); setjmp(ef1.jb); ie. we don't need the extra pointer variable, the pointer in the record should suffice. [not sure where to put this in email] I tried figuring out how to determine the current vs. topmost-in-current-function scope. I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, so I kinda hoped it'd just use the correct scope. Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. Even if it doesn't get used, that might be ok, since there is tracking of use, and declaring it might not cause any waste if it isn't used. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 12:14:09 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? Hi Jay, I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Jan 30, 2011, at 5:45 AM, Jay K wrote: I'm trying to fix the inadvertant change I made where FOR i := 1 TO 10 DO TRY FINALLY END will call alloca 10 times. I'm not familiar with the frontend, and it is hard for me to understand. It seems like it is probably very well factored, written by someone who really knew what they were doing, but it is dense, and not verbosely commented. Something like this?? But this doesn't work. What am I missing? And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? ? 1.txt Index: src/misc/Marker.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v retrieving revision 1.1.1.1 diff -u -r1.1.1.1 Marker.i3 --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 @@ -64,9 +64,12 @@ PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); (* generate the call to acquire or release a mutex *) -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T); (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) PROCEDURE Reset (); +PROCEDURE NewJumpbuf (): Variable.T; + END Marker. Index: src/misc/Marker.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v retrieving revision 1.9 diff -u -r1.9 Marker.m3 --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 @@ -10,7 +10,7 @@ MODULE Marker; IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; TYPE Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, @@ -231,8 +231,10 @@ END; END CallFinallyHandler; -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T) = VAR new: BOOLEAN; + label := CG.Next_label (); BEGIN (* int setjmp(void* ); *) IF (setjmp = NIL) THEN @@ -262,13 +264,33 @@ Target.Word.size, Target.Word.align, Target.Word.cg_type, 0); END; - - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) + + (* IF jumpuf # NIL THEN + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); + * END + *) + + CG.Load_nil (); + Variable.Load (jumpbuf); + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); + CG.Start_call_direct (alloca, 0, Target.Address.cg_type); CG.Load_int (Target.Word.cg_type, Jumpbuf_size); CG.Pop_param (Target.Word.cg_type); CG.Call_direct (alloca, Target.Address.cg_type); + + (* FUTURE: We should actually have a Variable + * for the entire EF1, including initializing the jumpbuf + * in it. That would save us this extra load/store, + * and save a local variable. + *) + Variable.LoadLValue (jumpbuf); + CG.Swap (); + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); + Variable.Load (jumpbuf); + CG.Store_addr (frame, M3RT.EF1_jmpbuf); + CG.Set_label (label); (* setmp(frame.jmpbuf) *) CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); @@ -806,5 +828,15 @@ tos := 0; END Reset; +PROCEDURE NewJumpbuf (): Variable.T = + VAR jumpbuf: Variable.T; + BEGIN + jumpbuf := Variable.New (M3ID.NoID, TRUE); + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, + readonly := FALSE, open_array_ok := FALSE, + needs_init := TRUE); + RETURN jumpbuf; + END NewJumpbuf; + BEGIN END Marker. Index: src/stmts/TryFinStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v retrieving revision 1.6 diff -u -r1.6 TryFinStmt.m3 --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -10,6 +10,7 @@ IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; +IMPORT Variable; FROM Stmt IMPORT Outcome; TYPE @@ -20,6 +21,7 @@ viaProc : BOOLEAN; scope : Scope.T; handler : HandlerProc; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -63,6 +65,11 @@ RETURN p; END Parse; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + RETURN NOT (Target.Has_stack_walker OR p.viaProc); + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; BEGIN @@ -95,6 +102,9 @@ END; END; TryStmt.PopHandler (); + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE HandlerName (uid: INTEGER): TEXT = @@ -106,11 +116,16 @@ END HandlerName; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN - IF Target.Has_stack_walker THEN RETURN Compile1 (p); - ELSIF p.viaProc THEN RETURN Compile2 (p); - ELSE RETURN Compile3 (p); + IF Target.Has_stack_walker THEN result := Compile1 (p); + ELSIF p.viaProc THEN result := Compile2 (p); + ELSE usesSetjmp := TRUE; + result := Compile3 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -302,7 +317,9 @@ l := CG.Next_label (3); CG.Set_label (l, barrier := TRUE); Marker.PushFrame (frame, M3RT.HandlerClass.Finally); - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) Marker.PushFinally (l, l+1, frame); Index: src/stmts/TryStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v retrieving revision 1.3 diff -u -r1.3 TryStmt.m3 --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -22,6 +22,7 @@ hasElse : BOOLEAN; elseBody : Stmt.T; handled : ESet.T; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -153,6 +154,14 @@ p.handles := h3; END ReverseHandlers; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + IF (p.handles = NIL) AND (NOT p.hasElse) THEN + RETURN FALSE; + END; + RETURN NOT Target.Has_stack_walker; + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR h: Handler; handled: ESet.T; BEGIN @@ -182,6 +191,10 @@ WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; Stmt.TypeCheck (p.elseBody, cs); PopHandler (); + + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = @@ -245,14 +258,19 @@ END CheckHandler; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN IF (p.handles = NIL) AND (NOT p.hasElse) THEN - RETURN Stmt.Compile (p.body); - END; - IF Target.Has_stack_walker - THEN RETURN Compile1 (p); - ELSE RETURN Compile2 (p); + result := Stmt.Compile (p.body); + ELSIF Target.Has_stack_walker THEN + result := Compile1 (p); + ELSE + usesSetjmp := TRUE; + result := Compile2 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -423,7 +441,9 @@ END; (* capture the machine state *) - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) oc := Stmt.Compile (p.body); Thanks, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 04:39:36 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 31 Jan 2011 22:39:36 -0500 Subject: [M3devel] link failure on NT386 In-Reply-To: References: Message-ID: Why can't we just have a run-time hook that resolves differently on different targets? Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. On Jan 31, 2011, at 7:53 PM, Jay K wrote: > Oops, I was just wondering to self if I had fixed that. > I can get to it fairly soon. > We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling > convention (which imho is a good reason to > make it a separate m3cg call...) > (The input parameter is in eax, and the output is in esp.) > (Tony, you really don't want a separate m3cg call?) > > > - Jay > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 31 Jan 2011 19:48:21 -0500 > Subject: [M3devel] link failure on NT386 > > Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. > Do you have a plan to fix? > Anything I can do to help? > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 04:45:53 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 31 Jan 2011 22:45:53 -0500 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: , , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, , Message-ID: <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu> I'd like to do what you envisage. It is a significant improvement, and should not require too much effort: 1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3) 2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. On Jan 31, 2011, at 7:55 PM, Jay K wrote: > ps: I'm happy to report that my available time here > has become significantly reduced. Partly with that in mind, > but even w/o it, I am willing to go back to the old way. > m3front is full of mystery and I'm not sure I'm keen on understanding it. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 15:26:20 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I think we need to have a way to get the scope of the containing procedure. > TryStmt.Parse is not the right place. > If I get some time soon I can take a look at this. > > On Jan 31, 2011, at 2:19 PM, Jay K wrote: > > Right..I also tried Scope.Insert, didn't seem to help. > > But I think I might be doing it too late. I'll try doing it in Parse instead of Check. > Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. > I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat > more closely resembles the old pattern: > > IF Foo() THEN > RETURN > ELSE TRY FINALLY > > in the old way would use the stack, so we could too. > But the allocation is slower/larger now (alloca possibly aligning up more than needed). > Anyway, this point isn't the problem. The problem is my inability to work with m3front. > I'll look at it a bit more -- i.e. as I said, moving the code to Parse. > > The other improvement I'd like to make: > > currently, if I had it working, what I'm aiming for is: > struct EF1{ ... jmpbuf* jb } ef1; > jmpbuf* jb = 0; > > jb = jb ? jb : alloca(sizeof(*jb)); > ef1.jb = jb; > setjmp(ef1.jb); > > but it'd be better to have: > > struct EF1{ ... jmpbuf* jb } ef1; > ef1.jb = 0; > > ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); > setjmp(ef1.jb); > > > ie. we don't need the extra pointer variable, the pointer in the record should suffice. > > > [not sure where to put this in email] > I tried figuring out how to determine the current vs. topmost-in-current-function scope. > I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps > some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, > so I kinda hoped it'd just use the correct scope. > > > Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. > Even if it doesn't get used, that might be ok, since there is tracking of use, and > declaring it might not cause any waste if it isn't used. > > > Thanks, > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 12:14:09 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > Hi Jay, > > I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On Jan 30, 2011, at 5:45 AM, Jay K wrote: > > I'm trying to fix the inadvertant change I made where > FOR i := 1 TO 10 DO TRY FINALLY END > > will call alloca 10 times. > > I'm not familiar with the frontend, and it is hard for me to understand. > It seems like it is probably very well factored, written by someone who really knew what they were doing, > but it is dense, and not verbosely commented. > > Something like this?? But this doesn't work. What am I missing? > And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? > > > ? 1.txt > Index: src/misc/Marker.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v > retrieving revision 1.1.1.1 > diff -u -r1.1.1.1 Marker.i3 > --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 > +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 > @@ -64,9 +64,12 @@ > PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); > (* generate the call to acquire or release a mutex *) > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T); > (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) > > PROCEDURE Reset (); > > +PROCEDURE NewJumpbuf (): Variable.T; > + > END Marker. > Index: src/misc/Marker.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v > retrieving revision 1.9 > diff -u -r1.9 Marker.m3 > --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 > +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,7 +10,7 @@ > MODULE Marker; > > IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; > -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; > +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; > > TYPE > Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, > @@ -231,8 +231,10 @@ > END; > END CallFinallyHandler; > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T) = > VAR new: BOOLEAN; > + label := CG.Next_label (); > BEGIN > (* int setjmp(void* ); *) > IF (setjmp = NIL) THEN > @@ -262,13 +264,33 @@ > Target.Word.size, Target.Word.align, > Target.Word.cg_type, 0); > END; > - > - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) > + > + (* IF jumpuf # NIL THEN > + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); > + * END > + *) > + > + CG.Load_nil (); > + Variable.Load (jumpbuf); > + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); > + > CG.Start_call_direct (alloca, 0, Target.Address.cg_type); > CG.Load_int (Target.Word.cg_type, Jumpbuf_size); > CG.Pop_param (Target.Word.cg_type); > CG.Call_direct (alloca, Target.Address.cg_type); > + > + (* FUTURE: We should actually have a Variable > + * for the entire EF1, including initializing the jumpbuf > + * in it. That would save us this extra load/store, > + * and save a local variable. > + *) > + Variable.LoadLValue (jumpbuf); > + CG.Swap (); > + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); > + Variable.Load (jumpbuf); > + > CG.Store_addr (frame, M3RT.EF1_jmpbuf); > + CG.Set_label (label); > > (* setmp(frame.jmpbuf) *) > CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); > @@ -806,5 +828,15 @@ > tos := 0; > END Reset; > > +PROCEDURE NewJumpbuf (): Variable.T = > + VAR jumpbuf: Variable.T; > + BEGIN > + jumpbuf := Variable.New (M3ID.NoID, TRUE); > + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, > + readonly := FALSE, open_array_ok := FALSE, > + needs_init := TRUE); > + RETURN jumpbuf; > + END NewJumpbuf; > + > BEGIN > END Marker. > Index: src/stmts/TryFinStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v > retrieving revision 1.6 > diff -u -r1.6 TryFinStmt.m3 > --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 > +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,6 +10,7 @@ > > IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; > IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; > +IMPORT Variable; > FROM Stmt IMPORT Outcome; > > TYPE > @@ -20,6 +21,7 @@ > viaProc : BOOLEAN; > scope : Scope.T; > handler : HandlerProc; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -63,6 +65,11 @@ > RETURN p; > END Parse; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + RETURN NOT (Target.Has_stack_walker OR p.viaProc); > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; > BEGIN > @@ -95,6 +102,9 @@ > END; > END; > TryStmt.PopHandler (); > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE HandlerName (uid: INTEGER): TEXT = > @@ -106,11 +116,16 @@ > END HandlerName; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > - IF Target.Has_stack_walker THEN RETURN Compile1 (p); > - ELSIF p.viaProc THEN RETURN Compile2 (p); > - ELSE RETURN Compile3 (p); > + IF Target.Has_stack_walker THEN result := Compile1 (p); > + ELSIF p.viaProc THEN result := Compile2 (p); > + ELSE usesSetjmp := TRUE; > + result := Compile3 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -302,7 +317,9 @@ > l := CG.Next_label (3); > CG.Set_label (l, barrier := TRUE); > Marker.PushFrame (frame, M3RT.HandlerClass.Finally); > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > Marker.PushFinally (l, l+1, frame); > Index: src/stmts/TryStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v > retrieving revision 1.3 > diff -u -r1.3 TryStmt.m3 > --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 > +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -22,6 +22,7 @@ > hasElse : BOOLEAN; > elseBody : Stmt.T; > handled : ESet.T; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -153,6 +154,14 @@ > p.handles := h3; > END ReverseHandlers; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + IF (p.handles = NIL) AND (NOT p.hasElse) THEN > + RETURN FALSE; > + END; > + RETURN NOT Target.Has_stack_walker; > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR h: Handler; handled: ESet.T; > BEGIN > @@ -182,6 +191,10 @@ > WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; > Stmt.TypeCheck (p.elseBody, cs); > PopHandler (); > + > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = > @@ -245,14 +258,19 @@ > END CheckHandler; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > IF (p.handles = NIL) AND (NOT p.hasElse) THEN > - RETURN Stmt.Compile (p.body); > - END; > - IF Target.Has_stack_walker > - THEN RETURN Compile1 (p); > - ELSE RETURN Compile2 (p); > + result := Stmt.Compile (p.body); > + ELSIF Target.Has_stack_walker THEN > + result := Compile1 (p); > + ELSE > + usesSetjmp := TRUE; > + result := Compile2 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -423,7 +441,9 @@ > END; > > (* capture the machine state *) > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > oc := Stmt.Compile (p.body); > > Thanks, > - Jay > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 08:12:13 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 07:12:13 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: References: , , Message-ID: Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. Why are the atomics in m3cg? I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. alloca is simpler just by virtue of it only taking one parameter. Why aren't and/or/xor/add runtime hooks, or special functions? Or the set operations? I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. And that there may be a desire to in general avoid growing m3cg. esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. I'm also wishy washy. I can see it is easy to implement easily any way. I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. At some point, I realize, you can model just about everything as function calls, except, calling functions. :) I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. Since anything implemented in Modula-3 and exposed must be in some interface. However runtime hooks can be implemented in C as well. It depends somewhat if they are only called by generated code or also hand written Modula-3. You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, there is just relying on m3front and m3core agreeing. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:39:36 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] link failure on NT386 Why can't we just have a run-time hook that resolves differently on different targets?Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. On Jan 31, 2011, at 7:53 PM, Jay K wrote:Oops, I was just wondering to self if I had fixed that. I can get to it fairly soon. We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling convention (which imho is a good reason to make it a separate m3cg call...) (The input parameter is in eax, and the output is in esp.) (Tony, you really don't want a separate m3cg call?) - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier.Do you have a plan to fix?Anything I can do to help?--Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 08:16:40 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 07:16:40 +0000 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu> References: , , , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, , , , , , <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu> Message-ID: These look like good hints. I wouldn't know to make such slightly large changes, that the existing infrastructure doesn't suffice. I'll have a go at it..as my now-very-limited time permits. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:45:53 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I'd like to do what you envisage. It is a significant improvement, and should not require too much effort:1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3)2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. On Jan 31, 2011, at 7:55 PM, Jay K wrote:ps: I'm happy to report that my available time here has become significantly reduced. Partly with that in mind, but even w/o it, I am willing to go back to the old way. m3front is full of mystery and I'm not sure I'm keen on understanding it. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 15:26:20 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I think we need to have a way to get the scope of the containing procedure.TryStmt.Parse is not the right place.If I get some time soon I can take a look at this. On Jan 31, 2011, at 2:19 PM, Jay K wrote:Right..I also tried Scope.Insert, didn't seem to help. But I think I might be doing it too late. I'll try doing it in Parse instead of Check. Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat more closely resembles the old pattern: IF Foo() THEN RETURN ELSE TRY FINALLY in the old way would use the stack, so we could too. But the allocation is slower/larger now (alloca possibly aligning up more than needed). Anyway, this point isn't the problem. The problem is my inability to work with m3front. I'll look at it a bit more -- i.e. as I said, moving the code to Parse. The other improvement I'd like to make: currently, if I had it working, what I'm aiming for is: struct EF1{ ... jmpbuf* jb } ef1; jmpbuf* jb = 0; jb = jb ? jb : alloca(sizeof(*jb)); ef1.jb = jb; setjmp(ef1.jb); but it'd be better to have: struct EF1{ ... jmpbuf* jb } ef1; ef1.jb = 0; ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); setjmp(ef1.jb); ie. we don't need the extra pointer variable, the pointer in the record should suffice. [not sure where to put this in email] I tried figuring out how to determine the current vs. topmost-in-current-function scope. I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, so I kinda hoped it'd just use the correct scope. Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. Even if it doesn't get used, that might be ok, since there is tracking of use, and declaring it might not cause any waste if it isn't used. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 12:14:09 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? Hi Jay, I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. Antony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Jan 30, 2011, at 5:45 AM, Jay K wrote:I'm trying to fix the inadvertant change I made where FOR i := 1 TO 10 DO TRY FINALLY END will call alloca 10 times. I'm not familiar with the frontend, and it is hard for me to understand. It seems like it is probably very well factored, written by someone who really knew what they were doing, but it is dense, and not verbosely commented. Something like this?? But this doesn't work. What am I missing? And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? ? 1.txt Index: src/misc/Marker.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v retrieving revision 1.1.1.1 diff -u -r1.1.1.1 Marker.i3 --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 @@ -64,9 +64,12 @@ PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); (* generate the call to acquire or release a mutex *) -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T); (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) PROCEDURE Reset (); +PROCEDURE NewJumpbuf (): Variable.T; + END Marker. Index: src/misc/Marker.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v retrieving revision 1.9 diff -u -r1.9 Marker.m3 --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 @@ -10,7 +10,7 @@ MODULE Marker; IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; TYPE Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, @@ -231,8 +231,10 @@ END; END CallFinallyHandler; -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T) = VAR new: BOOLEAN; + label := CG.Next_label (); BEGIN (* int setjmp(void* ); *) IF (setjmp = NIL) THEN @@ -262,13 +264,33 @@ Target.Word.size, Target.Word.align, Target.Word.cg_type, 0); END; - - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) + + (* IF jumpuf # NIL THEN + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); + * END + *) + + CG.Load_nil (); + Variable.Load (jumpbuf); + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); + CG.Start_call_direct (alloca, 0, Target.Address.cg_type); CG.Load_int (Target.Word.cg_type, Jumpbuf_size); CG.Pop_param (Target.Word.cg_type); CG.Call_direct (alloca, Target.Address.cg_type); + + (* FUTURE: We should actually have a Variable + * for the entire EF1, including initializing the jumpbuf + * in it. That would save us this extra load/store, + * and save a local variable. + *) + Variable.LoadLValue (jumpbuf); + CG.Swap (); + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); + Variable.Load (jumpbuf); + CG.Store_addr (frame, M3RT.EF1_jmpbuf); + CG.Set_label (label); (* setmp(frame.jmpbuf) *) CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); @@ -806,5 +828,15 @@ tos := 0; END Reset; +PROCEDURE NewJumpbuf (): Variable.T = + VAR jumpbuf: Variable.T; + BEGIN + jumpbuf := Variable.New (M3ID.NoID, TRUE); + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, + readonly := FALSE, open_array_ok := FALSE, + needs_init := TRUE); + RETURN jumpbuf; + END NewJumpbuf; + BEGIN END Marker. Index: src/stmts/TryFinStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v retrieving revision 1.6 diff -u -r1.6 TryFinStmt.m3 --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -10,6 +10,7 @@ IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; +IMPORT Variable; FROM Stmt IMPORT Outcome; TYPE @@ -20,6 +21,7 @@ viaProc : BOOLEAN; scope : Scope.T; handler : HandlerProc; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -63,6 +65,11 @@ RETURN p; END Parse; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + RETURN NOT (Target.Has_stack_walker OR p.viaProc); + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; BEGIN @@ -95,6 +102,9 @@ END; END; TryStmt.PopHandler (); + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE HandlerName (uid: INTEGER): TEXT = @@ -106,11 +116,16 @@ END HandlerName; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN - IF Target.Has_stack_walker THEN RETURN Compile1 (p); - ELSIF p.viaProc THEN RETURN Compile2 (p); - ELSE RETURN Compile3 (p); + IF Target.Has_stack_walker THEN result := Compile1 (p); + ELSIF p.viaProc THEN result := Compile2 (p); + ELSE usesSetjmp := TRUE; + result := Compile3 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -302,7 +317,9 @@ l := CG.Next_label (3); CG.Set_label (l, barrier := TRUE); Marker.PushFrame (frame, M3RT.HandlerClass.Finally); - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) Marker.PushFinally (l, l+1, frame); Index: src/stmts/TryStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v retrieving revision 1.3 diff -u -r1.3 TryStmt.m3 --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -22,6 +22,7 @@ hasElse : BOOLEAN; elseBody : Stmt.T; handled : ESet.T; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -153,6 +154,14 @@ p.handles := h3; END ReverseHandlers; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + IF (p.handles = NIL) AND (NOT p.hasElse) THEN + RETURN FALSE; + END; + RETURN NOT Target.Has_stack_walker; + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR h: Handler; handled: ESet.T; BEGIN @@ -182,6 +191,10 @@ WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; Stmt.TypeCheck (p.elseBody, cs); PopHandler (); + + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = @@ -245,14 +258,19 @@ END CheckHandler; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN IF (p.handles = NIL) AND (NOT p.hasElse) THEN - RETURN Stmt.Compile (p.body); - END; - IF Target.Has_stack_walker - THEN RETURN Compile1 (p); - ELSE RETURN Compile2 (p); + result := Stmt.Compile (p.body); + ELSIF Target.Has_stack_walker THEN + result := Compile1 (p); + ELSE + usesSetjmp := TRUE; + result := Compile2 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -423,7 +441,9 @@ END; (* capture the machine state *) - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) oc := Stmt.Compile (p.body); Thanks, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 15:46:22 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 1 Feb 2011 09:46:22 -0500 Subject: [M3devel] link failure on NT386 In-Reply-To: References: Message-ID: <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu> The atomics are not necessarily functions. They approximate real instructions generated inline. Shall I just go ahead and add it? Sent from my iPad On 01/02/2011, at 2:12 AM, Jay K wrote: > Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? > In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. > Why are the atomics in m3cg? > I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. > alloca is simpler just by virtue of it only taking one parameter. > > Why aren't and/or/xor/add runtime hooks, or special functions? > Or the set operations? > > I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. > > And that there may be a desire to in general avoid growing m3cg. > esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. > > I'm also wishy washy. I can see it is easy to implement easily any way. > I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. > > > At some point, I realize, you can model just about everything as function calls, except, calling functions. :) > > I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. > Since anything implemented in Modula-3 and exposed must be in some interface. > However runtime hooks can be implemented in C as well. > It depends somewhat if they are only called by generated code or also hand written Modula-3. > You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, > there is just relying on m3front and m3core agreeing. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:39:36 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] link failure on NT386 > > Why can't we just have a run-time hook that resolves differently on different targets? > Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. > > On Jan 31, 2011, at 7:53 PM, Jay K wrote: > > Oops, I was just wondering to self if I had fixed that. > I can get to it fairly soon. > We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling > convention (which imho is a good reason to > make it a separate m3cg call...) > (The input parameter is in eax, and the output is in esp.) > (Tony, you really don't want a separate m3cg call?) > > > - Jay > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 31 Jan 2011 19:48:21 -0500 > Subject: [M3devel] link failure on NT386 > > Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. > Do you have a plan to fix? > Anything I can do to help? > --Randy Coleburn > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 15:48:02 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 1 Feb 2011 09:48:02 -0500 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu> <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu> Message-ID: Let me know if you need some help. Sent from my iPad On 01/02/2011, at 2:16 AM, Jay K wrote: > These look like good hints. I wouldn't know to make such slightly large changes, > that the existing infrastructure doesn't suffice. > I'll have a go at it..as my now-very-limited time permits. > > Thanks, > - Jay > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:45:53 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I'd like to do what you envisage. It is a significant improvement, and should not require too much effort: > 1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3) > 2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. > > On Jan 31, 2011, at 7:55 PM, Jay K wrote: > > ps: I'm happy to report that my available time here > has become significantly reduced. Partly with that in mind, > but even w/o it, I am willing to go back to the old way. > m3front is full of mystery and I'm not sure I'm keen on understanding it. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 15:26:20 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I think we need to have a way to get the scope of the containing procedure. > TryStmt.Parse is not the right place. > If I get some time soon I can take a look at this. > > On Jan 31, 2011, at 2:19 PM, Jay K wrote: > > Right..I also tried Scope.Insert, didn't seem to help. > > But I think I might be doing it too late. I'll try doing it in Parse instead of Check. > Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. > I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat > more closely resembles the old pattern: > > IF Foo() THEN > RETURN > ELSE TRY FINALLY > > in the old way would use the stack, so we could too. > But the allocation is slower/larger now (alloca possibly aligning up more than needed). > Anyway, this point isn't the problem. The problem is my inability to work with m3front. > I'll look at it a bit more -- i.e. as I said, moving the code to Parse. > > The other improvement I'd like to make: > > currently, if I had it working, what I'm aiming for is: > struct EF1{ ... jmpbuf* jb } ef1; > jmpbuf* jb = 0; > > jb = jb ? jb : alloca(sizeof(*jb)); > ef1.jb = jb; > setjmp(ef1.jb); > > but it'd be better to have: > > struct EF1{ ... jmpbuf* jb } ef1; > ef1.jb = 0; > > ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); > setjmp(ef1.jb); > > > ie. we don't need the extra pointer variable, the pointer in the record should suffice. > > > [not sure where to put this in email] > I tried figuring out how to determine the current vs. topmost-in-current-function scope. > I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps > some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, > so I kinda hoped it'd just use the correct scope. > > > Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. > Even if it doesn't get used, that might be ok, since there is tracking of use, and > declaring it might not cause any waste if it isn't used. > > > Thanks, > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 12:14:09 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > Hi Jay, > > I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On Jan 30, 2011, at 5:45 AM, Jay K wrote: > > I'm trying to fix the inadvertant change I made where > FOR i := 1 TO 10 DO TRY FINALLY END > > will call alloca 10 times. > > I'm not familiar with the frontend, and it is hard for me to understand. > It seems like it is probably very well factored, written by someone who really knew what they were doing, > but it is dense, and not verbosely commented. > > Something like this?? But this doesn't work. What am I missing? > And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? > > > ? 1.txt > Index: src/misc/Marker.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v > retrieving revision 1.1.1.1 > diff -u -r1.1.1.1 Marker.i3 > --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 > +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 > @@ -64,9 +64,12 @@ > PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); > (* generate the call to acquire or release a mutex *) > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T); > (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) > > PROCEDURE Reset (); > > +PROCEDURE NewJumpbuf (): Variable.T; > + > END Marker. > Index: src/misc/Marker.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v > retrieving revision 1.9 > diff -u -r1.9 Marker.m3 > --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 > +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,7 +10,7 @@ > MODULE Marker; > > IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; > -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; > +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; > > TYPE > Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, > @@ -231,8 +231,10 @@ > END; > END CallFinallyHandler; > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T) = > VAR new: BOOLEAN; > + label := CG.Next_label (); > BEGIN > (* int setjmp(void* ); *) > IF (setjmp = NIL) THEN > @@ -262,13 +264,33 @@ > Target.Word.size, Target.Word.align, > Target.Word.cg_type, 0); > END; > - > - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) > + > + (* IF jumpuf # NIL THEN > + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); > + * END > + *) > + > + CG.Load_nil (); > + Variable.Load (jumpbuf); > + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); > + > CG.Start_call_direct (alloca, 0, Target.Address.cg_type); > CG.Load_int (Target.Word.cg_type, Jumpbuf_size); > CG.Pop_param (Target.Word.cg_type); > CG.Call_direct (alloca, Target.Address.cg_type); > + > + (* FUTURE: We should actually have a Variable > + * for the entire EF1, including initializing the jumpbuf > + * in it. That would save us this extra load/store, > + * and save a local variable. > + *) > + Variable.LoadLValue (jumpbuf); > + CG.Swap (); > + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); > + Variable.Load (jumpbuf); > + > CG.Store_addr (frame, M3RT.EF1_jmpbuf); > + CG.Set_label (label); > > (* setmp(frame.jmpbuf) *) > CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); > @@ -806,5 +828,15 @@ > tos := 0; > END Reset; > > +PROCEDURE NewJumpbuf (): Variable.T = > + VAR jumpbuf: Variable.T; > + BEGIN > + jumpbuf := Variable.New (M3ID.NoID, TRUE); > + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, > + readonly := FALSE, open_array_ok := FALSE, > + needs_init := TRUE); > + RETURN jumpbuf; > + END NewJumpbuf; > + > BEGIN > END Marker. > Index: src/stmts/TryFinStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v > retrieving revision 1.6 > diff -u -r1.6 TryFinStmt.m3 > --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 > +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,6 +10,7 @@ > > IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; > IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; > +IMPORT Variable; > FROM Stmt IMPORT Outcome; > > TYPE > @@ -20,6 +21,7 @@ > viaProc : BOOLEAN; > scope : Scope.T; > handler : HandlerProc; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -63,6 +65,11 @@ > RETURN p; > END Parse; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + RETURN NOT (Target.Has_stack_walker OR p.viaProc); > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; > BEGIN > @@ -95,6 +102,9 @@ > END; > END; > TryStmt.PopHandler (); > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE HandlerName (uid: INTEGER): TEXT = > @@ -106,11 +116,16 @@ > END HandlerName; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > - IF Target.Has_stack_walker THEN RETURN Compile1 (p); > - ELSIF p.viaProc THEN RETURN Compile2 (p); > - ELSE RETURN Compile3 (p); > + IF Target.Has_stack_walker THEN result := Compile1 (p); > + ELSIF p.viaProc THEN result := Compile2 (p); > + ELSE usesSetjmp := TRUE; > + result := Compile3 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -302,7 +317,9 @@ > l := CG.Next_label (3); > CG.Set_label (l, barrier := TRUE); > Marker.PushFrame (frame, M3RT.HandlerClass.Finally); > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > Marker.PushFinally (l, l+1, frame); > Index: src/stmts/TryStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v > retrieving revision 1.3 > diff -u -r1.3 TryStmt.m3 > --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 > +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -22,6 +22,7 @@ > hasElse : BOOLEAN; > elseBody : Stmt.T; > handled : ESet.T; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -153,6 +154,14 @@ > p.handles := h3; > END ReverseHandlers; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + IF (p.handles = NIL) AND (NOT p.hasElse) THEN > + RETURN FALSE; > + END; > + RETURN NOT Target.Has_stack_walker; > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR h: Handler; handled: ESet.T; > BEGIN > @@ -182,6 +191,10 @@ > WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; > Stmt.TypeCheck (p.elseBody, cs); > PopHandler (); > + > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = > @@ -245,14 +258,19 @@ > END CheckHandler; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > IF (p.handles = NIL) AND (NOT p.hasElse) THEN > - RETURN Stmt.Compile (p.body); > - END; > - IF Target.Has_stack_walker > - THEN RETURN Compile1 (p); > - ELSE RETURN Compile2 (p); > + result := Stmt.Compile (p.body); > + ELSIF Target.Has_stack_walker THEN > + result := Compile1 (p); > + ELSE > + usesSetjmp := TRUE; > + result := Compile2 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -423,7 +441,9 @@ > END; > > (* capture the machine state *) > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > oc := Stmt.Compile (p.body); > > Thanks, > - Jay > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 20:20:50 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 19:20:50 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu> References: , <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu> Message-ID: Modeling something as a function in m3cg does not necessarily correlate to generating a function call. E.g. the existing m3_alloca. And vice versa. e.g some of the set operations. Almost anything can be modeled as a function call in m3cg, but not end up generating one. Many things can be not modeled as a function call, but end up generating one. gcc has a huge number of "builtin functions". So does Visual C++. In Visual C++, basically any "new instruction" (mmx, sse2/3/4, avx) ends up modeled as a function, but generates roughly one instruction per function calls. It's a slightly high level assembly -- i.e. you get to chose the important algorithmic instructions, the compiler does register allocation, stack/local allocation, etc. I can add it -- surely my diff from a few weeks ago wasn't terrible? I'm ok either way. I'm more curious to understand than to "do". I admit. - Jay CC: m3devel at elegosoft.com From: hosking at cs.purdue.edu Subject: Re: [M3devel] link failure on NT386 Date: Tue, 1 Feb 2011 09:46:22 -0500 To: jay.krell at cornell.edu The atomics are not necessarily functions. They approximate real instructions generated inline.Shall I just go ahead and add it? Sent from my iPad On 01/02/2011, at 2:12 AM, Jay K wrote: Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. Why are the atomics in m3cg? I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. alloca is simpler just by virtue of it only taking one parameter. Why aren't and/or/xor/add runtime hooks, or special functions? Or the set operations? I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. And that there may be a desire to in general avoid growing m3cg. esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. I'm also wishy washy. I can see it is easy to implement easily any way. I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. At some point, I realize, you can model just about everything as function calls, except, calling functions. :) I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. Since anything implemented in Modula-3 and exposed must be in some interface. However runtime hooks can be implemented in C as well. It depends somewhat if they are only called by generated code or also hand written Modula-3. You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, there is just relying on m3front and m3core agreeing. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:39:36 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] link failure on NT386 Why can't we just have a run-time hook that resolves differently on different targets?Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. On Jan 31, 2011, at 7:53 PM, Jay K wrote:Oops, I was just wondering to self if I had fixed that. I can get to it fairly soon. We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling convention (which imho is a good reason to make it a separate m3cg call...) (The input parameter is in eax, and the output is in esp.) (Tony, you really don't want a separate m3cg call?) - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier.Do you have a plan to fix?Anything I can do to help?--Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 21:01:44 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 1 Feb 2011 15:01:44 -0500 Subject: [M3devel] link failure on NT386 In-Reply-To: References: <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu> Message-ID: <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu> RTHooks makes most sense. Sent from my iPhone On Feb 1, 2011, at 2:20 PM, Jay K wrote: > Modeling something as a function in m3cg does not necessarily correlate to generating a function call. > E.g. the existing m3_alloca. > And vice versa. e.g some of the set operations. > Almost anything can be modeled as a function call in m3cg, but not end up generating one. > Many things can be not modeled as a function call, but end up generating one. > gcc has a huge number of "builtin functions". So does Visual C++. > In Visual C++, basically any "new instruction" (mmx, sse2/3/4, avx) ends up modeled > as a function, but generates roughly one instruction per function calls. > It's a slightly high level assembly -- i.e. you get to chose the important algorithmic instructions, > the compiler does register allocation, stack/local allocation, etc. > > I can add it -- surely my diff from a few weeks ago wasn't terrible? > > I'm ok either way. > I'm more curious to understand than to "do". I admit. > > - Jay > > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > Subject: Re: [M3devel] link failure on NT386 > Date: Tue, 1 Feb 2011 09:46:22 -0500 > To: jay.krell at cornell.edu > > The atomics are not necessarily functions. They approximate real instructions generated inline. > Shall I just go ahead and add it? > > Sent from my iPad > > On 01/02/2011, at 2:12 AM, Jay K wrote: > > Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? > In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. > Why are the atomics in m3cg? > I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. > alloca is simpler just by virtue of it only taking one parameter. > > Why aren't and/or/xor/add runtime hooks, or special functions? > Or the set operations? > > I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. > > And that there may be a desire to in general avoid growing m3cg. > esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. > > I'm also wishy washy. I can see it is easy to implement easily any way. > I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. > > > At some point, I realize, you can model just about everything as function calls, except, calling functions. :) > > I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. > Since anything implemented in Modula-3 and exposed must be in some interface. > However runtime hooks can be implemented in C as well. > It depends somewhat if they are only called by generated code or also hand written Modula-3. > You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, > there is just relying on m3front and m3core agreeing. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:39:36 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] link failure on NT386 > > Why can't we just have a run-time hook that resolves differently on different targets? > Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. > > On Jan 31, 2011, at 7:53 PM, Jay K wrote: > > Oops, I was just wondering to self if I had fixed that. > I can get to it fairly soon. > We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling > convention (which imho is a good reason to > make it a separate m3cg call...) > (The input parameter is in eax, and the output is in esp.) > (Tony, you really don't want a separate m3cg call?) > > > - Jay > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 31 Jan 2011 19:48:21 -0500 > Subject: [M3devel] link failure on NT386 > > Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. > Do you have a plan to fix? > Anything I can do to help? > --Randy Coleburn > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 22:12:05 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 21:12:05 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu> References: , , , , <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu>, , <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu> Message-ID: Why? You do realize, though it doesn't really matter, that the function cannot be implemented in Modula-3, or C, and isn't implemented on most platforms. It is only implemented on NT, in assembly. Maybe RTHooks is the holder for most function calls from generated code? The interface between any library and m3front? No matter how and if they are implemented? (it could perhaps be implemented in C, void* m3_alloca(size_t n) { return alloca(n); } but I have a "problem" with that, not sure it is a real problem or not, that it strongly resembles returning the address of a local, and if tools had enough visibility into our code, they would flag it as a bug and/or misoptimize it.) As well, this still requires the backend to recognize and special-case the function. As well, it seems like atomics could fit the same model. Why are they in m3cg? I'm trying not to insist on anything in particular, but I'd like to understand why the different options. m3cg offers the most direct, easily implemented, easily optimized, special-case free approach, though it makes m3cg "wider" and m3cg is kind of scary and large. But making special-case functions is just a way to hide the interface widening..in a way that is easier to remove or omit per-target if applicable perhaps.. - Jay From: hosking at cs.purdue.edu Date: Tue, 1 Feb 2011 15:01:44 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] link failure on NT386 RTHooks makes most sense. Sent from my iPhone On Feb 1, 2011, at 2:20 PM, Jay K wrote: Modeling something as a function in m3cg does not necessarily correlate to generating a function call. E.g. the existing m3_alloca. And vice versa. e.g some of the set operations. Almost anything can be modeled as a function call in m3cg, but not end up generating one. Many things can be not modeled as a function call, but end up generating one. gcc has a huge number of "builtin functions". So does Visual C++. In Visual C++, basically any "new instruction" (mmx, sse2/3/4, avx) ends up modeled as a function, but generates roughly one instruction per function calls. It's a slightly high level assembly -- i.e. you get to chose the important algorithmic instructions, the compiler does register allocation, stack/local allocation, etc. I can add it -- surely my diff from a few weeks ago wasn't terrible? I'm ok either way. I'm more curious to understand than to "do". I admit. - Jay CC: m3devel at elegosoft.com From: hosking at cs.purdue.edu Subject: Re: [M3devel] link failure on NT386 Date: Tue, 1 Feb 2011 09:46:22 -0500 To: jay.krell at cornell.edu The atomics are not necessarily functions. They approximate real instructions generated inline. Shall I just go ahead and add it? Sent from my iPad On 01/02/2011, at 2:12 AM, Jay K wrote: Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. Why are the atomics in m3cg? I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. alloca is simpler just by virtue of it only taking one parameter. Why aren't and/or/xor/add runtime hooks, or special functions? Or the set operations? I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. And that there may be a desire to in general avoid growing m3cg. esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. I'm also wishy washy. I can see it is easy to implement easily any way. I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. At some point, I realize, you can model just about everything as function calls, except, calling functions. :) I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. Since anything implemented in Modula-3 and exposed must be in some interface. However runtime hooks can be implemented in C as well. It depends somewhat if they are only called by generated code or also hand written Modula-3. You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, there is just relying on m3front and m3core agreeing. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:39:36 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] link failure on NT386 Why can't we just have a run-time hook that resolves differently on different targets? Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. On Jan 31, 2011, at 7:53 PM, Jay K wrote: Oops, I was just wondering to self if I had fixed that. I can get to it fairly soon. We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling convention (which imho is a good reason to make it a separate m3cg call...) (The input parameter is in eax, and the output is in esp.) (Tony, you really don't want a separate m3cg call?) - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. Do you have a plan to fix? Anything I can do to help? --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Feb 2 00:57:37 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 1 Feb 2011 18:57:37 -0500 Subject: [M3devel] link failure on NT386 In-Reply-To: References: , , , , <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu>, , <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu> Message-ID: I still don't understand what you are proposing? M3CG.alloca? On Feb 1, 2011, at 4:12 PM, Jay K wrote: > Why? You do realize, though it doesn't really matter, that the function cannot be implemented in Modula-3, or C, and isn't implemented on most platforms. It is only implemented on NT, in assembly. > Maybe RTHooks is the holder for most function calls from generated code? > The interface between any library and m3front? > No matter how and if they are implemented? > > (it could perhaps be implemented in C, void* m3_alloca(size_t n) { return alloca(n); } but I have a "problem" with that, not sure it is a real problem or not, that it strongly resembles returning the address of a local, and if tools had enough visibility into our code, they would flag it as a bug and/or misoptimize it.) > > As well, this still requires the backend to recognize and special-case the function. > As well, it seems like atomics could fit the same model. Why are they in m3cg? > > I'm trying not to insist on anything in particular, but I'd like to understand why the different options. > m3cg offers the most direct, easily implemented, easily optimized, special-case free approach, though it makes m3cg "wider" and m3cg is kind of scary and large. But making special-case functions is just a way to hide the interface widening..in a way that is easier to remove or omit per-target if applicable perhaps.. > > > - Jay > > From: hosking at cs.purdue.edu > Date: Tue, 1 Feb 2011 15:01:44 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] link failure on NT386 > > RTHooks makes most sense. > > Sent from my iPhone > > On Feb 1, 2011, at 2:20 PM, Jay K wrote: > > Modeling something as a function in m3cg does not necessarily correlate to generating a function call. > E.g. the existing m3_alloca. > And vice versa. e.g some of the set operations. > Almost anything can be modeled as a function call in m3cg, but not end up generating one. > Many things can be not modeled as a function call, but end up generating one. > gcc has a huge number of "builtin functions". So does Visual C++. > In Visual C++, basically any "new instruction" (mmx, sse2/3/4, avx) ends up modeled > as a function, but generates roughly one instruction per function calls. > It's a slightly high level assembly -- i.e. you get to chose the important algorithmic instructions, > the compiler does register allocation, stack/local allocation, etc. > > I can add it -- surely my diff from a few weeks ago wasn't terrible? > > I'm ok either way. > I'm more curious to understand than to "do". I admit. > > - Jay > > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > Subject: Re: [M3devel] link failure on NT386 > Date: Tue, 1 Feb 2011 09:46:22 -0500 > To: jay.krell at cornell.edu > > The atomics are not necessarily functions. They approximate real instructions generated inline. > Shall I just go ahead and add it? > > Sent from my iPad > > On 01/02/2011, at 2:12 AM, Jay K wrote: > > Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? > In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. > Why are the atomics in m3cg? > I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. > alloca is simpler just by virtue of it only taking one parameter. > > Why aren't and/or/xor/add runtime hooks, or special functions? > Or the set operations? > > I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. > > And that there may be a desire to in general avoid growing m3cg. > esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. > > I'm also wishy washy. I can see it is easy to implement easily any way. > I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. > > > At some point, I realize, you can model just about everything as function calls, except, calling functions. :) > > I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. > Since anything implemented in Modula-3 and exposed must be in some interface. > However runtime hooks can be implemented in C as well. > It depends somewhat if they are only called by generated code or also hand written Modula-3. > You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, > there is just relying on m3front and m3core agreeing. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:39:36 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] link failure on NT386 > > Why can't we just have a run-time hook that resolves differently on different targets? > Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. > > On Jan 31, 2011, at 7:53 PM, Jay K wrote: > > Oops, I was just wondering to self if I had fixed that. > I can get to it fairly soon. > We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling > convention (which imho is a good reason to > make it a separate m3cg call...) > (The input parameter is in eax, and the output is in esp.) > (Tony, you really don't want a separate m3cg call?) > > > - Jay > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 31 Jan 2011 19:48:21 -0500 > Subject: [M3devel] link failure on NT386 > > Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. > Do you have a plan to fix? > Anything I can do to help? > --Randy Coleburn > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Feb 2 01:10:46 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 2 Feb 2011 00:10:46 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: References: , , , , , , , , <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu>, , , , <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu>, , Message-ID: I still don't understand what you are proposing? M3CG.alloca? Yes. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Thu Feb 3 03:02:27 2011 From: hendrik at topoi.pooq.com (Hendrik Boom) Date: Wed, 2 Feb 2011 21:02:27 -0500 Subject: [M3devel] About Juno-2 and automating debugging software In-Reply-To: <22454.41538.qm@web29715.mail.ird.yahoo.com> References: <459675.89026.qm@web29709.mail.ird.yahoo.com> <22454.41538.qm@web29715.mail.ird.yahoo.com> Message-ID: <20110203020227.GB1094@topoi.pooq.com> On Mon, Jan 31, 2011 at 04:20:58AM +0000, Daniel Alejandro Benavides D. wrote: > Hi all: > yes, I believe is doable especially if you think this issue comes again in another form, the AI guys thinking what went wrong, well nothing went wrong as I realize now, there is a lot of research certainly, but in the end is just the same thing like before, and you can see this like the reuse of concepts of prior knowledge as is mentioned between other by Dr Andrew Tanenbaum, it just happens when something like this: see, the GPGPU GPU stuff alike is a nother come of the concept of explicit parallelism and its implication in computing fields, surely more developed in terms of the hardware this was I believe, but at the end is just another way of calling Transputers in earlier times, is just like it, then it was not an abandoned thing like before, just a reuse of the concept, and in practical terms this what it is the Occam Occam2 and alike languages; there is one in particular of my interest here the one built for drawing using CSP of Hoare, the language > is called Armadillo see a slide show here, specially check for the Computer magazine backcover in p. 6 in: > https://lsl.unal.edu.co/fl18/conferencias/09.20_ArmadilloTortugav02.pdf The TUNES website is a good place to look for plans and musings in the direction of formally verified programming: http://tunes.org/ It's a project that aims to use formal verification for security, and has produces a lot of interesting theorizing about the issues, but not much practical code. At least, that was the case last time I looked at the web site seriously a few years ago. -- hendrik From dabenavidesd at yahoo.es Sun Feb 6 01:07:56 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 6 Feb 2011 00:07:56 +0000 (GMT) Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: <20110205221326.967B12474007@birch.elegosoft.com> Message-ID: <692616.32152.qm@web29703.mail.ird.yahoo.com> Hi all: it seems that by denying itself from being well behaved program is checking for the unmatched traces or good behaviour programs, sort of analyzing many times more (incorrect traces with correct one) than checking the traces are well behaved and just checking wheh its not, better to check when its not, and not asumming the traces are ill anyway, then most of the time is spend in good standing than in bad or ill behaved program nalysis. The question how do you get those traces in good state, for instance checking by itself several times you would gain a pattern is repeatable, opposite of and ill corrupt and unmatched execution traces. In that case it makes a trace pattern you are in good stands but of course if it doens't then you have sort of a problem there. The next step is where to get the break point before it goes into an unrepeatable trace, that is when you must have an static analyzer which run a simulation and verifies program is well stated at each assert of it. Then it seems like you were getting the opposite direction going to see where it is not working yet at each part it makes asserts it, sort of looking for not well behaved errors but rather irrepeatable ones, which I think is harder. A error one you get by denying it's safe guarded makes illegal steps in the program you state or forbidden ones, is an unsafe error, a safe error is if you get it to pass it but you can't get it works by its results (sort of visbile) so getting in the right results but not in its good states are plausible to find, but ill ones, sort of what you are looking for i.e. looking for forbidden states that are ill-behaved and get wrong results are almost the harder path to get it such a bug Just to let you know if you see this point. --- El s?b, 5/2/11, Mika Nystrom escribi?: > De: Mika Nystrom > Asunto: [M3commit] CVS Update: cm3 > Para: m3commit at elegosoft.com > Fecha: s?bado, 5 de febrero, 2011 18:13 > CVSROOT: /usr/cvs > Changes by: mika at birch. > 11/02/05 23:13:26 > > Modified files: > cm3/m3-libs/m3core/tests/thread/src/: > Main.m3 > > Log message: > removed a lot of unnecessary heap memory > allocation, which could interfere with the threading tests. > > Most output to Stdio.stdout, not sure if > this is right (compared with Stdio.stderr). > > From hendrik at topoi.pooq.com Sun Feb 6 01:49:35 2011 From: hendrik at topoi.pooq.com (Hendrik Boom) Date: Sat, 5 Feb 2011 19:49:35 -0500 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: <692616.32152.qm@web29703.mail.ird.yahoo.com> References: <20110205221326.967B12474007@birch.elegosoft.com> <692616.32152.qm@web29703.mail.ird.yahoo.com> Message-ID: <20110206004935.GA31175@topoi.pooq.com> On Sun, Feb 06, 2011 at 12:07:56AM +0000, Daniel Alejandro Benavides D. wrote: > Hi all: ... ... > Just to let you know if you see this point. It sounds as if you're saying it's possible to prove a program is faulty by running it, but it's not possible to say it's correct. And concurrency makes things nonrepeatable, especially when there's bugs. -- hendrik From dabenavidesd at yahoo.es Sun Feb 6 02:01:16 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 6 Feb 2011 01:01:16 +0000 (GMT) Subject: [M3devel] About Juno-2 and automating debugging software In-Reply-To: <20110203020227.GB1094@topoi.pooq.com> Message-ID: <749746.64343.qm@web29720.mail.ird.yahoo.com> Hi all: thanks for your asnwer I came to see this today: http://www.espresso-soft.org/main.php?button=supervisors expand the last (show/hide) abstract selector. I think definitively this guys looks better things than many of us !! Or not? In a sense is good I guess, but what if not? Just hope to see what it brings to it. :) --- El mi?, 2/2/11, Hendrik Boom escribi?: > De: Hendrik Boom > Asunto: Re: [M3devel] About Juno-2 and automating debugging software > Para: m3devel at elegosoft.com > Fecha: mi?rcoles, 2 de febrero, 2011 21:02 > On Mon, Jan 31, 2011 at 04:20:58AM > +0000, Daniel Alejandro Benavides D. wrote: > > Hi all: > > yes, I believe is doable especially if you think this > issue comes again in another form, the AI guys thinking what > went wrong, well nothing went wrong as I realize now, there > is a lot of research certainly, but in the end is just the > same thing like before, and you can see this like the reuse > of concepts of prior knowledge as is mentioned between other > by Dr Andrew Tanenbaum, it just happens when something like > this: see, the GPGPU GPU stuff alike is a nother come of the > concept of explicit parallelism and its implication in > computing fields, surely more developed in terms of the > hardware this was I believe, but at the end is just another > way of calling Transputers in earlier times, is just like > it, then it was not an abandoned thing like before, just a > reuse of the concept, and in practical terms this what it is > the Occam Occam2 and alike languages; there is one in > particular of my interest here the one built for drawing > using CSP of Hoare, the language > > is called Armadillo see a slide show here, > specially check for the Computer magazine backcover in p. 6 > in: > > https://lsl.unal.edu.co/fl18/conferencias/09.20_ArmadilloTortugav02.pdf > > The TUNES website is a good place to look for plans and > musings in the > direction of formally verified programming: > > http://tunes.org/ > > It's a project that aims to use formal verification for > security, and > has produces a lot of interesting theorizing about the > issues, but not > much practical code. At least, that was the case last > time I looked > at the web site seriously a few years ago. > > -- hendrik > From dabenavidesd at yahoo.es Sun Feb 6 02:14:01 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 6 Feb 2011 01:14:01 +0000 (GMT) Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: <20110206004935.GA31175@topoi.pooq.com> Message-ID: <879742.59049.qm@web29710.mail.ird.yahoo.com> Hi all: yes in a sense, as we don't have the static checker or any static analyzer for its check yet, that is what it is for now, or Am I wrong? Surely, they showed that is possible to find; Damian Dominguez with Dr Nelson to see a thread deadlock in object oriented code but with a special tool very primitive, but in general we don't have that, I mean we just can get where a thread implementation makes this things illegal, not when the program is illegal by itself, that's the aim for me now. Better to think as you say even better. I don't dispute the last point though you can get a tool to analyze a sample of outputs and make a distribution of it thats what I think a professor told me a day when looking for one of threading without locking, just yielding its control, here surely the case is very different. Thanks in advance --- El s?b, 5/2/11, Hendrik Boom escribi?: > De: Hendrik Boom > Asunto: Re: [M3devel] [M3commit] CVS Update: cm3 > Para: m3devel at elegosoft.com > Fecha: s?bado, 5 de febrero, 2011 19:49 > On Sun, Feb 06, 2011 at 12:07:56AM > +0000, Daniel Alejandro Benavides D. wrote: > > Hi all: > ... > ... > > Just to let you know if you see this point. > > It sounds as if you're saying it's possible to prove a > program is faulty > by running it, but it's not possible to say it's correct. > > And concurrency makes things nonrepeatable, especially when > there's > bugs. > > -- hendrik > From jay.krell at cornell.edu Tue Feb 8 15:06:34 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 8 Feb 2011 14:06:34 +0000 Subject: [M3devel] assert in Modula-3 C code? Message-ID: It was recently suspect that asserts aren't enabled in the C code in m3core.h I don't think so. I think there might just be a missing check of a return value in m3core/thread/*.m3 However, if we want to dispell this fear now and forever, we could do e.g.: m3core.h: M3_DLL_LOCAL void __stdcall M3C__AssertFailed (const char* file, unsigned line, const char* expression); #define M3_ASSERT(expr) ((expr) ? ((void)0) ? M3C__AssertFailed ( \ __FILE__, __LINE__, #expression)) hand.c: M3_DLL_LOCAL void __stdcall M3C__AssertFailed (const char* file, unsigned line, const char* expression) { fprintf (stderr, "%s(%u): assertion failed: %s\n", file, line, expression); abort (); } This might slightly reduce quality on some platforms, e.g.: Assertion failed: (0), function main, file 1.c, line 2. Abort trap jbook2:libm3 jay$ cat 1.c #include int main() { assert(0); return 0; } that is -- see how it printed the file name? Anyway, we could get that back too, like #if __GNUC__ > whatever #define M3_FUNCTION something #elif _MSC_VER > something #define M3_FUNCTION __FUNCTION__ #else #define M3_FUNCTION 0 #endif ? But I think probably using standard assert is adequate or preferable. We could #undef NDEBUG in m3core.h, in case someone added to the C compiler on the config files. or #ifdef NDEBUG, #error. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 8 15:27:57 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 8 Feb 2011 09:27:57 -0500 Subject: [M3devel] assert in Modula-3 C code? In-Reply-To: References: Message-ID: I think I was mistaken. It does appear to be turned on. On Feb 8, 2011, at 9:06 AM, Jay K wrote: > It was recently suspect that asserts aren't enabled in the C code in m3core.h > I don't think so. I think there might just be a missing check of a return value in m3core/thread/*.m3 > > However, if we want to dispell this fear now and forever, we could do e.g.: > > m3core.h: > > M3_DLL_LOCAL > void > __stdcall > M3C__AssertFailed (const char* file, unsigned line, const char* expression); > > #define M3_ASSERT(expr) ((expr) ? ((void)0) ? M3C__AssertFailed ( \ > __FILE__, __LINE__, #expression)) > > > > hand.c: > > M3_DLL_LOCAL > void > __stdcall > M3C__AssertFailed (const char* file, unsigned line, const char* expression) > { > fprintf (stderr, "%s(%u): assertion failed: %s\n", file, line, expression); > abort (); > } > > This might slightly reduce quality on some platforms, e.g.: > > Assertion failed: (0), function main, file 1.c, line 2. > Abort trap > jbook2:libm3 jay$ cat 1.c > #include > int main() { assert(0); return 0; } > > > that is -- see how it printed the file name? Anyway, we could get that back too, like > #if __GNUC__ > whatever > #define M3_FUNCTION something > #elif _MSC_VER > something > #define M3_FUNCTION __FUNCTION__ > #else > #define M3_FUNCTION 0 > #endif > > > ? > > But I think probably using standard assert is adequate or preferable. > We could #undef NDEBUG in m3core.h, in case someone added to the C compiler on the config files. > or #ifdef NDEBUG, #error. > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 8 23:41:44 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 8 Feb 2011 22:41:44 +0000 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: ,,, , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, ,,, , , , , , , <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu>, Message-ID: "Are you sure?" I mean, are you sure the right thing wouldn't be to declare Variables in Try*.Parse or Try*.Check? Possibly they'll even be at a high enough scope as to be correct? ie: they don't really have to be at the function scope, as long as they are just outside the try and the except/finally...depending on how except/finally are generated..ok, this is the less important point. I can see putting them at function-scope being easy enough and more clearly correct. But I do still wonder about the approach using Variable.T and such, instead of something "custom" involving Cg.declare_local and separate hooks in ProcBody esp. for initialization. (Though I can see a "custom" approach having other benefits: e.g. doing a multiplication jmpbuf_size times number of trys and making one larger call to alloca. Furthermore, the approach that uses just one jmpbuf for the entire function, plus a local integer, would more reasonly need a special access to the function scope.) I have a bit more time to spend here now maybe..but I'm tempted to put in a temporary hack of using a 1K jmpbuf for all targets..at least that keeps the per-target code out... and an assert m3core that it is large enough (as was there already). We could also pick some liberal values per-processor, like 64 bytes for x86, 128 for amd64, 512 for sparc, 1K for powerpc/alpha/mips/other, along with the assertion. (powerpc does tend to need quite large jmpbuf, alpha/mips/other are rare and can suffer until a better solution is in place, 64 for x86 is still bloated but not so bad temporarily..)) Again, this is just a stopgap. The fact that very long running loops with a try allocate more and more stack is quite bad. I didn't realize it at the time, sorry. I also think the frontend should declare the EF1 type correctly and itself, so as to initialize the pointer in it, instead of having a separate pointer local variable that gets copied to it. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Tue, 1 Feb 2011 07:16:40 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? These look like good hints. I wouldn't know to make such slightly large changes, that the existing infrastructure doesn't suffice. I'll have a go at it..as my now-very-limited time permits. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:45:53 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I'd like to do what you envisage. It is a significant improvement, and should not require too much effort: 1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3) 2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. On Jan 31, 2011, at 7:55 PM, Jay K wrote: ps: I'm happy to report that my available time here has become significantly reduced. Partly with that in mind, but even w/o it, I am willing to go back to the old way. m3front is full of mystery and I'm not sure I'm keen on understanding it. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 15:26:20 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I think we need to have a way to get the scope of the containing procedure. TryStmt.Parse is not the right place. If I get some time soon I can take a look at this. On Jan 31, 2011, at 2:19 PM, Jay K wrote: Right..I also tried Scope.Insert, didn't seem to help. But I think I might be doing it too late. I'll try doing it in Parse instead of Check. Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat more closely resembles the old pattern: IF Foo() THEN RETURN ELSE TRY FINALLY in the old way would use the stack, so we could too. But the allocation is slower/larger now (alloca possibly aligning up more than needed). Anyway, this point isn't the problem. The problem is my inability to work with m3front. I'll look at it a bit more -- i.e. as I said, moving the code to Parse. The other improvement I'd like to make: currently, if I had it working, what I'm aiming for is: struct EF1{ ... jmpbuf* jb } ef1; jmpbuf* jb = 0; jb = jb ? jb : alloca(sizeof(*jb)); ef1.jb = jb; setjmp(ef1.jb); but it'd be better to have: struct EF1{ ... jmpbuf* jb } ef1; ef1.jb = 0; ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); setjmp(ef1.jb); ie. we don't need the extra pointer variable, the pointer in the record should suffice. [not sure where to put this in email] I tried figuring out how to determine the current vs. topmost-in-current-function scope. I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, so I kinda hoped it'd just use the correct scope. Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. Even if it doesn't get used, that might be ok, since there is tracking of use, and declaring it might not cause any waste if it isn't used. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 12:14:09 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? Hi Jay, I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Jan 30, 2011, at 5:45 AM, Jay K wrote: I'm trying to fix the inadvertant change I made where FOR i := 1 TO 10 DO TRY FINALLY END will call alloca 10 times. I'm not familiar with the frontend, and it is hard for me to understand. It seems like it is probably very well factored, written by someone who really knew what they were doing, but it is dense, and not verbosely commented. Something like this?? But this doesn't work. What am I missing? And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? ? 1.txt Index: src/misc/Marker.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v retrieving revision 1.1.1.1 diff -u -r1.1.1.1 Marker.i3 --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 @@ -64,9 +64,12 @@ PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); (* generate the call to acquire or release a mutex *) -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T); (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) PROCEDURE Reset (); +PROCEDURE NewJumpbuf (): Variable.T; + END Marker. Index: src/misc/Marker.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v retrieving revision 1.9 diff -u -r1.9 Marker.m3 --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 @@ -10,7 +10,7 @@ MODULE Marker; IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; TYPE Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, @@ -231,8 +231,10 @@ END; END CallFinallyHandler; -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T) = VAR new: BOOLEAN; + label := CG.Next_label (); BEGIN (* int setjmp(void* ); *) IF (setjmp = NIL) THEN @@ -262,13 +264,33 @@ Target.Word.size, Target.Word.align, Target.Word.cg_type, 0); END; - - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) + + (* IF jumpuf # NIL THEN + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); + * END + *) + + CG.Load_nil (); + Variable.Load (jumpbuf); + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); + CG.Start_call_direct (alloca, 0, Target.Address.cg_type); CG.Load_int (Target.Word.cg_type, Jumpbuf_size); CG.Pop_param (Target.Word.cg_type); CG.Call_direct (alloca, Target.Address.cg_type); + + (* FUTURE: We should actually have a Variable + * for the entire EF1, including initializing the jumpbuf + * in it. That would save us this extra load/store, + * and save a local variable. + *) + Variable.LoadLValue (jumpbuf); + CG.Swap (); + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); + Variable.Load (jumpbuf); + CG.Store_addr (frame, M3RT.EF1_jmpbuf); + CG.Set_label (label); (* setmp(frame.jmpbuf) *) CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); @@ -806,5 +828,15 @@ tos := 0; END Reset; +PROCEDURE NewJumpbuf (): Variable.T = + VAR jumpbuf: Variable.T; + BEGIN + jumpbuf := Variable.New (M3ID.NoID, TRUE); + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, + readonly := FALSE, open_array_ok := FALSE, + needs_init := TRUE); + RETURN jumpbuf; + END NewJumpbuf; + BEGIN END Marker. Index: src/stmts/TryFinStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v retrieving revision 1.6 diff -u -r1.6 TryFinStmt.m3 --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -10,6 +10,7 @@ IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; +IMPORT Variable; FROM Stmt IMPORT Outcome; TYPE @@ -20,6 +21,7 @@ viaProc : BOOLEAN; scope : Scope.T; handler : HandlerProc; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -63,6 +65,11 @@ RETURN p; END Parse; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + RETURN NOT (Target.Has_stack_walker OR p.viaProc); + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; BEGIN @@ -95,6 +102,9 @@ END; END; TryStmt.PopHandler (); + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE HandlerName (uid: INTEGER): TEXT = @@ -106,11 +116,16 @@ END HandlerName; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN - IF Target.Has_stack_walker THEN RETURN Compile1 (p); - ELSIF p.viaProc THEN RETURN Compile2 (p); - ELSE RETURN Compile3 (p); + IF Target.Has_stack_walker THEN result := Compile1 (p); + ELSIF p.viaProc THEN result := Compile2 (p); + ELSE usesSetjmp := TRUE; + result := Compile3 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -302,7 +317,9 @@ l := CG.Next_label (3); CG.Set_label (l, barrier := TRUE); Marker.PushFrame (frame, M3RT.HandlerClass.Finally); - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) Marker.PushFinally (l, l+1, frame); Index: src/stmts/TryStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v retrieving revision 1.3 diff -u -r1.3 TryStmt.m3 --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -22,6 +22,7 @@ hasElse : BOOLEAN; elseBody : Stmt.T; handled : ESet.T; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -153,6 +154,14 @@ p.handles := h3; END ReverseHandlers; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + IF (p.handles = NIL) AND (NOT p.hasElse) THEN + RETURN FALSE; + END; + RETURN NOT Target.Has_stack_walker; + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR h: Handler; handled: ESet.T; BEGIN @@ -182,6 +191,10 @@ WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; Stmt.TypeCheck (p.elseBody, cs); PopHandler (); + + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = @@ -245,14 +258,19 @@ END CheckHandler; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN IF (p.handles = NIL) AND (NOT p.hasElse) THEN - RETURN Stmt.Compile (p.body); - END; - IF Target.Has_stack_walker - THEN RETURN Compile1 (p); - ELSE RETURN Compile2 (p); + result := Stmt.Compile (p.body); + ELSIF Target.Has_stack_walker THEN + result := Compile1 (p); + ELSE + usesSetjmp := TRUE; + result := Compile2 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -423,7 +441,9 @@ END; (* capture the machine state *) - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) oc := Stmt.Compile (p.body); Thanks, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Feb 9 15:18:59 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 9 Feb 2011 09:18:59 -0500 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: , , , , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, , , , , , , , , , <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu>, Message-ID: <65642FC4-4F9C-4F8B-817E-EE32B93D4E11@cs.purdue.edu> That is overkill and not particularly modular. I'd prefer to keep the changes as localised as possible. On Feb 8, 2011, at 5:41 PM, Jay K wrote: > "Are you sure?" > I mean, are you sure the right thing wouldn't be to declare Variables in Try*.Parse or Try*.Check? > Possibly they'll even be at a high enough scope as to be correct? > > > ie: they don't really have to be at the function scope, as long as they are just outside the try and the except/finally...depending on how except/finally are generated..ok, this is the less important point. I can see putting them at function-scope being easy enough and more clearly correct. But I do still wonder about the approach using Variable.T and such, instead of something "custom" involving Cg.declare_local and separate hooks in ProcBody esp. for initialization. > > > (Though I can see a "custom" approach having other benefits: e.g. doing a multiplication jmpbuf_size times number of trys and making one larger call to alloca. Furthermore, the approach that uses just one jmpbuf for the entire function, plus a local integer, would more reasonly need a special access to the function scope.) > > > I have a bit more time to spend here now maybe..but I'm tempted to put in a temporary hack of using a 1K jmpbuf for all targets..at least that keeps the per-target code out... and an assert m3core that it is large enough (as was there already). > > > > We could also pick some liberal values per-processor, like 64 bytes for x86, 128 for amd64, 512 for sparc, 1K for powerpc/alpha/mips/other, along with the assertion. (powerpc does tend to need quite large jmpbuf, alpha/mips/other are rare and can suffer until a better solution is in place, 64 for x86 is still bloated but not so bad temporarily..)) Again, this is just a stopgap. The fact that very long running loops with a try allocate more and more stack is quite bad. I didn't realize it at the time, sorry. > > > I also think the frontend should declare the EF1 type correctly and itself, so as to initialize the pointer in it, instead of having a separate pointer local variable that gets copied to it. > > > - Jay > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Tue, 1 Feb 2011 07:16:40 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > These look like good hints. I wouldn't know to make such slightly large changes, > that the existing infrastructure doesn't suffice. > I'll have a go at it..as my now-very-limited time permits. > > Thanks, > - Jay > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:45:53 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I'd like to do what you envisage. It is a significant improvement, and should not require too much effort: > 1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3) > 2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. > > On Jan 31, 2011, at 7:55 PM, Jay K wrote: > > ps: I'm happy to report that my available time here > has become significantly reduced. Partly with that in mind, > but even w/o it, I am willing to go back to the old way. > m3front is full of mystery and I'm not sure I'm keen on understanding it. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 15:26:20 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I think we need to have a way to get the scope of the containing procedure. > TryStmt.Parse is not the right place. > If I get some time soon I can take a look at this. > > On Jan 31, 2011, at 2:19 PM, Jay K wrote: > > Right..I also tried Scope.Insert, didn't seem to help. > > But I think I might be doing it too late. I'll try doing it in Parse instead of Check. > Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. > I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat > more closely resembles the old pattern: > > IF Foo() THEN > RETURN > ELSE TRY FINALLY > > in the old way would use the stack, so we could too. > But the allocation is slower/larger now (alloca possibly aligning up more than needed). > Anyway, this point isn't the problem. The problem is my inability to work with m3front. > I'll look at it a bit more -- i.e. as I said, moving the code to Parse. > > The other improvement I'd like to make: > > currently, if I had it working, what I'm aiming for is: > struct EF1{ ... jmpbuf* jb } ef1; > jmpbuf* jb = 0; > > jb = jb ? jb : alloca(sizeof(*jb)); > ef1.jb = jb; > setjmp(ef1.jb); > > but it'd be better to have: > > struct EF1{ ... jmpbuf* jb } ef1; > ef1.jb = 0; > > ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); > setjmp(ef1.jb); > > > ie. we don't need the extra pointer variable, the pointer in the record should suffice. > > > [not sure where to put this in email] > I tried figuring out how to determine the current vs. topmost-in-current-function scope. > I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps > some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, > so I kinda hoped it'd just use the correct scope. > > > Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. > Even if it doesn't get used, that might be ok, since there is tracking of use, and > declaring it might not cause any waste if it isn't used. > > > Thanks, > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 12:14:09 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > Hi Jay, > > I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On Jan 30, 2011, at 5:45 AM, Jay K wrote: > > I'm trying to fix the inadvertant change I made where > FOR i := 1 TO 10 DO TRY FINALLY END > > will call alloca 10 times. > > I'm not familiar with the frontend, and it is hard for me to understand. > It seems like it is probably very well factored, written by someone who really knew what they were doing, > but it is dense, and not verbosely commented. > > Something like this?? But this doesn't work. What am I missing? > And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? > > > ? 1.txt > Index: src/misc/Marker.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v > retrieving revision 1.1.1.1 > diff -u -r1.1.1.1 Marker.i3 > --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 > +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 > @@ -64,9 +64,12 @@ > PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); > (* generate the call to acquire or release a mutex *) > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T); > (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) > > PROCEDURE Reset (); > > +PROCEDURE NewJumpbuf (): Variable.T; > + > END Marker. > Index: src/misc/Marker.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v > retrieving revision 1.9 > diff -u -r1.9 Marker.m3 > --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 > +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,7 +10,7 @@ > MODULE Marker; > > IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; > -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; > +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; > > TYPE > Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, > @@ -231,8 +231,10 @@ > END; > END CallFinallyHandler; > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T) = > VAR new: BOOLEAN; > + label := CG.Next_label (); > BEGIN > (* int setjmp(void* ); *) > IF (setjmp = NIL) THEN > @@ -262,13 +264,33 @@ > Target.Word.size, Target.Word.align, > Target.Word.cg_type, 0); > END; > - > - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) > + > + (* IF jumpuf # NIL THEN > + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); > + * END > + *) > + > + CG.Load_nil (); > + Variable.Load (jumpbuf); > + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); > + > CG.Start_call_direct (alloca, 0, Target.Address.cg_type); > CG.Load_int (Target.Word.cg_type, Jumpbuf_size); > CG.Pop_param (Target.Word.cg_type); > CG.Call_direct (alloca, Target.Address.cg_type); > + > + (* FUTURE: We should actually have a Variable > + * for the entire EF1, including initializing the jumpbuf > + * in it. That would save us this extra load/store, > + * and save a local variable. > + *) > + Variable.LoadLValue (jumpbuf); > + CG.Swap (); > + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); > + Variable.Load (jumpbuf); > + > CG.Store_addr (frame, M3RT.EF1_jmpbuf); > + CG.Set_label (label); > > (* setmp(frame.jmpbuf) *) > CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); > @@ -806,5 +828,15 @@ > tos := 0; > END Reset; > > +PROCEDURE NewJumpbuf (): Variable.T = > + VAR jumpbuf: Variable.T; > + BEGIN > + jumpbuf := Variable.New (M3ID.NoID, TRUE); > + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, > + readonly := FALSE, open_array_ok := FALSE, > + needs_init := TRUE); > + RETURN jumpbuf; > + END NewJumpbuf; > + > BEGIN > END Marker. > Index: src/stmts/TryFinStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v > retrieving revision 1.6 > diff -u -r1.6 TryFinStmt.m3 > --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 > +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,6 +10,7 @@ > > IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; > IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; > +IMPORT Variable; > FROM Stmt IMPORT Outcome; > > TYPE > @@ -20,6 +21,7 @@ > viaProc : BOOLEAN; > scope : Scope.T; > handler : HandlerProc; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -63,6 +65,11 @@ > RETURN p; > END Parse; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + RETURN NOT (Target.Has_stack_walker OR p.viaProc); > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; > BEGIN > @@ -95,6 +102,9 @@ > END; > END; > TryStmt.PopHandler (); > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE HandlerName (uid: INTEGER): TEXT = > @@ -106,11 +116,16 @@ > END HandlerName; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > - IF Target.Has_stack_walker THEN RETURN Compile1 (p); > - ELSIF p.viaProc THEN RETURN Compile2 (p); > - ELSE RETURN Compile3 (p); > + IF Target.Has_stack_walker THEN result := Compile1 (p); > + ELSIF p.viaProc THEN result := Compile2 (p); > + ELSE usesSetjmp := TRUE; > + result := Compile3 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -302,7 +317,9 @@ > l := CG.Next_label (3); > CG.Set_label (l, barrier := TRUE); > Marker.PushFrame (frame, M3RT.HandlerClass.Finally); > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > Marker.PushFinally (l, l+1, frame); > Index: src/stmts/TryStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v > retrieving revision 1.3 > diff -u -r1.3 TryStmt.m3 > --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 > +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -22,6 +22,7 @@ > hasElse : BOOLEAN; > elseBody : Stmt.T; > handled : ESet.T; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -153,6 +154,14 @@ > p.handles := h3; > END ReverseHandlers; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + IF (p.handles = NIL) AND (NOT p.hasElse) THEN > + RETURN FALSE; > + END; > + RETURN NOT Target.Has_stack_walker; > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR h: Handler; handled: ESet.T; > BEGIN > @@ -182,6 +191,10 @@ > WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; > Stmt.TypeCheck (p.elseBody, cs); > PopHandler (); > + > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = > @@ -245,14 +258,19 @@ > END CheckHandler; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > IF (p.handles = NIL) AND (NOT p.hasElse) THEN > - RETURN Stmt.Compile (p.body); > - END; > - IF Target.Has_stack_walker > - THEN RETURN Compile1 (p); > - ELSE RETURN Compile2 (p); > + result := Stmt.Compile (p.body); > + ELSIF Target.Has_stack_walker THEN > + result := Compile1 (p); > + ELSE > + usesSetjmp := TRUE; > + result := Compile2 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -423,7 +441,9 @@ > END; > > (* capture the machine state *) > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > oc := Stmt.Compile (p.body); > > Thanks, > - Jay > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Wed Feb 9 23:00:57 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Wed, 09 Feb 2011 14:00:57 -0800 Subject: [M3devel] threading on Windows? Message-ID: <20110209220057.790671A2078@async.async.caltech.edu> Hi m3devel, I'm just curious if anyone out there who's running CM3 on Windows has had a chance to try my thread testing program yet. Current status on Unix (Linux, FreeBSD) appears to be that there are problems in pthreads (but I think only under heavy load) and that user threading works perfectly. So I now wonder how things are on the third threading platform (Windows). The thread testing program is at m3-libs/m3core/tests/thread . (Note however that there is a memory leak in TRY-EXCEPT in the current head so don't try to update everything to the CVS head. The previous release should be OK.) Also has anyone else noticed that debugging information has recently broken? m3gdb is very unhappy on all platforms for me... Mika From rcolebur at SCIRES.COM Wed Feb 9 23:25:20 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Wed, 9 Feb 2011 17:25:20 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <20110209220057.790671A2078@async.async.caltech.edu> References: <20110209220057.790671A2078@async.async.caltech.edu> Message-ID: Mika: Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. As soon as I can get the build problem resolved, I'll try out your test program. Regards, Randy Coleburn -----Original Message----- From: Mika Nystrom [mailto:mika at async.caltech.edu] Sent: Wednesday, February 09, 2011 5:01 PM To: m3devel at elegosoft.com Subject: [M3devel] threading on Windows? Hi m3devel, I'm just curious if anyone out there who's running CM3 on Windows has had a chance to try my thread testing program yet. Current status on Unix (Linux, FreeBSD) appears to be that there are problems in pthreads (but I think only under heavy load) and that user threading works perfectly. So I now wonder how things are on the third threading platform (Windows). The thread testing program is at m3-libs/m3core/tests/thread . (Note however that there is a memory leak in TRY-EXCEPT in the current head so don't try to update everything to the CVS head. The previous release should be OK.) Also has anyone else noticed that debugging information has recently broken? m3gdb is very unhappy on all platforms for me... Mika From jay.krell at cornell.edu Wed Feb 9 23:58:30 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 9 Feb 2011 22:58:30 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, Message-ID: Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Thu Feb 10 00:22:56 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, Message-ID: I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 10 03:01:22 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 10 Feb 2011 02:01:22 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , Message-ID: In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Thu Feb 10 04:28:45 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Wed, 9 Feb 2011 22:28:45 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , Message-ID: Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 10 05:40:03 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 10 Feb 2011 04:40:03 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , , Message-ID: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -0500 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Feb 10 06:48:20 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 10 Feb 2011 00:48:20 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , , Message-ID: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: > Of course not. > It is a made-up function that the frontend generates a call to. > That it never previously did. > Using the general function call interface. > You need to add a special case, in the general function call code, to do something different and specific > for this specific function. > > > - Jay > > From: rcolebur at SCIRES.COM > To: jay.krell at cornell.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 22:28:45 -0500 > Subject: Re: [M3devel] threading on Windows? > > Jay: > > I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. > > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 9:01 PM > To: Coleburn, Randy; m3devel > Subject: RE: [M3devel] threading on Windows? > > In m3back/src/M3x86.m3. > > - Jay > > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 18:22:56 -0500 > Subject: Re: [M3devel] threading on Windows? > > I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. > All I know is that everything was working fine until I checked out the HEAD repository. > When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? > Are we talking Modula-3 code, C code, Assembler, what? What package/module? > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 5:59 PM > To: Coleburn, Randy; Mika Nystrom; m3devel > Subject: RE: [M3devel] threading on Windows? > > Maybe someone else can do it? > The fix is: special case calls to the function "m3_alloca". > Change it to call "_chkstk" (or maybe "chkstk", whatever works). > The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. > To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. > > - Jay > > > From: rcolebur at SCIRES.COM > > To: mika at async.caltech.edu; m3devel at elegosoft.com > > Date: Wed, 9 Feb 2011 17:25:20 -0500 > > Subject: Re: [M3devel] threading on Windows? > > > > Mika: > > > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > > > As soon as I can get the build problem resolved, I'll try out your test program. > > > > Regards, > > Randy Coleburn > > > > -----Original Message----- > > From: Mika Nystrom [mailto:mika at async.caltech.edu] > > Sent: Wednesday, February 09, 2011 5:01 PM > > To: m3devel at elegosoft.com > > Subject: [M3devel] threading on Windows? > > > > Hi m3devel, > > > > I'm just curious if anyone out there who's running CM3 on Windows has had > > a chance to try my thread testing program yet. Current status on Unix > > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > > I think only under heavy load) and that user threading works perfectly. > > So I now wonder how things are on the third threading platform (Windows). > > The thread testing program is at m3-libs/m3core/tests/thread . > > > > (Note however that there is a memory leak in TRY-EXCEPT in the current > > head so don't try to update everything to the CVS head. The previous > > release should be OK.) > > > > Also has anyone else noticed that debugging information has recently > > broken? m3gdb is very unhappy on all platforms for me... > > > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 10 12:01:12 2011 From: jay.krell at cornell.edu (jay.krell at cornell.edu) Date: Thu, 10 Feb 2011 03:01:12 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu> Message-ID: <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: > What happened to fixing this with runtime hooks? > > > On Feb 9, 2011, at 11:40 PM, Jay K wrote: > >> Of course not. >> It is a made-up function that the frontend generates a call to. >> That it never previously did. >> Using the general function call interface. >> You need to add a special case, in the general function call code, to do something different and specific >> for this specific function. >> >> >> - Jay >> >> From: rcolebur at SCIRES.COM >> To: jay.krell at cornell.edu; m3devel at elegosoft.com >> Date: Wed, 9 Feb 2011 22:28:45 -050 >> Subject: Re: [M3devel] threading on Windows? >> >> Jay: >> >> I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. >> >> Regards, >> Randy >> >> From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K >> Sent: Wednesday, February 09, 2011 9:01 PM >> To: Coleburn, Randy; m3devel >> Subject: RE: [M3devel] threading on Windows? >> >> In m3back/src/M3x86.m3. >> >> - Jay >> >> >> From: rcolebur at SCIRES.COM >> To: m3devel at elegosoft.com >> Date: Wed, 9 Feb 2011 18:22:56 -0500 >> Subject: Re: [M3devel] threading on Windows? >> >> I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. >> All I know is that everything was working fine until I checked out the HEAD repository. >> When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? >> Are we talking Modula-3 code, C code, Assembler, what? What package/module? >> Regards, >> Randy >> >> From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K >> Sent: Wednesday, February 09, 2011 5:59 PM >> To: Coleburn, Randy; Mika Nystrom; m3devel >> Subject: RE: [M3devel] threading on Windows? >> >> Maybe someone else can do it? >> The fix is: special case calls to the function "m3_alloca". >> Change it to call "_chkstk" (or maybe "chkstk", whatever works). >> The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. >> To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. >> >> - Jay >> >> > From: rcolebur at SCIRES.COM >> > To: mika at async.caltech.edu; m3devel at elegosoft.com >> > Date: Wed, 9 Feb 2011 17:25:20 -0500 >> > Subject: Re: [M3devel] threading on Windows? >> > >> > Mika: >> > >> > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. >> > >> > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. >> > >> > As soon as I can get the build problem resolved, I'll try out your test program. >> > >> > Regards, >> > Randy Coleburn >> > >> > -----Original Message----- >> > From: Mika Nystrom [mailto:mika at async.caltech.edu] >> > Sent: Wednesday, February 09, 2011 5:01 PM >> > To: m3devel at elegosoft.com >> > Subject: [M3devel] threading on Windows? >> > >> > Hi m3devel, >> > >> > I'm just curious if anyone out there who's running CM3 on Windows has had >> > a chance to try my thread testing program yet. Current status on Unix >> > (Linux, FreeBSD) appears to be that there are problems in pthreads (but >> > I think only under heavy load) and that user threading works perfectly. >> > So I now wonder how things are on the third threading platform (Windows). >> > The thread testing program is at m3-libs/m3core/tests/thread . >> > >> > (Note however that there is a memory leak in TRY-EXCEPT in the current >> > head so don't try to update everything to the CVS head. The previous >> > release should be OK.) >> > >> > Also has anyone else noticed that debugging information has recently >> > broken? m3gdb is very unhappy on all platforms for me... >> > >> > Mika > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Feb 10 21:32:41 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 10 Feb 2011 15:32:41 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> Message-ID: <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: > 1. I never understood why this any better. > 2. It how things are currently. > 3. Why not m3cg.alloca? (but see #2) > > - Jay/iPad > > On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: > >> What happened to fixing this with runtime hooks? >> >> >> On Feb 9, 2011, at 11:40 PM, Jay K wrote: >> >>> Of course not. >>> It is a made-up function that the frontend generates a call to. >>> That it never previously did. >>> Using the general function call interface. >>> You need to add a special case, in the general function call code, to do something different and specific >>> for this specific function. >>> >>> >>> - Jay >>> >>> From: rcolebur at SCIRES.COM >>> To: jay.krell at cornell.edu; m3devel at elegosoft.com >>> Date: Wed, 9 Feb 2011 22:28:45 -050 >>> Subject: Re: [M3devel] threading on Windows? >>> >>> Jay: >>> >>> I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. >>> >>> Regards, >>> Randy >>> >>> From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K >>> Sent: Wednesday, February 09, 2011 9:01 PM >>> To: Coleburn, Randy; m3devel >>> Subject: RE: [M3devel] threading on Windows? >>> >>> In m3back/src/M3x86.m3. >>> >>> - Jay >>> >>> >>> From: rcolebur at SCIRES.COM >>> To: m3devel at elegosoft.com >>> Date: Wed, 9 Feb 2011 18:22:56 -0500 >>> Subject: Re: [M3devel] threading on Windows? >>> >>> I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. >>> All I know is that everything was working fine until I checked out the HEAD repository. >>> When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? >>> Are we talking Modula-3 code, C code, Assembler, what? What package/module? >>> Regards, >>> Randy >>> >>> From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K >>> Sent: Wednesday, February 09, 2011 5:59 PM >>> To: Coleburn, Randy; Mika Nystrom; m3devel >>> Subject: RE: [M3devel] threading on Windows? >>> >>> Maybe someone else can do it? >>> The fix is: special case calls to the function "m3_alloca". >>> Change it to call "_chkstk" (or maybe "chkstk", whatever works). >>> The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. >>> To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. >>> >>> - Jay >>> >>> > From: rcolebur at SCIRES.COM >>> > To: mika at async.caltech.edu; m3devel at elegosoft.com >>> > Date: Wed, 9 Feb 2011 17:25:20 -0500 >>> > Subject: Re: [M3devel] threading on Windows? >>> > >>> > Mika: >>> > >>> > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. >>> > >>> > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. >>> > >>> > As soon as I can get the build problem resolved, I'll try out your test program. >>> > >>> > Regards, >>> > Randy Coleburn >>> > >>> > -----Original Message----- >>> > From: Mika Nystrom [mailto:mika at async.caltech.edu] >>> > Sent: Wednesday, February 09, 2011 5:01 PM >>> > To: m3devel at elegosoft.com >>> > Subject: [M3devel] threading on Windows? >>> > >>> > Hi m3devel, >>> > >>> > I'm just curious if anyone out there who's running CM3 on Windows has had >>> > a chance to try my thread testing program yet. Current status on Unix >>> > (Linux, FreeBSD) appears to be that there are problems in pthreads (but >>> > I think only under heavy load) and that user threading works perfectly. >>> > So I now wonder how things are on the third threading platform (Windows). >>> > The thread testing program is at m3-libs/m3core/tests/thread . >>> > >>> > (Note however that there is a memory leak in TRY-EXCEPT in the current >>> > head so don't try to update everything to the CVS head. The previous >>> > release should be OK.) >>> > >>> > Also has anyone else noticed that debugging information has recently >>> > broken? m3gdb is very unhappy on all platforms for me... >>> > >>> > Mika >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Thu Feb 10 21:59:25 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 10 Feb 2011 15:59:25 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> Message-ID: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am "in the dark", both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I'm seeing several different symbols that don't resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking > wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Feb 10 22:25:51 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 10 Feb 2011 16:25:51 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> Message-ID: <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu> Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: > Jay / Tony: > > I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. > > I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. > > I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. > > BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. > > Regards, > Randy > > From: Tony Hosking [mailto:hosking at cs.purdue.edu] > Sent: Thursday, February 10, 2011 3:33 PM > To: jay.krell at cornell.edu > Cc: Coleburn, Randy; m3devel > Subject: Re: [M3devel] threading on Windows? > > Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. > > On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: > > > 1. I never understood why this any better. > 2. It how things are currently. > 3. Why not m3cg.alloca? (but see #2) > > - Jay/iPad > > On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: > > What happened to fixing this with runtime hooks? > > > On Feb 9, 2011, at 11:40 PM, Jay K wrote: > > > Of course not. > It is a made-up function that the frontend generates a call to. > That it never previously did. > Using the general function call interface. > You need to add a special case, in the general function call code, to do something different and specific > for this specific function. > > > - Jay > > From: rcolebur at SCIRES.COM > To: jay.krell at cornell.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 22:28:45 -050 > Subject: Re: [M3devel] threading on Windows? > > Jay: > > I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. > > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 9:01 PM > To: Coleburn, Randy; m3devel > Subject: RE: [M3devel] threading on Windows? > > In m3back/src/M3x86.m3. > > - Jay > > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 18:22:56 -0500 > Subject: Re: [M3devel] threading on Windows? > > I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. > All I know is that everything was working fine until I checked out the HEAD repository. > When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? > Are we talking Modula-3 code, C code, Assembler, what? What package/module? > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 5:59 PM > To: Coleburn, Randy; Mika Nystrom; m3devel > Subject: RE: [M3devel] threading on Windows? > > Maybe someone else can do it? > The fix is: special case calls to the function "m3_alloca". > Change it to call "_chkstk" (or maybe "chkstk", whatever works). > The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. > To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. > > - Jay > > > From: rcolebur at SCIRES.COM > > To: mika at async.caltech.edu; m3devel at elegosoft.com > > Date: Wed, 9 Feb 2011 17:25:20 -0500 > > Subject: Re: [M3devel] threading on Windows? > > > > Mika: > > > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > > > As soon as I can get the build problem resolved, I'll try out your test program. > > > > Regards, > > Randy Coleburn > > > > -----Original Message----- > > From: Mika Nystrom [mailto:mika at async.caltech.edu] > > Sent: Wednesday, February 09, 2011 5:01 PM > > To: m3devel at elegosoft.com > > Subject: [M3devel] threading on Windows? > > > > Hi m3devel, > > > > I'm just curious if anyone out there who's running CM3 on Windows has had > > a chance to try my thread testing program yet. Current status on Unix > > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > > I think only under heavy load) and that user threading works perfectly. > > So I now wonder how things are on the third threading platform (Windows). > > The thread testing program is at m3-libs/m3core/tests/thread . > > > > (Note however that there is a memory leak in TRY-EXCEPT in the current > > head so don't try to update everything to the CVS head. The previous > > release should be OK.) > > > > Also has anyone else noticed that debugging information has recently > > broken? m3gdb is very unhappy on all platforms for me... > > > > Mika > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 10 23:11:55 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 10 Feb 2011 22:11:55 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> Message-ID: Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 11 00:13:57 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Thu, 10 Feb 2011 23:13:57 +0000 (GMT) Subject: [M3devel] threading on Windows? In-Reply-To: <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> Message-ID: <295326.94632.qm@web29715.mail.ird.yahoo.com> Hi all: by the way this must be a top one if you want cross platform compatibility rather than built-in platform specific fixes, why just we can make another thread runtime and put everything on it, e.g make this threading behavior consistent with the rest of pthread unsupported platforms (say DECthreads, or any other thread safe interface, was this ever supported i.e for SPIN trends threads or as now Pthreads are as thread safe?), at lower layer level, i.e like it was with the user threads, perhaps it is like the counter example you would need to test more deeply in the implementation not other ones' implementations, perhaps? Is there any test invalid currently, or any of it correct, if there isn't a "safe" pattern in the code why do you test unsafe threaded code (i.e all behave odly bad or all behave the same bad? This is to make another system thread safe implementation, like it was SPIN, sadly uniprocessor platform, this is better to test in an uniprocessor rather than multiprocessor indeed as its a simulation of the real code is running on it), still very impressive bench marks time in the user level or kernel level for DEC system SPIN , is like the same? for both user level ones and besides it behaves like it was a Mach 3.0 kernel user or a DECThreads client, as it had it underneath it layered, this proves a better runtime makes even better for correctness and efficiency, at least is what I see in the benchmarks, who knows with SMP systems, but just a point on it. Besides this SPIN trends threads were more abstract and more specialized e.g allow to select scheduling policy (currently at compile time in pthreads) and timing selected values. I wonder how anything of this would be manageable on a SMP machine, still worth the value to check as it is really faster than anything on the market for those days, see: http://qstream.org/~krasic/cs508-2006/summaries/paper17/SpinOS.ppt http://www.cs.cornell.edu/People/egs/papers/spin-strands.pdf Yet another hint for your thread testing, there is a static checker for threading, it used thread states in object oriented programs written in path expression constraints and proved to check threads synchronization consistency, one would be able to compile it to thread primitives to check for semantic analysis and optimization, just in case the references see below at the end of this paragraph, I must say this looks very interesting so your work would be to create annotations not trace them, or trace them if needed for create them, but still very promising if one is capable of creating the abstract annotations to some extent ahead of time or later if in execution time, as it was in some intelligent debuggers, at some extent should be the call graphs make the path expressions, perhaps abstract as through a Trestle Locking level abstraction and a thread safe condition? http://www.informatik.fh-luebeck.de/Seehusen/Publications/OS92/osna.ps http://130.203.133.150/viewdoc/versions;jsessionid=6B086982E4C53F40A01EDA48A003EE3A?doi=10.1.1.34.1092 (let it http://www.cs.rug.nl/~wim/pub/whh217.ps.gz ) and http://www.cs.rug.nl/~wim/pub/whh233.ps.gz and so http://books.google.com/books?id=Hb_pOpI8mJ8C&lpg=PP1&dq=Concurrent%20programming&pg=PA221#v=onepage&q&f=false http://www.cs.brown.edu/people/spr/fieldbook/fieldbook.html Thanks in advance --- El jue, 10/2/11, jay.krell at cornell.edu escribi?: De: jay.krell at cornell.edu Asunto: Re: [M3devel] threading on Windows? Para: "Tony Hosking" CC: "m3devel" , "Jay K" Fecha: jueves, 10 de febrero, 2011 06:01 1. I never understood why this any better. 2. It how things are currently.3. Why not m3cg.alloca? (but see #2) ?- Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. ? ? ?- Jay? ? From:?rcolebur at SCIRES.COM To:?jay.krell at cornell.edu;?m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay:?I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file.?Regards,Randy?From:?jayk123 at hotmail.com?[mailto:jayk123 at hotmail.com]?On Behalf Of?Jay K Sent:?Wednesday, February 09, 2011 9:01 PM To:?Coleburn, Randy; m3devel Subject:?RE: [M3devel] threading on Windows??In m3back/src/M3x86.m3. ? ?- Jay ?From:?rcolebur at SCIRES.COM To:?m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows?I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve.All I know is that everything was working fine until I checked out the HEAD repository.When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case??Are we talking Modula-3 code, C code, Assembler, what?? What package/module?Regards,Randy?From:?jayk123 at hotmail.com?[mailto:jayk123 at hotmail.com]?On Behalf Of?Jay K Sent:?Wednesday, February 09, 2011 5:59 PM To:?Coleburn, Randy; Mika Nystrom; m3devel Subject:?RE: [M3devel] threading on Windows??Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed?in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. ? ?- Jay ? > From:?rcolebur at SCIRES.COM > To:?mika at async.caltech.edu;?m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? >? > Mika: >? > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. >? > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. >? > As soon as I can get the build problem resolved, I'll try out your test program. >? > Regards, > Randy Coleburn >? > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu]? > Sent: Wednesday, February 09, 2011 5:01 PM > To:?m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? >? > Hi m3devel, >? > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . >? > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) >? > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... >? > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 11 00:40:48 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Thu, 10 Feb 2011 23:40:48 +0000 (GMT) Subject: [M3devel] threading on Windows? In-Reply-To: Message-ID: <161929.78691.qm@web29701.mail.ird.yahoo.com> Hi all: is useful to say here that the compiler linker approach at some extent was done in the same way as was in Modula-3, now Open Watcom, used same runtime structure and required as far as I know no better enlightment in the original versions of it, I read it some quite time ago in its sources, hard to tell you where it was, but if OpenWatcom sold it as a Windows solution in its runtime why not in the Modula-3 one, indeed how it would be to port its linker to work for Modula-3 and tune it for the? required? works of it. Is better to adapt to a better portable (Linux, etc) solution than? to adapt to a? non-portable still solution as I believe. Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] threading on Windows? Para: "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 17:11 Why is it better? ?Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) ?- Jay/iPad ? You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. ? ? Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. ? ? As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. ? ? Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. ? ? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Feb 11 01:36:12 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 11 Feb 2011 00:36:12 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <161929.78691.qm@web29701.mail.ird.yahoo.com> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> Message-ID: Daniel, you generally make very little sense to me. Please consider additional English training. "Modula-3" defines and implements its own threading library, it's own I/O library, it's own traced and untraced memory allocator (traced == garbage collected), it's own GUI library (Trestle). It is its own portabilty layer, akin to various other options, e.g. using pthreads and then implementing pthreads on Windows, using Tk or Qt, etc., and depending on them to be ported. etc. Yes, I am deliberately confusing "language" and "library". Which are people interested in? One, the other, or both? Now, granted, we are portable between essentially only two platforms Win32/NT and "Posix", where "Posix" is defined as including pthreads and X Windows, and even in a sense gcc. (Even if Modula-3 predates Posix, the dependencies are pretty light, such that they have mostly always been almost consistent across the board -- open/read/write/close, socket/accept/connect/recv/send/listen, a few variations for signals). (Personally I would like to change the backend to generate C...) One could "get out of this (library) business" and target another intermediate portability layer, i.e. write "bindings" for Tk, Qt, etc., and then use Modula-3 "just" as a language, and not an associated large set of libraries. In general, it is desired for portability layers to be thin in terms of performance and possibly state. Ideally, given the choice of writing to Win32 threads and getting the smallest fastest program that runs only on Wondows, or pthreads and getting the smallest fastest program that runs on Posix, nobody should shy away from using Modula-3 threads, since they run pretty much just as well as Win32 and pthreads. Pretty much. Actually condition variables can't be done super well on Win32 prior to Vista, but they are done in a pretty ok way now. Still, threads and mutexes at least are very thin layers. Implementing threading more completely yourself, on many systems, precludes you from actually running on more than one processor. Win7's "user mode scheduling" offers an option with more control and still the ability to run on multiple processors. But I suspect we aren't very interested, at least not currently. - Jay Date: Thu, 10 Feb 2011 23:40:48 +0000 From: dabenavidesd at yahoo.es To: hosking at cs.purdue.edu; jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Hi all: is useful to say here that the compiler linker approach at some extent was done in the same way as was in Modula-3, now Open Watcom, used same runtime structure and required as far as I know no better enlightment in the original versions of it, I read it some quite time ago in its sources, hard to tell you where it was, but if OpenWatcom sold it as a Windows solution in its runtime why not in the Modula-3 one, indeed how it would be to port its linker to work for Modula-3 and tune it for the required works of it. Is better to adapt to a better portable (Linux, etc) solution than to adapt to a non-portable still solution as I believe. Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] threading on Windows? Para: "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 17:11 Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 11 02:08:31 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Fri, 11 Feb 2011 01:08:31 +0000 (GMT) Subject: [M3devel] threading on Windows? In-Reply-To: Message-ID: <355223.44215.qm@web29702.mail.ird.yahoo.com> Hi all: thanks for your suggestion yet I learn also from you and others so I try to keep improving it, I may read some English grammar too, this is the best I can do for now, sorry for that and thanks for being open about it (other suggestions are most welcome). Please think about this, I believe this is transforming very quickly and we don't know pretty much yet where it would be in some not short time, see my point, 1.2 billions of smart phones are being ship, 3 million more than those of PCs, if you see this, this is a turn break point, you need to be able to see also this market becomes more important than some may think. Still about the PCs, the current trend is too much research being done at the GPGPU thing, with yet another revolution of this days in computation, this is my point precisely, will be able to tackle the opportunity of the change, or be overwhelmed by it, and just wait for another time. I just say in behalf of the things I have said that could be more common than we thought, yet there are good opportunities, who thought this was another revolution to come, I did not believe it but I'm more turned to think this is another "trend" in computing. And I think you would not believe but in a country like mine they thought to construct hardware assisted graphics chips, it didn't turn out to be real, but many things came after this, we have self configurable hardware, what else you one want? Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: RE: [M3devel] threading on Windows? Para: dabenavidesd at yahoo.es, "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 19:36 Daniel, you generally make very little sense to me. Please consider?additional?English training. ? ? "Modula-3" defines?and implements its?own threading library,?it's own I/O library, it's own traced and untraced memory allocator (traced == garbage collected), it's own GUI library (Trestle). ? It is its own portabilty layer, akin to various other options, e.g. using pthreads and?then implementing pthreads on Windows, using Tk or Qt, etc., and depending on them to be ported. etc. Yes, I am deliberately confusing "language" and "library". Which are people interested in? One, the other, or both? ? ? Now, granted, we are portable between essentially only two platforms Win32/NT and "Posix", where "Posix" is defined as including pthreads and X Windows, and even in a sense gcc. ? (Even if Modula-3 predates Posix, the dependencies are pretty light, such that they have mostly always been almost consistent across the board -- open/read/write/close, socket/accept/connect/recv/send/listen, a few variations for signals). ? (Personally I would like to change the backend to generate C...) ? ? One could "get out of this (library) business" and target another intermediate portability layer, i.e. write "bindings" for Tk, Qt, etc., and then use Modula-3 "just" as a language, and not an?associated large set of libraries. ? ? In general, it is desired for portability layers to be thin in terms of performance and possibly state. Ideally, given the choice of writing to Win32 threads and getting the smallest fastest program that runs only on Wondows, or pthreads and getting the smallest fastest program that runs on Posix, nobody should shy away from using Modula-3 threads, since they run pretty much just as well as Win32 and pthreads. Pretty much. Actually condition variables can't be done super well on Win32 prior to Vista, but they are done in a pretty ok way now. Still, threads and mutexes at least are very thin layers. ? ? Implementing threading more completely yourself, on many systems, precludes you from actually running on more than one processor. Win7's "user mode scheduling" offers an option with more control and still the ability to run on multiple processors. But I suspect we aren't very interested, at least not currently. ? ? ? ?- Jay ? Date: Thu, 10 Feb 2011 23:40:48 +0000 From: dabenavidesd at yahoo.es To: hosking at cs.purdue.edu; jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Hi all: is useful to say here that the compiler linker approach at some extent was done in the same way as was in Modula-3, now Open Watcom, used same runtime structure and required as far as I know no better enlightment in the original versions of it, I read it some quite time ago in its sources, hard to tell you where it was, but if OpenWatcom sold it as a Windows solution in its runtime why not in the Modula-3 one, indeed how it would be to port its linker to work for Modula-3 and tune it for the? required? works of it. Is better to adapt to a better portable (Linux, etc) solution than? to adapt to a? non-portable still solution as I believe. Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] threading on Windows? Para: "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 17:11 #yiv1956618167 .yiv1956618167ExternalClass #yiv1956618167ecxyiv295862095 .yiv1956618167ecxyiv295862095hmmessage P {padding:0px;} #yiv1956618167 .yiv1956618167ExternalClass #yiv1956618167ecxyiv295862095 .yiv1956618167ecxyiv295862095hmmessage {font-size:10pt;font-family:Tahoma;} Why is it better? ?Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) ?- Jay/iPad ? You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. ? ? Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. ? ? As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. ? ? Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. ? ? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 11 03:51:10 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Fri, 11 Feb 2011 02:51:10 +0000 (GMT) Subject: [M3devel] threading on Windows? In-Reply-To: Message-ID: <332041.74062.qm@web29718.mail.ird.yahoo.com> Hi all: Jay let me make a statement, perhaps if some can follow me to agree in this, a claim from a? Professor here, very pro FOSS anyway: "The cost of hardware are always descending and growing up in technology capabilities, this is not the case in software terms, there is little improvement in the techniques and yet they are more excessive expensive each time it has a new version". I would agree if one looks to some companies, like in M$ fields, this is the case certainly, as Win ships in every machine human has seen almost, ok but in other responsible houses this was not the case, Digital Equipment Corporation if I may say so, they had very good machines, Professor told you could specify very precisely what do you would want of it and they shipped and were very endurable. Of software though they didn't make a revolution they did the best achievements in the software I have seen so far, runtime capabilities of this compiler I have see so far anything like this and specially static analysis tools, very impressive, even some people would tell and have told certainly already and certainly they are greater than I am. Thanks in advance Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: RE: [M3devel] threading on Windows? Para: dabenavidesd at yahoo.es, "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 19:36 Daniel, you generally make very little sense to me. Please consider?additional?English training. ? ? "Modula-3" defines?and implements its?own threading library,?it's own I/O library, it's own traced and untraced memory allocator (traced == garbage collected), it's own GUI library (Trestle). ? It is its own portabilty layer, akin to various other options, e.g. using pthreads and?then implementing pthreads on Windows, using Tk or Qt, etc., and depending on them to be ported. etc. Yes, I am deliberately confusing "language" and "library". Which are people interested in? One, the other, or both? ? ? Now, granted, we are portable between essentially only two platforms Win32/NT and "Posix", where "Posix" is defined as including pthreads and X Windows, and even in a sense gcc. ? (Even if Modula-3 predates Posix, the dependencies are pretty light, such that they have mostly always been almost consistent across the board -- open/read/write/close, socket/accept/connect/recv/send/listen, a few variations for signals). ? (Personally I would like to change the backend to generate C...) ? ? One could "get out of this (library) business" and target another intermediate portability layer, i.e. write "bindings" for Tk, Qt, etc., and then use Modula-3 "just" as a language, and not an?associated large set of libraries. ? ? In general, it is desired for portability layers to be thin in terms of performance and possibly state. Ideally, given the choice of writing to Win32 threads and getting the smallest fastest program that runs only on Wondows, or pthreads and getting the smallest fastest program that runs on Posix, nobody should shy away from using Modula-3 threads, since they run pretty much just as well as Win32 and pthreads. Pretty much. Actually condition variables can't be done super well on Win32 prior to Vista, but they are done in a pretty ok way now. Still, threads and mutexes at least are very thin layers. ? ? Implementing threading more completely yourself, on many systems, precludes you from actually running on more than one processor. Win7's "user mode scheduling" offers an option with more control and still the ability to run on multiple processors. But I suspect we aren't very interested, at least not currently. ? ? ? ?- Jay ? Date: Thu, 10 Feb 2011 23:40:48 +0000 From: dabenavidesd at yahoo.es To: hosking at cs.purdue.edu; jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Hi all: is useful to say here that the compiler linker approach at some extent was done in the same way as was in Modula-3, now Open Watcom, used same runtime structure and required as far as I know no better enlightment in the original versions of it, I read it some quite time ago in its sources, hard to tell you where it was, but if OpenWatcom sold it as a Windows solution in its runtime why not in the Modula-3 one, indeed how it would be to port its linker to work for Modula-3 and tune it for the? required? works of it. Is better to adapt to a better portable (Linux, etc) solution than? to adapt to a? non-portable still solution as I believe. Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] threading on Windows? Para: "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 17:11 #yiv863291047 .yiv863291047ExternalClass #yiv863291047ecxyiv295862095 .yiv863291047ecxyiv295862095hmmessage P {padding:0px;} #yiv863291047 .yiv863291047ExternalClass #yiv863291047ecxyiv295862095 .yiv863291047ecxyiv295862095hmmessage {font-size:10pt;font-family:Tahoma;} Why is it better? ?Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) ?- Jay/iPad ? You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. ? ? Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. ? ? As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. ? ? Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. ? ? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Feb 11 22:26:52 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Fri, 11 Feb 2011 16:26:52 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu> Message-ID: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new "heap-based jmpbuf allocation" have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate "alloca" routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I "do" anything, I'll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the "heap based jumpbuf allocation scheme" was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am "in the dark", both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I'm seeing several different symbols that don't resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking > wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Feb 11 22:29:43 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 11 Feb 2011 16:29:43 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu> Message-ID: <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: > Tony: > > Thank you very much for this explanation. > > So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: > 1. Space leak problem. > 2. Binding to appropriate ?alloca? routine on Windows platform. > > Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. > > I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. > > Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. > > Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. > > Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. > > If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. > > Regards, > Randy Coleburn > > From: Tony Hosking [mailto:hosking at cs.purdue.edu] > Sent: Thursday, February 10, 2011 4:26 PM > To: Coleburn, Randy > Cc: m3devel > Subject: Re: [M3devel] threading on Windows? > > Hi Randy, > > A far as I know, the changes are as follows. > > TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. > > The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. > > To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. > > -- Tony > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: > > > Jay / Tony: > > I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. > > I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. > > I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. > > BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. > > Regards, > Randy > > From: Tony Hosking [mailto:hosking at cs.purdue.edu] > Sent: Thursday, February 10, 2011 3:33 PM > To: jay.krell at cornell.edu > Cc: Coleburn, Randy; m3devel > Subject: Re: [M3devel] threading on Windows? > > Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. > > On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: > > > > 1. I never understood why this any better. > 2. It how things are currently. > 3. Why not m3cg.alloca? (but see #2) > > - Jay/iPad > > On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: > > What happened to fixing this with runtime hooks? > > > On Feb 9, 2011, at 11:40 PM, Jay K wrote: > > > > Of course not. > It is a made-up function that the frontend generates a call to. > That it never previously did. > Using the general function call interface. > You need to add a special case, in the general function call code, to do something different and specific > for this specific function. > > > - Jay > > From: rcolebur at SCIRES.COM > To: jay.krell at cornell.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 22:28:45 -050 > Subject: Re: [M3devel] threading on Windows? > > Jay: > > I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. > > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 9:01 PM > To: Coleburn, Randy; m3devel > Subject: RE: [M3devel] threading on Windows? > > In m3back/src/M3x86.m3. > > - Jay > > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 18:22:56 -0500 > Subject: Re: [M3devel] threading on Windows? > > I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. > All I know is that everything was working fine until I checked out the HEAD repository. > When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? > Are we talking Modula-3 code, C code, Assembler, what? What package/module? > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 5:59 PM > To: Coleburn, Randy; Mika Nystrom; m3devel > Subject: RE: [M3devel] threading on Windows? > > Maybe someone else can do it? > The fix is: special case calls to the function "m3_alloca". > Change it to call "_chkstk" (or maybe "chkstk", whatever works). > The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. > To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. > > - Jay > > > From: rcolebur at SCIRES.COM > > To: mika at async.caltech.edu; m3devel at elegosoft.com > > Date: Wed, 9 Feb 2011 17:25:20 -0500 > > Subject: Re: [M3devel] threading on Windows? > > > > Mika: > > > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > > > As soon as I can get the build problem resolved, I'll try out your test program. > > > > Regards, > > Randy Coleburn > > > > -----Original Message----- > > From: Mika Nystrom [mailto:mika at async.caltech.edu] > > Sent: Wednesday, February 09, 2011 5:01 PM > > To: m3devel at elegosoft.com > > Subject: [M3devel] threading on Windows? > > > > Hi m3devel, > > > > I'm just curious if anyone out there who's running CM3 on Windows has had > > a chance to try my thread testing program yet. Current status on Unix > > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > > I think only under heavy load) and that user threading works perfectly. > > So I now wonder how things are on the third threading platform (Windows). > > The thread testing program is at m3-libs/m3core/tests/thread . > > > > (Note however that there is a memory leak in TRY-EXCEPT in the current > > head so don't try to update everything to the CVS head. The previous > > release should be OK.) > > > > Also has anyone else noticed that debugging information has recently > > broken? m3gdb is very unhappy on all platforms for me... > > > > Mika > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Feb 11 22:51:37 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 11 Feb 2011 21:51:37 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> Message-ID: It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - Jay From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote:Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely:1. Space leak problem.2. Binding to appropriate ?alloca? routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards,Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- TonyAntony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. Regards,Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler.On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently.3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote:What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows?Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows?I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve.All I know is that everything was working fine until I checked out the HEAD repository.When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case??Are we talking Modula-3 code, C code, Assembler, what? What package/module?Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 00:09:17 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 11 Feb 2011 23:09:17 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> Message-ID: > me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows It is totally compatible. Someone just has to write a few lines of code to implement it. - Jay From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate ?alloca? routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Sat Feb 12 00:17:15 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Fri, 11 Feb 2011 18:17:15 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> Message-ID: Jay: Ok, I don't mean to start a new issue or upset anyone. I see you checked in some new changes. I updated to latest HEAD and tried to build, but again I get unresolved symbols beginning with m3-libs\sysutils (see attached sysutils.lst) So, not sure what you mean by "it should be ok now". Are you saying you do, or do not, want me to work further on the solving the "problem," regardless of how we describe it ? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, February 11, 2011 4:52 PM To: Tony; Coleburn, Randy Cc: m3devel Subject: RE: [M3devel] threading on Windows? It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - Jay ________________________________ From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new "heap-based jmpbuf allocation" have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate "alloca" routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I "do" anything, I'll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the "heap based jumpbuf allocation scheme" was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am "in the dark", both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I'm seeing several different symbols that don't resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking > wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sysutils.lst Type: application/octet-stream Size: 4059 bytes Desc: sysutils.lst URL: From mika at async.caltech.edu Sat Feb 12 02:57:15 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 17:57:15 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> Message-ID: <20110212015715.A43471A2078@async.async.caltech.edu> Hello m3devel, I'm attempting to test the CVS head as of Jay's latest checkins with the thread tester. I'm afraid there seems to be something wrong. I get a seg fault when running a few threads that do the following: PROCEDURE TApply(cl : Closure) : REFANY = BEGIN Thread.Pause(InitPause); LOOP TRY WITH now = Time.Now() DO times1[cl.id]:= FLOOR(now); IF now < 0.0d0 THEN RAISE X END END EXCEPT X => <*ASSERT FALSE*> END END END TApply; The crash is as follows: (gdb) run -tests tryexcept Starting program: /home/mika/cm3-cvs/cm3/m3-libs/m3core/tests/thread/LINUXLIBC6/threadtest -tests tryexcept [Thread debugging using libthread_db enabled] Writing file...done Creating tryexcept threads...done running...printing oldest/median age/newest . Program received signal SIGSEGV, Segmentation fault. 0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 53 { (gdb) where #0 0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 #1 0x0808cb57 in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 #2 0x0804b875 in Main__TApply (cl=) at ../src/Main.m3:205 #3 0x08086c5e in ThreadPosix__RunThread () at ../src/thread/POSIX/ThreadPosix.m3:993 #4 0x400c343b in makecontext () from /lib/libc.so.6 #5 0x00000000 in ?? () (gdb) Note I am using user threads since pthreads are known to be buggy at the moment. I would like to find one configuration (any configuration!!) that lets me run a multithreaded program under AMD64_LINUX. I don't care what version of the compiler it is, whether the threads are user or pthreads, how many things I have to hack. Does anyone know what I need to do? Mika From mika at async.caltech.edu Sat Feb 12 03:08:36 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 18:08:36 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: <20110212015715.A43471A2078@async.async.caltech.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu> Message-ID: <20110212020836.97F7A1A207A@async.async.caltech.edu> I forgot to add that I was running that program on the platform LINUXLIBC6. On FreeBSD4 I get the following bizarre behavior (program hangs): (gdb) run -tests tryexcept Starting program: /big/home/mika/cm3-writable/cm3/m3-libs/m3core/tests/thread/FreeBSD4/threadtest -tests tryexcept Writing file...done Creating tryexcept threads...done running...printing oldest/median age/newest .^C Program received signal SIGINT, Interrupt. [Switching to Thread 1 (LWP 100315)] 0x681093f3 in pthread_testcancel () from /usr/lib/libpthread.so.1 (gdb) where #0 0x681093f3 in pthread_testcancel () from /usr/lib/libpthread.so.1 #1 0x680f917c in sigaction () from /usr/lib/libpthread.so.1 #2 0x680f9d03 in sigaction () from /usr/lib/libpthread.so.1 #3 0x680f850b in sigaction () from /usr/lib/libpthread.so.1 #4 0x680f828c in sigaction () from /usr/lib/libpthread.so.1 #5 0x6810192e in pthread_mutexattr_init () from /usr/lib/libpthread.so.1 #6 0x00000000 in ?? () (gdb) Even though I am using ThreadPosix, not ThreadPthread... Also on AMD64_LINUX, the following: (gdb) run -tests tryexcept Starting program: /home/mika/cm3/cm3/m3-libs/m3core/tests/thread/AMD64_LINUX/threadtest -tests tryexcept [Thread debugging using libthread_db enabled] Writing file...done Creating tryexcept threads...done running...printing oldest/median age/newest .[New Thread 0x2b7877844620 (LWP 24086)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x2b7877844620 (LWP 24086)] 0x00007fff44dff5b1 in ?? () (gdb) where #0 0x00007fff44dff5b1 in ?? () #1 0x00007fff44dff6a5 in gettimeofday () #2 0x00002b787589690a in gettimeofday () from /lib/libc.so.6 #3 0x000000000045148f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:58 #4 0x000000000045139e in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 #5 0x0000000000404da7 in Main__TApply (cl=Cannot access memory at address 0x2b7974014e68 ) at ../src/Main.m3:205 #6 0x000000000044b3fb in ThreadPosix__RunThread () at ../src/thread/POSIX/ThreadPosix.m3:993 #7 0x00002b787584b7b0 in ?? () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () (gdb) AMD64_FREEBSD seems to work, oddly enough, but slowly. I wonder if my compiler buidling has failed somehow (I am using the Python script following a recipe provided by Jay after modifying m3core/src/thread/m3makefile ) Mika Mika Nystrom writes: > >Hello m3devel, > >I'm attempting to test the CVS head as of Jay's latest checkins with >the thread tester. > >I'm afraid there seems to be something wrong. > >I get a seg fault when running a few threads that do the following: > >PROCEDURE TApply(cl : Closure) : REFANY = > BEGIN > Thread.Pause(InitPause); > LOOP > TRY > WITH now = Time.Now() DO > times1[cl.id]:= FLOOR(now); > IF now < 0.0d0 THEN RAISE X END > END > EXCEPT X => <*ASSERT FALSE*> > END > END > END TApply; > >The crash is as follows: > >(gdb) run -tests tryexcept >Starting program: /home/mika/cm3-cvs/cm3/m3-libs/m3core/tests/thread/LINUXLIBC >6/threadtest -tests tryexcept >[Thread debugging using libthread_db enabled] >Writing file...done >Creating tryexcept threads...done >running...printing oldest/median age/newest >. >Program received signal SIGSEGV, Segmentation fault. >0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 >53 { >(gdb) where >#0 0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 >#1 0x0808cb57 in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 >#2 0x0804b875 in Main__TApply (cl=) > at ../src/Main.m3:205 >#3 0x08086c5e in ThreadPosix__RunThread () > at ../src/thread/POSIX/ThreadPosix.m3:993 >#4 0x400c343b in makecontext () from /lib/libc.so.6 >#5 0x00000000 in ?? () >(gdb) > >Note I am using user threads since pthreads are known to be buggy at the >moment. > >I would like to find one configuration (any configuration!!) that lets >me run a multithreaded program under AMD64_LINUX. I don't care what >version of the compiler it is, whether the threads are user or pthreads, >how many things I have to hack. Does anyone know what I need to do? > > Mika From mika at async.caltech.edu Sat Feb 12 03:50:13 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 18:50:13 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: <20110212020836.97F7A1A207A@async.async.caltech.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu> <20110212020836.97F7A1A207A@async.async.caltech.edu> Message-ID: <20110212025013.916EA1A2078@async.async.caltech.edu> Another odd datapoint is that even rather aggressive testing seems not to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user threads, of course. So status right now of user threads seems to be: FreeBSD4 : strange failure in libpthread as reported by gdb LINUXLIBC6 : seg fault AMD64_LINUX : seg fault AMD64_FREEBSD : OK I386_DARWIN : OK I am never, ever 100% certain of my bootstrapping procedure so it's possible I've done something wrong... however I'm using upgrade.py and when I look for Thread\*.*o files I only see the ThreadPosix file and not ThreadPThread. I'll try bootstrapping again :-) Mika Mika Nystrom writes: > >I forgot to add that I was running that program on the platform LINUXLIBC6. > >On FreeBSD4 I get the following bizarre behavior (program hangs): > >(gdb) run -tests tryexcept >Starting program: /big/home/mika/cm3-writable/cm3/m3-libs/m3core/tests/thread/ >FreeBSD4/threadtest -tests tryexcept >Writing file...done >Creating tryexcept threads...done >running...printing oldest/median age/newest >.^C >Program received signal SIGINT, Interrupt. >[Switching to Thread 1 (LWP 100315)] >0x681093f3 in pthread_testcancel () from /usr/lib/libpthread.so.1 > >(gdb) where >#0 0x681093f3 in pthread_testcancel () from /usr/lib/libpthread.so.1 >#1 0x680f917c in sigaction () from /usr/lib/libpthread.so.1 >#2 0x680f9d03 in sigaction () from /usr/lib/libpthread.so.1 >#3 0x680f850b in sigaction () from /usr/lib/libpthread.so.1 >#4 0x680f828c in sigaction () from /usr/lib/libpthread.so.1 >#5 0x6810192e in pthread_mutexattr_init () from /usr/lib/libpthread.so.1 >#6 0x00000000 in ?? () >(gdb) > >Even though I am using ThreadPosix, not ThreadPthread... > >Also on AMD64_LINUX, the following: > >(gdb) run -tests tryexcept >Starting program: /home/mika/cm3/cm3/m3-libs/m3core/tests/thread/AMD64_LINUX/t >hreadtest -tests tryexcept >[Thread debugging using libthread_db enabled] >Writing file...done >Creating tryexcept threads...done >running...printing oldest/median age/newest >.[New Thread 0x2b7877844620 (LWP 24086)] > >Program received signal SIGSEGV, Segmentation fault. >[Switching to Thread 0x2b7877844620 (LWP 24086)] >0x00007fff44dff5b1 in ?? () >(gdb) where >#0 0x00007fff44dff5b1 in ?? () >#1 0x00007fff44dff6a5 in gettimeofday () >#2 0x00002b787589690a in gettimeofday () from /lib/libc.so.6 >#3 0x000000000045148f in TimePosix__Now () > at ../src/time/POSIX/TimePosixC.c:58 >#4 0x000000000045139e in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 >#5 0x0000000000404da7 in Main__TApply (cl=Cannot access memory at address 0x2 >b7974014e68 >) at ../src/Main.m3:205 >#6 0x000000000044b3fb in ThreadPosix__RunThread () > at ../src/thread/POSIX/ThreadPosix.m3:993 >#7 0x00002b787584b7b0 in ?? () from /lib/libc.so.6 >#8 0x0000000000000000 in ?? () >(gdb) > >AMD64_FREEBSD seems to work, oddly enough, but slowly. > >I wonder if my compiler buidling has failed somehow (I am using the >Python script following a recipe provided by Jay after modifying >m3core/src/thread/m3makefile ) > > Mika > > >Mika Nystrom writes: >> >>Hello m3devel, >> >>I'm attempting to test the CVS head as of Jay's latest checkins with >>the thread tester. >> >>I'm afraid there seems to be something wrong. >> >>I get a seg fault when running a few threads that do the following: >> >>PROCEDURE TApply(cl : Closure) : REFANY = >> BEGIN >> Thread.Pause(InitPause); >> LOOP >> TRY >> WITH now = Time.Now() DO >> times1[cl.id]:= FLOOR(now); >> IF now < 0.0d0 THEN RAISE X END >> END >> EXCEPT X => <*ASSERT FALSE*> >> END >> END >> END TApply; >> >>The crash is as follows: >> >>(gdb) run -tests tryexcept >>Starting program: /home/mika/cm3-cvs/cm3/m3-libs/m3core/tests/thread/LINUXLIB >C >>6/threadtest -tests tryexcept >>[Thread debugging using libthread_db enabled] >>Writing file...done >>Creating tryexcept threads...done >>running...printing oldest/median age/newest >>. >>Program received signal SIGSEGV, Segmentation fault. >>0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 >>53 { >>(gdb) where >>#0 0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 >>#1 0x0808cb57 in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 >>#2 0x0804b875 in Main__TApply (cl=) >> at ../src/Main.m3:205 >>#3 0x08086c5e in ThreadPosix__RunThread () >> at ../src/thread/POSIX/ThreadPosix.m3:993 >>#4 0x400c343b in makecontext () from /lib/libc.so.6 >>#5 0x00000000 in ?? () >>(gdb) >> >>Note I am using user threads since pthreads are known to be buggy at the >>moment. >> >>I would like to find one configuration (any configuration!!) that lets >>me run a multithreaded program under AMD64_LINUX. I don't care what >>version of the compiler it is, whether the threads are user or pthreads, >>how many things I have to hack. Does anyone know what I need to do? >> >> Mika From mika at async.caltech.edu Sat Feb 12 04:05:25 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 19:05:25 -0800 Subject: [M3devel] pthread_atfork Message-ID: <20110212030525.822411A2078@async.async.caltech.edu> Tony, What's RTProcessC.c doing? I note the following cryptic comment: /* NOTE: Even userthreads now depends * on availability of pthreads. * This can be fixed if need be. */ now further down we can read: #if defined(_WIN32) \ || defined(__vms) \ || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) return 0; #else while (1) { int i = pthread_atfork(prepare, parent, child); if (i != EAGAIN) return i; sleep(0); } #endif so on FreeBSD 5 (what I'm running on my "FreeBSD4" system), RTProcess.RegisterForkHandlers does nothing? Hmmm..... Mika From mika at async.caltech.edu Sat Feb 12 07:48:12 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 22:48:12 -0800 Subject: [M3devel] threading on Windows? Message-ID: <20110212064812.286E61A2078@async.async.caltech.edu> Well, with the following changes: re-bootstrapping on the two Linux systems and removal of the -pthread flag on FreeBSD4, all is well. User threading works on all the architectures listed below, at least as far as the thread testing program can test it. (It seems that because of the way pthread_atfork is coded, pthread is not needed at all on FreeBSD4.) Mika Mika Nystrom writes: >Another odd datapoint is that even rather aggressive testing seems not >to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user >threads, of course. > >So status right now of user threads seems to be: > >FreeBSD4 : strange failure in libpthread as reported by gdb >LINUXLIBC6 : seg fault >AMD64_LINUX : seg fault >AMD64_FREEBSD : OK >I386_DARWIN : OK From jay.krell at cornell.edu Sat Feb 12 11:24:01 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 10:24:01 +0000 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212030525.822411A2078@async.async.caltech.edu> References: <20110212030525.822411A2078@async.async.caltech.edu> Message-ID: I wrote that. As I recall, FreeBSD < 6 doesn't have pthread_atfork. I think that was based on reading the man pages. So, yes. What would you suggest? All this atfork stuff came about as part of getting cvsupd to work. With pthreads/kernel threads. It historically worked, with Modula-3 user threads. Modula-3 user threads have this funny property that all threads survive work. With pthreads/kernel threads, fork gives you a new process with just one thread. (Solaris has fork1 and forkall, and fork maybe used to be forkall, but now it is fork1). cvsupd is one of those slightly unusual programs that does "fork and then do more work" as supposed to the more typical "fork and exec". Which is to say..I don't know. Maybe, don't expect pthreads to work on FreeBSD < 6? At least not in a process that ever calls fork? Or does "fork and then more work"? In reality, i think a lot of libraries have trouble with "fork and then do more work", but I don't know. I recall vague notices in the Apple/Darwin manpages not to assume this works -- the obvious implication that there is plenty of use of pthreads e.g. mutexes, but relatively little use of pthread_atfork. You could perhaps replace all uses of fork with a function that called the fork handlers, then fork, then the other handlers. That is probably what I meant in the comment about this being easy to fix. You might though then run into the problem that callers of fork can't do much before exec...oh, that's vfork I bet actually. - Jay > To: hosking at cs.purdue.edu > Date: Fri, 11 Feb 2011 19:05:25 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: [M3devel] pthread_atfork > > Tony, > > What's RTProcessC.c doing? > > I note the following cryptic comment: > > /* NOTE: Even userthreads now depends > * on availability of pthreads. > * This can be fixed if need be. > */ > > now further down we can read: > > #if defined(_WIN32) \ > || defined(__vms) \ > || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > return 0; > #else > while (1) > { > int i = pthread_atfork(prepare, parent, child); > if (i != EAGAIN) > return i; > sleep(0); > } > #endif > > so on FreeBSD 5 (what I'm running on my "FreeBSD4" system), > RTProcess.RegisterForkHandlers does nothing? > > Hmmm..... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 11:39:06 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 10:39:06 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , , , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, , <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , , , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , , , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu>, , Message-ID: TextUtils.mo : error LNK2001: unresolved external symbol __alloca TextUtils.mo : error LNK2001: unresolved external symbol _Csetjmp__Jumpbuf_size sysutils.dll : fatal error LNK1120: 2 unresolved externals Most likely you didn't build it correctly. Go back to a working cm3/m3core/libm3 and run upgrade.py. There should be no longer any references to alloca of Csetjmp__Jumpbuf_size. Really, Csetjmp__Jumpbuf_size should never have been a problem. That likely also indicates some sort of incorrect bootstrap. > Are you saying you do, or do not, want me to work further on the solving the ?problem,? regardless of how we describe it ? I still want alloca to work, either via a function call or a specific m3cg method. (I still don't see that RTHooks.Alloca makes better sense; I still don't see a way to avoid hacking the compiler, given the custom calling convention.) It will be difficult to test the fix until/unless m3front is changed back. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Fri, 11 Feb 2011 18:17:15 -0500 Subject: Re: [M3devel] threading on Windows? Jay: Ok, I don?t mean to start a new issue or upset anyone. I see you checked in some new changes. I updated to latest HEAD and tried to build, but again I get unresolved symbols beginning with m3-libs\sysutils (see attached sysutils.lst) So, not sure what you mean by ?it should be ok now?. Are you saying you do, or do not, want me to work further on the solving the ?problem,? regardless of how we describe it ? Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, February 11, 2011 4:52 PM To: Tony; Coleburn, Randy Cc: m3devel Subject: RE: [M3devel] threading on Windows? It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - JayFrom: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely:1. Space leak problem.2. Binding to appropriate ?alloca? routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards,Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- TonyAntony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. Regards,Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler.On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently.3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote:What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows?Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows?I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve.All I know is that everything was working fine until I checked out the HEAD repository.When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case??Are we talking Modula-3 code, C code, Assembler, what? What package/module?Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Sat Feb 12 13:02:11 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Sat, 12 Feb 2011 13:02:11 +0100 Subject: [M3devel] pthread_atfork In-Reply-To: References: <20110212030525.822411A2078@async.async.caltech.edu> Message-ID: <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com> I think it would be acceptable to say that pthread threading is broken on FreeBSD6 (as it is not officially supported any more). All the stuff below should be documented in the 'known problems' section. Especially the different behaviours of fork for user and pthreads. This should also be mentioned in the M3 interfaces (source code)! Also that to use use threads with FreeBSD requires to remove the -pthread flag, as Mika noted. BTW, do we have a chapter explaining what to do to use user threads instead of the default pthreads? Olaf PS: Hint: this would be a great task for anybody lurking on the lists and trying to get involved :-) Quoting Jay K : > > I wrote that. > > As I recall, FreeBSD < 6 doesn't have pthread_atfork. > > I think that was based on reading the man pages. > > So, yes. What would you suggest? > > > > > > All this atfork stuff came about as part of getting cvsupd to work. > > With pthreads/kernel threads. It historically worked, with > Modula-3 user threads. > > Modula-3 user threads have this funny property that all threads > survive work. > > With pthreads/kernel threads, fork gives you a new process with > just one thread. > > (Solaris has fork1 and forkall, and fork maybe used to be > forkall, but now it is fork1). > > > > > > cvsupd is one of those slightly unusual programs that does "fork and > then do more work" > > as supposed to the more typical "fork and exec". > > > > Which is to say..I don't know. Maybe, don't expect pthreads to work > on FreeBSD < 6? > > At least not in a process that ever calls fork? Or does "fork and > then more work"? > > > > > > In reality, i think a lot of libraries have trouble with "fork and > then do more work", but > > I don't know. I recall vague notices in the Apple/Darwin manpages > not to assume > > this works -- the obvious implication that there is plenty of use of > pthreads e.g. mutexes, > > but relatively little use of pthread_atfork. > > > > > > You could perhaps replace all uses of fork with a function that > called the fork handlers, > > then fork, then the other handlers. That is probably what I meant in > the comment > > about this being easy to fix. > > You might though then run into the problem that callers of fork can't do much > > before exec...oh, that's vfork I bet actually. > > > > > > - Jay > > > > > > >> To: hosking at cs.purdue.edu >> Date: Fri, 11 Feb 2011 19:05:25 -0800 >> From: mika at async.caltech.edu >> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >> Subject: [M3devel] pthread_atfork >> >> Tony, >> >> What's RTProcessC.c doing? >> >> I note the following cryptic comment: >> >> /* NOTE: Even userthreads now depends >> * on availability of pthreads. >> * This can be fixed if need be. >> */ >> >> now further down we can read: >> >> #if defined(_WIN32) \ >> || defined(__vms) \ >> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) >> return 0; >> #else >> while (1) >> { >> int i = pthread_atfork(prepare, parent, child); >> if (i != EAGAIN) >> return i; >> sleep(0); >> } >> #endif >> >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system), >> RTProcess.RegisterForkHandlers does nothing? >> >> Hmmm..... >> >> Mika > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Sat Feb 12 14:38:33 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 13:38:33 +0000 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com> References: <20110212030525.822411A2078@async.async.caltech.edu>, , <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com> Message-ID: > Especially the different behaviours of fork for user and pthreads The behavior has been unified. User threads no longer survive fork. Except perhaps on FreeBSD 5? (Off by one somewhere in the discussion!) It looks like it is ok on FreeBSD 6. I think we could fix it on FreeBSD 5. At least if we changed all of our fork() to be RTProcess.Fork() and called the handlers in there, at least on FreeBSD 5. I believe the difference is only noticable if you fork-and-do-work. Not if you fork-and-exec. I am very tempted to fix this, so that nobody would be discussing where it did or didn't work. (Though, fork() would still vary in behavior, but we'd never use it directly..) Threads not surviving fork is new with using pthreads. Threads never survived fork on Windows. For user threads, the change was a bit late, didn't occur until we worked on getting cvsupd to work with pthreads. - Jay > Date: Sat, 12 Feb 2011 13:02:11 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] pthread_atfork > > I think it would be acceptable to say that pthread threading is > broken on FreeBSD6 (as it is not officially supported any more). > All the stuff below should be documented in the 'known problems' > section. Especially the different behaviours of fork for > user and pthreads. This should also be mentioned in the M3 > interfaces (source code)! > > Also that to use use threads with FreeBSD requires to remove the > -pthread flag, as Mika noted. > > BTW, do we have a chapter explaining what to do to use user threads > instead of the default pthreads? > > Olaf > > PS: Hint: this would be a great task for anybody lurking on the lists > and trying to get involved :-) > > Quoting Jay K : > > > > > I wrote that. > > > > As I recall, FreeBSD < 6 doesn't have pthread_atfork. > > > > I think that was based on reading the man pages. > > > > So, yes. What would you suggest? > > > > > > > > > > > > All this atfork stuff came about as part of getting cvsupd to work. > > > > With pthreads/kernel threads. It historically worked, with > > Modula-3 user threads. > > > > Modula-3 user threads have this funny property that all threads > > survive work. > > > > With pthreads/kernel threads, fork gives you a new process with > > just one thread. > > > > (Solaris has fork1 and forkall, and fork maybe used to be > > forkall, but now it is fork1). > > > > > > > > > > > > cvsupd is one of those slightly unusual programs that does "fork and > > then do more work" > > > > as supposed to the more typical "fork and exec". > > > > > > > > Which is to say..I don't know. Maybe, don't expect pthreads to work > > on FreeBSD < 6? > > > > At least not in a process that ever calls fork? Or does "fork and > > then more work"? > > > > > > > > > > > > In reality, i think a lot of libraries have trouble with "fork and > > then do more work", but > > > > I don't know. I recall vague notices in the Apple/Darwin manpages > > not to assume > > > > this works -- the obvious implication that there is plenty of use of > > pthreads e.g. mutexes, > > > > but relatively little use of pthread_atfork. > > > > > > > > > > > > You could perhaps replace all uses of fork with a function that > > called the fork handlers, > > > > then fork, then the other handlers. That is probably what I meant in > > the comment > > > > about this being easy to fix. > > > > You might though then run into the problem that callers of fork can't do much > > > > before exec...oh, that's vfork I bet actually. > > > > > > > > > > > > - Jay > > > > > > > > > > > > > >> To: hosking at cs.purdue.edu > >> Date: Fri, 11 Feb 2011 19:05:25 -0800 > >> From: mika at async.caltech.edu > >> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > >> Subject: [M3devel] pthread_atfork > >> > >> Tony, > >> > >> What's RTProcessC.c doing? > >> > >> I note the following cryptic comment: > >> > >> /* NOTE: Even userthreads now depends > >> * on availability of pthreads. > >> * This can be fixed if need be. > >> */ > >> > >> now further down we can read: > >> > >> #if defined(_WIN32) \ > >> || defined(__vms) \ > >> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > >> return 0; > >> #else > >> while (1) > >> { > >> int i = pthread_atfork(prepare, parent, child); > >> if (i != EAGAIN) > >> return i; > >> sleep(0); > >> } > >> #endif > >> > >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system), > >> RTProcess.RegisterForkHandlers does nothing? > >> > >> Hmmm..... > >> > >> Mika > > > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 15:09:59 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 14:09:59 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <20110212064812.286E61A2078@async.async.caltech.edu> References: <20110212064812.286E61A2078@async.async.caltech.edu> Message-ID: > >LINUXLIBC6 : seg fault I've seen this now. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb3200b90 (LWP 18154)] 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) at ../src/runtime/common/RTCollector.m3:409 409 IF hdr.typecode = RT0.TextLitTypecode THEN RETURN END; 1: x/i $pc 0xb6dee58f : mov (%eax),%eax (gdb) bt 9 #0 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) at ../src/runtime/common/RTCollector.m3:409 #1 0xb6deaa46 in RTHeapMap__Walk (x=0xb6a20014, pc=0xb7738648, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:202 #2 0xb6dea341 in RTHeapMap__DoWalkRef (t=0xb7738694, a=0xb6a20014, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:62 #3 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739c54, a=0xb6a2000c, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 #4 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739e68, a=0xb6a2000c, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 #5 0xb6dea2bf in RTHeapMap__WalkRef (h=0xb6a20008, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:47 #6 0xb6df0636 in RTCollector__CleanBetween (h=0xb6a20008, he=0xb6a30000, clean=0 '\0') at ../src/runtime/common/RTCollector.m3:1091 #7 0xb6df0473 in RTCollector__CleanPage (page=0xb6a20000) at ../src/runtime/common/RTCollector.m3:1064 #8 0xb6defc96 in RTCollector__CollectSomeInStateZero () at ../src/runtime/common/RTCollector.m3:885 (More stack frames follow...) (gdb) p $eax $2 = 2097148 (gdb) p/x $eax $3 = 0x1ffffc (gdb) x/x $eax+4 0x200000: Cannot access memory at address 0x200000 (gdb) x/x $eax 0x1ffffc: Cannot access memory at address 0x1ffffc Maybe I should try with a correctly sized jmpbuf to assure it isn't stack overflow? But notice that the next page doesn't work either. Actually I tried adding a lot to $eax and still not accessible. Maybe maybe maybe maybe I'll dig into this. But I'm very busy lately and maybe indefinitely into future.. Thanks, - Jay > To: jay.krell at cornell.edu; hosking at cs.purdue.edu > Date: Fri, 11 Feb 2011 22:48:12 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] threading on Windows? > > Well, with the following changes: re-bootstrapping on the two Linux > systems and removal of the -pthread flag on FreeBSD4, all is well. > User threading works on all the architectures listed below, at least > as far as the thread testing program can test it. (It seems that because > of the way pthread_atfork is coded, pthread is not needed at all on > FreeBSD4.) > > Mika > > Mika Nystrom writes: > >Another odd datapoint is that even rather aggressive testing seems not > >to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user > >threads, of course. > > > >So status right now of user threads seems to be: > > > >FreeBSD4 : strange failure in libpthread as reported by gdb > >LINUXLIBC6 : seg fault > >AMD64_LINUX : seg fault > >AMD64_FREEBSD : OK > >I386_DARWIN : OK -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 15:10:13 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 14:10:13 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110212064812.286E61A2078@async.async.caltech.edu>, Message-ID: (ps: using kernel/pthreads) From: jay.krell at cornell.edu To: mika at async.caltech.edu; hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] threading on Windows? Date: Sat, 12 Feb 2011 14:09:59 +0000 > >LINUXLIBC6 : seg fault I've seen this now. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb3200b90 (LWP 18154)] 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) at ../src/runtime/common/RTCollector.m3:409 409 IF hdr.typecode = RT0.TextLitTypecode THEN RETURN END; 1: x/i $pc 0xb6dee58f : mov (%eax),%eax (gdb) bt 9 #0 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) at ../src/runtime/common/RTCollector.m3:409 #1 0xb6deaa46 in RTHeapMap__Walk (x=0xb6a20014, pc=0xb7738648, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:202 #2 0xb6dea341 in RTHeapMap__DoWalkRef (t=0xb7738694, a=0xb6a20014, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:62 #3 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739c54, a=0xb6a2000c, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 #4 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739e68, a=0xb6a2000c, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 #5 0xb6dea2bf in RTHeapMap__WalkRef (h=0xb6a20008, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:47 #6 0xb6df0636 in RTCollector__CleanBetween (h=0xb6a20008, he=0xb6a30000, clean=0 '\0') at ../src/runtime/common/RTCollector.m3:1091 #7 0xb6df0473 in RTCollector__CleanPage (page=0xb6a20000) at ../src/runtime/common/RTCollector.m3:1064 #8 0xb6defc96 in RTCollector__CollectSomeInStateZero () at ../src/runtime/common/RTCollector.m3:885 (More stack frames follow...) (gdb) p $eax $2 = 2097148 (gdb) p/x $eax $3 = 0x1ffffc (gdb) x/x $eax+4 0x200000: Cannot access memory at address 0x200000 (gdb) x/x $eax 0x1ffffc: Cannot access memory at address 0x1ffffc Maybe I should try with a correctly sized jmpbuf to assure it isn't stack overflow? But notice that the next page doesn't work either. Actually I tried adding a lot to $eax and still not accessible. Maybe maybe maybe maybe I'll dig into this. But I'm very busy lately and maybe indefinitely into future.. Thanks, - Jay > To: jay.krell at cornell.edu; hosking at cs.purdue.edu > Date: Fri, 11 Feb 2011 22:48:12 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] threading on Windows? > > Well, with the following changes: re-bootstrapping on the two Linux > systems and removal of the -pthread flag on FreeBSD4, all is well. > User threading works on all the architectures listed below, at least > as far as the thread testing program can test it. (It seems that because > of the way pthread_atfork is coded, pthread is not needed at all on > FreeBSD4.) > > Mika > > Mika Nystrom writes: > >Another odd datapoint is that even rather aggressive testing seems not > >to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user > >threads, of course. > > > >So status right now of user threads seems to be: > > > >FreeBSD4 : strange failure in libpthread as reported by gdb > >LINUXLIBC6 : seg fault > >AMD64_LINUX : seg fault > >AMD64_FREEBSD : OK > >I386_DARWIN : OK -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 12 17:59:30 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 08:59:30 -0800 Subject: [M3devel] pthread_atfork In-Reply-To: References: <20110212030525.822411A2078@async.async.caltech.edu> Message-ID: <20110212165930.DC32C1A2078@async.async.caltech.edu> Hi Jay, I think "fork and do work" qualifies as a bug in the application, if the application wants to be portable. Why can't it fork and exec instead? If it wants to do more work it should just fork and exec another copy of itself. fork() is also not part of Modula-3... Process.Create() is, however. Don't tell me you broke that as part of "fixing" things for cvsupd... There is still one aspect of your comment I don't understand. What does the call to pthread_atfork actually accomplish when running with user threads? You say below that "don't expect pthreads to work on FreeBSD < 6 [using the fork-and-do-work pattern]". Fair enough. But why would user threads work? We're skipping pthread_atfork for them too. Do we not need pthread_atfork with user threads? What you're saying suggests that FreeBSD >= 6 and all other platforms need pthread_atfork with user threads. Is that really what you mean? At the moment I'm just concerned with getting user threads to work as I know there are bugs with pthreads. I also think we should not waste time and effort making the "fork and do work" pattern work unless it's really, really necessary (which I really don't see why it would be). Fix the application instead of introducing bugs in the libraries and language implementation! I've been trying for a long, long time to switch from PM3 to CM3 (almost ten years, I think). These bugs from "enhancements" have stopped me every time. Maybe things are finally working well enough... (some comments inline in your quoted text below) Mika Jay K writes: >--_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >I wrote that. > >As I recall=2C FreeBSD < 6 doesn't have pthread_atfork. > >I think that was based on reading the man pages. > >So=2C yes. What would you suggest? > > > > > >All this atfork stuff came about as part of getting cvsupd to work. > > With pthreads/kernel threads. It historically worked=2C with Modula-3 use= >r threads. > > Modula-3 user threads have this funny property that all threads survive w= >ork. you mean survive fork don't you? > > With pthreads/kernel threads=2C fork gives you a new process with just on= >e thread. > > (Solaris has fork1 and forkall=2C and fork maybe used to be forkall=2C b= >ut now it is fork1). > > > > > >cvsupd is one of those slightly unusual programs that does "fork and then d= >o more work" > >as supposed to the more typical "fork and exec". > > > >Which is to say..I don't know. Maybe=2C don't expect pthreads to work on Fr= >eeBSD < 6? > >At least not in a process that ever calls fork? Or does "fork and then more= > work"? > > > > > >In reality=2C i think a lot of libraries have trouble with "fork and then d= >o more work"=2C but > >I don't know. I recall vague notices in the Apple/Darwin manpages not to as= >sume > >this works -- the obvious implication that there is plenty of use of pthrea= >ds e.g. mutexes=2C > >but relatively little use of pthread_atfork. > > > > > >You could perhaps replace all uses of fork with a function that called the = >fork handlers=2C > >then fork=2C then the other handlers. That is probably what I meant in the = >comment > >about this being easy to fix. > >You might though then run into the problem that callers of fork can't do mu= >ch > >before exec...oh=2C that's vfork I bet actually. I don't understand this paragraph either :( > > > > > > - Jay > > > > > > >> To: hosking at cs.purdue.edu >> Date: Fri=2C 11 Feb 2011 19:05:25 -0800 >> From: mika at async.caltech.edu >> CC: m3devel at elegosoft.com=3B jay.krell at cornell.edu >> Subject: [M3devel] pthread_atfork >>=20 >> Tony=2C >>=20 >> What's RTProcessC.c doing? >>=20 >> I note the following cryptic comment: >>=20 >> /* NOTE: Even userthreads now depends >> * on availability of pthreads. >> * This can be fixed if need be. >> */ >>=20 >> now further down we can read: >>=20 >> #if defined(_WIN32) \ >> || defined(__vms) \ >> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) >> return 0=3B >> #else >> while (1) >> { >> int i =3D pthread_atfork(prepare=2C parent=2C child)=3B >> if (i !=3D EAGAIN) >> return i=3B >> sleep(0)=3B >> } >> #endif >>=20 >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=2C >> RTProcess.RegisterForkHandlers does nothing? >>=20 >> Hmmm..... >>=20 >> Mika > = > >--_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >I wrote that.
>As I recall=2C FreeBSD <=3B 6 doesn't have pthread_atfork.
>I think that was based on reading the man pages.
>So=2C yes. What would you suggest?
>
>
>All this atfork stuff came about as part of getting cvsupd to work.
> =3B With pthreads/kernel threads. It historically worked=2C with Modul= >a-3 user threads.
> =3B Modula-3 user threads have this funny property that all threads su= >rvive work.
> =3B With pthreads/kernel threads=2C fork gives you a new process with = >just one thread.
> =3B =3B (Solaris has fork1 and forkall=2C and fork maybe used to b= >e forkall=2C but now it is fork1).
>
>
>cvsupd is one of those slightly unusual programs that does "fork and then d= >o more work"
>as supposed to the more typical "fork and exec".
>
>Which is to say..I don't know. Maybe=2C don't expect pthreads to work on Fr= >eeBSD <=3B 6?
>At least not in a process that ever calls fork? Or does "fork and then more= > work"?
>
>
>In reality=2C i think a lot of libraries have trouble with "fork and then d= >o more work"=2C but
>I don't know. I recall vague notices in the Apple/Darwin manpages not to as= >sume
>this works -- the obvious implication that there is plenty of use of pthrea= >ds e.g. mutexes=2C
>but relatively little use of pthread_atfork.
>
>
>You could perhaps replace all uses of fork with a function that called the = >fork handlers=2C
>then fork=2C then the other handlers. That is probably what I meant in the = >comment
>about this being easy to fix.
>You might though then run into the problem that callers of fork can't do mu= >ch
>before exec...oh=2C that's vfork I bet actually.
>
>
> =3B- Jay






>=3B To: hosking at cs.purdue.edur>>=3B Date: Fri=2C 11 Feb 2011 19:05:25 -0800
>=3B From: mika at async= >.caltech.edu
>=3B CC: m3devel at elegosoft.com=3B jay.krell at cornell.edur>>=3B Subject: [M3devel] pthread_atfork
>=3B
>=3B Tony=2C
= >>=3B
>=3B What's RTProcessC.c doing?
>=3B
>=3B I note th= >e following cryptic comment:
>=3B
>=3B /* NOTE: Even userthreads= > now depends
>=3B * on availability of pthreads.
>=3B * This ca= >n be fixed if need be.
>=3B */
>=3B
>=3B now further down = >we can read:
>=3B
>=3B #if defined(_WIN32) \
>=3B |= >| defined(__vms) \
>=3B || (defined(__FreeBSD__) &=3B&= >=3B (__FreeBSD__ <=3B 6))
>=3B return 0=3B
>=3B #else
&g= >t=3B while (1)
>=3B {
>=3B int i =3D pthread_atfork= >(prepare=2C parent=2C child)=3B
>=3B if (i !=3D EAGAIN)
>= >=3B return i=3B
>=3B sleep(0)=3B
>=3B }
>= >=3B #endif
>=3B
>=3B so on FreeBSD 5 (what I'm running on my "Fr= >eeBSD4" system)=2C
>=3B RTProcess.RegisterForkHandlers does nothing?r>>=3B
>=3B Hmmm.....
>=3B
>=3B Mika
> = > >= > >--_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_-- From wagner at elegosoft.com Sat Feb 12 20:55:27 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Sat, 12 Feb 2011 20:55:27 +0100 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212165930.DC32C1A2078@async.async.caltech.edu> References: <20110212030525.822411A2078@async.async.caltech.edu> <20110212165930.DC32C1A2078@async.async.caltech.edu> Message-ID: <20110212205527.e6xetojo0ssgocg4@mail.elegosoft.com> Well, I think actually the fork-and-do-work pattern has been the default for a long time with the M3 user threads implementation. It wasn't specified though, and I think no application except CVSup did use it. Jay is not responsible for any breakage here, he just tried to fix CVSup and establish a consistent behaviour after our change to system pthreads on most platforms. Olaf Quoting Mika Nystrom : > Hi Jay, > > I think "fork and do work" qualifies as a bug in the application, if > the application wants to be portable. Why can't it fork and exec instead? > If it wants to do more work it should just fork and exec another copy > of itself. > > fork() is also not part of Modula-3... Process.Create() is, however. > Don't tell me you broke that as part of "fixing" things for cvsupd... > > There is still one aspect of your comment I don't understand. What does > the call to pthread_atfork actually accomplish when running with user > threads? You say below that "don't expect pthreads to work on FreeBSD < > 6 [using the fork-and-do-work pattern]". Fair enough. But why would > user threads work? We're skipping pthread_atfork for them too. > > Do we not need pthread_atfork with user threads? What you're saying > suggests that FreeBSD >= 6 and all other platforms need pthread_atfork > with user threads. Is that really what you mean? > > At the moment I'm just concerned with getting user threads to work as > I know there are bugs with pthreads. > > I also think we should not waste time and effort making the "fork and do > work" pattern work unless it's really, really necessary (which I really > don't see why it would be). Fix the application instead of introducing > bugs in the libraries and language implementation! > > I've been trying for a long, long time to switch from PM3 to CM3 (almost > ten years, I think). These bugs from "enhancements" have stopped me > every time. Maybe things are finally working well enough... > > (some comments inline in your quoted text below) > > Mika > > > Jay K writes: >> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >> Content-Type: text/plain; charset="iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> >> I wrote that. >> >> As I recall=2C FreeBSD < 6 doesn't have pthread_atfork. >> >> I think that was based on reading the man pages. >> >> So=2C yes. What would you suggest? >> >> >> >> >> >> All this atfork stuff came about as part of getting cvsupd to work. >> >> With pthreads/kernel threads. It historically worked=2C with Modula-3 use= >> r threads. >> >> Modula-3 user threads have this funny property that all threads survive w= >> ork. > > you mean survive fork don't you? > >> >> With pthreads/kernel threads=2C fork gives you a new process with just on= >> e thread. >> >> (Solaris has fork1 and forkall=2C and fork maybe used to be forkall=2C b= >> ut now it is fork1). >> >> >> >> >> >> cvsupd is one of those slightly unusual programs that does "fork and then d= >> o more work" >> >> as supposed to the more typical "fork and exec". >> >> >> >> Which is to say..I don't know. Maybe=2C don't expect pthreads to work on Fr= >> eeBSD < 6? >> >> At least not in a process that ever calls fork? Or does "fork and then more= >> work"? >> >> >> >> >> >> In reality=2C i think a lot of libraries have trouble with "fork and then d= >> o more work"=2C but >> >> I don't know. I recall vague notices in the Apple/Darwin manpages not to as= >> sume >> >> this works -- the obvious implication that there is plenty of use of pthrea= >> ds e.g. mutexes=2C >> >> but relatively little use of pthread_atfork. >> >> >> >> >> >> You could perhaps replace all uses of fork with a function that called the = >> fork handlers=2C >> >> then fork=2C then the other handlers. That is probably what I meant in the = >> comment >> >> about this being easy to fix. >> >> You might though then run into the problem that callers of fork can't do mu= >> ch >> >> before exec...oh=2C that's vfork I bet actually. > > I don't understand this paragraph either :( > >> >> >> >> >> >> - Jay >> >> >> >> >> >> >>> To: hosking at cs.purdue.edu >>> Date: Fri=2C 11 Feb 2011 19:05:25 -0800 >>> From: mika at async.caltech.edu >>> CC: m3devel at elegosoft.com=3B jay.krell at cornell.edu >>> Subject: [M3devel] pthread_atfork >>> =20 >>> Tony=2C >>> =20 >>> What's RTProcessC.c doing? >>> =20 >>> I note the following cryptic comment: >>> =20 >>> /* NOTE: Even userthreads now depends >>> * on availability of pthreads. >>> * This can be fixed if need be. >>> */ >>> =20 >>> now further down we can read: >>> =20 >>> #if defined(_WIN32) \ >>> || defined(__vms) \ >>> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) >>> return 0=3B >>> #else >>> while (1) >>> { >>> int i =3D pthread_atfork(prepare=2C parent=2C child)=3B >>> if (i !=3D EAGAIN) >>> return i=3B >>> sleep(0)=3B >>> } >>> #endif >>> =20 >>> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=2C >>> RTProcess.RegisterForkHandlers does nothing? >>> =20 >>> Hmmm..... >>> =20 >>> Mika >> = >> >> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >> Content-Type: text/html; charset="iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> >> >> >> >> >> I wrote that.
>> As I recall=2C FreeBSD <=3B 6 doesn't have pthread_atfork.
>> I think that was based on reading the man pages.
>> So=2C yes. What would you suggest?
>>
>>
>> All this atfork stuff came about as part of getting cvsupd to work.
>>  =3B With pthreads/kernel threads. It historically worked=2C with Modul= >> a-3 user threads.
>>  =3B Modula-3 user threads have this funny property that all threads su= >> rvive work.
>>  =3B With pthreads/kernel threads=2C fork gives you a new process with = >> just one thread.
>>  =3B =3B (Solaris has fork1 and forkall=2C and fork maybe used to b= >> e forkall=2C but now it is fork1).
>>
>>
>> cvsupd is one of those slightly unusual programs that does "fork and then d= >> o more work"
>> as supposed to the more typical "fork and exec".
>>
>> Which is to say..I don't know. Maybe=2C don't expect pthreads to work on Fr= >> eeBSD <=3B 6?
>> At least not in a process that ever calls fork? Or does "fork and then more= >> work"?
>>
>>
>> In reality=2C i think a lot of libraries have trouble with "fork and then d= >> o more work"=2C but
>> I don't know. I recall vague notices in the Apple/Darwin manpages not to as= >> sume
>> this works -- the obvious implication that there is plenty of use of pthrea= >> ds e.g. mutexes=2C
>> but relatively little use of pthread_atfork.
>>
>>
>> You could perhaps replace all uses of fork with a function that called the = >> fork handlers=2C
>> then fork=2C then the other handlers. That is probably what I meant in the = >> comment
>> about this being easy to fix.
>> You might though then run into the problem that callers of fork can't do mu= >> ch
>> before exec...oh=2C that's vfork I bet actually.
>>
>>
>>  =3B- Jay






>=3B To: hosking at cs.purdue.edu> r>>=3B Date: Fri=2C 11 Feb 2011 19:05:25 -0800
>=3B From: mika at async= >> .caltech.edu
>=3B CC: m3devel at elegosoft.com=3B jay.krell at cornell.edu> r>>=3B Subject: [M3devel] pthread_atfork
>=3B
>=3B Tony=2C
= >> >=3B
>=3B What's RTProcessC.c doing?
>=3B
>=3B I note th= >> e following cryptic comment:
>=3B
>=3B /* NOTE: Even userthreads= >> now depends
>=3B * on availability of pthreads.
>=3B * This ca= >> n be fixed if need be.
>=3B */
>=3B
>=3B now further down = >> we can read:
>=3B
>=3B #if defined(_WIN32) \
>=3B |= >> | defined(__vms) \
>=3B || (defined(__FreeBSD__) &=3B&= >> =3B (__FreeBSD__ <=3B 6))
>=3B return 0=3B
>=3B #else
&g= >> t=3B while (1)
>=3B {
>=3B int i =3D pthread_atfork= >> (prepare=2C parent=2C child)=3B
>=3B if (i !=3D EAGAIN)
>= >> =3B return i=3B
>=3B sleep(0)=3B
>=3B }
>= >> =3B #endif
>=3B
>=3B so on FreeBSD 5 (what I'm running on my "Fr= >> eeBSD4" system)=2C
>=3B RTProcess.RegisterForkHandlers does nothing?> r>>=3B
>=3B Hmmm.....
>=3B
>=3B Mika
>> = >> >> = >> >> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_-- > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Sat Feb 12 21:25:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 12:25:50 -0800 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212205527.e6xetojo0ssgocg4@mail.elegosoft.com> References: <20110212030525.822411A2078@async.async.caltech.edu> <20110212165930.DC32C1A2078@async.async.caltech.edu> <20110212205527.e6xetojo0ssgocg4@mail.elegosoft.com> Message-ID: <20110212202550.542D41A2078@async.async.caltech.edu> How do you do "fork-and-do-work" from Modula-3? All my programs that create new Unix processes do it with Process.Create, which as far as I know does a fork-and-exec... I'm not sure what if anything is broken here, either. But then again I didn't fully understand Jay's last mail on the subject. Mika Olaf Wagner writes: >Well, I think actually the fork-and-do-work pattern has been >the default for a long time with the M3 user threads implementation. >It wasn't specified though, and I think no application except >CVSup did use it. > >Jay is not responsible for any breakage here, he just tried to fix >CVSup and establish a consistent behaviour after our change to >system pthreads on most platforms. > >Olaf > >Quoting Mika Nystrom : > >> Hi Jay, >> >> I think "fork and do work" qualifies as a bug in the application, if >> the application wants to be portable. Why can't it fork and exec instead? >> If it wants to do more work it should just fork and exec another copy >> of itself. >> >> fork() is also not part of Modula-3... Process.Create() is, however. >> Don't tell me you broke that as part of "fixing" things for cvsupd... >> >> There is still one aspect of your comment I don't understand. What does >> the call to pthread_atfork actually accomplish when running with user >> threads? You say below that "don't expect pthreads to work on FreeBSD < >> 6 [using the fork-and-do-work pattern]". Fair enough. But why would >> user threads work? We're skipping pthread_atfork for them too. >> >> Do we not need pthread_atfork with user threads? What you're saying >> suggests that FreeBSD >=3D 6 and all other platforms need pthread_atfork >> with user threads. Is that really what you mean? >> >> At the moment I'm just concerned with getting user threads to work as >> I know there are bugs with pthreads. >> >> I also think we should not waste time and effort making the "fork and do >> work" pattern work unless it's really, really necessary (which I really >> don't see why it would be). Fix the application instead of introducing >> bugs in the libraries and language implementation! >> >> I've been trying for a long, long time to switch from PM3 to CM3 (almost >> ten years, I think). These bugs from "enhancements" have stopped me >> every time. Maybe things are finally working well enough... >> >> (some comments inline in your quoted text below) >> >> Mika >> >> >> Jay K writes: >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >>> Content-Type: text/plain; charset=3D"iso-8859-1" >>> Content-Transfer-Encoding: quoted-printable >>> >>> >>> I wrote that. >>> >>> As I recall=3D2C FreeBSD < 6 doesn't have pthread_atfork. >>> >>> I think that was based on reading the man pages. >>> >>> So=3D2C yes. What would you suggest? >>> >>> >>> >>> >>> >>> All this atfork stuff came about as part of getting cvsupd to work. >>> >>> With pthreads/kernel threads. It historically worked=3D2C with Modula-3 = >use=3D >>> r threads. >>> >>> Modula-3 user threads have this funny property that all threads survive = >w=3D >>> ork. >> >> you mean survive fork don't you? >> >>> >>> With pthreads/kernel threads=3D2C fork gives you a new process with just= > on=3D >>> e thread. >>> >>> (Solaris has fork1 and forkall=3D2C and fork maybe used to be forkall= >=3D2C b=3D >>> ut now it is fork1). >>> >>> >>> >>> >>> >>> cvsupd is one of those slightly unusual programs that does "fork and then= > d=3D >>> o more work" >>> >>> as supposed to the more typical "fork and exec". >>> >>> >>> >>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o= >n Fr=3D >>> eeBSD < 6? >>> >>> At least not in a process that ever calls fork? Or does "fork and then mo= >re=3D >>> work"? >>> >>> >>> >>> >>> >>> In reality=3D2C i think a lot of libraries have trouble with "fork and th= >en d=3D >>> o more work"=3D2C but >>> >>> I don't know. I recall vague notices in the Apple/Darwin manpages not to = >as=3D >>> sume >>> >>> this works -- the obvious implication that there is plenty of use of pthr= >ea=3D >>> ds e.g. mutexes=3D2C >>> >>> but relatively little use of pthread_atfork. >>> >>> >>> >>> >>> >>> You could perhaps replace all uses of fork with a function that called th= >e =3D >>> fork handlers=3D2C >>> >>> then fork=3D2C then the other handlers. That is probably what I meant in = >the =3D >>> comment >>> >>> about this being easy to fix. >>> >>> You might though then run into the problem that callers of fork can't do = >mu=3D >>> ch >>> >>> before exec...oh=3D2C that's vfork I bet actually. >> >> I don't understand this paragraph either :( >> >>> >>> >>> >>> >>> >>> - Jay >>> >>> >>> >>> >>> >>> >>>> To: hosking at cs.purdue.edu >>>> Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800 >>>> From: mika at async.caltech.edu >>>> CC: m3devel at elegosoft.com=3D3B jay.krell at cornell.edu >>>> Subject: [M3devel] pthread_atfork >>>> =3D20 >>>> Tony=3D2C >>>> =3D20 >>>> What's RTProcessC.c doing? >>>> =3D20 >>>> I note the following cryptic comment: >>>> =3D20 >>>> /* NOTE: Even userthreads now depends >>>> * on availability of pthreads. >>>> * This can be fixed if need be. >>>> */ >>>> =3D20 >>>> now further down we can read: >>>> =3D20 >>>> #if defined(_WIN32) \ >>>> || defined(__vms) \ >>>> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) >>>> return 0=3D3B >>>> #else >>>> while (1) >>>> { >>>> int i =3D3D pthread_atfork(prepare=3D2C parent=3D2C child)=3D3B >>>> if (i !=3D3D EAGAIN) >>>> return i=3D3B >>>> sleep(0)=3D3B >>>> } >>>> #endif >>>> =3D20 >>>> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=3D2C >>>> RTProcess.RegisterForkHandlers does nothing? >>>> =3D20 >>>> Hmmm..... >>>> =3D20 >>>> Mika >>> =09=09 =09 =09=09 =3D >>> >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >>> Content-Type: text/html; charset=3D"iso-8859-1" >>> Content-Transfer-Encoding: quoted-printable >>> >>> >>> >>> >>> >>> >>> I wrote that.
>>> As I recall=3D2C FreeBSD <=3D3B 6 doesn't have pthread_atfork.
>>> I think that was based on reading the man pages.
>>> So=3D2C yes. What would you suggest?
>>>
>>>
>>> All this atfork stuff came about as part of getting cvsupd to work.
>>>  =3D3B With pthreads/kernel threads. It historically worked=3D2C with= > Modul=3D >>> a-3 user threads.
>>>  =3D3B Modula-3 user threads have this funny property that all thread= >s su=3D >>> rvive work.
>>>  =3D3B With pthreads/kernel threads=3D2C fork gives you a new process= > with =3D >>> just one thread.
>>>  =3D3B =3D3B (Solaris has fork1 and forkall=3D2C and fork maybe u= >sed to b=3D >>> e forkall=3D2C but now it is fork1).
>>>
>>>
>>> cvsupd is one of those slightly unusual programs that does "fork and then= > d=3D >>> o more work"
>>> as supposed to the more typical "fork and exec".
>>>
>>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o= >n Fr=3D >>> eeBSD <=3D3B 6?
>>> At least not in a process that ever calls fork? Or does "fork and then mo= >re=3D >>> work"?
>>>
>>>
>>> In reality=3D2C i think a lot of libraries have trouble with "fork and th= >en d=3D >>> o more work"=3D2C but
>>> I don't know. I recall vague notices in the Apple/Darwin manpages not to = >as=3D >>> sume
>>> this works -- the obvious implication that there is plenty of use of pthr= >ea=3D >>> ds e.g. mutexes=3D2C
>>> but relatively little use of pthread_atfork.
>>>
>>>
>>> You could perhaps replace all uses of fork with a function that called th= >e =3D >>> fork handlers=3D2C
>>> then fork=3D2C then the other handlers. That is probably what I meant in = >the =3D >>> comment
>>> about this being easy to fix.
>>> You might though then run into the problem that callers of fork can't do = >mu=3D >>> ch
>>> before exec...oh=3D2C that's vfork I bet actually.
>>>
>>>
>>>  =3D3B- Jay






>=3D3B To: hosking at cs.purdue= >.edu>> r>>=3D3B Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800
>=3D3B From: mi= >ka at async=3D >>> .caltech.edu
>=3D3B CC: m3devel at elegosoft.com=3D3B jay.krell at cornell= >.edu>> r>>=3D3B Subject: [M3devel] pthread_atfork
>=3D3B
>=3D3B Ton= >y=3D2C
=3D >>> >=3D3B
>=3D3B What's RTProcessC.c doing?
>=3D3B
>=3D3B= > I note th=3D >>> e following cryptic comment:
>=3D3B
>=3D3B /* NOTE: Even usert= >hreads=3D >>> now depends
>=3D3B * on availability of pthreads.
>=3D3B * Th= >is ca=3D >>> n be fixed if need be.
>=3D3B */
>=3D3B
>=3D3B now furth= >er down =3D >>> we can read:
>=3D3B
>=3D3B #if defined(_WIN32) \
>=3D3B = > |=3D >>> | defined(__vms) \
>=3D3B || (defined(__FreeBSD__) &=3D3B= >&=3D >>> =3D3B (__FreeBSD__ <=3D3B 6))
>=3D3B return 0=3D3B
>=3D3B= > #else
&g=3D >>> t=3D3B while (1)
>=3D3B {
>=3D3B int i =3D3D pthr= >ead_atfork=3D >>> (prepare=3D2C parent=3D2C child)=3D3B
>=3D3B if (i !=3D3D EAGA= >IN)
>=3D >>> =3D3B return i=3D3B
>=3D3B sleep(0)=3D3B
>=3D3B = > }
>=3D >>> =3D3B #endif
>=3D3B
>=3D3B so on FreeBSD 5 (what I'm running o= >n my "Fr=3D >>> eeBSD4" system)=3D2C
>=3D3B RTProcess.RegisterForkHandlers does noth= >ing?>> r>>=3D3B
>=3D3B Hmmm.....
>=3D3B
>=3D3B Mika
>>> =09 =09 =09=09 =3D >>> >>> =3D >>> >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_-- >> > > > >--=20 >Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa= >ny >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz= >: Berlin >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Sat Feb 12 21:43:02 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 12:43:02 -0800 Subject: [M3devel] Performance issues with Process.Wait under userthreads Message-ID: <20110212204302.19C781A2078@async.async.caltech.edu> Hi again m3devel (especially Tony), I have finally taken a bite of a problem that has been annoying me for a very, very long time. Under user threads, the code for Process.Wait is as follows (see ThreadPosix.m3): PROCEDURE WaitProcess (pid: int; VAR status: int): int = (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) CONST Delay = 0.1D0; BEGIN LOOP WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO IF r # 0 THEN RETURN r END; END; Pause(Delay); END; END WaitProcess; It inserts a 0.1 second delay after each failed waitpid. This is extremely annoying for programs that start a long sequence of child processes and wait for them in sequence. Namely, the compiler itself. As a result the cm3 compiler (and PM3's m3build) are normally very very slow when using user threads. For about the last ten years, I've had a hacked up m3build (for my PM3 installation) that skips the Pause and busy-waits instead. Note there is another problem here. Since the Modula-3 runtime ignores SIGCHLD, no zombie processes are created since the Unix system automatically reaps the child processes. I can see this would be a problem since PIDs are eventually reused and ... couldn't Uexec.waitpid wind up referring to the wrong process?? I will further note that the comment in Process.i3 reads as follows: PROCEDURE Wait(p: T): ExitCode; Wait until the process with handle p terminates, then free the operating system resources associated with the process and return an exit code indicating the reason for its termination. It is a checked runtime error to call Wait twice on the same process handle. I am going to take this as fair warning that Process.Create *may* use resources that are not going to be released until Process.Wait has been called. I have modified (in my local copy of CM3) the system as follows. I have come up with a semi-general mechanism for immediately unblocking a thread on the receipt of a unix signal. 1. The system relies on changing ThreadPosix.XPause such that if a signal is allowed to wake up a threads, that fact is recorded in a new field in the thread's descriptor record (of type ThreadPosix.T). 2. On receipt of a waited-for unix signal, a mask is set and control is passed to the thread scheduler which maintains the non-zero mask for exactly one iteration through the thread ring. 3. If a thread is paused and waiting for EITHER a signal or some time, the thread is released for running and the thread's waiting state is cleared. The changes are more or less as follows: 1. I have added a new field of type "int" to ThreadPosix.T: (* if state = pausing, the time at which we can restart *) waitingForTime: Time.T; + (* if state = pausing, the signal that truncates the pause *) + waitingForSig: int := -1; + (* true if we are waiting during an AlertWait or AlertJoin or AlertPause *) alertable: BOOLEAN := FALSE; 2. Modifications to pause: + PROCEDURE SigPause(n: LONGREAL; sig: int)= + <*FATAL Alerted*> + VAR until := n + Time.Now (); + BEGIN + XPause(until, FALSE, sig); + END SigPause; + PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= VAR until := n + Time.Now (); BEGIN XPause(until, TRUE); END AlertPause; ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) ! RAISES {Alerted} = BEGIN INC (inCritical); self.waitingForTime := until; self.alertable := alertable; + IF sig # -1 THEN + self.waitingForSig := sig + END; ICannotRun (State.pausing); DEC (inCritical); InternalYield (); 3. The received-signals mask: ! CONST MaxSigs = 64; ! TYPE Sig = [ 0..MaxSigs-1 ]; ! ! (* in order to listen to other signals, they have to be enabled in ! allow_sigvtalrm as well *) ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); ! ! gotSigs := SET OF Sig { }; ! ValueOfSIGCHLD() is a C function used to get the value of the SIGCHLD constant without guessing at it (in ThreadPosixC.c). 4. changes to the signal handler: ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); ! ! INC(inCritical); ! (* mark signal as being delivered *) ! IF sig >= 0 AND sig < MaxSigs THEN ! gotSigs := gotSigs + SET OF Sig { sig } ! END; ! DEC(inCritical); ! ! IF inCritical = 0 AND heapState.inCritical = 0 THEN ! InternalYield () ! END; END switch_thread; Note that I don't know if INC/DEC(inCritical) does exactly the right thing here. 5. changes to the scheduler: a. thread wakeup IF t.alertable AND t.alertPending THEN CanRun (t); EXIT; + + ELSIF t.waitingForSig IN gotSigs THEN + t.waitingForSig := -1; + CanRun(t); + EXIT; ELSIF t.waitingForTime <= now THEN CanRun (t); EXIT; ! b. clearing the mask END; END; + gotSigs := SET OF Sig {}; + IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN IF perfOn THEN PerfRunning (t.id); END; (* At least one thread wants to run; transfer to it *) 6. changes to WaitProcess (Process.Wait): PROCEDURE WaitProcess (pid: int; VAR status: int): int = (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) ! CONST Delay = 10.0D0; BEGIN LOOP WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO IF r # 0 THEN RETURN r END; END; ! SigPause(Delay,SIGCHLD); END; END WaitProcess; 7. install signal handler even if program is single-threaded: BEGIN + (* we need to call set up the signal handler for other reasons than + just thread switching now *) + setup_sigvtalrm (switch_thread); END ThreadPosix. 8. modify signal handler in ThreadPosixC.c to catch SIGCHLD: sigemptyset(&ThreadSwitchSignal); sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); + sigaddset(&ThreadSwitchSignal, SIGCHLD); act.sa_handler = handler; act.sa_flags = SA_RESTART; sigemptyset(&(act.sa_mask)); if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); + if (sigaction (SIGCHLD, &act, NULL)) abort(); I'll send the complete diff in a separate message for those who want to study it more closely. I propose the above changes for inclusion in the current CM3 repository. Mika From mika at async.caltech.edu Sat Feb 12 21:44:06 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 12:44:06 -0800 Subject: [M3devel] SIGCHLD diff Message-ID: <20110212204406.88A661A2078@async.async.caltech.edu> Index: ThreadPosix.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v retrieving revision 1.16 diff -c -r1.16 ThreadPosix.i3 *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 *************** *** 39,42 **** --- 39,45 ---- (*---------------------------------------------------------------------------*) + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> + PROCEDURE ValueOfSIGCHLD(): int; + END ThreadPosix. Index: ThreadPosix.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v retrieving revision 1.73 diff -c -r1.73 ThreadPosix.m3 *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 *************** *** 97,102 **** --- 97,105 ---- (* if state = pausing, the time at which we can restart *) waitingForTime: Time.T; + (* if state = pausing, the signal that truncates the pause *) + waitingForSig: int := -1; + (* true if we are waiting during an AlertWait or AlertJoin or AlertPause *) alertable: BOOLEAN := FALSE; *************** *** 147,152 **** --- 150,160 ---- defaultStackSize := 3000; + (* note that even though the "heavy machinery" is only used for + multipleThreads, we still need to set up the signal handler so + that we can catch signals from other sources than thread switching. + e.g., we use SIGCHLD to speed up Process.Wait *) + VAR stats: RECORD n_forks := 0; *************** *** 511,527 **** XPause(until, FALSE); END Pause; PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= VAR until := n + Time.Now (); BEGIN XPause(until, TRUE); END AlertPause; ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE) RAISES {Alerted} = BEGIN INC (inCritical); self.waitingForTime := until; self.alertable := alertable; ICannotRun (State.pausing); DEC (inCritical); InternalYield (); --- 519,546 ---- XPause(until, FALSE); END Pause; + PROCEDURE SigPause(n: LONGREAL; sig: int)= + <*FATAL Alerted*> + VAR until := n + Time.Now (); + BEGIN + XPause(until, FALSE, sig); + END SigPause; + PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= VAR until := n + Time.Now (); BEGIN XPause(until, TRUE); END AlertPause; ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) ! RAISES {Alerted} = BEGIN INC (inCritical); self.waitingForTime := until; self.alertable := alertable; + IF sig # -1 THEN + self.waitingForSig := sig + END; ICannotRun (State.pausing); DEC (inCritical); InternalYield (); *************** *** 703,712 **** END; END StartSwitching; ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); ! IF inCritical = 0 AND heapState.inCritical = 0 THEN InternalYield () END; END switch_thread; (*------------------------------------------------------------- scheduler ---*) --- 722,750 ---- END; END StartSwitching; ! CONST MaxSigs = 64; ! TYPE Sig = [ 0..MaxSigs-1 ]; ! ! (* in order to listen to other signals, they have to be enabled in ! allow_sigvtalrm as well *) ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); ! ! gotSigs := SET OF Sig { }; ! ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); ! ! INC(inCritical); ! (* mark signal as being delivered *) ! IF sig >= 0 AND sig < MaxSigs THEN ! gotSigs := gotSigs + SET OF Sig { sig } ! END; ! DEC(inCritical); ! ! IF inCritical = 0 AND heapState.inCritical = 0 THEN ! InternalYield () ! END; END switch_thread; (*------------------------------------------------------------- scheduler ---*) *************** *** 782,792 **** IF t.alertable AND t.alertPending THEN CanRun (t); EXIT; ELSIF t.waitingForTime <= now THEN CanRun (t); EXIT; ! ELSIF NOT somePausing THEN earliest := t.waitingForTime; somePausing := TRUE; --- 820,835 ---- IF t.alertable AND t.alertPending THEN CanRun (t); EXIT; + + ELSIF t.waitingForSig IN gotSigs THEN + t.waitingForSig := -1; + CanRun(t); + EXIT; ELSIF t.waitingForTime <= now THEN CanRun (t); EXIT; ! ELSIF NOT somePausing THEN earliest := t.waitingForTime; somePausing := TRUE; *************** *** 886,891 **** --- 929,936 ---- END; END; + gotSigs := SET OF Sig {}; + IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN IF perfOn THEN PerfRunning (t.id); END; (* At least one thread wants to run; transfer to it *) *************** *** 948,960 **** PROCEDURE WaitProcess (pid: int; VAR status: int): int = (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) ! CONST Delay = 0.1D0; BEGIN LOOP WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO IF r # 0 THEN RETURN r END; END; ! Pause(Delay); END; END WaitProcess; --- 993,1005 ---- PROCEDURE WaitProcess (pid: int; VAR status: int): int = (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) ! CONST Delay = 10.0D0; BEGIN LOOP WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO IF r # 0 THEN RETURN r END; END; ! SigPause(Delay,SIGCHLD); END; END WaitProcess; *************** *** 1361,1364 **** --- 1406,1412 ---- VAR debug := RTParams.IsPresent ("debugthreads"); BEGIN + (* we need to call set up the signal handler for other reasons than + just thread switching now *) + setup_sigvtalrm (switch_thread); END ThreadPosix. Index: ThreadPosixC.c =================================================================== RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v retrieving revision 1.59 diff -c -r1.59 ThreadPosixC.c *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 *************** *** 78,88 **** --- 78,97 ---- sigemptyset(&ThreadSwitchSignal); sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); + sigaddset(&ThreadSwitchSignal, SIGCHLD); act.sa_handler = handler; act.sa_flags = SA_RESTART; sigemptyset(&(act.sa_mask)); if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); + if (sigaction (SIGCHLD, &act, NULL)) abort(); + } + + int + __cdecl + ThreadPosix__value_of_SIGCHLD(void) + { + return SIGCHLD; } void From dabenavidesd at yahoo.es Sat Feb 12 22:08:13 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sat, 12 Feb 2011 21:08:13 +0000 (GMT) Subject: [M3devel] pthread_atfork In-Reply-To: <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com> Message-ID: <651549.53394.qm@web29714.mail.ird.yahoo.com> Hi all: by the way you told, do user space interfaces need to be linked differently from user space threads library than the one in kernel threads, so the behaviour can be selected better at runtime if more than one available processor (i.e a virtualized one, perhaps two if there are virtualized GPU, GPGPU, etc) is available on demand (in example the collector has its own user space thread allocated control and one in kernel space that user can control from the Runtime), could it work in such a way to allow a Thread to live in spaces that are accesible through like an NetObj.T, thus allowing it to communciate them through such an abstraction? Perhaps this could allow to run the kernel threads with Modula-3 governance and if so put them to work or switch to kernel them to accelerate its process according to its demands and such debug them in a special API like the one in the SPIN OS, fairly more comprenhensive thread implementation of Mach, Cthreads (aka C Pthreads calls) and own strends. Also scheduling could be handled to manage both user/kernel level synchronization, the threads and also allow to change from having a specialized hardware-assisted virtualized processor to a non hardware-assissted virtualized processor (like an NetObj.T again), which is like the same situation for Modula-3 with just one processor so in case it needs, it does so. In any case thread and object migration are safely defined over Obliq. Granted, this is same situation like was before the enter of virtualization hardware but again the whole idea of hardware-assisted virtualization was not new, even there is an article in which a PDP-10 is claimed has requirements to make it virtual in hardware (and I believe in software too). So SPIN could be a platform, like for hardware assissted virtualization and/or operating-system level virtualization, if so, we would be able to trace system calls for a given OS client running on it, rather than barely, so it makes the implementation with hardware-assisted source-level debugger ready, which is related to where programs are segfaulting now (no system-level breakpoints, that is, no hardware assisted source-level debugger for virtualization user, different from Modula-3 situation in current user-level space threading or SPIN gdb DEC TTD Topaz Tele Debugger at source-level service and not to forget the ldb cross-platform source-level remote debugger) in pthread current implementations, likewise the whole idea of customizable scheduler and kernel allocator weren't and aren't new nor bad at all, neither for us nor were for SPINers; remember the CSA Occam2 TRAC sourced OS SuperSetPlus machine which had from 16 up to 256 individual processors independently executable as according to their product info add it could be connected over LAN up to 16 Mac, PC, and Sun workstations, pretty much like a compilation and execution engine in one, pretty and good mussle to have to see running. They offered besides Occam2, C, C++, Modula-2, Pascal and FORTRAN, prolog and Ada with source level debuggers and Operating Systems as their Software products too. I hope this would be an scenario in some years to come, at least for any of us, if solet's see where and what this takes to/from us. The main question will be whether this guys will allow enogh detail to drive their GPGPU or CPU, I hope this history doesn't repeat like it is and wasn't in the past of machine OSes sources, i.e braking traditional development etc. I'm sorry for the long of the message. Thanks in advance --- El s?b, 12/2/11, Olaf Wagner escribi?: > De: Olaf Wagner > Asunto: Re: [M3devel] pthread_atfork > Para: m3devel at elegosoft.com > Fecha: s?bado, 12 de febrero, 2011 07:02 > I think it would be acceptable to say > that pthread threading is > broken on FreeBSD6 (as it is not officially supported any > more). > All the stuff below should be documented in the 'known > problems' > section. Especially the different behaviours of fork for > user and pthreads. This should also be mentioned in the M3 > interfaces (source code)! > > Also that to use use threads with FreeBSD requires to > remove the > -pthread flag, as Mika noted. > > BTW, do we have a chapter explaining what to do to use user > threads > instead of the default pthreads? > > Olaf > > PS: Hint: this would be a great task for anybody lurking on > the lists > and trying to get involved :-) > > Quoting Jay K : > > > > > I wrote that. > > > > As I recall, FreeBSD < 6 doesn't have > pthread_atfork. > > > > I think that was based on reading the man pages. > > > > So, yes. What would you suggest? > > > > > > > > > > > > All this atfork stuff came about as part of getting > cvsupd to work. > > > > With pthreads/kernel threads. It > historically worked, with > > Modula-3 user threads. > > > > Modula-3 user threads have this funny > property that all threads > > survive work. > > > > With pthreads/kernel threads, fork > gives you a new process with > > just one thread. > > > > (Solaris has fork1 and forkall, and fork > maybe used to be > > forkall, but now it is fork1). > > > > > > > > > > > > cvsupd is one of those slightly unusual programs that > does "fork and > > then do more work" > > > > as supposed to the more typical "fork and exec". > > > > > > > > Which is to say..I don't know. Maybe, don't expect > pthreads to work > > on FreeBSD < 6? > > > > At least not in a process that ever calls fork? Or > does "fork and > > then more work"? > > > > > > > > > > > > In reality, i think a lot of libraries have trouble > with "fork and > > then do more work", but > > > > I don't know. I recall vague notices in the > Apple/Darwin manpages > > not to assume > > > > this works -- the obvious implication that there is > plenty of use of > > pthreads e.g. mutexes, > > > > but relatively little use of pthread_atfork. > > > > > > > > > > > > You could perhaps replace all uses of fork with a > function that > > called the fork handlers, > > > > then fork, then the other handlers. That is probably > what I meant in > > the comment > > > > about this being easy to fix. > > > > You might though then run into the problem that > callers of fork can't do much > > > > before exec...oh, that's vfork I bet actually. > > > > > > > > > > > > - Jay > > > > > > > > > > > > > >> To: hosking at cs.purdue.edu > >> Date: Fri, 11 Feb 2011 19:05:25 -0800 > >> From: mika at async.caltech.edu > >> CC: m3devel at elegosoft.com; > jay.krell at cornell.edu > >> Subject: [M3devel] pthread_atfork > >> > >> Tony, > >> > >> What's RTProcessC.c doing? > >> > >> I note the following cryptic comment: > >> > >> /* NOTE: Even userthreads now depends > >> * on availability of pthreads. > >> * This can be fixed if need be. > >> */ > >> > >> now further down we can read: > >> > >> #if defined(_WIN32) \ > >> || > defined(__vms) \ > >> || > (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > >> return 0; > >> #else > >> while (1) > >> { > >> int i = > pthread_atfork(prepare, parent, child); > >> if (i != EAGAIN) > >> return i; > >> sleep(0); > >> } > >> #endif > >> > >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" > system), > >> RTProcess.RegisterForkHandlers does nothing? > >> > >> Hmmm..... > >> > >> Mika > > > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 > 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | > Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | > USt-IdNr: DE163214194 > > From jay.krell at cornell.edu Sat Feb 12 22:45:18 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 21:45:18 +0000 Subject: [M3devel] pthread_atfork In-Reply-To: <651549.53394.qm@web29714.mail.ird.yahoo.com> References: <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com>, <651549.53394.qm@web29714.mail.ird.yahoo.com> Message-ID: > so the behaviour can be selected better at runtime if more than one available processor I would like user vs. kernel threads to be selectable at link and/or run-time, but Tony has nixed the small changes required to make it so. It has also been suggested you could copy around different m3core.so files to switch, but that isn't the case either. Thread.T has a different typeid in the two systems. Anyway, the better behavior is almost always kernel threads. You should really just generally stick with them. You do forever ongoing debate on these matters, and the "pendulum" has seemingly swung many times, so I can't really say everyone agrees in kernel threads always. Windows 7 adds "user mode scheduling", which is still kernel threads -- there is still a one-to-one mapping between some usermode state and kernelmode state, but the scheduling decisions are made in usermode. I believe pthreads on OpenVMS use the "m:n" approach. But Linux, Solaris by default, Darwin, I suspect but am not sure FreeBSD and NetBSD, Win32 all use the simple "1:1" model. Where every thread is a kernel thread. All scheduling is done by the kernel. Win32 also has "fibers" but these are fairly...surprisingly difficult to use, but in a way that Tony et. al. wouldn't be surprised by. When you use fibers, you have to write your own scheduler, and your own locking primitives. Normal Win32 functions like GetCurrentThreadId(), EnterCriticalSection() become sort of useless/broken in the face of fibers. Basically the same as what it takes to implement Modula-3 user threads. Though you can run on multiple processors I believe fairly easily. Hm. The "m:n" model is actually not difficult to implement on Win32 and w/ pthreads. But still, probably not a good idea. - Jay > Date: Sat, 12 Feb 2011 21:08:13 +0000 > From: dabenavidesd at yahoo.es > To: m3devel at elegosoft.com; wagner at elegosoft.com > Subject: Re: [M3devel] pthread_atfork > > Hi all: > by the way you told, do user space interfaces need to be linked differently from user space threads library than the one in kernel threads, so the behaviour can be selected better at runtime if more than one available processor (i.e a virtualized one, perhaps two if there are virtualized GPU, GPGPU, etc) is available on demand (in example the collector has its own user space thread allocated control and one in kernel space that user can control from the Runtime), could it work in such a way to allow a Thread to live in spaces that are accesible through like an NetObj.T, thus allowing it to communciate them through such an abstraction? > Perhaps this could allow to run the kernel threads with Modula-3 governance and if so put them to work or switch to kernel them to accelerate its process according to its demands and such debug them in a special API like the one in the SPIN OS, fairly more comprenhensive thread implementation of Mach, Cthreads (aka C Pthreads calls) and own strends. Also scheduling could be handled to manage both user/kernel level synchronization, the threads and also allow to change from having a specialized hardware-assisted virtualized processor to a non hardware-assissted virtualized processor (like an NetObj.T again), which is like the same situation for Modula-3 with just one processor so in case it needs, it does so. In any case thread and object migration are safely defined over Obliq. > Granted, this is same situation like was before the enter of virtualization hardware but again the whole idea of hardware-assisted virtualization was not new, even there is an article in which a PDP-10 is claimed has requirements to make it virtual in hardware (and I believe in software too). So SPIN could be a platform, like for hardware assissted virtualization and/or operating-system level virtualization, if so, we would be able to trace system calls for a given OS client running on it, rather than barely, so it makes the implementation with hardware-assisted source-level debugger ready, which is related to where programs are segfaulting now (no system-level breakpoints, that is, no hardware assisted source-level debugger for virtualization user, different from Modula-3 situation in current user-level space threading or SPIN gdb DEC TTD Topaz Tele Debugger at source-level service and not to forget the ldb cross-platform source-level remote debugger) > in pthread current implementations, likewise the whole idea of customizable scheduler and kernel allocator weren't and aren't new nor bad at all, neither for us nor were for SPINers; remember the CSA Occam2 TRAC sourced OS SuperSetPlus machine which had from 16 up to 256 individual processors independently executable as according to their product info add it could be connected over LAN up to 16 Mac, PC, and Sun workstations, pretty much like a compilation and execution engine in one, pretty and good mussle to have to see running. They offered besides Occam2, C, C++, Modula-2, Pascal and FORTRAN, prolog and Ada with source level debuggers and Operating Systems as their Software products too. I hope this would be an scenario in some years to come, at least for any of us, if solet's see where and what this takes to/from us. > The main question will be whether this guys will allow enogh detail to drive their GPGPU or CPU, I hope this history doesn't repeat like it is and wasn't in the past of machine OSes sources, i.e braking traditional development etc. > I'm sorry for the long of the message. > Thanks in advance > > --- El s?b, 12/2/11, Olaf Wagner escribi?: > > > De: Olaf Wagner > > Asunto: Re: [M3devel] pthread_atfork > > Para: m3devel at elegosoft.com > > Fecha: s?bado, 12 de febrero, 2011 07:02 > > I think it would be acceptable to say > > that pthread threading is > > broken on FreeBSD6 (as it is not officially supported any > > more). > > All the stuff below should be documented in the 'known > > problems' > > section. Especially the different behaviours of fork for > > user and pthreads. This should also be mentioned in the M3 > > interfaces (source code)! > > > > Also that to use use threads with FreeBSD requires to > > remove the > > -pthread flag, as Mika noted. > > > > BTW, do we have a chapter explaining what to do to use user > > threads > > instead of the default pthreads? > > > > Olaf > > > > PS: Hint: this would be a great task for anybody lurking on > > the lists > > and trying to get involved :-) > > > > Quoting Jay K : > > > > > > > > I wrote that. > > > > > > As I recall, FreeBSD < 6 doesn't have > > pthread_atfork. > > > > > > I think that was based on reading the man pages. > > > > > > So, yes. What would you suggest? > > > > > > > > > > > > > > > > > > All this atfork stuff came about as part of getting > > cvsupd to work. > > > > > > With pthreads/kernel threads. It > > historically worked, with > > > Modula-3 user threads. > > > > > > Modula-3 user threads have this funny > > property that all threads > > > survive work. > > > > > > With pthreads/kernel threads, fork > > gives you a new process with > > > just one thread. > > > > > > (Solaris has fork1 and forkall, and fork > > maybe used to be > > > forkall, but now it is fork1). > > > > > > > > > > > > > > > > > > cvsupd is one of those slightly unusual programs that > > does "fork and > > > then do more work" > > > > > > as supposed to the more typical "fork and exec". > > > > > > > > > > > > Which is to say..I don't know. Maybe, don't expect > > pthreads to work > > > on FreeBSD < 6? > > > > > > At least not in a process that ever calls fork? Or > > does "fork and > > > then more work"? > > > > > > > > > > > > > > > > > > In reality, i think a lot of libraries have trouble > > with "fork and > > > then do more work", but > > > > > > I don't know. I recall vague notices in the > > Apple/Darwin manpages > > > not to assume > > > > > > this works -- the obvious implication that there is > > plenty of use of > > > pthreads e.g. mutexes, > > > > > > but relatively little use of pthread_atfork. > > > > > > > > > > > > > > > > > > You could perhaps replace all uses of fork with a > > function that > > > called the fork handlers, > > > > > > then fork, then the other handlers. That is probably > > what I meant in > > > the comment > > > > > > about this being easy to fix. > > > > > > You might though then run into the problem that > > callers of fork can't do much > > > > > > before exec...oh, that's vfork I bet actually. > > > > > > > > > > > > > > > > > > - Jay > > > > > > > > > > > > > > > > > > > > >> To: hosking at cs.purdue.edu > > >> Date: Fri, 11 Feb 2011 19:05:25 -0800 > > >> From: mika at async.caltech.edu > > >> CC: m3devel at elegosoft.com; > > jay.krell at cornell.edu > > >> Subject: [M3devel] pthread_atfork > > >> > > >> Tony, > > >> > > >> What's RTProcessC.c doing? > > >> > > >> I note the following cryptic comment: > > >> > > >> /* NOTE: Even userthreads now depends > > >> * on availability of pthreads. > > >> * This can be fixed if need be. > > >> */ > > >> > > >> now further down we can read: > > >> > > >> #if defined(_WIN32) \ > > >> || > > defined(__vms) \ > > >> || > > (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > > >> return 0; > > >> #else > > >> while (1) > > >> { > > >> int i = > > pthread_atfork(prepare, parent, child); > > >> if (i != EAGAIN) > > >> return i; > > >> sleep(0); > > >> } > > >> #endif > > >> > > >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" > > system), > > >> RTProcess.RegisterForkHandlers does nothing? > > >> > > >> Hmmm..... > > >> > > >> Mika > > > > > > > > > > > -- > > Olaf Wagner -- elego Software Solutions GmbH > > > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > > phone: +49 30 23 45 86 96 mobile: +49 177 2345 > > 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | > > Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > > Handelregister: Amtsgericht Charlottenburg HRB 77719 | > > USt-IdNr: DE163214194 > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 22:50:28 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 21:50:28 +0000 Subject: [M3devel] Performance issues with Process.Wait under userthreads In-Reply-To: <20110212204302.19C781A2078@async.async.caltech.edu> References: <20110212204302.19C781A2078@async.async.caltech.edu> Message-ID: I don't believe there is a good solution here, other than using pthreads. Is there? Maybe tune the wait down smaller? I understand it stinks either way. I don't understand the SIGCHLD stuff, at a quick glance. Nor your changes. However you might be the only user of user threads. Is there any downside to your change? (Other than more code, ok.) Any loss of performance in any situation? Any loss of portability? - Jay > To: m3devel at elegosoft.com > Date: Sat, 12 Feb 2011 12:43:02 -0800 > From: mika at async.caltech.edu > Subject: [M3devel] Performance issues with Process.Wait under userthreads > > Hi again m3devel (especially Tony), > > I have finally taken a bite of a problem that has been annoying me > for a very, very long time. > > Under user threads, the code for Process.Wait is as follows (see > ThreadPosix.m3): > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > CONST Delay = 0.1D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > Pause(Delay); > END; > END WaitProcess; > > It inserts a 0.1 second delay after each failed waitpid. This is extremely > annoying for programs that start a long sequence of child processes and > wait for them in sequence. Namely, the compiler itself. As a result > the cm3 compiler (and PM3's m3build) are normally very very slow when > using user threads. For about the last ten years, I've had a hacked > up m3build (for my PM3 installation) that skips the Pause and busy-waits > instead. > > Note there is another problem here. Since the Modula-3 runtime ignores > SIGCHLD, no zombie processes are created since the Unix system automatically > reaps the child processes. I can see this would be a problem since PIDs > are eventually reused and ... couldn't Uexec.waitpid wind up referring to > the wrong process?? > > I will further note that the comment in Process.i3 reads as follows: > > PROCEDURE Wait(p: T): ExitCode; > > Wait until the process with handle p terminates, then free the operating system resources associated with the process and return an exit code indicating the reason for its termination. It is a checked runtime error to call Wait twice on the same process handle. > > I am going to take this as fair warning that Process.Create *may* use > resources that are not going to be released until Process.Wait has been > called. > > I have modified (in my local copy of CM3) the system as follows. > I have come up with a semi-general mechanism for immediately unblocking > a thread on the receipt of a unix signal. > > 1. The system relies on changing > ThreadPosix.XPause such that if a signal is allowed to wake up a threads, > that fact is recorded in a new field in the thread's descriptor > record (of type ThreadPosix.T). > > 2. On receipt of a waited-for unix signal, a mask is set and control > is passed to the thread scheduler which maintains the non-zero mask for > exactly one iteration through the thread ring. > > 3. If a thread is paused and waiting for EITHER a signal or some time, > the thread is released for running and the thread's waiting state is > cleared. > > The changes are more or less as follows: > > 1. I have added a new field of type "int" to ThreadPosix.T: > > (* if state = pausing, the time at which we can restart *) > waitingForTime: Time.T; > > + (* if state = pausing, the signal that truncates the pause *) > + waitingForSig: int := -1; > + > (* true if we are waiting during an AlertWait or AlertJoin > or AlertPause *) > alertable: BOOLEAN := FALSE; > > > 2. Modifications to pause: > > + PROCEDURE SigPause(n: LONGREAL; sig: int)= > + <*FATAL Alerted*> > + VAR until := n + Time.Now (); > + BEGIN > + XPause(until, FALSE, sig); > + END SigPause; > + > PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= > VAR until := n + Time.Now (); > BEGIN > XPause(until, TRUE); > END AlertPause; > > ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) > ! RAISES {Alerted} = > BEGIN > INC (inCritical); > self.waitingForTime := until; > self.alertable := alertable; > + IF sig # -1 THEN > + self.waitingForSig := sig > + END; > ICannotRun (State.pausing); > DEC (inCritical); > InternalYield (); > > 3. The received-signals mask: > > ! CONST MaxSigs = 64; > ! TYPE Sig = [ 0..MaxSigs-1 ]; > ! > ! (* in order to listen to other signals, they have to be enabled in > ! allow_sigvtalrm as well *) > ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); > ! > ! gotSigs := SET OF Sig { }; > ! > > ValueOfSIGCHLD() is a C function used to get the value of the SIGCHLD > constant without guessing at it (in ThreadPosixC.c). > > 4. changes to the signal handler: > > ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); > ! > ! INC(inCritical); > ! (* mark signal as being delivered *) > ! IF sig >= 0 AND sig < MaxSigs THEN > ! gotSigs := gotSigs + SET OF Sig { sig } > ! END; > ! DEC(inCritical); > ! > ! IF inCritical = 0 AND heapState.inCritical = 0 THEN > ! InternalYield () > ! END; > END switch_thread; > > Note that I don't know if INC/DEC(inCritical) does exactly the right > thing here. > > 5. changes to the scheduler: > > a. thread wakeup > IF t.alertable AND t.alertPending THEN > CanRun (t); > EXIT; > + > + ELSIF t.waitingForSig IN gotSigs THEN > + t.waitingForSig := -1; > + CanRun(t); > + EXIT; > > ELSIF t.waitingForTime <= now THEN > CanRun (t); > EXIT; > ! > > b. clearing the mask > > END; > END; > > + gotSigs := SET OF Sig {}; > + > IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN > IF perfOn THEN PerfRunning (t.id); END; > (* At least one thread wants to run; transfer to it *) > > 6. changes to WaitProcess (Process.Wait): > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > ! CONST Delay = 10.0D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > ! SigPause(Delay,SIGCHLD); > END; > END WaitProcess; > > 7. install signal handler even if program is single-threaded: > > BEGIN > + (* we need to call set up the signal handler for other reasons than > + just thread switching now *) > + setup_sigvtalrm (switch_thread); > END ThreadPosix. > > 8. modify signal handler in ThreadPosixC.c to catch SIGCHLD: > > sigemptyset(&ThreadSwitchSignal); > sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); > + sigaddset(&ThreadSwitchSignal, SIGCHLD); > > act.sa_handler = handler; > act.sa_flags = SA_RESTART; > sigemptyset(&(act.sa_mask)); > if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); > + if (sigaction (SIGCHLD, &act, NULL)) abort(); > > I'll send the complete diff in a separate message for those who want > to study it more closely. > > I propose the above changes for inclusion in the current CM3 repository. > > Mika > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 23:08:42 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 22:08:42 +0000 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212202550.542D41A2078@async.async.caltech.edu> References: <20110212030525.822411A2078@async.async.caltech.edu>, , <20110212165930.DC32C1A2078@async.async.caltech.edu>, <20110212205527.e6xetojo0ssgocg4@mail.elegosoft.com>, <20110212202550.542D41A2078@async.async.caltech.edu> Message-ID: "fork() and do more work" is portable, to Posix, which is many systems. It is easily done in Modula-3 by calling fork(). Given a choice between fixing fork() and do more work, vs. changing cvsupd to not depend on it, and given what I know about fixing fork() and do more work, I chose fork() and do more work. Or just abandon cvsupd. cvsupd is far from the only user of this pattern, in the wider non-Modula-3 world. > But why would user threads work? We're skipping pthread_atfork for them too. It is quite possible that fork and do more work + user threads + FreeBSD < 6, doesn't work. It is quite possible that it works intermittently. Let me try to explain what pthread_atfork is for. Though the opengroup spec does discuss it in detail and is clear (Bing for "opengroup pthread_atfork"; the first hit is it). In a multithreaded system, arbitrary mutexes are held at arbitrary times. Aribrary libraries spawn arbitrary worker threads, and use mutexes therein. As well, such programs might fork and do more work. There is not necessarily any central arbitrator among worker threads and the thread that calls fork(). That is, when you fork, you might be holding multiple mutexes. There isn't a way for the forking thread to request all other threads to enter some quiescent state in which no mutexes are held. If you fork and do more work, the new process has only one thread, the thread that called fork(). But any mutex ownership at the time of fork() is inherited. As a result, attemps to use mutexes in the new child process will have a high propensity to hang/deadlock. Therefore...go back..and do introduce an arbitrator. The implementation of fork() is therefore integrated with the implementation of pthreads. pthread_atfork establishes a set of callbacks. 3 sets actually. fork() calls them at the appropriate times -- before fork in the parent, and after fork in both the parent and the child. What the callbacks are meant to do is acquire all locks in the parent before the fork, and release them all in the parent and child after the work. And possibly reinitialize globals in the child after the fork. In the Modula-3 user thread system, threads exist by virtue of having memory allocated for their stack, small memory allocated for their processor context/registers/etc., and by being on a global linked list. Scheduling is triggered by an occasional timer. Historically, upon fork(), all that memory and the global linked list were unchanged. Therefore the child process would continue to have all the same threads as the parent. But compare this to pthreads. Solaris has an option to be either way, but in everything else I've seen -- OpenBSD, FreeBSD, NetBSD, Darwin, Linux -- and I think Cygwin, HP-UX, Irix, AIX, and probably the Posix spec -- fork() is specified as creating a child process with only one thread. Solaris has fork1() and forkall() and the behavior of plain fork() has varied from release to release. So, again, upon fork(), historical Modula-3 user threads retain all threads. If a thread held a mutex at fork() time, that's ok, it will continue running in the child and eventually release the mutex. No hang. If fork() is followed shortly by exec, well, then indeed, even with user threads, alll threads are gone, globals get reinitialized (to zero), there is a new thread. I guess you just have to be a bit careful what you do between fork() and exec(). Notice that things like Process.Create() suspend scheduling and probably after that avoid allocating any memory and possibly avoid touching garbage collected memory and probably avoid taking any locks. That is, upon suspending scheduling, they don't depend on any mutex to need to be released by any other thread. The various threads/mutexes can be in an arbitrary state. In the child they will go away shortly (upon exec). In the parent, scheduling will shortly resume. Does some of this make sense? > It is quite possible that fork and do more work + user threads + FreeBSD < 6, doesn't work. Historically it did work -- child process retained all the threads. These days I expect..I expect you still get that same behavior. We could fix it, I believe, by never calling fork() directly. Have a wrapper Process.Fork(). On most systems, it would just call fork. But on OpenVMS and FreeBSD < 6...there is also the Register function. On most systems, it'd just call pthread_atfork. On OpenVMS and FreeBSD < 6, it should maintain the lists itself. And then Process.Fork on those systems, make the call outs. Indeed, the OpenVMS/FreeBSD<6 implementation could be the one and only implementation. If fork() was disallowed and Process.Fork() the only allowed replacement. Or, it could be tested that way, and then actually commited the other way. I very well might do this. - Jay > To: wagner at elegosoft.com > Date: Sat, 12 Feb 2011 12:25:50 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] pthread_atfork > > How do you do "fork-and-do-work" from Modula-3? > > All my programs that create new Unix processes do it with Process.Create, > which as far as I know does a fork-and-exec... > > I'm not sure what if anything is broken here, either. But then again I > didn't fully understand Jay's last mail on the subject. > > Mika > > Olaf Wagner writes: > >Well, I think actually the fork-and-do-work pattern has been > >the default for a long time with the M3 user threads implementation. > >It wasn't specified though, and I think no application except > >CVSup did use it. > > > >Jay is not responsible for any breakage here, he just tried to fix > >CVSup and establish a consistent behaviour after our change to > >system pthreads on most platforms. > > > >Olaf > > > >Quoting Mika Nystrom : > > > >> Hi Jay, > >> > >> I think "fork and do work" qualifies as a bug in the application, if > >> the application wants to be portable. Why can't it fork and exec instead? > >> If it wants to do more work it should just fork and exec another copy > >> of itself. > >> > >> fork() is also not part of Modula-3... Process.Create() is, however. > >> Don't tell me you broke that as part of "fixing" things for cvsupd... > >> > >> There is still one aspect of your comment I don't understand. What does > >> the call to pthread_atfork actually accomplish when running with user > >> threads? You say below that "don't expect pthreads to work on FreeBSD < > >> 6 [using the fork-and-do-work pattern]". Fair enough. But why would > >> user threads work? We're skipping pthread_atfork for them too. > >> > >> Do we not need pthread_atfork with user threads? What you're saying > >> suggests that FreeBSD >=3D 6 and all other platforms need pthread_atfork > >> with user threads. Is that really what you mean? > >> > >> At the moment I'm just concerned with getting user threads to work as > >> I know there are bugs with pthreads. > >> > >> I also think we should not waste time and effort making the "fork and do > >> work" pattern work unless it's really, really necessary (which I really > >> don't see why it would be). Fix the application instead of introducing > >> bugs in the libraries and language implementation! > >> > >> I've been trying for a long, long time to switch from PM3 to CM3 (almost > >> ten years, I think). These bugs from "enhancements" have stopped me > >> every time. Maybe things are finally working well enough... > >> > >> (some comments inline in your quoted text below) > >> > >> Mika > >> > >> > >> Jay K writes: > >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ > >>> Content-Type: text/plain; charset=3D"iso-8859-1" > >>> Content-Transfer-Encoding: quoted-printable > >>> > >>> > >>> I wrote that. > >>> > >>> As I recall=3D2C FreeBSD < 6 doesn't have pthread_atfork. > >>> > >>> I think that was based on reading the man pages. > >>> > >>> So=3D2C yes. What would you suggest? > >>> > >>> > >>> > >>> > >>> > >>> All this atfork stuff came about as part of getting cvsupd to work. > >>> > >>> With pthreads/kernel threads. It historically worked=3D2C with Modula-3 = > >use=3D > >>> r threads. > >>> > >>> Modula-3 user threads have this funny property that all threads survive = > >w=3D > >>> ork. > >> > >> you mean survive fork don't you? > >> > >>> > >>> With pthreads/kernel threads=3D2C fork gives you a new process with just= > > on=3D > >>> e thread. > >>> > >>> (Solaris has fork1 and forkall=3D2C and fork maybe used to be forkall= > >=3D2C b=3D > >>> ut now it is fork1). > >>> > >>> > >>> > >>> > >>> > >>> cvsupd is one of those slightly unusual programs that does "fork and then= > > d=3D > >>> o more work" > >>> > >>> as supposed to the more typical "fork and exec". > >>> > >>> > >>> > >>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o= > >n Fr=3D > >>> eeBSD < 6? > >>> > >>> At least not in a process that ever calls fork? Or does "fork and then mo= > >re=3D > >>> work"? > >>> > >>> > >>> > >>> > >>> > >>> In reality=3D2C i think a lot of libraries have trouble with "fork and th= > >en d=3D > >>> o more work"=3D2C but > >>> > >>> I don't know. I recall vague notices in the Apple/Darwin manpages not to = > >as=3D > >>> sume > >>> > >>> this works -- the obvious implication that there is plenty of use of pthr= > >ea=3D > >>> ds e.g. mutexes=3D2C > >>> > >>> but relatively little use of pthread_atfork. > >>> > >>> > >>> > >>> > >>> > >>> You could perhaps replace all uses of fork with a function that called th= > >e =3D > >>> fork handlers=3D2C > >>> > >>> then fork=3D2C then the other handlers. That is probably what I meant in = > >the =3D > >>> comment > >>> > >>> about this being easy to fix. > >>> > >>> You might though then run into the problem that callers of fork can't do = > >mu=3D > >>> ch > >>> > >>> before exec...oh=3D2C that's vfork I bet actually. > >> > >> I don't understand this paragraph either :( > >> > >>> > >>> > >>> > >>> > >>> > >>> - Jay > >>> > >>> > >>> > >>> > >>> > >>> > >>>> To: hosking at cs.purdue.edu > >>>> Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800 > >>>> From: mika at async.caltech.edu > >>>> CC: m3devel at elegosoft.com=3D3B jay.krell at cornell.edu > >>>> Subject: [M3devel] pthread_atfork > >>>> =3D20 > >>>> Tony=3D2C > >>>> =3D20 > >>>> What's RTProcessC.c doing? > >>>> =3D20 > >>>> I note the following cryptic comment: > >>>> =3D20 > >>>> /* NOTE: Even userthreads now depends > >>>> * on availability of pthreads. > >>>> * This can be fixed if need be. > >>>> */ > >>>> =3D20 > >>>> now further down we can read: > >>>> =3D20 > >>>> #if defined(_WIN32) \ > >>>> || defined(__vms) \ > >>>> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > >>>> return 0=3D3B > >>>> #else > >>>> while (1) > >>>> { > >>>> int i =3D3D pthread_atfork(prepare=3D2C parent=3D2C child)=3D3B > >>>> if (i !=3D3D EAGAIN) > >>>> return i=3D3B > >>>> sleep(0)=3D3B > >>>> } > >>>> #endif > >>>> =3D20 > >>>> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=3D2C > >>>> RTProcess.RegisterForkHandlers does nothing? > >>>> =3D20 > >>>> Hmmm..... > >>>> =3D20 > >>>> Mika > >>> =09=09 =09 =09=09 =3D > >>> > >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ > >>> Content-Type: text/html; charset=3D"iso-8859-1" > >>> Content-Transfer-Encoding: quoted-printable > >>> > >>> > >>> > >>> > >>> > >>> > >>> I wrote that.
> >>> As I recall=3D2C FreeBSD <=3D3B 6 doesn't have pthread_atfork.
> >>> I think that was based on reading the man pages.
> >>> So=3D2C yes. What would you suggest?
> >>>
> >>>
> >>> All this atfork stuff came about as part of getting cvsupd to work.
> >>>  =3D3B With pthreads/kernel threads. It historically worked=3D2C with= > > Modul=3D > >>> a-3 user threads.
> >>>  =3D3B Modula-3 user threads have this funny property that all thread= > >s su=3D > >>> rvive work.
> >>>  =3D3B With pthreads/kernel threads=3D2C fork gives you a new process= > > with =3D > >>> just one thread.
> >>>  =3D3B =3D3B (Solaris has fork1 and forkall=3D2C and fork maybe u= > >sed to b=3D > >>> e forkall=3D2C but now it is fork1).
> >>>
> >>>
> >>> cvsupd is one of those slightly unusual programs that does "fork and then= > > d=3D > >>> o more work"
> >>> as supposed to the more typical "fork and exec".
> >>>
> >>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o= > >n Fr=3D > >>> eeBSD <=3D3B 6?
> >>> At least not in a process that ever calls fork? Or does "fork and then mo= > >re=3D > >>> work"?
> >>>
> >>>
> >>> In reality=3D2C i think a lot of libraries have trouble with "fork and th= > >en d=3D > >>> o more work"=3D2C but
> >>> I don't know. I recall vague notices in the Apple/Darwin manpages not to = > >as=3D > >>> sume
> >>> this works -- the obvious implication that there is plenty of use of pthr= > >ea=3D > >>> ds e.g. mutexes=3D2C
> >>> but relatively little use of pthread_atfork.
> >>>
> >>>
> >>> You could perhaps replace all uses of fork with a function that called th= > >e =3D > >>> fork handlers=3D2C
> >>> then fork=3D2C then the other handlers. That is probably what I meant in = > >the =3D > >>> comment
> >>> about this being easy to fix.
> >>> You might though then run into the problem that callers of fork can't do = > >mu=3D > >>> ch
> >>> before exec...oh=3D2C that's vfork I bet actually.
> >>>
> >>>
> >>>  =3D3B- Jay






>=3D3B To: hosking at cs.purdue= > >.edu >>> r>>=3D3B Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800
>=3D3B From: mi= > >ka at async=3D > >>> .caltech.edu
>=3D3B CC: m3devel at elegosoft.com=3D3B jay.krell at cornell= > >.edu >>> r>>=3D3B Subject: [M3devel] pthread_atfork
>=3D3B
>=3D3B Ton= > >y=3D2C
=3D > >>> >=3D3B
>=3D3B What's RTProcessC.c doing?
>=3D3B
>=3D3B= > > I note th=3D > >>> e following cryptic comment:
>=3D3B
>=3D3B /* NOTE: Even usert= > >hreads=3D > >>> now depends
>=3D3B * on availability of pthreads.
>=3D3B * Th= > >is ca=3D > >>> n be fixed if need be.
>=3D3B */
>=3D3B
>=3D3B now furth= > >er down =3D > >>> we can read:
>=3D3B
>=3D3B #if defined(_WIN32) \
>=3D3B = > > |=3D > >>> | defined(__vms) \
>=3D3B || (defined(__FreeBSD__) &=3D3B= > >&=3D > >>> =3D3B (__FreeBSD__ <=3D3B 6))
>=3D3B return 0=3D3B
>=3D3B= > > #else
&g=3D > >>> t=3D3B while (1)
>=3D3B {
>=3D3B int i =3D3D pthr= > >ead_atfork=3D > >>> (prepare=3D2C parent=3D2C child)=3D3B
>=3D3B if (i !=3D3D EAGA= > >IN)
>=3D > >>> =3D3B return i=3D3B
>=3D3B sleep(0)=3D3B
>=3D3B = > > }
>=3D > >>> =3D3B #endif
>=3D3B
>=3D3B so on FreeBSD 5 (what I'm running o= > >n my "Fr=3D > >>> eeBSD4" system)=3D2C
>=3D3B RTProcess.RegisterForkHandlers does noth= > >ing? >>> r>>=3D3B
>=3D3B Hmmm.....
>=3D3B
>=3D3B Mika
> >>> =09 =09 =09=09 =3D > >>> > >>> =3D > >>> > >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_-- > >> > > > > > > > >--=20 > >Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa= > >ny > >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz= > >: Berlin > >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sun Feb 13 00:27:43 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 12 Feb 2011 18:27:43 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <20110212015715.A43471A2078@async.async.caltech.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu> Message-ID: <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> Mika, I have no idea what the issue is here. But I do note that AMD64_LINUX with pthreads used to run just fine. I ran a bunch or stress tests when first ported to that target and all seemed good. What is preventing you using pthreads on this target? -- Tony On Feb 11, 2011, at 8:57 PM, Mika Nystrom wrote: > I would like to find one configuration (any configuration!!) that lets > me run a multithreaded program under AMD64_LINUX. I don't care what > version of the compiler it is, whether the threads are user or pthreads, > how many things I have to hack. Does anyone know what I need to do? From jay.krell at cornell.edu Sun Feb 13 00:37:12 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 23:37:12 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu>, <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> Message-ID: I assume he means "the thread tester should work"? But agreed, the vast vast vast vast majority of things are ok and have been ok for the vast vast vast majority of Modula-3's existance. Even in recent years. pthreads and user threads. Infinite loops with TRY in January 2011 would use infinite stack. Understood. Finite loops would merely use more than they should and reclaim upon function return. cvsupd didn't work for a while with pthreads. Because of its use of "fork and do more work" and its assumption that all threads survive fork. Relatively few programs do that and they were all broken by pthreads. The new thread tester probably needs more attention. - Jay > Subject: Re: [M3devel] threading on Windows? > From: hosking at cs.purdue.edu > Date: Sat, 12 Feb 2011 18:27:43 -0500 > CC: jay.krell at cornell.edu; m3devel at elegosoft.com > To: mika at async.caltech.edu > > Mika, > > I have no idea what the issue is here. But I do note that AMD64_LINUX with pthreads used to run just fine. I ran a bunch or stress tests when first ported to that target and all seemed good. What is preventing you using pthreads on this target? > > -- Tony > > On Feb 11, 2011, at 8:57 PM, Mika Nystrom wrote: > > > I would like to find one configuration (any configuration!!) that lets > > me run a multithreaded program under AMD64_LINUX. I don't care what > > version of the compiler it is, whether the threads are user or pthreads, > > how many things I have to hack. Does anyone know what I need to do? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 13 00:39:15 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 15:39:15 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu> <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> Message-ID: <20110212233915.A8EEE1A2078@async.async.caltech.edu> Tony, I have yet to find a single system on which my thread testing program actually runs properly under pthreads. I have however managed to get it to work everywhere with user threading. My last email to you on the issue said the following: read: segfault after a long time fork: OK alloc: deadlock creat: appears OK for a long time then hangs hard (can't get into debugger) lock: OK I think Jay has now fixed the "read" problem, which had nothing to do with threading but was a TRY-EXCEPT issue. The fact that the runtime deadlocks when NEWing objects I think is the most serious problem. Could you try it, please? If you run "threadtest -n 10 -tests all" you should see something like this (the following is AMD64_FREEBSD with user threading---the numbers denote how much threads are lagging, in seconds). The program is utterly standard Modula-3. /big/home/mika/cm3-writable/cm3/m3-libs/m3core/tests/thread/AMD64_FREEBSD/threadtest -n 10 -tests all Writing file...done Creating read threads...done Creating nxread threads...done Creating tryexcept threads...done Creating fork threads...done Creating alloc threads...done Creating creat threads...done Creating lock threads...done running...printing oldest/median age/newest ..........laziest thread is 56/13/0 (tests: read 1/1/0 nxread 3/2/2 tryexcept 12/8/3 fork 13/13/13 alloc 17/15/13 creat 38/17/17 lock 56/18/17) ..........laziest thread is 57/13/0 (tests: read 2/1/0 nxread 3/2/2 tryexcept 12/8/3 fork 13/13/13 alloc 17/15/13 creat 36/17/17 lock 57/36/17) ..........laziest thread is 40/13/0 (tests: read 1/1/0 nxread 3/2/1 tryexcept 12/8/3 fork 13/13/13 alloc 17/15/13 creat 37/17/17 lock 40/37/17) ..........laziest thread is 34/14/0 (tests: read 1/0/0 nxread 3/2/1 tryexcept 13/9/3 fork 14/14/14 alloc 17/15/14 creat 31/17/17 lock 34/31/17) ..........laziest thread is 56/13/0 (tests: read 2/1/0 nxread 3/2/2 tryexcept 12/8/3 fork 13/13/13 alloc 16/15/13 creat 37/16/16 lock 56/18/16) ..........laziest thread is 37/14/0 (tests: read 1/0/0 nxread 3/2/1 tryexcept 13/8/3 fork 14/14/14 alloc 18/15/14 creat 37/19/19 lock 37/19/19) ..........laziest thread is 34/15/0 (tests: read 2/1/0 nxread 3/3/2 tryexcept 14/9/3 fork 15/15/15 alloc 16/16/15 creat 34/17/17 lock 34/17/17) ..........laziest thread is 34/14/0 (tests: read 1/1/0 nxread 3/2/1 tryexcept 13/8/3 fork 14/14/14 alloc 16/15/14 creat 34/16/16 lock 34/16/16) ..........laziest thread is 35/14/0 (tests: read 2/1/0 nxread 3/2/2 tryexcept 13/9/3 fork 14/14/14 alloc 17/16/14 creat 35/17/17 lock 35/17/17) ..........laziest thread is 37/14/0 (tests: read 2/1/0 nxread 3/2/2 tryexcept 13/8/3 fork 14/14/14 alloc 16/15/14 creat 37/17/17 lock 37/17/17) All tests complete. Congratulations. Mika Tony Hosking writes: >Mika, > >I have no idea what the issue is here. But I do note that AMD64_LINUX = >with pthreads used to run just fine. I ran a bunch or stress tests when = >first ported to that target and all seemed good. What is preventing you = >using pthreads on this target? > >-- Tony > >On Feb 11, 2011, at 8:57 PM, Mika Nystrom wrote: > >> I would like to find one configuration (any configuration!!) that lets >> me run a multithreaded program under AMD64_LINUX. I don't care what >> version of the compiler it is, whether the threads are user or = >pthreads, >> how many things I have to hack. Does anyone know what I need to do? From mika at async.caltech.edu Sun Feb 13 01:33:57 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 16:33:57 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu>, <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> Message-ID: <20110213003358.00D2A1A2078@async.async.caltech.edu> Jay K writes: >--_879a340b-b2bb-415e-936c-be8d5cdd855c_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >I assume he means "the thread tester should work"? >But agreed=2C the vast vast vast vast majority of things are ok and have be= >en ok for the vast vast vast majority of Modula-3's existance. > Even in recent years. pthreads and user threads. > Infinite loops with TRY in January 2011 would use infinite stack. Underst= >ood. > Finite loops would merely use more than they should and reclaim upon func= >tion return. >cvsupd didn't work for a while with pthreads. Because of its use of "fork a= >nd do more work" and its assumption >that all threads survive fork. Relatively few programs do that and they wer= >e all broken by pthreads. >=20 >The new thread tester probably needs more attention. >=20 > - Jay My thread tester wasn't developed "ex nihilo". It's based on analyzing the failure modes of a larger application that I am prevented from distributing the source code for, both for reasons of confidentiality and practicality. I have purposely written the tester to be "obviously correct" and I would like nothing more than for someone to show me where I have made a mistake and that pthreads are working as they should. For example, here is the "alloc" thread: PROCEDURE AApply(cl : Closure) : REFANY = BEGIN Thread.Pause(InitPause); LOOP VAR arr := NEW(REF ARRAY OF INTEGER, 1025); BEGIN FOR i := FIRST(arr^)+1 TO LAST(arr^) DO arr[i] := arr[i] - arr[i-1] END; times1[cl.id] := FLOOR(Time.Now()) END END END AApply; Is there anything here that is wrong or bad? When I tried pthreads just now on AMD64_LINUX, running two instances of AApply at the same time caused the runtime to deadlock. [BTW, I have never seen a problem with user threads except for building issues and the TRY-EXCEPT thing you just fixed.] Unfortunately, since just about every Modula-3 program allocates memory, my analysis is that because it appears you can deadlock the runtime by allocating memory, nothing is *really* OK with pthreads. Please tell me I'm wrong! Mika P.S. backtraces of deadlock: Thread 3 (Thread 0x40862950 (LWP 19974)): #0 0x00002b9d49f730e1 in nanosleep () from /lib/libpthread.so.0 #1 0x000000000045806d in ThreadPThread__Nanosleep (req=Cannot access memory at address 0x140861b08 ) at ../src/thread/PTHREAD/ThreadPThreadC.c:500 #2 0x0000000000454294 in ThreadPThread__CommonSleep () at ../src/thread/PTHREAD/ThreadPThread.m3:551 #3 0x0000000000456438 in ThreadPThread__StopWorld () at ../src/thread/PTHREAD/ThreadPThread.m3:1086 #4 0x0000000000455787 in RTThread__SuspendOthers () at ../src/thread/PTHREAD/ThreadPThread.m3:812 #5 0x000000000043970c in RTCollector__CollectSomeInStateZero () at ../src/runtime/common/RTCollector.m3:746 #6 0x00000000004396b8 in RTCollector__CollectSome () at ../src/runtime/common/RTCollector.m3:720 #7 0x00000000004390b9 in RTHeapRep__CollectEnough () at ../src/runtime/common/RTCollector.m3:654 #8 0x0000000000436111 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=Cannot access memory at address 0x140861dc8 ) at ../src/runtime/common/RTAllocator.m3:363 #9 0x000000000043596f in RTAllocator__GetOpenArray (M3_Eic7CK_def=Cannot access memory at address 0x140861eb8 ) at ../src/runtime/common/RTAllocator.m3:292 #10 0x0000000000434ab1 in RTHooks__AllocateOpenArray (M3_AJWxb1_defn=Cannot access memory at address 0x140861f38 ) at ../src/runtime/common/RTAllocator.m3:141 #11 0x0000000000407030 in Main__AApply (M3_AP7a1g_cl=Cannot access memory at address 0x140861f88 ) at ../src/Main.m3:234 #12 0x0000000000453b3f in ThreadPThread__RunThread (M3_DMxDjQ_me=Cannot access memory at address 0x140862028 ) at ../src/thread/PTHREAD/ThreadPThread.m3:475 #13 0x00000000004537d3 in ThreadPThread__ThreadBase (M3_AJWxb1_param=Cannot access memory at address 0x1408620d8 ) at ../src/thread/PTHREAD/ThreadPThread.m3:447 #14 0x00002b9d49f6bfc7 in start_thread () from /lib/libpthread.so.0 #15 0x00002b9d4b3bd64d in clone () from /lib/libc.so.6 #16 0x0000000000000000 in ?? () Thread 2 (Thread 0x40661950 (LWP 19973)): #0 0x00002b9d49f72384 in __lll_lock_wait () from /lib/libpthread.so.0 #1 0x00002b9d49f6dbf0 in _L_lock_102 () from /lib/libpthread.so.0 #2 0x00002b9d49f6d4fe in pthread_mutex_lock () from /lib/libpthread.so.0 #3 0x000000000045815e in ThreadPThread__pthread_mutex_lock (m=Cannot access memory at address 0x140660d68 ) at ../src/thread/PTHREAD/ThreadPThreadC.c:557 #4 0x00000000004575af in RTOS__LockHeap () at ../src/thread/PTHREAD/ThreadPThread.m3:1434 #5 0x0000000000436107 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=Cannot access memory at address 0x140660dc8 ) at ../src/runtime/common/RTAllocator.m3:361 #6 0x000000000043596f in RTAllocator__GetOpenArray (M3_Eic7CK_def=Cannot access memory at address 0x140660eb8 ) at ../src/runtime/common/RTAllocator.m3:292 #7 0x0000000000434ab1 in RTHooks__AllocateOpenArray (M3_AJWxb1_defn=Cannot access memory at address 0x140660f38 ) at ../src/runtime/common/RTAllocator.m3:141 #8 0x0000000000407030 in Main__AApply (M3_AP7a1g_cl=Cannot access memory at address 0x140660f88 ) at ../src/Main.m3:234 #9 0x0000000000453b3f in ThreadPThread__RunThread (M3_DMxDjQ_me=Cannot access memory at address 0x140661028 ) at ../src/thread/PTHREAD/ThreadPThread.m3:475 #10 0x00000000004537d3 in ThreadPThread__ThreadBase (M3_AJWxb1_param=Cannot access memory at address 0x1406610d8 ) at ../src/thread/PTHREAD/ThreadPThread.m3:447 #11 0x00002b9d49f6bfc7 in start_thread () from /lib/libpthread.so.0 #12 0x00002b9d4b3bd64d in clone () from /lib/libc.so.6 #13 0x0000000000000000 in ?? () Thread 1 (Thread 0x2b9d4d32a1a0 (LWP 19965)): #0 0x00002b9d4b3202a2 in sigsuspend () from /lib/libc.so.6 #1 0x0000000000457ba9 in ThreadPThread__sigsuspend () at ../src/thread/PTHREAD/ThreadPThreadC.c:141 #2 0x0000000000456a0d in ThreadPThread__SignalHandler (M3_DLS2Hj_sig=Cannot access memory at address 0x80001dde057c ) at ../src/thread/PTHREAD/ThreadPThread.m3:1205 #3 #4 0x00002b9d49f6ffad in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #5 0x00000000004580b3 in ThreadPThread__pthread_cond_timedwait (cond=Cannot access memory at address 0x80001dde0a88 ) at ../src/thread/PTHREAD/ThreadPThreadC.c:515 #6 0x00000000004544cf in ThreadPThread__XPause (M3_DMxDjQ_self=Cannot access memory at address 0x80001dde0ab8 ) at ../src/thread/PTHREAD/ThreadPThread.m3:580 #7 0x00000000004545b2 in Thread__Pause (M3_CtKayy_n=Cannot access memory at address 0x80001dde0b08 ) at ../src/thread/PTHREAD/ThreadPThread.m3:595 #8 0x00000000004097ee in Main_M3 (M3_AcxOUs_mode=Cannot access memory at address 0x80001dde0bc8 ) at ../src/Main.m3:449 #9 0x0000000000445799 in RTLinker__RunMainBody (M3_DjPxE3_m=Cannot access memory at address 0x80001dde0f58 ) at ../src/runtime/common/RTLinker.m3:406 #10 0x00000000004448f2 in RTLinker__AddUnitI (M3_DjPxE3_m=Cannot access memory at address 0x80001dde0ff8 ) at ../src/runtime/common/RTLinker.m3:113 #11 0x0000000000444990 in RTLinker__AddUnit (M3_DjPxE5_b=Cannot access memory at address 0x80001dde1018 ) at ../src/runtime/common/RTLinker.m3:122 #12 0x000000000040652c in main (argc=Cannot access memory at address 0x80001dde1068 ) at _m3main.mc:4 From mika at async.caltech.edu Sun Feb 13 07:29:55 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 22:29:55 -0800 Subject: [M3devel] Performance issues with Process.Wait under userthreads In-Reply-To: References: <20110212204302.19C781A2078@async.async.caltech.edu> Message-ID: <20110213062955.4F84B1A2078@async.async.caltech.edu> Jay K writes: >--_5ec25af1-29a7-4f8a-b7ad-e61068941159_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >I don't believe there is a good solution here=2C other than using pthreads. >Is there? Maybe tune the wait down smaller? >I understand it stinks either way. Well as I keep saying, I'm looking for a reliable runtime. I really need this---and soon. I can't keep futzing around with PM3 forever. I'd hope pthreads is it but for now the only thing that passes my tests is user threading. We (group at Caltech) determined many years ago that tuning the wait does nothing, unless you set it to zero (busy waiting). Possibly this has to do with the time quantum of the user threads scheduler, I don't know. >=20 >=20 >I don't understand the SIGCHLD stuff=2C at a quick glance. The idea is this. We have SIGVTALRM coming in already to do threadswitching. So I am enabling SIGCHLD and then taking SIGCHLD the way SIGVTALRM is taken, that is, in the scheduler. The other change is that a paused thread can wait for a condition such as "either after X delay OR after Y signal" and then we allow the paused thread to become runnable on the Y signal rather than after the X delay. >Nor your changes. >However you might be the only user of user threads. >Is there any downside to your change? > (Other than more code=2C ok.) >Any loss of performance in any situation? >Any loss of portability? I am not sure. Is SIGCHLD completely standard? I am worried about one thing. The thread switcher turns signals back on as its first action. If a large number of children exit at the same time, I fear that we get a very deeply nested signal stack, which doesn't sound good. Perhaps signals should not be turned back on immediately. Or SIGCHLD should be set to SIG_IGN and then re-enabled? (Simulating old-fashioned "unreliable" SysV signals!) I'm not sure. In any case I was just trying to parallelize the compiler and ran into some very odd things related to having lots of child processes running around... The SIGCHLD change speeds up the CM3 compiler by about a factor of 3 when using user threads. Mika >=20 >=20 > - Jay >=20 >> To: m3devel at elegosoft.com >> Date: Sat=2C 12 Feb 2011 12:43:02 -0800 >> From: mika at async.caltech.edu >> Subject: [M3devel] Performance issues with Process.Wait under userthreads >>=20 >> Hi again m3devel (especially Tony)=2C=20 >>=20 >> I have finally taken a bite of a problem that has been annoying me >> for a very=2C very long time. >>=20 >> Under user threads=2C the code for Process.Wait is as follows (see=20 >> ThreadPosix.m3): >>=20 >> PROCEDURE WaitProcess (pid: int=3B VAR status: int): int =3D >> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >> CONST Delay =3D 0.1D0=3B >> BEGIN >> LOOP >> WITH r =3D Uexec.waitpid(pid=2C ADR(status)=2C Uexec.WNOHANG) DO >> IF r # 0 THEN RETURN r END=3B >> END=3B >> Pause(Delay)=3B >> END=3B >> END WaitProcess=3B >>=20 >> It inserts a 0.1 second delay after each failed waitpid. This is extremel= >y >> annoying for programs that start a long sequence of child processes and >> wait for them in sequence. Namely=2C the compiler itself. As a result >> the cm3 compiler (and PM3's m3build) are normally very very slow when >> using user threads. For about the last ten years=2C I've had a hacked >> up m3build (for my PM3 installation) that skips the Pause and busy-waits >> instead.=20 >>=20 >> Note there is another problem here. Since the Modula-3 runtime ignores >> SIGCHLD=2C no zombie processes are created since the Unix system automati= >cally >> reaps the child processes. I can see this would be a problem since PIDs >> are eventually reused and ... couldn't Uexec.waitpid wind up referring to >> the wrong process?? >>=20 >> I will further note that the comment in Process.i3 reads as follows: >>=20 >> PROCEDURE Wait(p: T): ExitCode=3B >>=20 >> Wait until the process with handle p terminates=2C then free the operatin= >g system resources associated with the process and return an exit code indi= >cating the reason for its termination. It is a checked runtime error to cal= >l Wait twice on the same process handle.=20 >>=20 >> I am going to take this as fair warning that Process.Create *may* use >> resources that are not going to be released until Process.Wait has been >> called. >>=20 >> I have modified (in my local copy of CM3) the system as follows. >> I have come up with a semi-general mechanism for immediately unblocking >> a thread on the receipt of a unix signal.=20 >>=20 >> 1. The system relies on changing >> ThreadPosix.XPause such that if a signal is allowed to wake up a threads= >=2C >> that fact is recorded in a new field in the thread's descriptor >> record (of type ThreadPosix.T).=20 >>=20 >> 2. On receipt of a waited-for unix signal=2C a mask is set and control >> is passed to the thread scheduler which maintains the non-zero mask for >> exactly one iteration through the thread ring. >>=20 >> 3. If a thread is paused and waiting for EITHER a signal or some time=2C >> the thread is released for running and the thread's waiting state is=20 >> cleared. >>=20 >> The changes are more or less as follows: >>=20 >> 1. I have added a new field of type "int" to ThreadPosix.T: >>=20 >> (* if state =3D pausing=2C the time at which we can restart *) >> waitingForTime: Time.T=3B >>=20 >> + (* if state =3D pausing=2C the signal that truncates the pause *) >> + waitingForSig: int :=3D -1=3B >> + >> (* true if we are waiting during an AlertWait or AlertJoin >> or AlertPause *) >> alertable: BOOLEAN :=3D FALSE=3B >>=20 >>=20 >> 2. Modifications to pause: >>=20 >> + PROCEDURE SigPause(n: LONGREAL=3B sig: int)=3D >> + <*FATAL Alerted*> >> + VAR until :=3D n + Time.Now ()=3B >> + BEGIN >> + XPause(until=2C FALSE=2C sig)=3B >> + END SigPause=3B >> + >> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >> VAR until :=3D n + Time.Now ()=3B >> BEGIN >> XPause(until=2C TRUE)=3B >> END AlertPause=3B >>=20 >> ! PROCEDURE XPause (READONLY until: Time.T=3B alertable :=3D FALSE=3B sig= >:int :=3D -1) >> ! RAISES {Alerted} =3D >> BEGIN >> INC (inCritical)=3B >> self.waitingForTime :=3D until=3B >> self.alertable :=3D alertable=3B >> + IF sig # -1 THEN >> + self.waitingForSig :=3D sig >> + END=3B >> ICannotRun (State.pausing)=3B >> DEC (inCritical)=3B >> InternalYield ()=3B >>=20 >> 3. The received-signals mask: >>=20 >> ! CONST MaxSigs =3D 64=3B >> ! TYPE Sig =3D [ 0..MaxSigs-1 ]=3B >> ! >> ! (* in order to listen to other signals=2C they have to be enabled in >> ! allow_sigvtalrm as well *) >> ! VAR (*CONST*) SIGCHLD :=3D ValueOfSIGCHLD()=3B >> ! >> ! gotSigs :=3D SET OF Sig { }=3B >> ! >>=20 >> ValueOfSIGCHLD() is a C function used to get the value of the SIGCHLD >> constant without guessing at it (in ThreadPosixC.c). >>=20 >> 4. changes to the signal handler: >>=20 >> ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >> BEGIN >> allow_sigvtalrm ()=3B >> ! >> ! INC(inCritical)=3B >> ! (* mark signal as being delivered *) >> ! IF sig >=3D 0 AND sig < MaxSigs THEN >> ! gotSigs :=3D gotSigs + SET OF Sig { sig } >> ! END=3B >> ! DEC(inCritical)=3B >> ! >> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >> ! InternalYield () >> ! END=3B >> END switch_thread=3B >>=20 >> Note that I don't know if INC/DEC(inCritical) does exactly the right >> thing here. >>=20 >> 5. changes to the scheduler: >>=20 >> a. thread wakeup >> IF t.alertable AND t.alertPending THEN >> CanRun (t)=3B >> EXIT=3B >> + >> + ELSIF t.waitingForSig IN gotSigs THEN >> + t.waitingForSig :=3D -1=3B >> + CanRun(t)=3B >> + EXIT=3B >>=20 >> ELSIF t.waitingForTime <=3D now THEN >> CanRun (t)=3B >> EXIT=3B >> ! >>=20 >> b. clearing the mask >>=20 >> END=3B >> END=3B >>=20 >> + gotSigs :=3D SET OF Sig {}=3B >> + >> IF t.state =3D State.alive AND (scanned OR NOT someBlocking) THEN >> IF perfOn THEN PerfRunning (t.id)=3B END=3B >> (* At least one thread wants to run=3B transfer to it *) >>=20 >> 6. changes to WaitProcess (Process.Wait): >>=20 >> PROCEDURE WaitProcess (pid: int=3B VAR status: int): int =3D >> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >> ! CONST Delay =3D 10.0D0=3B >> BEGIN >> LOOP >> WITH r =3D Uexec.waitpid(pid=2C ADR(status)=2C Uexec.WNOHANG) DO >> IF r # 0 THEN RETURN r END=3B >> END=3B >> ! SigPause(Delay=2CSIGCHLD)=3B >> END=3B >> END WaitProcess=3B >>=20 >> 7. install signal handler even if program is single-threaded: >>=20 >> BEGIN >> + (* we need to call set up the signal handler for other reasons than >> + just thread switching now *) >> + setup_sigvtalrm (switch_thread)=3B >> END ThreadPosix. >>=20 >> 8. modify signal handler in ThreadPosixC.c to catch SIGCHLD: >>=20 >> sigemptyset(&ThreadSwitchSignal)=3B >> sigaddset(&ThreadSwitchSignal=2C SIG_TIMESLICE)=3B >> + sigaddset(&ThreadSwitchSignal=2C SIGCHLD)=3B >>=20 >> act.sa_handler =3D handler=3B >> act.sa_flags =3D SA_RESTART=3B >> sigemptyset(&(act.sa_mask))=3B >> if (sigaction (SIG_TIMESLICE=2C &act=2C NULL)) abort()=3B >> + if (sigaction (SIGCHLD=2C &act=2C NULL)) abort()=3B >>=20 >> I'll send the complete diff in a separate message for those who want >> to study it more closely. >>=20 >> I propose the above changes for inclusion in the current CM3 repository. >>=20 >> Mika >>=20 > = > >--_5ec25af1-29a7-4f8a-b7ad-e61068941159_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >I don't believe there is a good solution here=2C other than using pthreads.= >
>Is there? Maybe tune the wait down smaller?
>I understand it stinks either way.
> =3B
> =3B
>I don't =3Bunderstand the SIGCHLD stuff=2C at a quick glance.
>Nor your changes.
>However you might be the only user of user threads.
>Is there any downside to your change?
> =3B(Other than more code=2C ok.)
>Any loss of performance in any situation?
>Any loss of portability?
> =3B
> =3B
> =3B- Jay
 =3B
>>=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 12 Feb 2011 12:43:0= >2 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > Performance issues with Process.Wait under userthreads
>=3B
>= >=3B Hi again m3devel (especially Tony)=2C
>=3B
>=3B I have fina= >lly taken a bite of a problem that has been annoying me
>=3B for a ver= >y=2C very long time.
>=3B
>=3B Under user threads=2C the code fo= >r Process.Wait is as follows (see
>=3B ThreadPosix.m3):
>=3B >>=3B PROCEDURE WaitProcess (pid: int=3B VAR status: int): int =3D
>= >=3B (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *)
>=3B C= >ONST Delay =3D 0.1D0=3B
>=3B BEGIN
>=3B LOOP
>=3B WITH r =3D= > Uexec.waitpid(pid=2C ADR(status)=2C Uexec.WNOHANG) DO
>=3B IF r # 0 T= >HEN RETURN r END=3B
>=3B END=3B
>=3B Pause(Delay)=3B
>=3B EN= >D=3B
>=3B END WaitProcess=3B
>=3B
>=3B It inserts a 0.1 sec= >ond delay after each failed waitpid. This is extremely
>=3B annoying f= >or programs that start a long sequence of child processes and
>=3B wai= >t for them in sequence. Namely=2C the compiler itself. As a result
>= >=3B the cm3 compiler (and PM3's m3build) are normally very very slow whenR>>=3B using user threads. For about the last ten years=2C I've had a hac= >ked
>=3B up m3build (for my PM3 installation) that skips the Pause and= > busy-waits
>=3B instead.
>=3B
>=3B Note there is another = >problem here. Since the Modula-3 runtime ignores
>=3B SIGCHLD=2C no zo= >mbie processes are created since the Unix system automatically
>=3B re= >aps the child processes. I can see this would be a problem since PIDs
&g= >t=3B are eventually reused and ... couldn't Uexec.waitpid wind up referring= > to
>=3B the wrong process??
>=3B
>=3B I will further note = >that the comment in Process.i3 reads as follows:
>=3B
>=3B PROCE= >DURE Wait(p: T): ExitCode=3B
>=3B
>=3B Wait until the process wi= >th handle p terminates=2C then free the operating system resources associat= >ed with the process and return an exit code indicating the reason for its t= >ermination. It is a checked runtime error to call Wait twice on the same pr= >ocess handle.
>=3B
>=3B I am going to take this as fair warning= > that Process.Create *may* use
>=3B resources that are not going to be= > released until Process.Wait has been
>=3B called.
>=3B
>= >=3B I have modified (in my local copy of CM3) the system as follows.
>= >=3B I have come up with a semi-general mechanism for immediately unblocking= >
>=3B a thread on the receipt of a unix signal.
>=3B
>=3B = >1. The system relies on changing
>=3B ThreadPosix.XPause such that if = >a signal is allowed to wake up a threads=2C
>=3B that fact is recorded= > in a new field in the thread's descriptor
>=3B record (of type Thread= >Posix.T).
>=3B
>=3B 2. On receipt of a waited-for unix signal= >=2C a mask is set and control
>=3B is passed to the thread scheduler w= >hich maintains the non-zero mask for
>=3B exactly one iteration throug= >h the thread ring.
>=3B
>=3B 3. If a thread is paused and waitin= >g for EITHER a signal or some time=2C
>=3B the thread is released for = >running and the thread's waiting state is
>=3B cleared.
>=3B >>=3B The changes are more or less as follows:
>=3B
>=3B 1. I = >have added a new field of type "int" to ThreadPosix.T:
>=3B
>=3B= > (* if state =3D pausing=2C the time at which we can restart *)
>=3B w= >aitingForTime: Time.T=3B
>=3B
>=3B + (* if state =3D pausing=2C = >the signal that truncates the pause *)
>=3B + waitingForSig: int :=3D = >-1=3B
>=3B +
>=3B (* true if we are waiting during an AlertWait o= >r AlertJoin
>=3B or AlertPause *)
>=3B alertable: BOOLEAN :=3D FA= >LSE=3B
>=3B
>=3B
>=3B 2. Modifications to pause:
>=3B= >
>=3B + PROCEDURE SigPause(n: LONGREAL=3B sig: int)=3D
>=3B + &l= >t=3B*FATAL Alerted*>=3B
>=3B + VAR until :=3D n + Time.Now ()=3B
= >>=3B + BEGIN
>=3B + XPause(until=2C FALSE=2C sig)=3B
>=3B + END= > SigPause=3B
>=3B +
>=3B PROCEDURE AlertPause(n: LONGREAL) RAISES= > {Alerted}=3D
>=3B VAR until :=3D n + Time.Now ()=3B
>=3B BEGINR>>=3B XPause(until=2C TRUE)=3B
>=3B END AlertPause=3B
>=3B >>=3B ! PROCEDURE XPause (READONLY until: Time.T=3B alertable :=3D FALSE= >=3B sig:int :=3D -1)
>=3B ! RAISES {Alerted} =3D
>=3B BEGIN
&g= >t=3B INC (inCritical)=3B
>=3B self.waitingForTime :=3D until=3B
>= >=3B self.alertable :=3D alertable=3B
>=3B + IF sig # -1 THEN
>=3B= > + self.waitingForSig :=3D sig
>=3B + END=3B
>=3B ICannotRun (Sta= >te.pausing)=3B
>=3B DEC (inCritical)=3B
>=3B InternalYield ()=3B<= >BR>>=3B
>=3B 3. The received-signals mask:
>=3B
>=3B ! C= >ONST MaxSigs =3D 64=3B
>=3B ! TYPE Sig =3D [ 0..MaxSigs-1 ]=3B
>= >=3B !
>=3B ! (* in order to listen to other signals=2C they have to be= > enabled in
>=3B ! allow_sigvtalrm as well *)
>=3B ! VAR (*CONST*= >) SIGCHLD :=3D ValueOfSIGCHLD()=3B
>=3B !
>=3B ! gotSigs :=3D SET= > OF Sig { }=3B
>=3B !
>=3B
>=3B ValueOfSIGCHLD() is a C fun= >ction used to get the value of the SIGCHLD
>=3B constant without guess= >ing at it (in ThreadPosixC.c).
>=3B
>=3B 4. changes to the signa= >l handler:
>=3B
>=3B ! PROCEDURE switch_thread (sig: int) RAISES= > {Alerted} =3D
>=3B BEGIN
>=3B allow_sigvtalrm ()=3B
>=3B !<= >BR>>=3B ! INC(inCritical)=3B
>=3B ! (* mark signal as being delivere= >d *)
>=3B ! IF sig >=3B=3D 0 AND sig <=3B MaxSigs THEN
>=3B != > gotSigs :=3D gotSigs + SET OF Sig { sig }
>=3B ! END=3B
>=3B ! D= >EC(inCritical)=3B
>=3B !
>=3B ! IF inCritical =3D 0 AND heapState= >.inCritical =3D 0 THEN
>=3B ! InternalYield ()
>=3B ! END=3B
&= >gt=3B END switch_thread=3B
>=3B
>=3B Note that I don't know if I= >NC/DEC(inCritical) does exactly the right
>=3B thing here.
>=3B <= >BR>>=3B 5. changes to the scheduler:
>=3B
>=3B a. thread wakeu= >p
>=3B IF t.alertable AND t.alertPending THEN
>=3B CanRun (t)=3B<= >BR>>=3B EXIT=3B
>=3B +
>=3B + ELSIF t.waitingForSig IN gotSigs = >THEN
>=3B + t.waitingForSig :=3D -1=3B
>=3B + CanRun(t)=3B
>= >=3B + EXIT=3B
>=3B
>=3B ELSIF t.waitingForTime <=3B=3D now THE= >N
>=3B CanRun (t)=3B
>=3B EXIT=3B
>=3B !
>=3B
>= >=3B b. clearing the mask
>=3B
>=3B END=3B
>=3B END=3B
&g= >t=3B
>=3B + gotSigs :=3D SET OF Sig {}=3B
>=3B +
>=3B IF t.= >state =3D State.alive AND (scanned OR NOT someBlocking) THEN
>=3B IF p= >erfOn THEN PerfRunning (t.id)=3B END=3B
>=3B (* At least one thread wa= >nts to run=3B transfer to it *)
>=3B
>=3B 6. changes to WaitProc= >ess (Process.Wait):
>=3B
>=3B PROCEDURE WaitProcess (pid: int=3B= > VAR status: int): int =3D
>=3B (* ThreadPThread.m3 and ThreadPosix.m3= > are very similar. *)
>=3B ! CONST Delay =3D 10.0D0=3B
>=3B BEGIN= >
>=3B LOOP
>=3B WITH r =3D Uexec.waitpid(pid=2C ADR(status)=2C Ue= >xec.WNOHANG) DO
>=3B IF r # 0 THEN RETURN r END=3B
>=3B END=3B>>=3B ! SigPause(Delay=2CSIGCHLD)=3B
>=3B END=3B
>=3B END WaitP= >rocess=3B
>=3B
>=3B 7. install signal handler even if program is= > single-threaded:
>=3B
>=3B BEGIN
>=3B + (* we need to call= > set up the signal handler for other reasons than
>=3B + just thread s= >witching now *)
>=3B + setup_sigvtalrm (switch_thread)=3B
>=3B EN= >D ThreadPosix.
>=3B
>=3B 8. modify signal handler in ThreadPosix= >C.c to catch SIGCHLD:
>=3B
>=3B sigemptyset(&=3BThreadSwitchS= >ignal)=3B
>=3B sigaddset(&=3BThreadSwitchSignal=2C SIG_TIMESLICE)= >=3B
>=3B + sigaddset(&=3BThreadSwitchSignal=2C SIGCHLD)=3B
>= >=3B
>=3B act.sa_handler =3D handler=3B
>=3B act.sa_flags =3D SA_= >RESTART=3B
>=3B sigemptyset(&=3B(act.sa_mask))=3B
>=3B if (sig= >action (SIG_TIMESLICE=2C &=3Bact=2C NULL)) abort()=3B
>=3B + if (si= >gaction (SIGCHLD=2C &=3Bact=2C NULL)) abort()=3B
>=3B
>=3B I'= >ll send the complete diff in a separate message for those who want
>= >=3B to study it more closely.
>=3B
>=3B I propose the above chan= >ges for inclusion in the current CM3 repository.
>=3B
>=3B Mika<= >BR>>=3B
>= > >--_5ec25af1-29a7-4f8a-b7ad-e61068941159_-- From mika at async.caltech.edu Sun Feb 13 08:04:14 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 23:04:14 -0800 Subject: [M3devel] More on threading Message-ID: <20110213070414.DD0341A2078@async.async.caltech.edu> Hi again m3devel, Well I thought I would relate another story having to do with threading and CM3. It's half-baked at the moment, but it only reflects a few hours' work on my part, sorry... I'm about to give up now because I don't know if the problem is in my code, or in threading, or what... maybe it is the atfork problem Jay talked about? Programmers spend a lot of time waiting for their compilers to run, so it would be nice if the compilation process were faster. After perusing the source code for the CM3 I realized that there is no feedback at all from the back-end and assembler steps to the rest of the compiler, and I believe that with the GCC back-end the vast majority of the runtime of the compiler is spent in these stages. So.... how hard could this be to parallelize? I decided to try making the two stages (cm3cg1 and as) into "promises" rather than running them on the spot. The QMachine adds a promise to call the back-end to a RefSeq.T called "promises", to which the Builder can also add promises. Then, after making all the promises, collect them and run them ("force them" a true Schemer would say) in parallel. Of course something goes wrong somewhere. With 10 threads I am able to run 31 seconds of CPU time in 19 seconds of wallclock time (using an average 160% of CPU) but then something goes wrong and my process is stuck in state "uxmxn". This is using user threads on AMD64_FREEBSD on a machine with four CPUs. Mika P.S. diff attached, it's a bit ugly, but maybe it works on some other system.... the code works just fine if you run one thread at a time. I.e., if you change IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END; to IF threads.size() > 0 THEN EVAL Thread.Join(threads.remlo()) END; Index: cm3/src/Builder.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/cm3/src/Builder.m3,v retrieving revision 1.36 diff -c -r1.36 Builder.m3 *** cm3/src/Builder.m3 24 Aug 2010 05:24:24 -0000 1.36 --- cm3/src/Builder.m3 13 Feb 2011 06:47:40 -0000 *************** *** 15,20 **** --- 15,22 ---- IMPORT QIdent; FROM Target IMPORT M3BackendMode_t, BackendAssembly, BackendModeStrings; FROM M3Path IMPORT OSKind, OSKindStrings; + IMPORT Pathname; + IMPORT RefSeq; TYPE UK = M3Unit.Kind; *************** *** 131,136 **** --- 133,139 ---- link_coverage : TEXT; (* coverage library *) m3_front_flags: Arg.List; (* configuration options for the front *) m3_options : Arg.List; (* misc. user options for the frontend *) + delayBackend := FALSE; END; TYPE *************** *** 932,937 **** --- 935,956 ---- (*------------------------------------------------------------ compilation --*) + TYPE MarkerPromise = QMachine.Promise BRANDED OBJECT OVERRIDES fulfil := FulfilNothing END; + + PROCEDURE FulfilNothing(<*UNUSED*>mp : MarkerPromise) = BEGIN END FulfilNothing; + + TYPE SeqClosure = Thread.Closure OBJECT seq : RefSeq.T; OVERRIDES apply := SeqApply END; + + PROCEDURE SeqApply(cl : SeqClosure) : REFANY = + BEGIN + FOR i := 0 TO cl.seq.size()-1 DO + WITH p = NARROW(cl.seq.get(i),QMachine.Promise) DO + p.fulfil() + END + END; + RETURN NIL + END SeqApply; + PROCEDURE CompileEverything (s: State; schedule: SourceList) = VAR u: M3Unit.T; BEGIN *************** *** 941,948 **** (* compile all the sources using the initial schedule *) FOR i := 0 TO LAST (schedule^) DO ! CompileOne (s, schedule[i]); END; FlushPending (s); (* recompile any interfaces where we goofed on the exports *) --- 960,1000 ---- (* compile all the sources using the initial schedule *) FOR i := 0 TO LAST (schedule^) DO ! s.delayBackend := TRUE; ! TRY ! CompileOne (s, schedule[i]); ! FINALLY ! s.delayBackend := FALSE; ! END; ! ! s.machine.promises.addhi(NEW(MarkerPromise)); ! ! END; ! ! VAR ! curSeq := NEW(RefSeq.T).init(); ! threads := NEW(RefSeq.T).init(); ! BEGIN ! FOR i := 0 TO s.machine.promises.size()-1 DO ! WITH p = s.machine.promises.get(i) DO ! curSeq.addhi(p); ! IF i = s.machine.promises.size()-1 OR ISTYPE(p,MarkerPromise) THEN ! WITH cl = NEW(SeqClosure, seq := curSeq) DO ! threads.addhi (Thread.Fork(cl)); ! ! IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END; ! ! curSeq := NEW(RefSeq.T).init() ! END ! END ! END ! END; ! WHILE threads.size() > 0 DO EVAL Thread.Join(threads.remlo()) END; END; + + EVAL s.machine.promises.init(); + + FlushPending (s); (* recompile any interfaces where we goofed on the exports *) *************** *** 1151,1156 **** --- 1203,1227 ---- END; END CompileM3; + TYPE + NotePromise = QMachine.Promise OBJECT + nam : Pathname.T; + OVERRIDES + fulfil := FulfilNP; + END; + + RemovePromise = QMachine.Promise OBJECT + nam : Pathname.T; + OVERRIDES + fulfil := FulfilRP; + END; + + PROCEDURE FulfilNP(np : NotePromise) = + BEGIN Utils.NoteTempFile(np.nam) END FulfilNP; + + PROCEDURE FulfilRP(rp : RemovePromise) = + BEGIN Utils.Remove(rp.nam) END FulfilRP; + PROCEDURE PushOneM3 (s: State; u: M3Unit.T): BOOLEAN = VAR tmpC, tmpS: TEXT; *************** *** 1191,1208 **** | 3 => (* -bootstrap, +m3back, +asm *) tmpC := TempCName (u); tmpS := TempSName (u); IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; IF RunM3 (s, u, tmpC) THEN ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) ! AND RunAsm (s, tmpS, u.object) THEN END; need_merge := TRUE; END; IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; | 6, (* +bootstrap, +m3back, -asm *) 7 => (* +bootstrap, +m3back, +asm *) --- 1262,1318 ---- | 3 => (* -bootstrap, +m3back, +asm *) + IF s.delayBackend THEN tmpC := TempCName (u); tmpS := TempSName (u); + (* IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; + *) + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(NotePromise, nam := tmpC)) + END; + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(NotePromise, nam := tmpS)) + END; + IF RunM3 (s, u, tmpC) THEN ! s.machine.record(TRUE); ! TRY ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) ! AND RunAsm (s, tmpS, u.object) THEN ! END; ! FINALLY ! s.machine.record(FALSE) END; + need_merge := TRUE; END; + (* IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; + *) + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpC)) + END; + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpS)) + END; + + ELSE + tmpC := TempCName (u); + tmpS := TempSName (u); + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; + IF RunM3 (s, u, tmpC) THEN + IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) + AND RunAsm (s, tmpS, u.object) THEN + END; + need_merge := TRUE; + END; + IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; + IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; + END | 6, (* +bootstrap, +m3back, -asm *) 7 => (* +bootstrap, +m3back, +asm *) Index: cm3/src/m3makefile =================================================================== RCS file: /usr/cvs/cm3/m3-sys/cm3/src/m3makefile,v retrieving revision 1.24 diff -c -r1.24 m3makefile *** cm3/src/m3makefile 8 Dec 2010 07:30:57 -0000 1.24 --- cm3/src/m3makefile 13 Feb 2011 06:47:42 -0000 *************** *** 29,34 **** --- 29,35 ---- end module ("Arg") module ("Builder") module ("Dirs") module ("M3Build") module ("M3Loc") Index: m3quake/src/QMachine.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.i3,v retrieving revision 1.6 diff -c -r1.6 QMachine.i3 *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -0000 1.6 --- m3quake/src/QMachine.i3 13 Feb 2011 06:47:43 -0000 *************** *** 8,13 **** --- 8,14 ---- IMPORT Thread, Wr, QValue, QCode; FROM Quake IMPORT Machine, Error, ID, IDMap; + IMPORT RefSeq; REVEAL T <: T_; *************** *** 15,20 **** --- 16,22 ---- T = Machine; T_ = OBJECT map: IDMap := NIL; (* READONLY *) + promises : RefSeq.T; METHODS init (map: IDMap): T; evaluate (s: QCode.Stream) RAISES {Error, Thread.Alerted}; *************** *** 37,42 **** --- 39,46 ---- set_wr (wr: Wr.T); exec_echo (b: BOOLEAN): BOOLEAN; trace (b: BOOLEAN); + + record(on : BOOLEAN); (* instead of performing certain acts, promise *) END; PROCEDURE PushBool (t: T; b: BOOLEAN); *************** *** 51,54 **** --- 55,63 ---- PROCEDURE GetEnv (default, v0, v1, v2, v3, v4: TEXT := NIL): TEXT; + TYPE Promise = OBJECT METHODS fulfil() RAISES { Error } END; + END QMachine. + + + Index: m3quake/src/QMachine.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3,v retrieving revision 1.35 diff -c -r1.35 QMachine.m3 *** m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35 --- m3quake/src/QMachine.m3 13 Feb 2011 06:47:44 -0000 *************** *** 16,21 **** --- 16,22 ---- IMPORT TextUtils, FSUtils, System, DirStack; (* sysutils *) IMPORT Compiler; IMPORT M3Path; + IMPORT RefSeq; CONST OnUnix = (Compiler.ThisOS = Compiler.OS.POSIX); *************** *** 44,49 **** --- 45,52 ---- shell : TEXT := NIL; sh_option : TEXT := NIL; tmp_dir : TEXT := NIL; + + doRecord := FALSE; OVERRIDES init := Init; evaluate := Evaluate; *************** *** 66,73 **** --- 69,81 ---- set_wr := SetWr; exec_echo := ExecEcho; trace := Trace; + + record := Record; END; + PROCEDURE Record(t : T; on : BOOLEAN) = + BEGIN t.doRecord := on END Record; + TYPE Registers = RECORD cp : QCode.Stream := NIL; (* code pointer *) *************** *** 139,144 **** --- 147,154 ---- t.globals := NEW (IntRefTbl.Default).init (); t.default_wr := Stdio.stdout; + t.promises := NEW(RefSeq.T).init(); + InitOSEnv (t); InitBuiltins (t); *************** *** 1555,1564 **** END; ELSE FlushIO (); ! Process.GetStandardFileHandles (stdin, stdout, stderr); ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), ! stdin := stdin, stdout := stdout, ! stderr := stderr); END; EXCEPT | Thread.Alerted => --- 1565,1594 ---- END; ELSE FlushIO (); ! IF t.doRecord THEN ! handle := NIL; ! WITH a = NEW(REF ARRAY OF TEXT, n_shell_args) DO ! a^ := SUBARRAY(args,0,n_shell_args); ! VAR wrx : Wr.T; BEGIN ! IF echo OR t.do_echo THEN ! wrx := wr ! ELSE ! wrx := NIL ! END; ! t.promises.addhi(NEW(ExecPromise, ! cmd := t.shell, ! wr := wrx, ! args := a, ! t := t, ! ignore_errors := ignore_errors)) ! END ! END ! ELSE ! Process.GetStandardFileHandles (stdin, stdout, stderr); ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), ! stdin := stdin, stdout := stdout, ! stderr := stderr); ! END; END; EXCEPT | Thread.Alerted => *************** *** 1573,1579 **** END; (* wait for everything to shutdown... *) ! exit_code := Process.Wait (handle); END; IF onlyTry THEN --- 1603,1613 ---- END; (* wait for everything to shutdown... *) ! IF handle = NIL THEN ! exit_code := 0 ! ELSE ! exit_code := Process.Wait (handle); ! END; (* else we're only promising *) END; IF onlyTry THEN *************** *** 1589,1594 **** --- 1623,1664 ---- END ExecCommand; + TYPE + ExecPromise = Promise OBJECT + cmd : TEXT; + args : REF ARRAY OF TEXT; + t : T; + wr : Wr.T; + ignore_errors : BOOLEAN; + OVERRIDES + fulfil := FulfilExecPromise; + END; + + PROCEDURE FulfilExecPromise(ep : ExecPromise) RAISES { Error } = + VAR + stdin, stdout, stderr: File.T; + BEGIN + Process.GetStandardFileHandles (stdin, stdout, stderr); + TRY + IF ep.wr # NIL THEN + Wr.PutText (ep.wr, ep.args[1]); + Wr.PutText (ep.wr, Wr.EOL); + FlushIO (); + END; + WITH handle = Process.Create (ep.cmd, ep.args^, + stdin := stdin, stdout := stdout, + stderr := stderr), + exit_code = Process.Wait(handle) DO + IF exit_code # 0 AND NOT ep.ignore_errors THEN + Err (ep.t, Fmt.F("exit %s: %s", Fmt.Int(exit_code), ep.cmd)) + END + END + EXCEPT + OSError.E (ec) => + Err (ep.t, Fmt.F ("exec failed%s *** %s", OSErr (ec), ep.cmd)); + END + END FulfilExecPromise; + PROCEDURE KillProcess (handle: Process.T) = BEGIN IF (handle # NIL) THEN From jay.krell at cornell.edu Sun Feb 13 08:41:23 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 13 Feb 2011 07:41:23 +0000 Subject: [M3devel] More on threading In-Reply-To: <20110213070414.DD0341A2078@async.async.caltech.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> Message-ID: Yes, gcc is slow and using as is wasteful. Yes this should all be multi-threaded. However it seems to me that m3front has many globals and therefore really making things multi-threaded isn't going to be easy. I think you are mostly wasting your time on user threads though. Have you tried the NT/x86 system? It is very fast. In the gcc system, each .mc file is run through m3cgc1 to produce an .ms file, which is run through as to produce an object file .mo. In the NT/x86 system, cm3 outputs .obj files directly. No .mc files. No m3cgc1 or as. Granted, I'd just as soon output .c files, and hope for a C compiler that doesn't run a separate assembler. My timing might loosen up a bit and I hope to spend some on the thread stress tester using pthreads. Next weekend. Am I right though that Darwin works fine? Solaris? FreeBSD? NetBSD? OpenBSD? Linux? I know, not Linux. You generally don't need any of these systems. All of the x86/AMD64 systems can run in a virtual machine. Solaris/sparc we have good access. - Jay > To: m3devel at elegosoft.com > Date: Sat, 12 Feb 2011 23:04:14 -0800 > From: mika at async.caltech.edu > CC: jay.krell at cornell.edu > Subject: [M3devel] More on threading > > > Hi again m3devel, > > Well I thought I would relate another story having to do with threading > and CM3. It's half-baked at the moment, but it only reflects a few hours' > work on my part, sorry... I'm about to give up now because I don't know > if the problem is in my code, or in threading, or what... maybe it is > the atfork problem Jay talked about? > > Programmers spend a lot of time waiting for their compilers to run, so > it would be nice if the compilation process were faster. After perusing > the source code for the CM3 I realized that there is no feedback at all > from the back-end and assembler steps to the rest of the compiler, and > I believe that with the GCC back-end the vast majority of the runtime > of the compiler is spent in these stages. > > So.... how hard could this be to parallelize? I decided to try making > the two stages (cm3cg1 and as) into "promises" rather than running them > on the spot. The QMachine adds a promise to call the back-end to a > RefSeq.T called "promises", to which the Builder can also add promises. > Then, after making all the promises, collect them and run them ("force > them" a true Schemer would say) in parallel. > > Of course something goes wrong somewhere. With 10 threads I am able to > run 31 seconds of CPU time in 19 seconds of wallclock time (using an > average 160% of CPU) but then something goes wrong and my process > is stuck in state "uxmxn". This is using user threads on AMD64_FREEBSD > on a machine with four CPUs. > > Mika > > P.S. diff attached, it's a bit ugly, but maybe it works on some other > system.... the code works just fine if you run one thread at a time. > I.e., if you change > > IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END; > > to > > IF threads.size() > 0 THEN EVAL Thread.Join(threads.remlo()) END; > > > > Index: cm3/src/Builder.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/cm3/src/Builder.m3,v > retrieving revision 1.36 > diff -c -r1.36 Builder.m3 > *** cm3/src/Builder.m3 24 Aug 2010 05:24:24 -0000 1.36 > --- cm3/src/Builder.m3 13 Feb 2011 06:47:40 -0000 > *************** > *** 15,20 **** > --- 15,22 ---- > IMPORT QIdent; > FROM Target IMPORT M3BackendMode_t, BackendAssembly, BackendModeStrings; > FROM M3Path IMPORT OSKind, OSKindStrings; > + IMPORT Pathname; > + IMPORT RefSeq; > > TYPE > UK = M3Unit.Kind; > *************** > *** 131,136 **** > --- 133,139 ---- > link_coverage : TEXT; (* coverage library *) > m3_front_flags: Arg.List; (* configuration options for the front *) > m3_options : Arg.List; (* misc. user options for the frontend *) > + delayBackend := FALSE; > END; > > TYPE > *************** > *** 932,937 **** > --- 935,956 ---- > > (*------------------------------------------------------------ compilation --*) > > + TYPE MarkerPromise = QMachine.Promise BRANDED OBJECT OVERRIDES fulfil := FulfilNothing END; > + > + PROCEDURE FulfilNothing(<*UNUSED*>mp : MarkerPromise) = BEGIN END FulfilNothing; > + > + TYPE SeqClosure = Thread.Closure OBJECT seq : RefSeq.T; OVERRIDES apply := SeqApply END; > + > + PROCEDURE SeqApply(cl : SeqClosure) : REFANY = > + BEGIN > + FOR i := 0 TO cl.seq.size()-1 DO > + WITH p = NARROW(cl.seq.get(i),QMachine.Promise) DO > + p.fulfil() > + END > + END; > + RETURN NIL > + END SeqApply; > + > PROCEDURE CompileEverything (s: State; schedule: SourceList) = > VAR u: M3Unit.T; > BEGIN > *************** > *** 941,948 **** > > (* compile all the sources using the initial schedule *) > FOR i := 0 TO LAST (schedule^) DO > ! CompileOne (s, schedule[i]); > END; > FlushPending (s); > > (* recompile any interfaces where we goofed on the exports *) > --- 960,1000 ---- > > (* compile all the sources using the initial schedule *) > FOR i := 0 TO LAST (schedule^) DO > ! s.delayBackend := TRUE; > ! TRY > ! CompileOne (s, schedule[i]); > ! FINALLY > ! s.delayBackend := FALSE; > ! END; > ! > ! s.machine.promises.addhi(NEW(MarkerPromise)); > ! > ! END; > ! > ! VAR > ! curSeq := NEW(RefSeq.T).init(); > ! threads := NEW(RefSeq.T).init(); > ! BEGIN > ! FOR i := 0 TO s.machine.promises.size()-1 DO > ! WITH p = s.machine.promises.get(i) DO > ! curSeq.addhi(p); > ! IF i = s.machine.promises.size()-1 OR ISTYPE(p,MarkerPromise) THEN > ! WITH cl = NEW(SeqClosure, seq := curSeq) DO > ! threads.addhi (Thread.Fork(cl)); > ! > ! IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END; > ! > ! curSeq := NEW(RefSeq.T).init() > ! END > ! END > ! END > ! END; > ! WHILE threads.size() > 0 DO EVAL Thread.Join(threads.remlo()) END; > END; > + > + EVAL s.machine.promises.init(); > + > + > FlushPending (s); > > (* recompile any interfaces where we goofed on the exports *) > *************** > *** 1151,1156 **** > --- 1203,1227 ---- > END; > END CompileM3; > > + TYPE > + NotePromise = QMachine.Promise OBJECT > + nam : Pathname.T; > + OVERRIDES > + fulfil := FulfilNP; > + END; > + > + RemovePromise = QMachine.Promise OBJECT > + nam : Pathname.T; > + OVERRIDES > + fulfil := FulfilRP; > + END; > + > + PROCEDURE FulfilNP(np : NotePromise) = > + BEGIN Utils.NoteTempFile(np.nam) END FulfilNP; > + > + PROCEDURE FulfilRP(rp : RemovePromise) = > + BEGIN Utils.Remove(rp.nam) END FulfilRP; > + > PROCEDURE PushOneM3 (s: State; u: M3Unit.T): BOOLEAN = > VAR > tmpC, tmpS: TEXT; > *************** > *** 1191,1208 **** > > > | 3 => (* -bootstrap, +m3back, +asm *) > tmpC := TempCName (u); > tmpS := TempSName (u); > IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; > IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; > IF RunM3 (s, u, tmpC) THEN > ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) > ! AND RunAsm (s, tmpS, u.object) THEN > END; > need_merge := TRUE; > END; > IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; > IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; > > | 6, (* +bootstrap, +m3back, -asm *) > 7 => (* +bootstrap, +m3back, +asm *) > --- 1262,1318 ---- > > > | 3 => (* -bootstrap, +m3back, +asm *) > + IF s.delayBackend THEN > tmpC := TempCName (u); > tmpS := TempSName (u); > + (* > IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; > IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; > + *) > + IF (NOT s.keep_files) THEN > + s.machine.promises.addhi(NEW(NotePromise, nam := tmpC)) > + END; > + IF (NOT s.keep_files) THEN > + s.machine.promises.addhi(NEW(NotePromise, nam := tmpS)) > + END; > + > IF RunM3 (s, u, tmpC) THEN > ! s.machine.record(TRUE); > ! TRY > ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) > ! AND RunAsm (s, tmpS, u.object) THEN > ! END; > ! FINALLY > ! s.machine.record(FALSE) > END; > + > need_merge := TRUE; > END; > + (* > IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; > IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; > + *) > + IF (NOT s.keep_files) THEN > + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpC)) > + END; > + IF (NOT s.keep_files) THEN > + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpS)) > + END; > + > + ELSE > + tmpC := TempCName (u); > + tmpS := TempSName (u); > + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; > + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; > + IF RunM3 (s, u, tmpC) THEN > + IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) > + AND RunAsm (s, tmpS, u.object) THEN > + END; > + need_merge := TRUE; > + END; > + IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; > + IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; > + END > > | 6, (* +bootstrap, +m3back, -asm *) > 7 => (* +bootstrap, +m3back, +asm *) > Index: cm3/src/m3makefile > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/cm3/src/m3makefile,v > retrieving revision 1.24 > diff -c -r1.24 m3makefile > *** cm3/src/m3makefile 8 Dec 2010 07:30:57 -0000 1.24 > --- cm3/src/m3makefile 13 Feb 2011 06:47:42 -0000 > *************** > *** 29,34 **** > --- 29,35 ---- > end > module ("Arg") > module ("Builder") > module ("Dirs") > module ("M3Build") > module ("M3Loc") > Index: m3quake/src/QMachine.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.i3,v > retrieving revision 1.6 > diff -c -r1.6 QMachine.i3 > *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -0000 1.6 > --- m3quake/src/QMachine.i3 13 Feb 2011 06:47:43 -0000 > *************** > *** 8,13 **** > --- 8,14 ---- > > IMPORT Thread, Wr, QValue, QCode; > FROM Quake IMPORT Machine, Error, ID, IDMap; > + IMPORT RefSeq; > > REVEAL > T <: T_; > *************** > *** 15,20 **** > --- 16,22 ---- > T = Machine; > T_ = OBJECT > map: IDMap := NIL; (* READONLY *) > + promises : RefSeq.T; > METHODS > init (map: IDMap): T; > evaluate (s: QCode.Stream) RAISES {Error, Thread.Alerted}; > *************** > *** 37,42 **** > --- 39,46 ---- > set_wr (wr: Wr.T); > exec_echo (b: BOOLEAN): BOOLEAN; > trace (b: BOOLEAN); > + > + record(on : BOOLEAN); (* instead of performing certain acts, promise *) > END; > > PROCEDURE PushBool (t: T; b: BOOLEAN); > *************** > *** 51,54 **** > --- 55,63 ---- > > PROCEDURE GetEnv (default, v0, v1, v2, v3, v4: TEXT := NIL): TEXT; > > + TYPE Promise = OBJECT METHODS fulfil() RAISES { Error } END; > + > END QMachine. > + > + > + > Index: m3quake/src/QMachine.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3,v > retrieving revision 1.35 > diff -c -r1.35 QMachine.m3 > *** m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35 > --- m3quake/src/QMachine.m3 13 Feb 2011 06:47:44 -0000 > *************** > *** 16,21 **** > --- 16,22 ---- > IMPORT TextUtils, FSUtils, System, DirStack; (* sysutils *) > IMPORT Compiler; > IMPORT M3Path; > + IMPORT RefSeq; > > CONST > OnUnix = (Compiler.ThisOS = Compiler.OS.POSIX); > *************** > *** 44,49 **** > --- 45,52 ---- > shell : TEXT := NIL; > sh_option : TEXT := NIL; > tmp_dir : TEXT := NIL; > + > + doRecord := FALSE; > OVERRIDES > init := Init; > evaluate := Evaluate; > *************** > *** 66,73 **** > --- 69,81 ---- > set_wr := SetWr; > exec_echo := ExecEcho; > trace := Trace; > + > + record := Record; > END; > > + PROCEDURE Record(t : T; on : BOOLEAN) = > + BEGIN t.doRecord := on END Record; > + > TYPE > Registers = RECORD > cp : QCode.Stream := NIL; (* code pointer *) > *************** > *** 139,144 **** > --- 147,154 ---- > t.globals := NEW (IntRefTbl.Default).init (); > t.default_wr := Stdio.stdout; > > + t.promises := NEW(RefSeq.T).init(); > + > InitOSEnv (t); > InitBuiltins (t); > > *************** > *** 1555,1564 **** > END; > ELSE > FlushIO (); > ! Process.GetStandardFileHandles (stdin, stdout, stderr); > ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), > ! stdin := stdin, stdout := stdout, > ! stderr := stderr); > END; > EXCEPT > | Thread.Alerted => > --- 1565,1594 ---- > END; > ELSE > FlushIO (); > ! IF t.doRecord THEN > ! handle := NIL; > ! WITH a = NEW(REF ARRAY OF TEXT, n_shell_args) DO > ! a^ := SUBARRAY(args,0,n_shell_args); > ! VAR wrx : Wr.T; BEGIN > ! IF echo OR t.do_echo THEN > ! wrx := wr > ! ELSE > ! wrx := NIL > ! END; > ! t.promises.addhi(NEW(ExecPromise, > ! cmd := t.shell, > ! wr := wrx, > ! args := a, > ! t := t, > ! ignore_errors := ignore_errors)) > ! END > ! END > ! ELSE > ! Process.GetStandardFileHandles (stdin, stdout, stderr); > ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), > ! stdin := stdin, stdout := stdout, > ! stderr := stderr); > ! END; > END; > EXCEPT > | Thread.Alerted => > *************** > *** 1573,1579 **** > END; > > (* wait for everything to shutdown... *) > ! exit_code := Process.Wait (handle); > END; > > IF onlyTry THEN > --- 1603,1613 ---- > END; > > (* wait for everything to shutdown... *) > ! IF handle = NIL THEN > ! exit_code := 0 > ! ELSE > ! exit_code := Process.Wait (handle); > ! END; (* else we're only promising *) > END; > > IF onlyTry THEN > *************** > *** 1589,1594 **** > --- 1623,1664 ---- > > END ExecCommand; > > + TYPE > + ExecPromise = Promise OBJECT > + cmd : TEXT; > + args : REF ARRAY OF TEXT; > + t : T; > + wr : Wr.T; > + ignore_errors : BOOLEAN; > + OVERRIDES > + fulfil := FulfilExecPromise; > + END; > + > + PROCEDURE FulfilExecPromise(ep : ExecPromise) RAISES { Error } = > + VAR > + stdin, stdout, stderr: File.T; > + BEGIN > + Process.GetStandardFileHandles (stdin, stdout, stderr); > + TRY > + IF ep.wr # NIL THEN > + Wr.PutText (ep.wr, ep.args[1]); > + Wr.PutText (ep.wr, Wr.EOL); > + FlushIO (); > + END; > + WITH handle = Process.Create (ep.cmd, ep.args^, > + stdin := stdin, stdout := stdout, > + stderr := stderr), > + exit_code = Process.Wait(handle) DO > + IF exit_code # 0 AND NOT ep.ignore_errors THEN > + Err (ep.t, Fmt.F("exit %s: %s", Fmt.Int(exit_code), ep.cmd)) > + END > + END > + EXCEPT > + OSError.E (ec) => > + Err (ep.t, Fmt.F ("exec failed%s *** %s", OSErr (ec), ep.cmd)); > + END > + END FulfilExecPromise; > + > PROCEDURE KillProcess (handle: Process.T) = > BEGIN > IF (handle # NIL) THEN > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 13 19:17:17 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 10:17:17 -0800 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu> Message-ID: <20110213181717.4F0E91A2078@async.async.caltech.edu> Jay K writes: >--_b09106d1-2a84-4ffe-876c-0b2e4cdfd0b0_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Yes=2C gcc is slow and using as is wasteful. >Yes this should all be multi-threaded. >However it seems to me that m3front has many globals and therefore >really making things multi-threaded isn't going to be easy. OK but as long as the back-end is taking up most of the cycles the property that the output of the front-end is used only by cmcg1 (m3back/m3cgc/whatever you call it) and the assembler means that what I did works for a pretty big speedup. In fact you can see that it works just fine (from the point of view of generating the correct output) since you can re-order running the backend with respect to all the steps done by the front-end. What you lose is (perhaps) some error messages, and of course, in general, any implicit dependencies between the error messages. I don't understand why the program gets stuck in uxmxn though. Maybe a problem with, sigh, user threads... >I think you are mostly wasting your time on user threads though. I hope so... I too would really like to use pthreads. > > >Have you tried the NT/x86 system? >It is very fast. > > >In the gcc system=2C each .mc file is run through m3cgc1 to produce an .ms = >file=2C >which is run through as to produce an object file .mo. > > >In the NT/x86 system=2C cm3 outputs .obj files directly. >No .mc files. No m3cgc1 or as. >Granted=2C I'd just as soon output .c files=2C and hope for a C compiler th= >at doesn't run a separate assembler. The NT compiler sounds faster but of course it's easier to parallelize the unix one using the method I outlined :-) Well unless you're using pthreads of course, then you can just fork a new thread for the integrated NT backend as long as it doesn't share too many variables with the front-end. But no I don't use Windows. I was asking on here how to get started with Modula-3 on Windows once but my machine was/is a Windows 2000 system and apparently CM3 no longer works on win2k..? I used to use PM3/Klagenfurt a lot on Windows (Cygwin) but thankfully my clients on that project let me switch the system to Unix so I don't have to touch that anymore. > > >My timing might loosen up a bit and I hope to spend some on the thread stre= >ss tester using pthreads. Next weekend. >Am I right though that Darwin works fine? >Solaris? FreeBSD? NetBSD? OpenBSD? Linux? I know=2C not Linux. >You generally don't need any of these systems. That would be great. I don't think pthreads works properly anywhere at the moment. My guess is that it never has, if you stress it hard enough. I first ran into problems on Darwin and then switched to AMD64 simply because I have better access to faster machines with AMD64. At first there were some obvious problems that Tony fixed quickly but then I kept running into odd behavior with my application and I added various other tests until I could reproduce everything I'd seen go wrong with my application, and I think at that point I was seeing problems on every platform (except user threads). The uxmxn problem I see with user threads on FreeBSD suggests that perhaps I need to add some more tests, however. Mika >All of the x86/AMD64 systems can run in a virtual machine. >Solaris/sparc we have good access. > > > - Jay > > >> To: m3devel at elegosoft.com >> Date: Sat=2C 12 Feb 2011 23:04:14 -0800 >> From: mika at async.caltech.edu >> CC: jay.krell at cornell.edu >> Subject: [M3devel] More on threading >>=20 >>=20 >> Hi again m3devel=2C >>=20 >> Well I thought I would relate another story having to do with threading >> and CM3. It's half-baked at the moment=2C but it only reflects a few hou= >rs' >> work on my part=2C sorry... I'm about to give up now because I don't know >> if the problem is in my code=2C or in threading=2C or what... maybe it is >> the atfork problem Jay talked about? >>=20 >> Programmers spend a lot of time waiting for their compilers to run=2C so >> it would be nice if the compilation process were faster. After perusing >> the source code for the CM3 I realized that there is no feedback at all >> from the back-end and assembler steps to the rest of the compiler=2C and >> I believe that with the GCC back-end the vast majority of the runtime >> of the compiler is spent in these stages. >>=20 >> So.... how hard could this be to parallelize? I decided to try making >> the two stages (cm3cg1 and as) into "promises" rather than running them >> on the spot. The QMachine adds a promise to call the back-end to a >> RefSeq.T called "promises"=2C to which the Builder can also add promises. >> Then=2C after making all the promises=2C collect them and run them ("forc= >e >> them" a true Schemer would say) in parallel. >>=20 >> Of course something goes wrong somewhere. With 10 threads I am able to >> run 31 seconds of CPU time in 19 seconds of wallclock time (using an >> average 160% of CPU) but then something goes wrong and my process >> is stuck in state "uxmxn". This is using user threads on AMD64_FREEBSD >> on a machine with four CPUs. >>=20 >> Mika >>=20 >> P.S. diff attached=2C it's a bit ugly=2C but maybe it works on some other >> system.... the code works just fine if you run one thread at a time. >> I.e.=2C if you change >>=20 >> IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END=3B >>=20 >> to >>=20 >> IF threads.size() > 0 THEN EVAL Thread.Join(threads.remlo()) END=3B >>=20 >>=20 >>=20 >> Index: cm3/src/Builder.m3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: /usr/cvs/cm3/m3-sys/cm3/src/Builder.m3=2Cv >> retrieving revision 1.36 >> diff -c -r1.36 Builder.m3 >> *** cm3/src/Builder.m3 24 Aug 2010 05:24:24 -0000 1.36 >> --- cm3/src/Builder.m3 13 Feb 2011 06:47:40 -0000 >> *************** >> *** 15=2C20 **** >> --- 15=2C22 ---- >> IMPORT QIdent=3B >> FROM Target IMPORT M3BackendMode_t=2C BackendAssembly=2C BackendModeStr= >ings=3B >> FROM M3Path IMPORT OSKind=2C OSKindStrings=3B >> + IMPORT Pathname=3B >> + IMPORT RefSeq=3B >> =20 >> TYPE >> UK =3D M3Unit.Kind=3B >> *************** >> *** 131=2C136 **** >> --- 133=2C139 ---- >> link_coverage : TEXT=3B (* coverage library *) >> m3_front_flags: Arg.List=3B (* configuration options for = >the front *) >> m3_options : Arg.List=3B (* misc. user options for the= > frontend *) >> + delayBackend :=3D FALSE=3B >> END=3B >> =20 >> TYPE >> *************** >> *** 932=2C937 **** >> --- 935=2C956 ---- >> =20 >> (*------------------------------------------------------------ compilat= >ion --*) >> =20 >> + TYPE MarkerPromise =3D QMachine.Promise BRANDED OBJECT OVERRIDES fulfil= > :=3D FulfilNothing END=3B >> +=20 >> + PROCEDURE FulfilNothing(<*UNUSED*>mp : MarkerPromise) =3D BEGIN END Ful= >filNothing=3B >> +=20 >> + TYPE SeqClosure =3D Thread.Closure OBJECT seq : RefSeq.T=3B OVERRIDES a= >pply :=3D SeqApply END=3B >> +=20 >> + PROCEDURE SeqApply(cl : SeqClosure) : REFANY =3D >> + BEGIN >> + FOR i :=3D 0 TO cl.seq.size()-1 DO >> + WITH p =3D NARROW(cl.seq.get(i)=2CQMachine.Promise) DO >> + p.fulfil() >> + END >> + END=3B >> + RETURN NIL >> + END SeqApply=3B >> +=20 >> PROCEDURE CompileEverything (s: State=3B schedule: SourceList) =3D >> VAR u: M3Unit.T=3B >> BEGIN >> *************** >> *** 941=2C948 **** >> =20 >> (* compile all the sources using the initial schedule *) >> FOR i :=3D 0 TO LAST (schedule^) DO >> ! CompileOne (s=2C schedule[i])=3B >> END=3B >> FlushPending (s)=3B >> =20 >> (* recompile any interfaces where we goofed on the exports *) >> --- 960=2C1000 ---- >> =20 >> (* compile all the sources using the initial schedule *) >> FOR i :=3D 0 TO LAST (schedule^) DO >> ! s.delayBackend :=3D TRUE=3B >> ! TRY >> ! CompileOne (s=2C schedule[i])=3B >> ! FINALLY >> ! s.delayBackend :=3D FALSE=3B >> ! END=3B >> !=20 >> ! s.machine.promises.addhi(NEW(MarkerPromise))=3B >> !=20 >> ! END=3B >> !=20 >> ! VAR >> ! curSeq :=3D NEW(RefSeq.T).init()=3B >> ! threads :=3D NEW(RefSeq.T).init()=3B >> ! BEGIN >> ! FOR i :=3D 0 TO s.machine.promises.size()-1 DO >> ! WITH p =3D s.machine.promises.get(i) DO >> ! curSeq.addhi(p)=3B >> ! IF i =3D s.machine.promises.size()-1 OR ISTYPE(p=2CMarkerProm= >ise) THEN >> ! WITH cl =3D NEW(SeqClosure=2C seq :=3D curSeq) DO >> ! threads.addhi (Thread.Fork(cl))=3B >> !=20 >> ! IF threads.size() > 10 THEN EVAL Thread.Join(threads.reml= >o()) END=3B >> ! =20 >> ! curSeq :=3D NEW(RefSeq.T).init() >> ! END >> ! END >> ! END >> ! END=3B >> ! WHILE threads.size() > 0 DO EVAL Thread.Join(threads.remlo()) END= >=3B >> END=3B >> + =20 >> + EVAL s.machine.promises.init()=3B >> +=20 >> +=20 >> FlushPending (s)=3B >> =20 >> (* recompile any interfaces where we goofed on the exports *) >> *************** >> *** 1151=2C1156 **** >> --- 1203=2C1227 ---- >> END=3B >> END CompileM3=3B >> =20 >> + TYPE >> + NotePromise =3D QMachine.Promise OBJECT >> + nam : Pathname.T=3B >> + OVERRIDES >> + fulfil :=3D FulfilNP=3B >> + END=3B >> +=20 >> + RemovePromise =3D QMachine.Promise OBJECT >> + nam : Pathname.T=3B >> + OVERRIDES >> + fulfil :=3D FulfilRP=3B >> + END=3B >> +=20 >> + PROCEDURE FulfilNP(np : NotePromise) =3D=20 >> + BEGIN Utils.NoteTempFile(np.nam) END FulfilNP=3B >> +=20 >> + PROCEDURE FulfilRP(rp : RemovePromise) =3D=20 >> + BEGIN Utils.Remove(rp.nam) END FulfilRP=3B >> +=20 >> PROCEDURE PushOneM3 (s: State=3B u: M3Unit.T): BOOLEAN =3D >> VAR >> tmpC=2C tmpS: TEXT=3B >> *************** >> *** 1191=2C1208 **** >> =20 >> =20 >> | 3 =3D> (* -bootstrap=2C +m3back=2C +asm *) >> tmpC :=3D TempCName (u)=3B >> tmpS :=3D TempSName (u)=3B >> IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3B >> IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END=3B >> IF RunM3 (s=2C u=2C tmpC) THEN >> ! IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.optimize) >> ! AND RunAsm (s=2C tmpS=2C u.object) THEN >> END=3B >> need_merge :=3D TRUE=3B >> END=3B >> IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END=3B >> IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END=3B >> =20 >> | 6=2C (* +bootstrap=2C +m3back=2C -asm *) >> 7 =3D> (* +bootstrap=2C +m3back=2C +asm *) >> --- 1262=2C1318 ---- >> =20 >> =20 >> | 3 =3D> (* -bootstrap=2C +m3back=2C +asm *) >> + IF s.delayBackend THEN >> tmpC :=3D TempCName (u)=3B >> tmpS :=3D TempSName (u)=3B >> + (* >> IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3B >> IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END=3B >> + *) >> + IF (NOT s.keep_files) THEN=20 >> + s.machine.promises.addhi(NEW(NotePromise=2C nam :=3D tmpC))=20 >> + END=3B >> + IF (NOT s.keep_files) THEN=20 >> + s.machine.promises.addhi(NEW(NotePromise=2C nam :=3D tmpS))=20 >> + END=3B >> +=20 >> IF RunM3 (s=2C u=2C tmpC) THEN >> ! s.machine.record(TRUE)=3B >> ! TRY >> ! IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.optimize) >> ! AND RunAsm (s=2C tmpS=2C u.object) THEN >> ! END=3B >> ! FINALLY >> ! s.machine.record(FALSE) >> END=3B >> +=20 >> need_merge :=3D TRUE=3B >> END=3B >> + (* >> IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END=3B >> IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END=3B >> + *) >> + IF (NOT s.keep_files) THEN=20 >> + s.machine.promises.addhi(NEW(RemovePromise=2C nam :=3D tmpC))= >=20 >> + END=3B >> + IF (NOT s.keep_files) THEN=20 >> + s.machine.promises.addhi(NEW(RemovePromise=2C nam :=3D tmpS))= >=20 >> + END=3B >> +=20 >> + ELSE >> + tmpC :=3D TempCName (u)=3B >> + tmpS :=3D TempSName (u)=3B >> + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3B >> + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END=3B >> + IF RunM3 (s=2C u=2C tmpC) THEN >> + IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.optimize) >> + AND RunAsm (s=2C tmpS=2C u.object) THEN >> + END=3B >> + need_merge :=3D TRUE=3B >> + END=3B >> + IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END=3B >> + IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END=3B >> + END >> =20 >> | 6=2C (* +bootstrap=2C +m3back=2C -asm *) >> 7 =3D> (* +bootstrap=2C +m3back=2C +asm *) >> Index: cm3/src/m3makefile >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: /usr/cvs/cm3/m3-sys/cm3/src/m3makefile=2Cv >> retrieving revision 1.24 >> diff -c -r1.24 m3makefile >> *** cm3/src/m3makefile 8 Dec 2010 07:30:57 -0000 1.24 >> --- cm3/src/m3makefile 13 Feb 2011 06:47:42 -0000 >> *************** >> *** 29=2C34 **** >> --- 29=2C35 ---- >> end >> module ("Arg") >> module ("Builder") >> module ("Dirs") >> module ("M3Build") >> module ("M3Loc") >> Index: m3quake/src/QMachine.i3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.i3=2Cv >> retrieving revision 1.6 >> diff -c -r1.6 QMachine.i3 >> *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -0000 1.6 >> --- m3quake/src/QMachine.i3 13 Feb 2011 06:47:43 -0000 >> *************** >> *** 8=2C13 **** >> --- 8=2C14 ---- >> =20 >> IMPORT Thread=2C Wr=2C QValue=2C QCode=3B >> FROM Quake IMPORT Machine=2C Error=2C ID=2C IDMap=3B >> + IMPORT RefSeq=3B >> =20 >> REVEAL >> T <: T_=3B >> *************** >> *** 15=2C20 **** >> --- 16=2C22 ---- >> T =3D Machine=3B >> T_ =3D OBJECT >> map: IDMap :=3D NIL=3B (* READONLY *) >> + promises : RefSeq.T=3B >> METHODS >> init (map: IDMap): T=3B >> evaluate (s: QCode.Stream) RAISES {Error=2C Th= >read.Alerted}=3B >> *************** >> *** 37=2C42 **** >> --- 39=2C46 ---- >> set_wr (wr: Wr.T)=3B >> exec_echo (b: BOOLEAN): BOOLEAN=3B >> trace (b: BOOLEAN)=3B >> + =20 >> + record(on : BOOLEAN)=3B (* instead of performing certain acts= >=2C promise *) >> END=3B >> =20 >> PROCEDURE PushBool (t: T=3B b: BOOLEAN)=3B >> *************** >> *** 51=2C54 **** >> --- 55=2C63 ---- >> =20 >> PROCEDURE GetEnv (default=2C v0=2C v1=2C v2=2C v3=2C v4: TEXT :=3D NIL)= >: TEXT=3B >> =20 >> + TYPE Promise =3D OBJECT METHODS fulfil() RAISES { Error } END=3B >> +=20 >> END QMachine. >> +=20 >> +=20 >> +=20 >> Index: m3quake/src/QMachine.m3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3=2Cv >> retrieving revision 1.35 >> diff -c -r1.35 QMachine.m3 >> *** m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35 >> --- m3quake/src/QMachine.m3 13 Feb 2011 06:47:44 -0000 >> *************** >> *** 16=2C21 **** >> --- 16=2C22 ---- >> IMPORT TextUtils=2C FSUtils=2C System=2C DirStack=3B (* sysutils *) >> IMPORT Compiler=3B >> IMPORT M3Path=3B >> + IMPORT RefSeq=3B >> =20 >> CONST >> OnUnix =3D (Compiler.ThisOS =3D Compiler.OS.POSIX)=3B >> *************** >> *** 44=2C49 **** >> --- 45=2C52 ---- >> shell : TEXT :=3D NIL=3B >> sh_option : TEXT :=3D NIL=3B >> tmp_dir : TEXT :=3D NIL=3B >> +=20 >> + doRecord :=3D FALSE=3B >> OVERRIDES >> init :=3D Init=3B >> evaluate :=3D Evaluate=3B >> *************** >> *** 66=2C73 **** >> --- 69=2C81 ---- >> set_wr :=3D SetWr=3B >> exec_echo :=3D ExecEcho=3B >> trace :=3D Trace=3B >> +=20 >> + record :=3D Record=3B >> END=3B >> =20 >> + PROCEDURE Record(t : T=3B on : BOOLEAN) =3D=20 >> + BEGIN t.doRecord :=3D on END Record=3B >> +=20 >> TYPE >> Registers =3D RECORD >> cp : QCode.Stream :=3D NIL=3B (* code pointer *) >> *************** >> *** 139=2C144 **** >> --- 147=2C154 ---- >> t.globals :=3D NEW (IntRefTbl.Default).init ()=3B >> t.default_wr :=3D Stdio.stdout=3B >> =20 >> + t.promises :=3D NEW(RefSeq.T).init()=3B >> +=20 >> InitOSEnv (t)=3B >> InitBuiltins (t)=3B >> =20 >> *************** >> *** 1555=2C1564 **** >> END=3B >> ELSE >> FlushIO ()=3B >> ! Process.GetStandardFileHandles (stdin=2C stdout=2C stderr)=3B >> ! handle :=3D Process.Create (t.shell=2C SUBARRAY (args=2C 0=2C= > n_shell_args)=2C >> ! stdin :=3D stdin=2C stdout :=3D std= >out=2C >> ! stderr :=3D stderr)=3B >> END=3B >> EXCEPT >> | Thread.Alerted =3D> >> --- 1565=2C1594 ---- >> END=3B >> ELSE >> FlushIO ()=3B >> ! IF t.doRecord THEN >> ! handle :=3D NIL=3B >> ! WITH a =3D NEW(REF ARRAY OF TEXT=2C n_shell_args) DO >> ! a^ :=3D SUBARRAY(args=2C0=2Cn_shell_args)=3B >> ! VAR wrx : Wr.T=3B BEGIN >> ! IF echo OR t.do_echo THEN >> ! wrx :=3D wr >> ! ELSE >> ! wrx :=3D NIL >> ! END=3B >> ! t.promises.addhi(NEW(ExecPromise=2C >> ! cmd :=3D t.shell=2C >> ! wr :=3D wrx=2C >> ! args :=3D a=2C >> ! t :=3D t=2C >> ! ignore_errors :=3D ignore_errors)) >> ! END >> ! END >> ! ELSE >> ! Process.GetStandardFileHandles (stdin=2C stdout=2C stderr)= >=3B >> ! handle :=3D Process.Create (t.shell=2C SUBARRAY (args=2C 0= >=2C n_shell_args)=2C >> ! stdin :=3D stdin=2C stdout :=3D s= >tdout=2C >> ! stderr :=3D stderr)=3B >> ! END=3B >> END=3B >> EXCEPT >> | Thread.Alerted =3D> >> *************** >> *** 1573=2C1579 **** >> END=3B >> =20 >> (* wait for everything to shutdown... *) >> ! exit_code :=3D Process.Wait (handle)=3B >> END=3B >> =20 >> IF onlyTry THEN >> --- 1603=2C1613 ---- >> END=3B >> =20 >> (* wait for everything to shutdown... *) >> ! IF handle =3D NIL THEN >> ! exit_code :=3D 0=20 >> ! ELSE >> ! exit_code :=3D Process.Wait (handle)=3B >> ! END=3B (* else we're only promising *) >> END=3B >> =20 >> IF onlyTry THEN >> *************** >> *** 1589=2C1594 **** >> --- 1623=2C1664 ---- >> =20 >> END ExecCommand=3B >> =20 >> + TYPE=20 >> + ExecPromise =3D Promise OBJECT >> + cmd : TEXT=3B >> + args : REF ARRAY OF TEXT=3B >> + t : T=3B >> + wr : Wr.T=3B >> + ignore_errors : BOOLEAN=3B >> + OVERRIDES >> + fulfil :=3D FulfilExecPromise=3B >> + END=3B >> +=20 >> + PROCEDURE FulfilExecPromise(ep : ExecPromise) RAISES { Error } =3D=20 >> + VAR >> + stdin=2C stdout=2C stderr: File.T=3B >> + BEGIN >> + Process.GetStandardFileHandles (stdin=2C stdout=2C stderr)=3B >> + TRY >> + IF ep.wr # NIL THEN >> + Wr.PutText (ep.wr=2C ep.args[1])=3B >> + Wr.PutText (ep.wr=2C Wr.EOL)=3B >> + FlushIO ()=3B >> + END=3B >> + WITH handle =3D Process.Create (ep.cmd=2C ep.args^=2C >> + stdin :=3D stdin=2C stdout :=3D std= >out=2C >> + stderr :=3D stderr)=2C >> + exit_code =3D Process.Wait(handle) DO >> + IF exit_code # 0 AND NOT ep.ignore_errors THEN >> + Err (ep.t=2C Fmt.F("exit %s: %s"=2C Fmt.Int(exit_code)=2C ep.= >cmd)) >> + END >> + END >> + EXCEPT >> + OSError.E (ec) =3D> >> + Err (ep.t=2C Fmt.F ("exec failed%s *** %s"=2C OSErr (ec)=2C e= >p.cmd))=3B >> + END >> + END FulfilExecPromise=3B >> +=20 >> PROCEDURE KillProcess (handle: Process.T) =3D >> BEGIN >> IF (handle # NIL) THEN >>=20 > = > >--_b09106d1-2a84-4ffe-876c-0b2e4cdfd0b0_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >Yes=2C gcc is slow and using as is wasteful.
Yes this should all be mult= >i-threaded.
However it seems to me that m3front has many globals and the= >refore
really making things multi-threaded isn't going to be easy.
I = >think you are mostly wasting your time on user threads though.


H= >ave you tried the NT/x86 system?
It is very fast.


In the gcc = >system=2C each .mc file is run through m3cgc1 to produce an .ms file=2C
= >which is run through as to produce an object file .mo.


In the NT= >/x86 system=2C cm3 outputs .obj files directly.
No .mc files. No m3cgc1 = >or as.
Granted=2C I'd just as soon output .c files=2C and hope for a C c= >ompiler that doesn't run a separate assembler.


My timing might l= >oosen up a bit and I hope to spend some on the thread stress tester using p= >threads. Next weekend.
Am I right though that Darwin works fine?
Sola= >ris? FreeBSD? NetBSD? OpenBSD? Linux? I know=2C not Linux.
You generally= > don't need any of these systems.
All of the x86/AMD64 systems can run i= >n a virtual machine.
Solaris/sparc we have good access.


 = >=3B- Jay


>=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C= > 12 Feb 2011 23:04:14 -0800
>=3B From: mika at async.caltech.edu
>= >=3B CC: jay.krell at cornell.edu
>=3B Subject: [M3devel] More on threadin= >g
>=3B
>=3B
>=3B Hi again m3devel=2C
>=3B
>=3B = >Well I thought I would relate another story having to do with threading
= >>=3B and CM3. It's half-baked at the moment=2C but it only reflects a fe= >w hours'
>=3B work on my part=2C sorry... I'm about to give up now bec= >ause I don't know
>=3B if the problem is in my code=2C or in threading= >=2C or what... maybe it is
>=3B the atfork problem Jay talked about?r>>=3B
>=3B Programmers spend a lot of time waiting for their compi= >lers to run=2C so
>=3B it would be nice if the compilation process wer= >e faster. After perusing
>=3B the source code for the CM3 I realized = >that there is no feedback at all
>=3B from the back-end and assembler = >steps to the rest of the compiler=2C and
>=3B I believe that with the = >GCC back-end the vast majority of the runtime
>=3B of the compiler is = >spent in these stages.
>=3B
>=3B So.... how hard could this be t= >o parallelize? I decided to try making
>=3B the two stages (cm3cg1 an= >d as) into "promises" rather than running them
>=3B on the spot. The = >QMachine adds a promise to call the back-end to a
>=3B RefSeq.T called= > "promises"=2C to which the Builder can also add promises.
>=3B Then= >=2C after making all the promises=2C collect them and run them ("force
&= >gt=3B them" a true Schemer would say) in parallel.
>=3B
>=3B Of = >course something goes wrong somewhere. With 10 threads I am able to
>= >=3B run 31 seconds of CPU time in 19 seconds of wallclock time (using an>>=3B average 160% of CPU) but then something goes wrong and my processr>>=3B is stuck in state "uxmxn". This is using user threads on AMD64_FR= >EEBSD
>=3B on a machine with four CPUs.
>=3B
>=3B Mika= >
>=3B
>=3B P.S. diff attached=2C it's a bit ugly=2C but maybe it= > works on some other
>=3B system.... the code works just fine if you = >run one thread at a time.
>=3B I.e.=2C if you change
>=3B
>= >=3B IF threads.size() >=3B 10 THEN EVAL Thread.Join(threads.remlo())= > END=3B
>=3B
>=3B to
>=3B
>=3B IF threads.size()= > >=3B 0 THEN EVAL Thread.Join(threads.remlo()) END=3B
>=3B
>= >=3B
>=3B
>=3B Index: cm3/src/Builder.m3
>=3B =3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=3B RCS file: /usr/cvs/cm3/m3= >-sys/cm3/src/Builder.m3=2Cv
>=3B retrieving revision 1.36
>=3B di= >ff -c -r1.36 Builder.m3
>=3B *** cm3/src/Builder.m3 24 Aug 2010 05: >24:= >24 -0000 1.36
>=3B --- cm3/src/Builder.m3 13 Feb 2011 06:47:40 -0 >000r>>=3B ***************
>=3B *** 15=2C20 ****
>=3B --- 15=2C22 -= >---
>=3B IMPORT QIdent=3B
>=3B FROM Target IMPORT M3BackendMo= >de_t=2C BackendAssembly=2C BackendModeStrings=3B
>=3B FROM M3Path IM= >PORT OSKind=2C OSKindStrings=3B
>=3B + IMPORT Pathname=3B
>=3B + = >IMPORT RefSeq=3B
>=3B
>=3B TYPE
>=3B UK =3D M3Unit.= >Kind=3B
>=3B ***************
>=3B *** 131=2C136 ****
>=3B --= >- 133=2C139 ----
>=3B link_coverage : TEXT=3B (* c= >overage library *)
>=3B m3_front_flags: Arg.List=3B (*= > configuration options for the front *)
>=3B m3_options : Arg= >.List=3B (* misc. user options for the frontend *)
>=3B + = > delayBackend :=3D FALSE=3B
>=3B END=3B
>=3B
>=3B = >TYPE
>=3B ***************
>=3B *** 932=2C937 ****
>=3B --- 9= >35=2C956 ----
>=3B
>=3B (*----------------------------------= >-------------------------- compilation --*)
>=3B
>=3B + TYPE M= >arkerPromise =3D QMachine.Promise BRANDED OBJECT OVERRIDES fulfil :=3D Fulf= >ilNothing END=3B
>=3B +
>=3B + PROCEDURE FulfilNothing(<=3B*UN= >USED*>=3Bmp : MarkerPromise) =3D BEGIN END FulfilNothing=3B
>=3B + <= >br>>=3B + TYPE SeqClosure =3D Thread.Closure OBJECT seq : RefSeq.T=3B OVE= >RRIDES apply :=3D SeqApply END=3B
>=3B +
>=3B + PROCEDURE SeqApp= >ly(cl : SeqClosure) : REFANY =3D
>=3B + BEGIN
>=3B + FOR i = >:=3D 0 TO cl.seq.size()-1 DO
>=3B + WITH p =3D NARROW(cl.seq.get= >(i)=2CQMachine.Promise) DO
>=3B + p.fulfil()
>=3B + = > END
>=3B + END=3B
>=3B + RETURN NIL
>=3B + END Se= >qApply=3B
>=3B +
>=3B PROCEDURE CompileEverything (s: State=3B= > schedule: SourceList) =3D
>=3B VAR u: M3Unit.T=3B
>=3B = >BEGIN
>=3B ***************
>=3B *** 941=2C948 ****
>=3B r>>=3B (* compile all the sources using the initial schedule *)
= >>=3B FOR i :=3D 0 TO LAST (schedule^) DO
>=3B ! CompileO= >ne (s=2C schedule[i])=3B
>=3B END=3B
>=3B FlushPendin= >g (s)=3B
>=3B
>=3B (* recompile any interfaces where we = >goofed on the exports *)
>=3B --- 960=2C1000 ----
>=3B
>= >=3B (* compile all the sources using the initial schedule *)
>= >=3B FOR i :=3D 0 TO LAST (schedule^) DO
>=3B ! s.delayBack= >end :=3D TRUE=3B
>=3B ! TRY
>=3B ! CompileOne (s=2C= > schedule[i])=3B
>=3B ! FINALLY
>=3B ! s.delayBacke= >nd :=3D FALSE=3B
>=3B ! END=3B
>=3B !
>=3B ! s.= >machine.promises.addhi(NEW(MarkerPromise))=3B
>=3B !
>=3B ! = >END=3B
>=3B !
>=3B ! VAR
>=3B ! curSeq :=3D NEW(= >RefSeq.T).init()=3B
>=3B ! threads :=3D NEW(RefSeq.T).init()=3B<= >br>>=3B ! BEGIN
>=3B ! FOR i :=3D 0 TO s.machine.promises.= >size()-1 DO
>=3B ! WITH p =3D s.machine.promises.get(i) DO
= >>=3B ! curSeq.addhi(p)=3B
>=3B ! IF i =3D s.mach= >ine.promises.size()-1 OR ISTYPE(p=2CMarkerPromise) THEN
>=3B ! = > WITH cl =3D NEW(SeqClosure=2C seq :=3D curSeq) DO
>=3B ! = > threads.addhi (Thread.Fork(cl))=3B
>=3B !
>=3B ! = > IF threads.size() >=3B 10 THEN EVAL Thread.Join(threads.remlo()) END= >=3B
>=3B !
>=3B ! curSeq :=3D NEW(Re= >fSeq.T).init()
>=3B ! END
>=3B ! END
>= >=3B ! END
>=3B ! END=3B
>=3B ! WHILE threads.= >size() >=3B 0 DO EVAL Thread.Join(threads.remlo()) END=3B
>=3B = > END=3B
>=3B +
>=3B + EVAL s.machine.promises.in= >it()=3B
>=3B +
>=3B +
>=3B FlushPending (s)=3B
&g= >t=3B
>=3B (* recompile any interfaces where we goofed on the = >exports *)
>=3B ***************
>=3B *** 1151=2C1156 ****
>= >=3B --- 1203=2C1227 ----
>=3B END=3B
>=3B END CompileM3= >=3B
>=3B
>=3B + TYPE
>=3B + NotePromise =3D QMachine.Pr= >omise OBJECT
>=3B + nam : Pathname.T=3B
>=3B + OVERRIDES>>=3B + fulfil :=3D FulfilNP=3B
>=3B + END=3B
>=3B +
= >>=3B + RemovePromise =3D QMachine.Promise OBJECT
>=3B + nam : = >Pathname.T=3B
>=3B + OVERRIDES
>=3B + fulfil :=3D FulfilRP= >=3B
>=3B + END=3B
>=3B +
>=3B + PROCEDURE FulfilNP(np : N= >otePromise) =3D
>=3B + BEGIN Utils.NoteTempFile(np.nam) END FulfilN= >P=3B
>=3B +
>=3B + PROCEDURE FulfilRP(rp : RemovePromise) =3D r>>=3B + BEGIN Utils.Remove(rp.nam) END FulfilRP=3B
>=3B +
>= >=3B PROCEDURE PushOneM3 (s: State=3B u: M3Unit.T): BOOLEAN =3D
>=3B= > VAR
>=3B tmpC=2C tmpS: TEXT=3B
>=3B ***************>>=3B *** 1191=2C1208 ****
>=3B
>=3B
>=3B | 3 = >=3D>=3B (* -bootstrap=2C +m3back=2C +asm *)
>=3B tmpC := >=3D TempCName (u)=3B
>=3B tmpS :=3D TempSName (u)=3B
>= >=3B IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3Br>>=3B IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END= >=3B
>=3B IF RunM3 (s=2C u=2C tmpC) THEN
>=3B ! = > IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.optimize)
>=3B ! = > AND RunAsm (s=2C tmpS=2C u.object) THEN
>=3B END= >=3B
>=3B need_merge :=3D TRUE=3B
>=3B END= >=3B
>=3B IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END= >=3B
>=3B IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END= >=3B
>=3B
>=3B | 6=2C (* +bootstrap=2C +m3back=2C -asm= > *)
>=3B 7 =3D>=3B (* +bootstrap=2C +m3back=2C +asm *)
&= >gt=3B --- 1262=2C1318 ----
>=3B
>=3B
>=3B | 3 = >=3D>=3B (* -bootstrap=2C +m3back=2C +asm *)
>=3B + IF s.delay= >Backend THEN
>=3B tmpC :=3D TempCName (u)=3B
>=3B = > tmpS :=3D TempSName (u)=3B
>=3B + (*
>=3B IF (NOT = >s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3B
>=3B IF = >(NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END=3B
>=3B + *)
= >>=3B + IF (NOT s.keep_files) THEN
>=3B + s.machin= >e.promises.addhi(NEW(NotePromise=2C nam :=3D tmpC))
>=3B + EN= >D=3B
>=3B + IF (NOT s.keep_files) THEN
>=3B + = >s.machine.promises.addhi(NEW(NotePromise=2C nam :=3D tmpS))
>=3B + = > END=3B
>=3B +
>=3B IF RunM3 (s=2C u=2C tmpC) THE= >N
>=3B ! s.machine.record(TRUE)=3B
>=3B ! TRY= >
>=3B ! IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.o= >ptimize)
>=3B ! AND RunAsm (s=2C tmpS=2C u.object) THEN>>=3B ! END=3B
>=3B ! FINALLY
>=3B ! = > s.machine.record(FALSE)
>=3B END=3B
>=3B + r>>=3B need_merge :=3D TRUE=3B
>=3B END=3B
= >>=3B + (*
>=3B IF (NOT s.keep_files) THEN Utils.Remove (tm= >pC) END=3B
>=3B IF (NOT s.keep_files) THEN Utils.Remove (tmp= >S) END=3B
>=3B + *)
>=3B + IF (NOT s.keep_files) THEN >>=3B + s.machine.promises.addhi(NEW(RemovePromise=2C nam :=3D = >tmpC))
>=3B + END=3B
>=3B + IF (NOT s.keep_files= >) THEN
>=3B + s.machine.promises.addhi(NEW(RemovePromise=2C= > nam :=3D tmpS))
>=3B + END=3B
>=3B +
>=3B + = > ELSE
>=3B + tmpC :=3D TempCName (u)=3B
>=3B + = > tmpS :=3D TempSName (u)=3B
>=3B + IF (NOT s.keep_files) TH= >EN Utils.NoteTempFile (tmpC) END=3B
>=3B + IF (NOT s.keep_fi= >les) THEN Utils.NoteTempFile (tmpS) END=3B
>=3B + IF RunM3 (= >s=2C u=2C tmpC) THEN
>=3B + IF RunM3Back (s=2C tmpC=2C tm= >pS=2C u.debug=2C u.optimize)
>=3B + AND RunAsm (s=2C tmpS= >=2C u.object) THEN
>=3B + END=3B
>=3B + n= >eed_merge :=3D TRUE=3B
>=3B + END=3B
>=3B + I= >F (NOT s.keep_files) THEN Utils.Remove (tmpC) END=3B
>=3B + = >IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END=3B
>=3B + END= >
>=3B
>=3B | 6=2C (* +bootstrap=2C +m3back=2C -asm *)= >
>=3B 7 =3D>=3B (* +bootstrap=2C +m3back=2C +asm *)
>= >=3B Index: cm3/src/m3makefile
>=3B =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D
>=3B RCS file: /usr/cvs/cm3/m3-sys/cm3/src/m3makefi= >le=2Cv
>=3B retrieving revision 1.24
>=3B diff -c -r1.24 m3makefi= >le
>=3B *** cm3/src/m3makefile 8 Dec 2010 07:30:57 -0000 1.24
>>= >=3B --- cm3/src/m3makefile 13 Feb 2011 06:47:42 -0000
>=3B ********** >*= >****
>=3B *** 29=2C34 ****
>=3B --- 29=2C35 ----
>=3B end<= >br>>=3B module ("Arg")
>=3B module ("Builder")
>=3B modul= >e ("Dirs")
>=3B module ("M3Build")
>=3B module ("M3Loc")
&= >gt=3B Index: m3quake/src/QMachine.i3
>=3B =3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D
>=3B RCS file: /usr/cvs/cm3/m3-sys/m3quake/sr= >c/QMachine.i3=2Cv
>=3B retrieving revision 1.6
>=3B diff -c -r1.6= > QMachine.i3
>=3B *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -00 >0= >0 1.6
>=3B --- m3quake/src/QMachine.i3 13 Feb 2011 06:47:43 -0 >000
&= >gt=3B ***************
>=3B *** 8=2C13 ****
>=3B --- 8=2C14 ----r>>=3B
>=3B IMPORT Thread=2C Wr=2C QValue=2C QCode=3B
>=3B= > FROM Quake IMPORT Machine=2C Error=2C ID=2C IDMap=3B
>=3B + IMPORT = >RefSeq=3B
>=3B
>=3B REVEAL
>=3B T <=3B: T_=3B
= >>=3B ***************
>=3B *** 15=2C20 ****
>=3B --- 16=2C22 ---= >-
>=3B T =3D Machine=3B
>=3B T_ =3D OBJECT
>=3B = > map: IDMap :=3D NIL=3B (* READONLY *)
>=3B + promises : RefSeq.T= >=3B
>=3B METHODS
>=3B init (map: IDMap): T=3B
&= >gt=3B evaluate (s: QCode.Stream) RAISES {Error= >=2C Thread.Alerted}=3B
>=3B ***************
>=3B *** 37=2C42 ****= >
>=3B --- 39=2C46 ----
>=3B set_wr (wr: Wr.T)=3B
>= >=3B exec_echo (b: BOOLEAN): BOOLEAN=3B
>=3B trace (b: = >BOOLEAN)=3B
>=3B +
>=3B + record(on : BOOLEAN)=3B (*= > instead of performing certain acts=2C promise *)
>=3B END=3B
&= >gt=3B
>=3B PROCEDURE PushBool (t: T=3B b: BOOLEAN)=3B
>=3B = >***************
>=3B *** 51=2C54 ****
>=3B --- 55=2C63 ----
&g= >t=3B
>=3B PROCEDURE GetEnv (default=2C v0=2C v1=2C v2=2C v3=2C v4= >: TEXT :=3D NIL): TEXT=3B
>=3B
>=3B + TYPE Promise =3D OBJECT = >METHODS fulfil() RAISES { Error } END=3B
>=3B +
>=3B END QMac= >hine.
>=3B +
>=3B +
>=3B +
>=3B Index: m3quake/src/Q= >Machine.m3
>=3B =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D>>=3B RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3=2Cv
>=3B= > retrieving revision 1.35
>=3B diff -c -r1.35 QMachine.m3
>=3B **= >* m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35
>=3B >--- m3qu= >ake/src/QMachine.m3 13 Feb 2011 06:47:44 -0000
>=3B ***************
= >>>=3B *** 16=2C21 ****
>=3B --- 16=2C22 ----
>=3B IMPORT Text= >Utils=2C FSUtils=2C System=2C DirStack=3B (* sysutils *)
>=3B IMPORT= > Compiler=3B
>=3B IMPORT M3Path=3B
>=3B + IMPORT RefSeq=3B
&= >gt=3B
>=3B CONST
>=3B OnUnix =3D (Compiler.ThisOS =3D Co= >mpiler.OS.POSIX)=3B
>=3B ***************
>=3B *** 44=2C49 ****>>=3B --- 45=2C52 ----
>=3B shell : TEXT :=3D NIL= >=3B
>=3B sh_option : TEXT :=3D NIL=3B
>=3B tm= >p_dir : TEXT :=3D NIL=3B
>=3B +
>=3B + doRecord := >=3D FALSE=3B
>=3B OVERRIDES
>=3B init :=3D Init=3B= >
>=3B evaluate :=3D Evaluate=3B
>=3B ***************
&g= >t=3B *** 66=2C73 ****
>=3B --- 69=2C81 ----
>=3B set_wr = >:=3D SetWr=3B
>=3B exec_echo :=3D ExecEcho=3B
>=3B tr= >ace :=3D Trace=3B
>=3B +
>=3B + record :=3D Record=3B>>=3B END=3B
>=3B
>=3B + PROCEDURE Record(t : T=3B on : = >BOOLEAN) =3D
>=3B + BEGIN t.doRecord :=3D on END Record=3B
>= >=3B +
>=3B TYPE
>=3B Registers =3D RECORD
>=3B = >cp : QCode.Stream :=3D NIL=3B (* code pointer *)
>=3B **************= >*
>=3B *** 139=2C144 ****
>=3B --- 147=2C154 ----
>=3B = > t.globals :=3D NEW (IntRefTbl.Default).init ()=3B
>=3B t.def= >ault_wr :=3D Stdio.stdout=3B
>=3B
>=3B + t.promises :=3D N= >EW(RefSeq.T).init()=3B
>=3B +
>=3B InitOSEnv (t)=3B
>= >=3B InitBuiltins (t)=3B
>=3B
>=3B ***************
>= >=3B *** 1555=2C1564 ****
>=3B END=3B
>=3B E= >LSE
>=3B FlushIO ()=3B
>=3B ! Process.GetSt= >andardFileHandles (stdin=2C stdout=2C stderr)=3B
>=3B ! hand= >le :=3D Process.Create (t.shell=2C SUBARRAY (args=2C 0=2C n_shell_args)=2C<= >br>>=3B ! stdin :=3D stdin=2C stdout = >:=3D stdout=2C
>=3B ! stderr :=3D = >stderr)=3B
>=3B END=3B
>=3B EXCEPT
>=3B = > | Thread.Alerted =3D>=3B
>=3B --- 1565=2C1594 ----
>=3B = > END=3B
>=3B ELSE
>=3B FlushIO ()= >=3B
>=3B ! IF t.doRecord THEN
>=3B ! handle= > :=3D NIL=3B
>=3B ! WITH a =3D NEW(REF ARRAY OF TEXT=2C n_= >shell_args) DO
>=3B ! a^ :=3D SUBARRAY(args=2C0=2Cn_shel= >l_args)=3B
>=3B ! VAR wrx : Wr.T=3B BEGIN
>=3B ! = > IF echo OR t.do_echo THEN
>=3B ! wrx := >=3D wr
>=3B ! ELSE
>=3B ! wrx := >=3D NIL
>=3B ! END=3B
>=3B ! t.pr= >omises.addhi(NEW(ExecPromise=2C
>=3B ! = > cmd :=3D t.shell=2C
>=3B ! = >wr :=3D wrx=2C
>=3B ! args :=3D a= >=2C
>=3B ! t :=3D t=2C
>=3B = >! ignore_errors :=3D ignore_errors))r>>=3B ! END
>=3B ! END
>=3B ! = > ELSE
>=3B ! Process.GetStandardFileHandles (stdin=2C s= >tdout=2C stderr)=3B
>=3B ! handle :=3D Process.Create (t.s= >hell=2C SUBARRAY (args=2C 0=2C n_shell_args)=2C
>=3B ! = > stdin :=3D stdin=2C stdout :=3D stdout=2C
>=3B = >! stderr :=3D stderr)=3B
>=3B ! = > END=3B
>=3B END=3B
>=3B EXCEPT
>= >=3B | Thread.Alerted =3D>=3B
>=3B ***************
>=3B = >*** 1573=2C1579 ****
>=3B END=3B
>=3B
>=3B = > (* wait for everything to shutdown... *)
>=3B ! exit_code :=3D= > Process.Wait (handle)=3B
>=3B END=3B
>=3B
>=3B = > IF onlyTry THEN
>=3B --- 1603=2C1613 ----
>=3B END=3B<= >br>>=3B
>=3B (* wait for everything to shutdown... *)
= >>=3B ! IF handle =3D NIL THEN
>=3B ! exit_code :=3D 0 = >
>=3B ! ELSE
>=3B ! exit_code :=3D Process.Wait (ha= >ndle)=3B
>=3B ! END=3B (* else we're only promising *)
>=3B= > END=3B
>=3B
>=3B IF onlyTry THEN
>=3B ******= >*********
>=3B *** 1589=2C1594 ****
>=3B --- 1623=2C1664 ----
= >>=3B
>=3B END ExecCommand=3B
>=3B
>=3B + TYPE r>>=3B + ExecPromise =3D Promise OBJECT
>=3B + cmd : TEXT=3Br>>=3B + args : REF ARRAY OF TEXT=3B
>=3B + t : T=3B
>= >=3B + wr : Wr.T=3B
>=3B + ignore_errors : BOOLEAN=3B
>=3B= > + OVERRIDES
>=3B + fulfil :=3D FulfilExecPromise=3B
>=3B += > END=3B
>=3B +
>=3B + PROCEDURE FulfilExecPromise(ep : ExecPro= >mise) RAISES { Error } =3D
>=3B + VAR
>=3B + stdin=2C stdo= >ut=2C stderr: File.T=3B
>=3B + BEGIN
>=3B + Process.GetStan= >dardFileHandles (stdin=2C stdout=2C stderr)=3B
>=3B + TRY
>= >=3B + IF ep.wr # NIL THEN
>=3B + Wr.PutText (ep.wr=2C ep= >.args[1])=3B
>=3B + Wr.PutText (ep.wr=2C Wr.EOL)=3B
>=3B = >+ FlushIO ()=3B
>=3B + END=3B
>=3B + WITH han= >dle =3D Process.Create (ep.cmd=2C ep.args^=2C
>=3B + = > stdin :=3D stdin=2C stdout :=3D stdout=2C
>=3B + = > stderr :=3D stderr)=2C
>=3B + = > exit_code =3D Process.Wait(handle) DO
>=3B + IF exit_code= > # 0 AND NOT ep.ignore_errors THEN
>=3B + Err (ep.t=2C Fmt.F= >("exit %s: %s"=2C Fmt.Int(exit_code)=2C ep.cmd))
>=3B + END>>=3B + END
>=3B + EXCEPT
>=3B + OSError.E (ec)= > =3D>=3B
>=3B + Err (ep.t=2C Fmt.F ("exec failed%s *** %s"= >=2C OSErr (ec)=2C ep.cmd))=3B
>=3B + END
>=3B + END FulfilE= >xecPromise=3B
>=3B +
>=3B PROCEDURE KillProcess (handle: Proce= >ss.T) =3D
>=3B BEGIN
>=3B IF (handle # NIL) THEN
>= >=3B
>= > >--_b09106d1-2a84-4ffe-876c-0b2e4cdfd0b0_-- From hosking at cs.purdue.edu Sun Feb 13 21:18:03 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:18:03 -0500 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110212204406.88A661A2078@async.async.caltech.edu> References: <20110212204406.88A661A2078@async.async.caltech.edu> Message-ID: Hi Mika, Why was this change needed (I haven't looked closely)? I don't think we diverged much from previous implementations of user threads, which presumably used to work for you. On Feb 12, 2011, at 3:44 PM, Mika Nystrom wrote: > > Index: ThreadPosix.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v > retrieving revision 1.16 > diff -c -r1.16 ThreadPosix.i3 > *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 > --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 > *************** > *** 39,42 **** > --- 39,45 ---- > > (*---------------------------------------------------------------------------*) > > + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> > + PROCEDURE ValueOfSIGCHLD(): int; > + > END ThreadPosix. > Index: ThreadPosix.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v > retrieving revision 1.73 > diff -c -r1.73 ThreadPosix.m3 > *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 > --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 > *************** > *** 97,102 **** > --- 97,105 ---- > (* if state = pausing, the time at which we can restart *) > waitingForTime: Time.T; > > + (* if state = pausing, the signal that truncates the pause *) > + waitingForSig: int := -1; > + > (* true if we are waiting during an AlertWait or AlertJoin > or AlertPause *) > alertable: BOOLEAN := FALSE; > *************** > *** 147,152 **** > --- 150,160 ---- > > defaultStackSize := 3000; > > + (* note that even though the "heavy machinery" is only used for > + multipleThreads, we still need to set up the signal handler so > + that we can catch signals from other sources than thread switching. > + e.g., we use SIGCHLD to speed up Process.Wait *) > + > VAR > stats: RECORD > n_forks := 0; > *************** > *** 511,527 **** > XPause(until, FALSE); > END Pause; > > PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= > VAR until := n + Time.Now (); > BEGIN > XPause(until, TRUE); > END AlertPause; > > ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE) RAISES {Alerted} = > BEGIN > INC (inCritical); > self.waitingForTime := until; > self.alertable := alertable; > ICannotRun (State.pausing); > DEC (inCritical); > InternalYield (); > --- 519,546 ---- > XPause(until, FALSE); > END Pause; > > + PROCEDURE SigPause(n: LONGREAL; sig: int)= > + <*FATAL Alerted*> > + VAR until := n + Time.Now (); > + BEGIN > + XPause(until, FALSE, sig); > + END SigPause; > + > PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= > VAR until := n + Time.Now (); > BEGIN > XPause(until, TRUE); > END AlertPause; > > ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) > ! RAISES {Alerted} = > BEGIN > INC (inCritical); > self.waitingForTime := until; > self.alertable := alertable; > + IF sig # -1 THEN > + self.waitingForSig := sig > + END; > ICannotRun (State.pausing); > DEC (inCritical); > InternalYield (); > *************** > *** 703,712 **** > END; > END StartSwitching; > > ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); > ! IF inCritical = 0 AND heapState.inCritical = 0 THEN InternalYield () END; > END switch_thread; > > (*------------------------------------------------------------- scheduler ---*) > --- 722,750 ---- > END; > END StartSwitching; > > ! CONST MaxSigs = 64; > ! TYPE Sig = [ 0..MaxSigs-1 ]; > ! > ! (* in order to listen to other signals, they have to be enabled in > ! allow_sigvtalrm as well *) > ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); > ! > ! gotSigs := SET OF Sig { }; > ! > ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); > ! > ! INC(inCritical); > ! (* mark signal as being delivered *) > ! IF sig >= 0 AND sig < MaxSigs THEN > ! gotSigs := gotSigs + SET OF Sig { sig } > ! END; > ! DEC(inCritical); > ! > ! IF inCritical = 0 AND heapState.inCritical = 0 THEN > ! InternalYield () > ! END; > END switch_thread; > > (*------------------------------------------------------------- scheduler ---*) > *************** > *** 782,792 **** > IF t.alertable AND t.alertPending THEN > CanRun (t); > EXIT; > > ELSIF t.waitingForTime <= now THEN > CanRun (t); > EXIT; > ! > ELSIF NOT somePausing THEN > earliest := t.waitingForTime; > somePausing := TRUE; > --- 820,835 ---- > IF t.alertable AND t.alertPending THEN > CanRun (t); > EXIT; > + > + ELSIF t.waitingForSig IN gotSigs THEN > + t.waitingForSig := -1; > + CanRun(t); > + EXIT; > > ELSIF t.waitingForTime <= now THEN > CanRun (t); > EXIT; > ! > ELSIF NOT somePausing THEN > earliest := t.waitingForTime; > somePausing := TRUE; > *************** > *** 886,891 **** > --- 929,936 ---- > END; > END; > > + gotSigs := SET OF Sig {}; > + > IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN > IF perfOn THEN PerfRunning (t.id); END; > (* At least one thread wants to run; transfer to it *) > *************** > *** 948,960 **** > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > ! CONST Delay = 0.1D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > ! Pause(Delay); > END; > END WaitProcess; > > --- 993,1005 ---- > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > ! CONST Delay = 10.0D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > ! SigPause(Delay,SIGCHLD); > END; > END WaitProcess; > > *************** > *** 1361,1364 **** > --- 1406,1412 ---- > VAR debug := RTParams.IsPresent ("debugthreads"); > > BEGIN > + (* we need to call set up the signal handler for other reasons than > + just thread switching now *) > + setup_sigvtalrm (switch_thread); > END ThreadPosix. > Index: ThreadPosixC.c > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v > retrieving revision 1.59 > diff -c -r1.59 ThreadPosixC.c > *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 > --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 > *************** > *** 78,88 **** > --- 78,97 ---- > > sigemptyset(&ThreadSwitchSignal); > sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); > + sigaddset(&ThreadSwitchSignal, SIGCHLD); > > act.sa_handler = handler; > act.sa_flags = SA_RESTART; > sigemptyset(&(act.sa_mask)); > if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); > + if (sigaction (SIGCHLD, &act, NULL)) abort(); > + } > + > + int > + __cdecl > + ThreadPosix__value_of_SIGCHLD(void) > + { > + return SIGCHLD; > } > > void From hosking at cs.purdue.edu Sun Feb 13 21:22:13 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:22:13 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110212064812.286E61A2078@async.async.caltech.edu> Message-ID: <35125A4E-2BB1-48B9-87EC-58CFCFA5D5AF@cs.purdue.edu> Looks like stack overflow to me. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 12, 2011, at 9:09 AM, Jay K wrote: > > >LINUXLIBC6 : seg fault > > I've seen this now. > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0xb3200b90 (LWP 18154)] > 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) > at ../src/runtime/common/RTCollector.m3:409 > 409 IF hdr.typecode = RT0.TextLitTypecode THEN RETURN END; > 1: x/i $pc > 0xb6dee58f : mov (%eax),%eax > (gdb) bt 9 > #0 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) > at ../src/runtime/common/RTCollector.m3:409 > #1 0xb6deaa46 in RTHeapMap__Walk (x=0xb6a20014, pc=0xb7738648, v=0xb6a9000c) > at ../src/runtime/common/RTHeapMap.m3:202 > #2 0xb6dea341 in RTHeapMap__DoWalkRef (t=0xb7738694, a=0xb6a20014, > v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:62 > #3 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739c54, a=0xb6a2000c, > v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 > #4 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739e68, a=0xb6a2000c, > v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 > #5 0xb6dea2bf in RTHeapMap__WalkRef (h=0xb6a20008, v=0xb6a9000c) > at ../src/runtime/common/RTHeapMap.m3:47 > #6 0xb6df0636 in RTCollector__CleanBetween (h=0xb6a20008, he=0xb6a30000, > clean=0 '\0') at ../src/runtime/common/RTCollector.m3:1091 > #7 0xb6df0473 in RTCollector__CleanPage (page=0xb6a20000) > at ../src/runtime/common/RTCollector.m3:1064 > #8 0xb6defc96 in RTCollector__CollectSomeInStateZero () > at ../src/runtime/common/RTCollector.m3:885 > (More stack frames follow...) > (gdb) p $eax > $2 = 2097148 > (gdb) p/x $eax > $3 = 0x1ffffc > (gdb) x/x $eax+4 > 0x200000: Cannot access memory at address 0x200000 > (gdb) x/x $eax > 0x1ffffc: Cannot access memory at address 0x1ffffc > > > Maybe I should try with a correctly sized jmpbuf to assure it isn't stack overflow? > But notice that the next page doesn't work either. Actually I tried adding > a lot to $eax and still not accessible. > > > Maybe maybe maybe maybe I'll dig into this. But I'm very busy lately and maybe indefinitely into future.. > > > Thanks, > - Jay > > > > To: jay.krell at cornell.edu; hosking at cs.purdue.edu > > Date: Fri, 11 Feb 2011 22:48:12 -0800 > > From: mika at async.caltech.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] threading on Windows? > > > > Well, with the following changes: re-bootstrapping on the two Linux > > systems and removal of the -pthread flag on FreeBSD4, all is well. > > User threading works on all the architectures listed below, at least > > as far as the thread testing program can test it. (It seems that because > > of the way pthread_atfork is coded, pthread is not needed at all on > > FreeBSD4.) > > > > Mika > > > > Mika Nystrom writes: > > >Another odd datapoint is that even rather aggressive testing seems not > > >to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user > > >threads, of course. > > > > > >So status right now of user threads seems to be: > > > > > >FreeBSD4 : strange failure in libpthread as reported by gdb > > >LINUXLIBC6 : seg fault > > >AMD64_LINUX : seg fault > > >AMD64_FREEBSD : OK > > >I386_DARWIN : OK -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sun Feb 13 21:32:05 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:32:05 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <20110213003358.00D2A1A2078@async.async.caltech.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu>, <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> <20110213003358.00D2A1A2078@async.async.caltech.edu> Message-ID: I am confused. This backtrace looks like it is from the release, which we know has problems. On Feb 12, 2011, at 7:33 PM, Mika Nystrom wrote: > P.S. backtraces of deadlock: > > Thread 3 (Thread 0x40862950 (LWP 19974)): > #0 0x00002b9d49f730e1 in nanosleep () from /lib/libpthread.so.0 > #1 0x000000000045806d in ThreadPThread__Nanosleep (req=Cannot access memory at address 0x140861b08 > ) at ../src/thread/PTHREAD/ThreadPThreadC.c:500 > #2 0x0000000000454294 in ThreadPThread__CommonSleep () at ../src/thread/PTHREAD/ThreadPThread.m3:551 > #3 0x0000000000456438 in ThreadPThread__StopWorld () at ../src/thread/PTHREAD/ThreadPThread.m3:1086 > #4 0x0000000000455787 in RTThread__SuspendOthers () at ../src/thread/PTHREAD/ThreadPThread.m3:812 > #5 0x000000000043970c in RTCollector__CollectSomeInStateZero () at ../src/runtime/common/RTCollector.m3:746 > #6 0x00000000004396b8 in RTCollector__CollectSome () at ../src/runtime/common/RTCollector.m3:720 > #7 0x00000000004390b9 in RTHeapRep__CollectEnough () at ../src/runtime/common/RTCollector.m3:654 > #8 0x0000000000436111 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=Cannot access memory at address 0x140861dc8 > ) at ../src/runtime/common/RTAllocator.m3:363 > #9 0x000000000043596f in RTAllocator__GetOpenArray (M3_Eic7CK_def=Cannot access memory at address 0x140861eb8 > ) at ../src/runtime/common/RTAllocator.m3:292 > #10 0x0000000000434ab1 in RTHooks__AllocateOpenArray (M3_AJWxb1_defn=Cannot access memory at address 0x140861f38 > ) at ../src/runtime/common/RTAllocator.m3:141 > #11 0x0000000000407030 in Main__AApply (M3_AP7a1g_cl=Cannot access memory at address 0x140861f88 > ) at ../src/Main.m3:234 > #12 0x0000000000453b3f in ThreadPThread__RunThread (M3_DMxDjQ_me=Cannot access memory at address 0x140862028 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:475 > #13 0x00000000004537d3 in ThreadPThread__ThreadBase (M3_AJWxb1_param=Cannot access memory at address 0x1408620d8 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:447 > #14 0x00002b9d49f6bfc7 in start_thread () from /lib/libpthread.so.0 > #15 0x00002b9d4b3bd64d in clone () from /lib/libc.so.6 > #16 0x0000000000000000 in ?? () > > Thread 2 (Thread 0x40661950 (LWP 19973)): > #0 0x00002b9d49f72384 in __lll_lock_wait () from /lib/libpthread.so.0 > #1 0x00002b9d49f6dbf0 in _L_lock_102 () from /lib/libpthread.so.0 > #2 0x00002b9d49f6d4fe in pthread_mutex_lock () from /lib/libpthread.so.0 > #3 0x000000000045815e in ThreadPThread__pthread_mutex_lock (m=Cannot access memory at address 0x140660d68 > ) at ../src/thread/PTHREAD/ThreadPThreadC.c:557 > #4 0x00000000004575af in RTOS__LockHeap () at ../src/thread/PTHREAD/ThreadPThread.m3:1434 > #5 0x0000000000436107 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=Cannot access memory at address 0x140660dc8 > ) at ../src/runtime/common/RTAllocator.m3:361 > #6 0x000000000043596f in RTAllocator__GetOpenArray (M3_Eic7CK_def=Cannot access memory at address 0x140660eb8 > ) at ../src/runtime/common/RTAllocator.m3:292 > #7 0x0000000000434ab1 in RTHooks__AllocateOpenArray (M3_AJWxb1_defn=Cannot access memory at address 0x140660f38 > ) at ../src/runtime/common/RTAllocator.m3:141 > #8 0x0000000000407030 in Main__AApply (M3_AP7a1g_cl=Cannot access memory at address 0x140660f88 > ) at ../src/Main.m3:234 > #9 0x0000000000453b3f in ThreadPThread__RunThread (M3_DMxDjQ_me=Cannot access memory at address 0x140661028 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:475 > #10 0x00000000004537d3 in ThreadPThread__ThreadBase (M3_AJWxb1_param=Cannot access memory at address 0x1406610d8 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:447 > #11 0x00002b9d49f6bfc7 in start_thread () from /lib/libpthread.so.0 > #12 0x00002b9d4b3bd64d in clone () from /lib/libc.so.6 > #13 0x0000000000000000 in ?? () > > Thread 1 (Thread 0x2b9d4d32a1a0 (LWP 19965)): > #0 0x00002b9d4b3202a2 in sigsuspend () from /lib/libc.so.6 > #1 0x0000000000457ba9 in ThreadPThread__sigsuspend () at ../src/thread/PTHREAD/ThreadPThreadC.c:141 > #2 0x0000000000456a0d in ThreadPThread__SignalHandler (M3_DLS2Hj_sig=Cannot access memory at address 0x80001dde057c > ) at ../src/thread/PTHREAD/ThreadPThread.m3:1205 > #3 > #4 0x00002b9d49f6ffad in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 > #5 0x00000000004580b3 in ThreadPThread__pthread_cond_timedwait (cond=Cannot access memory at address 0x80001dde0a88 > ) at ../src/thread/PTHREAD/ThreadPThreadC.c:515 > #6 0x00000000004544cf in ThreadPThread__XPause (M3_DMxDjQ_self=Cannot access memory at address 0x80001dde0ab8 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:580 > #7 0x00000000004545b2 in Thread__Pause (M3_CtKayy_n=Cannot access memory at address 0x80001dde0b08 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:595 > #8 0x00000000004097ee in Main_M3 (M3_AcxOUs_mode=Cannot access memory at address 0x80001dde0bc8 > ) at ../src/Main.m3:449 > #9 0x0000000000445799 in RTLinker__RunMainBody (M3_DjPxE3_m=Cannot access memory at address 0x80001dde0f58 > ) at ../src/runtime/common/RTLinker.m3:406 > #10 0x00000000004448f2 in RTLinker__AddUnitI (M3_DjPxE3_m=Cannot access memory at address 0x80001dde0ff8 > ) at ../src/runtime/common/RTLinker.m3:113 > #11 0x0000000000444990 in RTLinker__AddUnit (M3_DjPxE5_b=Cannot access memory at address 0x80001dde1018 > ) at ../src/runtime/common/RTLinker.m3:122 > #12 0x000000000040652c in main (argc=Cannot access memory at address 0x80001dde1068 > ) at _m3main.mc:4 From mika at async.caltech.edu Sun Feb 13 21:37:26 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 12:37:26 -0800 Subject: [M3devel] SIGCHLD diff In-Reply-To: References: <20110212204406.88A661A2078@async.async.caltech.edu> Message-ID: <20110213203726.82ADB1A2078@async.async.caltech.edu> Tony, The old user threads has an old problem with Process.Wait. It has a Thread.Pause(0.1d0) in it. This causes the compiler to run three times slower than it should. The m3build I've been using for years has that Thread.Pause removed (so it uses 100% CPU instead, busy-waiting...) But yeah user threads works the same as it used to (except for now calling pthread_atfork, which I'll return to in another email...) Mika Tony Hosking writes: >Hi Mika, Why was this change needed (I haven't looked closely)? I don't = >think we diverged much from previous implementations of user threads, = >which presumably used to work for you. > >On Feb 12, 2011, at 3:44 PM, Mika Nystrom wrote: > >>=20 >> Index: ThreadPosix.i3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: = >/usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v >> retrieving revision 1.16 >> diff -c -r1.16 ThreadPosix.i3 >> *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 >> --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 >> *************** >> *** 39,42 **** >> --- 39,45 ---- >>=20 >> = >(*------------------------------------------------------------------------= >---*) >>=20 >> + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> >> + PROCEDURE ValueOfSIGCHLD(): int; >> +=20 >> END ThreadPosix. >> Index: ThreadPosix.m3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: = >/usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v >> retrieving revision 1.73 >> diff -c -r1.73 ThreadPosix.m3 >> *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 >> --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 >> *************** >> *** 97,102 **** >> --- 97,105 ---- >> (* if state =3D pausing, the time at which we can restart *) >> waitingForTime: Time.T; >>=20 >> + (* if state =3D pausing, the signal that truncates the pause *) >> + waitingForSig: int :=3D -1; >> +=20 >> (* true if we are waiting during an AlertWait or AlertJoin=20 >> or AlertPause *) >> alertable: BOOLEAN :=3D FALSE; >> *************** >> *** 147,152 **** >> --- 150,160 ---- >>=20 >> defaultStackSize :=3D 3000; >>=20 >> + (* note that even though the "heavy machinery" is only used for=20 >> + multipleThreads, we still need to set up the signal handler so >> + that we can catch signals from other sources than thread = >switching. >> + e.g., we use SIGCHLD to speed up Process.Wait *) >> +=20 >> VAR >> stats: RECORD >> n_forks :=3D 0; >> *************** >> *** 511,527 **** >> XPause(until, FALSE); >> END Pause; >>=20 >> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >> VAR until :=3D n + Time.Now (); >> BEGIN >> XPause(until, TRUE); >> END AlertPause; >>=20 >> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D FALSE) = >RAISES {Alerted} =3D >> BEGIN >> INC (inCritical); >> self.waitingForTime :=3D until; >> self.alertable :=3D alertable; >> ICannotRun (State.pausing); >> DEC (inCritical); >> InternalYield (); >> --- 519,546 ---- >> XPause(until, FALSE); >> END Pause; >>=20 >> + PROCEDURE SigPause(n: LONGREAL; sig: int)=3D >> + <*FATAL Alerted*> >> + VAR until :=3D n + Time.Now (); >> + BEGIN >> + XPause(until, FALSE, sig); >> + END SigPause; >> +=20 >> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >> VAR until :=3D n + Time.Now (); >> BEGIN >> XPause(until, TRUE); >> END AlertPause; >>=20 >> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D FALSE; = >sig:int :=3D -1)=20 >> ! RAISES {Alerted} =3D >> BEGIN >> INC (inCritical); >> self.waitingForTime :=3D until; >> self.alertable :=3D alertable; >> + IF sig # -1 THEN >> + self.waitingForSig :=3D sig >> + END; >> ICannotRun (State.pausing); >> DEC (inCritical); >> InternalYield (); >> *************** >> *** 703,712 **** >> END; >> END StartSwitching; >>=20 >> ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} =3D >> BEGIN >> allow_sigvtalrm (); >> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN = >InternalYield () END; >> END switch_thread; >>=20 >> (*------------------------------------------------------------- = >scheduler ---*) >> --- 722,750 ---- >> END; >> END StartSwitching; >>=20 >> ! CONST MaxSigs =3D 64; >> ! TYPE Sig =3D [ 0..MaxSigs-1 ]; >> !=20 >> ! (* in order to listen to other signals, they have to be enabled in >> ! allow_sigvtalrm as well *) >> ! VAR (*CONST*) SIGCHLD :=3D ValueOfSIGCHLD(); >> ! =20 >> ! gotSigs :=3D SET OF Sig { }; >> !=20 >> ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >> BEGIN >> allow_sigvtalrm (); >> !=20 >> ! INC(inCritical); >> ! (* mark signal as being delivered *) >> ! IF sig >=3D 0 AND sig < MaxSigs THEN >> ! gotSigs :=3D gotSigs + SET OF Sig { sig } >> ! END; >> ! DEC(inCritical); >> !=20 >> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN=20 >> ! InternalYield ()=20 >> ! END; >> END switch_thread; >>=20 >> (*------------------------------------------------------------- = >scheduler ---*) >> *************** >> *** 782,792 **** >> IF t.alertable AND t.alertPending THEN >> CanRun (t); >> EXIT; >>=20 >> ELSIF t.waitingForTime <=3D now THEN >> CanRun (t); >> EXIT; >> ! =20 >> ELSIF NOT somePausing THEN >> earliest :=3D t.waitingForTime; >> somePausing :=3D TRUE; >> --- 820,835 ---- >> IF t.alertable AND t.alertPending THEN >> CanRun (t); >> EXIT; >> + =20 >> + ELSIF t.waitingForSig IN gotSigs THEN >> + t.waitingForSig :=3D -1; >> + CanRun(t); >> + EXIT; >>=20 >> ELSIF t.waitingForTime <=3D now THEN >> CanRun (t); >> EXIT; >> !=20 >> ELSIF NOT somePausing THEN >> earliest :=3D t.waitingForTime; >> somePausing :=3D TRUE; >> *************** >> *** 886,891 **** >> --- 929,936 ---- >> END; >> END; >>=20 >> + gotSigs :=3D SET OF Sig {}; >> +=20 >> IF t.state =3D State.alive AND (scanned OR NOT someBlocking) THEN >> IF perfOn THEN PerfRunning (t.id); END; >> (* At least one thread wants to run; transfer to it *) >> *************** >> *** 948,960 **** >>=20 >> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D >> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >> ! CONST Delay =3D 0.1D0; >> BEGIN >> LOOP >> WITH r =3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >> IF r # 0 THEN RETURN r END; >> END; >> ! Pause(Delay); >> END; >> END WaitProcess; >>=20 >> --- 993,1005 ---- >>=20 >> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D >> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >> ! CONST Delay =3D 10.0D0; >> BEGIN >> LOOP >> WITH r =3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >> IF r # 0 THEN RETURN r END; >> END; >> ! SigPause(Delay,SIGCHLD); >> END; >> END WaitProcess; >>=20 >> *************** >> *** 1361,1364 **** >> --- 1406,1412 ---- >> VAR debug :=3D RTParams.IsPresent ("debugthreads"); >>=20 >> BEGIN >> + (* we need to call set up the signal handler for other reasons = >than >> + just thread switching now *) >> + setup_sigvtalrm (switch_thread); >> END ThreadPosix. >> Index: ThreadPosixC.c >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: = >/usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v >> retrieving revision 1.59 >> diff -c -r1.59 ThreadPosixC.c >> *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 >> --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 >> *************** >> *** 78,88 **** >> --- 78,97 ---- >>=20 >> sigemptyset(&ThreadSwitchSignal); >> sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); >> + sigaddset(&ThreadSwitchSignal, SIGCHLD); >>=20 >> act.sa_handler =3D handler; >> act.sa_flags =3D SA_RESTART; >> sigemptyset(&(act.sa_mask)); >> if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); >> + if (sigaction (SIGCHLD, &act, NULL)) abort(); >> + } >> +=20 >> + int >> + __cdecl >> + ThreadPosix__value_of_SIGCHLD(void) >> + { >> + return SIGCHLD; >> } >>=20 >> void From hosking at cs.purdue.edu Sun Feb 13 21:37:32 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:37:32 -0500 Subject: [M3devel] Performance issues with Process.Wait under userthreads In-Reply-To: <20110212204302.19C781A2078@async.async.caltech.edu> References: <20110212204302.19C781A2078@async.async.caltech.edu> Message-ID: <80DB3538-EAFF-4C03-A117-75EE49BB5D2C@cs.purdue.edu> This all seems pretty reasonable. On Feb 12, 2011, at 3:43 PM, Mika Nystrom wrote: > Hi again m3devel (especially Tony), > > I have finally taken a bite of a problem that has been annoying me > for a very, very long time. > > Under user threads, the code for Process.Wait is as follows (see > ThreadPosix.m3): > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > CONST Delay = 0.1D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > Pause(Delay); > END; > END WaitProcess; > > It inserts a 0.1 second delay after each failed waitpid. This is extremely > annoying for programs that start a long sequence of child processes and > wait for them in sequence. Namely, the compiler itself. As a result > the cm3 compiler (and PM3's m3build) are normally very very slow when > using user threads. For about the last ten years, I've had a hacked > up m3build (for my PM3 installation) that skips the Pause and busy-waits > instead. > > Note there is another problem here. Since the Modula-3 runtime ignores > SIGCHLD, no zombie processes are created since the Unix system automatically > reaps the child processes. I can see this would be a problem since PIDs > are eventually reused and ... couldn't Uexec.waitpid wind up referring to > the wrong process?? > > I will further note that the comment in Process.i3 reads as follows: > > PROCEDURE Wait(p: T): ExitCode; > > Wait until the process with handle p terminates, then free the operating system resources associated with the process and return an exit code indicating the reason for its termination. It is a checked runtime error to call Wait twice on the same process handle. > > I am going to take this as fair warning that Process.Create *may* use > resources that are not going to be released until Process.Wait has been > called. > > I have modified (in my local copy of CM3) the system as follows. > I have come up with a semi-general mechanism for immediately unblocking > a thread on the receipt of a unix signal. > > 1. The system relies on changing > ThreadPosix.XPause such that if a signal is allowed to wake up a threads, > that fact is recorded in a new field in the thread's descriptor > record (of type ThreadPosix.T). > > 2. On receipt of a waited-for unix signal, a mask is set and control > is passed to the thread scheduler which maintains the non-zero mask for > exactly one iteration through the thread ring. > > 3. If a thread is paused and waiting for EITHER a signal or some time, > the thread is released for running and the thread's waiting state is > cleared. > > The changes are more or less as follows: > > 1. I have added a new field of type "int" to ThreadPosix.T: > > (* if state = pausing, the time at which we can restart *) > waitingForTime: Time.T; > > + (* if state = pausing, the signal that truncates the pause *) > + waitingForSig: int := -1; > + > (* true if we are waiting during an AlertWait or AlertJoin > or AlertPause *) > alertable: BOOLEAN := FALSE; > > > 2. Modifications to pause: > > + PROCEDURE SigPause(n: LONGREAL; sig: int)= > + <*FATAL Alerted*> > + VAR until := n + Time.Now (); > + BEGIN > + XPause(until, FALSE, sig); > + END SigPause; > + > PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= > VAR until := n + Time.Now (); > BEGIN > XPause(until, TRUE); > END AlertPause; > > ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) > ! RAISES {Alerted} = > BEGIN > INC (inCritical); > self.waitingForTime := until; > self.alertable := alertable; > + IF sig # -1 THEN > + self.waitingForSig := sig > + END; > ICannotRun (State.pausing); > DEC (inCritical); > InternalYield (); > > 3. The received-signals mask: > > ! CONST MaxSigs = 64; > ! TYPE Sig = [ 0..MaxSigs-1 ]; > ! > ! (* in order to listen to other signals, they have to be enabled in > ! allow_sigvtalrm as well *) > ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); > ! > ! gotSigs := SET OF Sig { }; > ! > > ValueOfSIGCHLD() is a C function used to get the value of the SIGCHLD > constant without guessing at it (in ThreadPosixC.c). > > 4. changes to the signal handler: > > ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); > ! > ! INC(inCritical); > ! (* mark signal as being delivered *) > ! IF sig >= 0 AND sig < MaxSigs THEN > ! gotSigs := gotSigs + SET OF Sig { sig } > ! END; > ! DEC(inCritical); > ! > ! IF inCritical = 0 AND heapState.inCritical = 0 THEN > ! InternalYield () > ! END; > END switch_thread; > > Note that I don't know if INC/DEC(inCritical) does exactly the right > thing here. > > 5. changes to the scheduler: > > a. thread wakeup > IF t.alertable AND t.alertPending THEN > CanRun (t); > EXIT; > + > + ELSIF t.waitingForSig IN gotSigs THEN > + t.waitingForSig := -1; > + CanRun(t); > + EXIT; > > ELSIF t.waitingForTime <= now THEN > CanRun (t); > EXIT; > ! > > b. clearing the mask > > END; > END; > > + gotSigs := SET OF Sig {}; > + > IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN > IF perfOn THEN PerfRunning (t.id); END; > (* At least one thread wants to run; transfer to it *) > > 6. changes to WaitProcess (Process.Wait): > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > ! CONST Delay = 10.0D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > ! SigPause(Delay,SIGCHLD); > END; > END WaitProcess; > > 7. install signal handler even if program is single-threaded: > > BEGIN > + (* we need to call set up the signal handler for other reasons than > + just thread switching now *) > + setup_sigvtalrm (switch_thread); > END ThreadPosix. > > 8. modify signal handler in ThreadPosixC.c to catch SIGCHLD: > > sigemptyset(&ThreadSwitchSignal); > sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); > + sigaddset(&ThreadSwitchSignal, SIGCHLD); > > act.sa_handler = handler; > act.sa_flags = SA_RESTART; > sigemptyset(&(act.sa_mask)); > if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); > + if (sigaction (SIGCHLD, &act, NULL)) abort(); > > I'll send the complete diff in a separate message for those who want > to study it more closely. > > I propose the above changes for inclusion in the current CM3 repository. > > Mika From hosking at cs.purdue.edu Sun Feb 13 21:44:11 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:44:11 -0500 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110213203726.82ADB1A2078@async.async.caltech.edu> References: <20110212204406.88A661A2078@async.async.caltech.edu> <20110213203726.82ADB1A2078@async.async.caltech.edu> Message-ID: Sorry, I missed you e-mail describing this change. Looks good. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: > Tony, > > The old user threads has an old problem with Process.Wait. It has a > Thread.Pause(0.1d0) in it. This causes the compiler to run three times > slower than it should. The m3build I've been using for years has > that Thread.Pause removed (so it uses 100% CPU instead, busy-waiting...) > > But yeah user threads works the same as it used to (except for > now calling pthread_atfork, which I'll return to in another email...) > > Mika > > Tony Hosking writes: >> Hi Mika, Why was this change needed (I haven't looked closely)? I don't = >> think we diverged much from previous implementations of user threads, = >> which presumably used to work for you. >> >> On Feb 12, 2011, at 3:44 PM, Mika Nystrom wrote: >> >>> =20 >>> Index: ThreadPosix.i3 >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> RCS file: = >> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v >>> retrieving revision 1.16 >>> diff -c -r1.16 ThreadPosix.i3 >>> *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 >>> --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 >>> *************** >>> *** 39,42 **** >>> --- 39,45 ---- >>> =20 >>> = >> (*------------------------------------------------------------------------= >> ---*) >>> =20 >>> + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> >>> + PROCEDURE ValueOfSIGCHLD(): int; >>> +=20 >>> END ThreadPosix. >>> Index: ThreadPosix.m3 >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> RCS file: = >> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v >>> retrieving revision 1.73 >>> diff -c -r1.73 ThreadPosix.m3 >>> *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 >>> --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 >>> *************** >>> *** 97,102 **** >>> --- 97,105 ---- >>> (* if state =3D pausing, the time at which we can restart *) >>> waitingForTime: Time.T; >>> =20 >>> + (* if state =3D pausing, the signal that truncates the pause *) >>> + waitingForSig: int :=3D -1; >>> +=20 >>> (* true if we are waiting during an AlertWait or AlertJoin=20 >>> or AlertPause *) >>> alertable: BOOLEAN :=3D FALSE; >>> *************** >>> *** 147,152 **** >>> --- 150,160 ---- >>> =20 >>> defaultStackSize :=3D 3000; >>> =20 >>> + (* note that even though the "heavy machinery" is only used for=20 >>> + multipleThreads, we still need to set up the signal handler so >>> + that we can catch signals from other sources than thread = >> switching. >>> + e.g., we use SIGCHLD to speed up Process.Wait *) >>> +=20 >>> VAR >>> stats: RECORD >>> n_forks :=3D 0; >>> *************** >>> *** 511,527 **** >>> XPause(until, FALSE); >>> END Pause; >>> =20 >>> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >>> VAR until :=3D n + Time.Now (); >>> BEGIN >>> XPause(until, TRUE); >>> END AlertPause; >>> =20 >>> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D FALSE) = >> RAISES {Alerted} =3D >>> BEGIN >>> INC (inCritical); >>> self.waitingForTime :=3D until; >>> self.alertable :=3D alertable; >>> ICannotRun (State.pausing); >>> DEC (inCritical); >>> InternalYield (); >>> --- 519,546 ---- >>> XPause(until, FALSE); >>> END Pause; >>> =20 >>> + PROCEDURE SigPause(n: LONGREAL; sig: int)=3D >>> + <*FATAL Alerted*> >>> + VAR until :=3D n + Time.Now (); >>> + BEGIN >>> + XPause(until, FALSE, sig); >>> + END SigPause; >>> +=20 >>> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >>> VAR until :=3D n + Time.Now (); >>> BEGIN >>> XPause(until, TRUE); >>> END AlertPause; >>> =20 >>> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D FALSE; = >> sig:int :=3D -1)=20 >>> ! RAISES {Alerted} =3D >>> BEGIN >>> INC (inCritical); >>> self.waitingForTime :=3D until; >>> self.alertable :=3D alertable; >>> + IF sig # -1 THEN >>> + self.waitingForSig :=3D sig >>> + END; >>> ICannotRun (State.pausing); >>> DEC (inCritical); >>> InternalYield (); >>> *************** >>> *** 703,712 **** >>> END; >>> END StartSwitching; >>> =20 >>> ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} =3D >>> BEGIN >>> allow_sigvtalrm (); >>> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN = >> InternalYield () END; >>> END switch_thread; >>> =20 >>> (*------------------------------------------------------------- = >> scheduler ---*) >>> --- 722,750 ---- >>> END; >>> END StartSwitching; >>> =20 >>> ! CONST MaxSigs =3D 64; >>> ! TYPE Sig =3D [ 0..MaxSigs-1 ]; >>> !=20 >>> ! (* in order to listen to other signals, they have to be enabled in >>> ! allow_sigvtalrm as well *) >>> ! VAR (*CONST*) SIGCHLD :=3D ValueOfSIGCHLD(); >>> ! =20 >>> ! gotSigs :=3D SET OF Sig { }; >>> !=20 >>> ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>> BEGIN >>> allow_sigvtalrm (); >>> !=20 >>> ! INC(inCritical); >>> ! (* mark signal as being delivered *) >>> ! IF sig >=3D 0 AND sig < MaxSigs THEN >>> ! gotSigs :=3D gotSigs + SET OF Sig { sig } >>> ! END; >>> ! DEC(inCritical); >>> !=20 >>> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN=20 >>> ! InternalYield ()=20 >>> ! END; >>> END switch_thread; >>> =20 >>> (*------------------------------------------------------------- = >> scheduler ---*) >>> *************** >>> *** 782,792 **** >>> IF t.alertable AND t.alertPending THEN >>> CanRun (t); >>> EXIT; >>> =20 >>> ELSIF t.waitingForTime <=3D now THEN >>> CanRun (t); >>> EXIT; >>> ! =20 >>> ELSIF NOT somePausing THEN >>> earliest :=3D t.waitingForTime; >>> somePausing :=3D TRUE; >>> --- 820,835 ---- >>> IF t.alertable AND t.alertPending THEN >>> CanRun (t); >>> EXIT; >>> + =20 >>> + ELSIF t.waitingForSig IN gotSigs THEN >>> + t.waitingForSig :=3D -1; >>> + CanRun(t); >>> + EXIT; >>> =20 >>> ELSIF t.waitingForTime <=3D now THEN >>> CanRun (t); >>> EXIT; >>> !=20 >>> ELSIF NOT somePausing THEN >>> earliest :=3D t.waitingForTime; >>> somePausing :=3D TRUE; >>> *************** >>> *** 886,891 **** >>> --- 929,936 ---- >>> END; >>> END; >>> =20 >>> + gotSigs :=3D SET OF Sig {}; >>> +=20 >>> IF t.state =3D State.alive AND (scanned OR NOT someBlocking) THEN >>> IF perfOn THEN PerfRunning (t.id); END; >>> (* At least one thread wants to run; transfer to it *) >>> *************** >>> *** 948,960 **** >>> =20 >>> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D >>> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >>> ! CONST Delay =3D 0.1D0; >>> BEGIN >>> LOOP >>> WITH r =3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >>> IF r # 0 THEN RETURN r END; >>> END; >>> ! Pause(Delay); >>> END; >>> END WaitProcess; >>> =20 >>> --- 993,1005 ---- >>> =20 >>> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D >>> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >>> ! CONST Delay =3D 10.0D0; >>> BEGIN >>> LOOP >>> WITH r =3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >>> IF r # 0 THEN RETURN r END; >>> END; >>> ! SigPause(Delay,SIGCHLD); >>> END; >>> END WaitProcess; >>> =20 >>> *************** >>> *** 1361,1364 **** >>> --- 1406,1412 ---- >>> VAR debug :=3D RTParams.IsPresent ("debugthreads"); >>> =20 >>> BEGIN >>> + (* we need to call set up the signal handler for other reasons = >> than >>> + just thread switching now *) >>> + setup_sigvtalrm (switch_thread); >>> END ThreadPosix. >>> Index: ThreadPosixC.c >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> RCS file: = >> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v >>> retrieving revision 1.59 >>> diff -c -r1.59 ThreadPosixC.c >>> *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 >>> --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 >>> *************** >>> *** 78,88 **** >>> --- 78,97 ---- >>> =20 >>> sigemptyset(&ThreadSwitchSignal); >>> sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); >>> + sigaddset(&ThreadSwitchSignal, SIGCHLD); >>> =20 >>> act.sa_handler =3D handler; >>> act.sa_flags =3D SA_RESTART; >>> sigemptyset(&(act.sa_mask)); >>> if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); >>> + if (sigaction (SIGCHLD, &act, NULL)) abort(); >>> + } >>> +=20 >>> + int >>> + __cdecl >>> + ThreadPosix__value_of_SIGCHLD(void) >>> + { >>> + return SIGCHLD; >>> } >>> =20 >>> void From hosking at cs.purdue.edu Sun Feb 13 21:46:26 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:46:26 -0500 Subject: [M3devel] More on threading In-Reply-To: <20110213181717.4F0E91A2078@async.async.caltech.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> Message-ID: <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> I know it worked properly at one point in time, when I stressed it very hard for some GC-related work I was doing. I need to see instances of failure (deadlock, crashes) captured in the debugger, running with the latest head code. I don't want to try and diagnose old versions of the system if at all possible. I may have some time to try your thread test program in the next couple of weeks. Please let me know workloads that reliably cause failure. On Feb 13, 2011, at 1:17 PM, Mika Nystrom wrote: > I don't think pthreads works properly anywhere at > the moment. My guess is that it never has, if you stress it hard enough. From mika at async.caltech.edu Sun Feb 13 22:15:51 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 13:15:51 -0800 Subject: [M3devel] More on threading In-Reply-To: <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> Message-ID: <20110213211551.2A60A1A2078@async.async.caltech.edu> I have bootstrapped so many versions of the compiler on so many systems in the past week I've probably lost track of where we are, yes. I'll do some testing on a bona fide head checkout and let you know what I find. Mika Tony Hosking writes: >I know it worked properly at one point in time, when I stressed it very = >hard for some GC-related work I was doing. >I need to see instances of failure (deadlock, crashes) captured in the = >debugger, running with the latest head code. >I don't want to try and diagnose old versions of the system if at all = >possible. >I may have some time to try your thread test program in the next couple = >of weeks. Please let me know workloads that reliably cause failure. > > >On Feb 13, 2011, at 1:17 PM, Mika Nystrom wrote: > >> I don't think pthreads works properly anywhere at >> the moment. My guess is that it never has, if you stress it hard = >enough. From mika at async.caltech.edu Sun Feb 13 23:08:04 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 14:08:04 -0800 Subject: [M3devel] SIGCHLD diff Message-ID: <20110213220804.7A9001A2078@async.async.caltech.edu> Hi Tony, I found a problem with the change. I would appreciate it if you and Jay could think about what I found and my solution (and anybody else is welcome to as well, of course). When fiddling with cm3 to get it to run in parallel, I pushed it up to fork up to 100 processes at once, just to see what happens. When I did that (with my SIGCHLD patch) I got a few errors such as illegal instruction and a bus error or two. I believe what was happening was that I was taking too many signals at once, because of the following code in ThreadPosix.m3: PROCEDURE switch_thread (sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); (* here *) INC(inCritical); (* mark signal as being delivered *) IF sig >= 0 AND sig < MaxSigs THEN gotSigs := gotSigs + SET OF Sig { sig } END; DEC(inCritical); IF inCritical = 0 AND heapState.inCritical = 0 THEN InternalYield () END; END switch_thread; I found it a bit odd that the first action of the signal handler is to enable signals. With VTALRM this is innocuous, but in my case, imagine what happens if you have a lot of exited child processes queued up. What happens is that your signal stack gets wound deeper and deeper, and each time you catch one SIGCHLD you ask for another. Eventually you run out of stack space. My change, which appears to fix this problem, is to split up the "allow" function as follows: PROCEDURE switch_thread (sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); (* VTALRM only *) INC(inCritical); IF inCritical = 1 THEN allow_othersigs (); END; (* SIGCHLD *) (* mark signal as being delivered *) IF sig >= 0 AND sig < MaxSigs THEN gotSigs := gotSigs + SET OF Sig { sig } END; DEC(inCritical); IF inCritical = 0 AND heapState.inCritical = 0 THEN InternalYield () END; END switch_thread; This way, SIGCHLD is not taken in nested signal handlers. But is there a problem here? Well, the fallback is that it takes 0.1 seconds to wake up on Process.Wait . Is there a chance that SIGCHLD signals are left pending? My argument is no, since in the "base" environment (i.e., not executing in a signal handler) SIGCHLD is enabled as a postcondition of allow_othersigs. All *other* occurrences of allow_sigvtalrm are of course replaced with allow_sigvtalrm(); allow_othersigs(); The goal is that SIGCHLD be enabled precisely when SIGVTALRM is except inside the signal handler itself. Mika Tony Hosking writes: >Sorry, I missed you e-mail describing this change. Looks good. > >Antony Hosking | Associate Professor | Computer Science | Purdue = >University >305 N. University Street | West Lafayette | IN 47907 | USA >Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > >On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: > >> Tony, >>=20 >> The old user threads has an old problem with Process.Wait. It has a=20= > >> Thread.Pause(0.1d0) in it. This causes the compiler to run three = >times >> slower than it should. The m3build I've been using for years has >> that Thread.Pause removed (so it uses 100% CPU instead, = >busy-waiting...) >>=20 >> But yeah user threads works the same as it used to (except for=20 >> now calling pthread_atfork, which I'll return to in another email...) >>=20 >> Mika >>=20 >> Tony Hosking writes: >>> Hi Mika, Why was this change needed (I haven't looked closely)? I = >don't =3D >>> think we diverged much from previous implementations of user threads, = >=3D >>> which presumably used to work for you. >>>=20 >>> On Feb 12, 2011, at 3:44 PM, Mika Nystrom wrote: >>>=20 >>>> =3D20 >>>> Index: ThreadPosix.i3 >>>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> =3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D= >3D=3D3D=3D3D=3D3D >>>> RCS file: =3D >>> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v >>>> retrieving revision 1.16 >>>> diff -c -r1.16 ThreadPosix.i3 >>>> *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 >>>> --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 >>>> *************** >>>> *** 39,42 **** >>>> --- 39,45 ---- >>>> =3D20 >>>> =3D >>> = >(*------------------------------------------------------------------------= >=3D >>> ---*) >>>> =3D20 >>>> + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> >>>> + PROCEDURE ValueOfSIGCHLD(): int; >>>> +=3D20 >>>> END ThreadPosix. >>>> Index: ThreadPosix.m3 >>>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> =3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D= >3D=3D3D=3D3D=3D3D >>>> RCS file: =3D >>> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v >>>> retrieving revision 1.73 >>>> diff -c -r1.73 ThreadPosix.m3 >>>> *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 >>>> --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 >>>> *************** >>>> *** 97,102 **** >>>> --- 97,105 ---- >>>> (* if state =3D3D pausing, the time at which we can restart *) >>>> waitingForTime: Time.T; >>>> =3D20 >>>> + (* if state =3D3D pausing, the signal that truncates the pause = >*) >>>> + waitingForSig: int :=3D3D -1; >>>> +=3D20 >>>> (* true if we are waiting during an AlertWait or AlertJoin=3D20 >>>> or AlertPause *) >>>> alertable: BOOLEAN :=3D3D FALSE; >>>> *************** >>>> *** 147,152 **** >>>> --- 150,160 ---- >>>> =3D20 >>>> defaultStackSize :=3D3D 3000; >>>> =3D20 >>>> + (* note that even though the "heavy machinery" is only used = >for=3D20 >>>> + multipleThreads, we still need to set up the signal handler = >so >>>> + that we can catch signals from other sources than thread =3D >>> switching. >>>> + e.g., we use SIGCHLD to speed up Process.Wait *) >>>> +=3D20 >>>> VAR >>>> stats: RECORD >>>> n_forks :=3D3D 0; >>>> *************** >>>> *** 511,527 **** >>>> XPause(until, FALSE); >>>> END Pause; >>>> =3D20 >>>> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D3D >>>> VAR until :=3D3D n + Time.Now (); >>>> BEGIN >>>> XPause(until, TRUE); >>>> END AlertPause; >>>> =3D20 >>>> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D3D FALSE) = >=3D >>> RAISES {Alerted} =3D3D >>>> BEGIN >>>> INC (inCritical); >>>> self.waitingForTime :=3D3D until; >>>> self.alertable :=3D3D alertable; >>>> ICannotRun (State.pausing); >>>> DEC (inCritical); >>>> InternalYield (); >>>> --- 519,546 ---- >>>> XPause(until, FALSE); >>>> END Pause; >>>> =3D20 >>>> + PROCEDURE SigPause(n: LONGREAL; sig: int)=3D3D >>>> + <*FATAL Alerted*> >>>> + VAR until :=3D3D n + Time.Now (); >>>> + BEGIN >>>> + XPause(until, FALSE, sig); >>>> + END SigPause; >>>> +=3D20 >>>> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D3D >>>> VAR until :=3D3D n + Time.Now (); >>>> BEGIN >>>> XPause(until, TRUE); >>>> END AlertPause; >>>> =3D20 >>>> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D3D FALSE; = >=3D >>> sig:int :=3D3D -1)=3D20 >>>> ! RAISES {Alerted} =3D3D >>>> BEGIN >>>> INC (inCritical); >>>> self.waitingForTime :=3D3D until; >>>> self.alertable :=3D3D alertable; >>>> + IF sig # -1 THEN >>>> + self.waitingForSig :=3D3D sig >>>> + END; >>>> ICannotRun (State.pausing); >>>> DEC (inCritical); >>>> InternalYield (); >>>> *************** >>>> *** 703,712 **** >>>> END; >>>> END StartSwitching; >>>> =3D20 >>>> ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} =3D3D= > >>>> BEGIN >>>> allow_sigvtalrm (); >>>> ! IF inCritical =3D3D 0 AND heapState.inCritical =3D3D 0 THEN =3D >>> InternalYield () END; >>>> END switch_thread; >>>> =3D20 >>>> (*------------------------------------------------------------- =3D >>> scheduler ---*) >>>> --- 722,750 ---- >>>> END; >>>> END StartSwitching; >>>> =3D20 >>>> ! CONST MaxSigs =3D3D 64; >>>> ! TYPE Sig =3D3D [ 0..MaxSigs-1 ]; >>>> !=3D20 >>>> ! (* in order to listen to other signals, they have to be enabled in >>>> ! allow_sigvtalrm as well *) >>>> ! VAR (*CONST*) SIGCHLD :=3D3D ValueOfSIGCHLD(); >>>> ! =3D20 >>>> ! gotSigs :=3D3D SET OF Sig { }; >>>> !=3D20 >>>> ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D3D >>>> BEGIN >>>> allow_sigvtalrm (); >>>> !=3D20 >>>> ! INC(inCritical); >>>> ! (* mark signal as being delivered *) >>>> ! IF sig >=3D3D 0 AND sig < MaxSigs THEN >>>> ! gotSigs :=3D3D gotSigs + SET OF Sig { sig } >>>> ! END; >>>> ! DEC(inCritical); >>>> !=3D20 >>>> ! IF inCritical =3D3D 0 AND heapState.inCritical =3D3D 0 THEN=3D20= > >>>> ! InternalYield ()=3D20 >>>> ! END; >>>> END switch_thread; >>>> =3D20 >>>> (*------------------------------------------------------------- =3D >>> scheduler ---*) >>>> *************** >>>> *** 782,792 **** >>>> IF t.alertable AND t.alertPending THEN >>>> CanRun (t); >>>> EXIT; >>>> =3D20 >>>> ELSIF t.waitingForTime <=3D3D now THEN >>>> CanRun (t); >>>> EXIT; >>>> ! =3D20 >>>> ELSIF NOT somePausing THEN >>>> earliest :=3D3D t.waitingForTime; >>>> somePausing :=3D3D TRUE; >>>> --- 820,835 ---- >>>> IF t.alertable AND t.alertPending THEN >>>> CanRun (t); >>>> EXIT; >>>> + =3D20 >>>> + ELSIF t.waitingForSig IN gotSigs THEN >>>> + t.waitingForSig :=3D3D -1; >>>> + CanRun(t); >>>> + EXIT; >>>> =3D20 >>>> ELSIF t.waitingForTime <=3D3D now THEN >>>> CanRun (t); >>>> EXIT; >>>> !=3D20 >>>> ELSIF NOT somePausing THEN >>>> earliest :=3D3D t.waitingForTime; >>>> somePausing :=3D3D TRUE; >>>> *************** >>>> *** 886,891 **** >>>> --- 929,936 ---- >>>> END; >>>> END; >>>> =3D20 >>>> + gotSigs :=3D3D SET OF Sig {}; >>>> +=3D20 >>>> IF t.state =3D3D State.alive AND (scanned OR NOT someBlocking) = >THEN >>>> IF perfOn THEN PerfRunning (t.id); END; >>>> (* At least one thread wants to run; transfer to it *) >>>> *************** >>>> *** 948,960 **** >>>> =3D20 >>>> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D3D >>>> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >>>> ! CONST Delay =3D3D 0.1D0; >>>> BEGIN >>>> LOOP >>>> WITH r =3D3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >>>> IF r # 0 THEN RETURN r END; >>>> END; >>>> ! Pause(Delay); >>>> END; >>>> END WaitProcess; >>>> =3D20 >>>> --- 993,1005 ---- >>>> =3D20 >>>> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D3D >>>> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >>>> ! CONST Delay =3D3D 10.0D0; >>>> BEGIN >>>> LOOP >>>> WITH r =3D3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >>>> IF r # 0 THEN RETURN r END; >>>> END; >>>> ! SigPause(Delay,SIGCHLD); >>>> END; >>>> END WaitProcess; >>>> =3D20 >>>> *************** >>>> *** 1361,1364 **** >>>> --- 1406,1412 ---- >>>> VAR debug :=3D3D RTParams.IsPresent ("debugthreads"); >>>> =3D20 >>>> BEGIN >>>> + (* we need to call set up the signal handler for other reasons =3D= > >>> than >>>> + just thread switching now *) >>>> + setup_sigvtalrm (switch_thread); >>>> END ThreadPosix. >>>> Index: ThreadPosixC.c >>>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> =3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D= >3D=3D3D=3D3D=3D3D >>>> RCS file: =3D >>> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v >>>> retrieving revision 1.59 >>>> diff -c -r1.59 ThreadPosixC.c >>>> *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 >>>> --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 >>>> *************** >>>> *** 78,88 **** >>>> --- 78,97 ---- >>>> =3D20 >>>> sigemptyset(&ThreadSwitchSignal); >>>> sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); >>>> + sigaddset(&ThreadSwitchSignal, SIGCHLD); >>>> =3D20 >>>> act.sa_handler =3D3D handler; >>>> act.sa_flags =3D3D SA_RESTART; >>>> sigemptyset(&(act.sa_mask)); >>>> if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); >>>> + if (sigaction (SIGCHLD, &act, NULL)) abort(); >>>> + } >>>> +=3D20 >>>> + int >>>> + __cdecl >>>> + ThreadPosix__value_of_SIGCHLD(void) >>>> + { >>>> + return SIGCHLD; >>>> } >>>> =3D20 >>>> void From mika at async.caltech.edu Sun Feb 13 23:27:51 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 14:27:51 -0800 Subject: [M3devel] More on threading In-Reply-To: <20110213181717.4F0E91A2078@async.async.caltech.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> Message-ID: <20110213222751.A98F31A2078@async.async.caltech.edu> All right, I have some more interesting observations on user threading. As well as on parallelizing CM3. They are connected because the latter is the source of the former. User threads on FreeBSD (FreeBSD7 /amd64) ======================= The fact that my process was getting stuck in "umtxn" (I think that is what top called it) turned out to be something inside fork() in libthr.so. Now, libthr.so is FreeBSD's name for pthreads, and this was a user threading program. This does not seem right. I tried the following: 1. remove the call to pthread_atfork No change in behavior. My parallelized CM3 still got stuck in umtxn. 2. remove the -pthread linking flag Success! Parallelized CM3 works! So, clearly -pthread is harmful to the health of Modula-3 programs running with user threading on FreeBSD. This is on FreeBSD 7.1, so it means pthread_atfork has to be removed as well. I'm not sure how to make the building of this as transparent as possible, but what's currently checked in is in any case wrong. Parallelizing CM3 ================= I found one bug in my earlier CM3 patch. Calls to Utils.* need to be monitored because there is a shared hash table in that module. New diff attached. With this diff the compiler runs correctly even when it is running 100 instances of the backend in parallel. Speedup is about 2.0 on a machine with four cores for a largish source directory. The max. number of subprocesses is set on line 985 of Builder.m3 . I am not claiming this patch is anywhere near "polished" enough to go into the CM3 tree at the moment. It's an experiment and starting point for further work... Mika Index: cm3/src/Builder.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/cm3/src/Builder.m3,v retrieving revision 1.36 diff -c -r1.36 Builder.m3 *** cm3/src/Builder.m3 24 Aug 2010 05:24:24 -0000 1.36 --- cm3/src/Builder.m3 13 Feb 2011 22:17:16 -0000 *************** *** 15,20 **** --- 15,22 ---- IMPORT QIdent; FROM Target IMPORT M3BackendMode_t, BackendAssembly, BackendModeStrings; FROM M3Path IMPORT OSKind, OSKindStrings; + IMPORT Pathname; + IMPORT RefSeq; TYPE UK = M3Unit.Kind; *************** *** 131,136 **** --- 133,139 ---- link_coverage : TEXT; (* coverage library *) m3_front_flags: Arg.List; (* configuration options for the front *) m3_options : Arg.List; (* misc. user options for the frontend *) + delayBackend := FALSE; END; TYPE *************** *** 932,937 **** --- 935,956 ---- (*------------------------------------------------------------ compilation --*) + TYPE MarkerPromise = QMachine.Promise BRANDED OBJECT OVERRIDES fulfil := FulfilNothing END; (* an empty promise, use as a marker *) + + PROCEDURE FulfilNothing(<*UNUSED*>mp : MarkerPromise) = BEGIN END FulfilNothing; + + TYPE SeqClosure = Thread.Closure OBJECT seq : RefSeq.T; OVERRIDES apply := SeqApply END; + + PROCEDURE SeqApply(cl : SeqClosure) : REFANY = + BEGIN (* force a Sequence of promises *) + FOR i := 0 TO cl.seq.size()-1 DO + WITH p = NARROW(cl.seq.get(i),QMachine.Promise) DO + p.fulfil() + END + END; + RETURN NIL + END SeqApply; + PROCEDURE CompileEverything (s: State; schedule: SourceList) = VAR u: M3Unit.T; BEGIN *************** *** 941,948 **** (* compile all the sources using the initial schedule *) FOR i := 0 TO LAST (schedule^) DO ! CompileOne (s, schedule[i]); END; FlushPending (s); (* recompile any interfaces where we goofed on the exports *) --- 960,1000 ---- (* compile all the sources using the initial schedule *) FOR i := 0 TO LAST (schedule^) DO ! s.delayBackend := TRUE; (* tell backend to promise instead of executing *) ! TRY ! CompileOne (s, schedule[i]); ! FINALLY ! s.delayBackend := FALSE; (* tell backend to go back to executing *) ! END; ! ! s.machine.promises.addhi(NEW(MarkerPromise)); (* promises, promises *) ! END; + + VAR + curSeq := NEW(RefSeq.T).init(); (* sequential chain *) + threads := NEW(RefSeq.T).init(); (* parallel chains *) + BEGIN + FOR i := 0 TO s.machine.promises.size()-1 DO + WITH p = s.machine.promises.get(i) DO + curSeq.addhi(p); + IF i = s.machine.promises.size()-1 OR ISTYPE(p,MarkerPromise) THEN (* mark as end of one sequential chain of promises *) + WITH cl = NEW(SeqClosure, seq := curSeq) DO + threads.addhi (Thread.Fork(cl)); (* start sequential chain *) + + IF threads.size() > 400 THEN EVAL Thread.Join(threads.remlo()) END; (* limit number of sequential chains executing in parallel *) + + curSeq := NEW(RefSeq.T).init() + END + END + END + END; + WHILE threads.size() > 0 DO EVAL Thread.Join(threads.remlo()) END; (* wait for all sequential chains to end *) + END; + + EVAL s.machine.promises.init(); (* clear out promises *) + + FlushPending (s); (* recompile any interfaces where we goofed on the exports *) *************** *** 1151,1156 **** --- 1203,1229 ---- END; END CompileM3; + TYPE + NotePromise = QMachine.Promise OBJECT + nam : Pathname.T; + OVERRIDES + fulfil := FulfilNP; + END; + + RemovePromise = QMachine.Promise OBJECT + nam : Pathname.T; + OVERRIDES + fulfil := FulfilRP; + END; + + VAR utilsMu := NEW(MUTEX); (* Utils.* fiddles with a table *) + + PROCEDURE FulfilNP(np : NotePromise) = + BEGIN LOCK utilsMu DO Utils.NoteTempFile(np.nam) END END FulfilNP; + + PROCEDURE FulfilRP(rp : RemovePromise) = + BEGIN LOCK utilsMu DO Utils.Remove(rp.nam) END END FulfilRP; + PROCEDURE PushOneM3 (s: State; u: M3Unit.T): BOOLEAN = VAR tmpC, tmpS: TEXT; *************** *** 1191,1208 **** | 3 => (* -bootstrap, +m3back, +asm *) tmpC := TempCName (u); tmpS := TempSName (u); IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; IF RunM3 (s, u, tmpC) THEN ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) ! AND RunAsm (s, tmpS, u.object) THEN END; need_merge := TRUE; END; IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; | 6, (* +bootstrap, +m3back, -asm *) 7 => (* +bootstrap, +m3back, +asm *) --- 1264,1320 ---- | 3 => (* -bootstrap, +m3back, +asm *) + IF s.delayBackend THEN tmpC := TempCName (u); tmpS := TempSName (u); + (* IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; + *) + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(NotePromise, nam := tmpC)) + END; + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(NotePromise, nam := tmpS)) + END; + IF RunM3 (s, u, tmpC) THEN ! s.machine.record(TRUE); ! TRY ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) ! AND RunAsm (s, tmpS, u.object) THEN ! END; ! FINALLY ! s.machine.record(FALSE) END; + need_merge := TRUE; END; + (* IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; + *) + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpC)) + END; + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpS)) + END; + + ELSE + tmpC := TempCName (u); + tmpS := TempSName (u); + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; + IF RunM3 (s, u, tmpC) THEN + IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) + AND RunAsm (s, tmpS, u.object) THEN + END; + need_merge := TRUE; + END; + IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; + IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; + END | 6, (* +bootstrap, +m3back, -asm *) 7 => (* +bootstrap, +m3back, +asm *) Index: m3quake/src/QMachine.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.i3,v retrieving revision 1.6 diff -c -r1.6 QMachine.i3 *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -0000 1.6 --- m3quake/src/QMachine.i3 13 Feb 2011 22:17:18 -0000 *************** *** 8,13 **** --- 8,14 ---- IMPORT Thread, Wr, QValue, QCode; FROM Quake IMPORT Machine, Error, ID, IDMap; + IMPORT RefSeq; REVEAL T <: T_; *************** *** 15,20 **** --- 16,22 ---- T = Machine; T_ = OBJECT map: IDMap := NIL; (* READONLY *) + promises : RefSeq.T; METHODS init (map: IDMap): T; evaluate (s: QCode.Stream) RAISES {Error, Thread.Alerted}; *************** *** 37,42 **** --- 39,46 ---- set_wr (wr: Wr.T); exec_echo (b: BOOLEAN): BOOLEAN; trace (b: BOOLEAN); + + record(on : BOOLEAN); (* instead of performing certain acts, promise *) END; PROCEDURE PushBool (t: T; b: BOOLEAN); *************** *** 51,54 **** --- 55,63 ---- PROCEDURE GetEnv (default, v0, v1, v2, v3, v4: TEXT := NIL): TEXT; + TYPE Promise = OBJECT METHODS fulfil() RAISES { Error } END; + END QMachine. + + + Index: m3quake/src/QMachine.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3,v retrieving revision 1.35 diff -c -r1.35 QMachine.m3 *** m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35 --- m3quake/src/QMachine.m3 13 Feb 2011 22:17:18 -0000 *************** *** 16,21 **** --- 16,22 ---- IMPORT TextUtils, FSUtils, System, DirStack; (* sysutils *) IMPORT Compiler; IMPORT M3Path; + IMPORT RefSeq; CONST OnUnix = (Compiler.ThisOS = Compiler.OS.POSIX); *************** *** 44,49 **** --- 45,52 ---- shell : TEXT := NIL; sh_option : TEXT := NIL; tmp_dir : TEXT := NIL; + + doRecord := FALSE; OVERRIDES init := Init; evaluate := Evaluate; *************** *** 66,73 **** --- 69,81 ---- set_wr := SetWr; exec_echo := ExecEcho; trace := Trace; + + record := Record; END; + PROCEDURE Record(t : T; on : BOOLEAN) = + BEGIN t.doRecord := on END Record; + TYPE Registers = RECORD cp : QCode.Stream := NIL; (* code pointer *) *************** *** 139,144 **** --- 147,154 ---- t.globals := NEW (IntRefTbl.Default).init (); t.default_wr := Stdio.stdout; + t.promises := NEW(RefSeq.T).init(); + InitOSEnv (t); InitBuiltins (t); *************** *** 1555,1564 **** END; ELSE FlushIO (); ! Process.GetStandardFileHandles (stdin, stdout, stderr); ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), ! stdin := stdin, stdout := stdout, ! stderr := stderr); END; EXCEPT | Thread.Alerted => --- 1565,1594 ---- END; ELSE FlushIO (); ! IF t.doRecord THEN ! handle := NIL; ! WITH a = NEW(REF ARRAY OF TEXT, n_shell_args) DO ! a^ := SUBARRAY(args,0,n_shell_args); ! VAR wrx : Wr.T; BEGIN ! IF echo OR t.do_echo THEN ! wrx := wr ! ELSE ! wrx := NIL ! END; ! t.promises.addhi(NEW(ExecPromise, ! cmd := t.shell, ! wr := wrx, ! args := a, ! t := t, ! ignore_errors := ignore_errors)) ! END ! END ! ELSE ! Process.GetStandardFileHandles (stdin, stdout, stderr); ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), ! stdin := stdin, stdout := stdout, ! stderr := stderr); ! END; END; EXCEPT | Thread.Alerted => *************** *** 1573,1579 **** END; (* wait for everything to shutdown... *) ! exit_code := Process.Wait (handle); END; IF onlyTry THEN --- 1603,1613 ---- END; (* wait for everything to shutdown... *) ! IF handle = NIL THEN ! exit_code := 0 ! ELSE ! exit_code := Process.Wait (handle); ! END; (* else we're only promising *) END; IF onlyTry THEN *************** *** 1589,1594 **** --- 1623,1664 ---- END ExecCommand; + TYPE + ExecPromise = Promise OBJECT + cmd : TEXT; + args : REF ARRAY OF TEXT; + t : T; + wr : Wr.T; + ignore_errors : BOOLEAN; + OVERRIDES + fulfil := FulfilExecPromise; + END; + + PROCEDURE FulfilExecPromise(ep : ExecPromise) RAISES { Error } = + VAR + stdin, stdout, stderr: File.T; + BEGIN + Process.GetStandardFileHandles (stdin, stdout, stderr); + TRY + IF ep.wr # NIL THEN + Wr.PutText (ep.wr, ep.args[1]); + Wr.PutText (ep.wr, Wr.EOL); + FlushIO (); + END; + WITH handle = Process.Create (ep.cmd, ep.args^, + stdin := stdin, stdout := stdout, + stderr := stderr), + exit_code = Process.Wait(handle) DO + IF exit_code # 0 AND NOT ep.ignore_errors THEN + Err (ep.t, Fmt.F("exit %s: %s", Fmt.Int(exit_code), ep.cmd)) + END + END + EXCEPT + OSError.E (ec) => + Err (ep.t, Fmt.F ("exec failed%s *** %s", OSErr (ec), ep.cmd)); + END + END FulfilExecPromise; + PROCEDURE KillProcess (handle: Process.T) = BEGIN IF (handle # NIL) THEN From mika at async.caltech.edu Mon Feb 14 00:15:02 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 15:15:02 -0800 Subject: [M3devel] bootstrapping problems In-Reply-To: References: <20110205215619.E65981A207A@async.async.caltech.edu> , <20110205234331.2F13A1A207A@async.async.caltech.edu> Message-ID: <20110213231502.7A5FA1A2078@async.async.caltech.edu> Jay sent me the following instructions for bootstrapping a while back: ./upgrade.py nocleangcc && ./do-cm3-all.py realclean skipgcc && ./do-cm3-all.py buildship skipgcc It seems reasonable enough for me. I unpacked the REL distribution, checked everything out, etc. However something is very wrong. The newer back-end does not seem compatible with the old m3front (not even backward-compatible), and it's getting called at the wrong time: ignoring ../src/m3overrides rm .M3SHIP rm .M3OVERRIDES rm .generic.tmp rm .generic.tmp inhale libsysutils.m3x inhale /usr/local/cm3/pkg/m3core/AMD64_LINUX/libm3core.a inhale /usr/local/cm3/pkg/libm3/AMD64_LINUX/libm3.a new source -> compiling EnvUtils.i3 m3front ../src/EnvUtils.i3 -w1 /home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 compilation terminated. m3_backend => 1 m3cc (aka cm3cg) failed compiling: EnvUtils.ic rm EnvUtils.is Why is it calling the new back end? This is the very first thing it does (it hasn't cleaned m3cc for some reason). The new back end doesn't like the output of the old compiler: (119)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>/home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 compilation terminated. But the old one in /usr/local/cm3/bin likes it just fine... (120)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is (121)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX> Is something going wrong with the bootstrapping process? I'm going to try to delete the newer back end and see if that fixes it. Mika From mika at async.caltech.edu Mon Feb 14 00:16:26 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 15:16:26 -0800 Subject: [M3devel] bootstrapping problems In-Reply-To: <20110213231502.7A5FA1A2078@async.async.caltech.edu> References: <20110205215619.E65981A207A@async.async.caltech.edu> , <20110205234331.2F13A1A207A@async.async.caltech.edu> <20110213231502.7A5FA1A2078@async.async.caltech.edu> Message-ID: <20110213231626.200DB1A2078@async.async.caltech.edu> I should probably not have "nocleangcc" and "skipgcc" in there huh... still a bit weird that the back-end isn't backward compatible? Mika Mika Nystrom writes: >Jay sent me the following instructions for bootstrapping a while back: > >./upgrade.py nocleangcc && > ./do-cm3-all.py realclean skipgcc && > ./do-cm3-all.py buildship skipgcc > >It seems reasonable enough for me. > >I unpacked the REL distribution, checked everything out, etc. > >However something is very wrong. The newer back-end does not seem >compatible with the old m3front (not even backward-compatible), and it's >getting called at the wrong time: > >ignoring ../src/m3overrides > >rm .M3SHIP >rm .M3OVERRIDES >rm .generic.tmp >rm .generic.tmp >inhale libsysutils.m3x >inhale /usr/local/cm3/pkg/m3core/AMD64_LINUX/libm3core.a >inhale /usr/local/cm3/pkg/libm3/AMD64_LINUX/libm3.a > >new source -> compiling EnvUtils.i3 >m3front ../src/EnvUtils.i3 -w1 >/home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-alig >n-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is >cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 >compilation terminated. > m3_backend => 1 >m3cc (aka cm3cg) failed compiling: EnvUtils.ic >rm EnvUtils.is > >Why is it calling the new back end? This is the very first thing it does (it >hasn't cleaned m3cc for some reason). > >The new back end doesn't like the output of the old compiler: > >(119)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>/home/mika/cm3/cm3/m3-sys >/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables >-quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is >cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 >compilation terminated. > >But the old one in /usr/local/cm3/bin likes it just fine... > >(120)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>cm3cg -gstabs+ -m64 -fPIC > -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o E >nvUtils.is >(121)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX> > >Is something going wrong with the bootstrapping process? I'm going to >try to delete the newer back end and see if that fixes it. > > Mika From jay.krell at cornell.edu Mon Feb 14 03:16:48 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 14 Feb 2011 02:16:48 +0000 Subject: [M3devel] bootstrapping problems In-Reply-To: <20110213231626.200DB1A2078@async.async.caltech.edu> References: <20110205215619.E65981A207A@async.async.caltech.edu>, , , <20110205234331.2F13A1A207A@async.async.caltech.edu>, , <20110213231502.7A5FA1A2078@async.async.caltech.edu>, <20110213231626.200DB1A2078@async.async.caltech.edu> Message-ID: /home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg That is the problem. For a long time, my config file were willing to run that. Instead of cm3cg in $PATH or cm3cg next to cm3. It bit me a bit too so I stopped doing it that way. The instructions are ok. Certainly the skipgcc. Nocleangcc depends on underlying correct incrementality, so might be a bit fragile. The backend has no need or obligation to be forward or backward compatible with any cm3/m3front. It need only be compatible with the one it goes with. The upgrading of cm3 and cm3cg is carefully coordinated to avoid any mismatch. Whenever the uprading has gone wrong, it has been fixed. Not any sort of compatibility restored. Granted, we could better use the version declaration and error right away clearly. I can make a change there. - Jay > To: m3devel at elegosoft.com > Date: Sun, 13 Feb 2011 15:16:26 -0800 > From: mika at async.caltech.edu > Subject: Re: [M3devel] bootstrapping problems > > I should probably not have "nocleangcc" and "skipgcc" in there huh... > > still a bit weird that the back-end isn't backward compatible? > > Mika > > Mika Nystrom writes: > >Jay sent me the following instructions for bootstrapping a while back: > > > >./upgrade.py nocleangcc && > > ./do-cm3-all.py realclean skipgcc && > > ./do-cm3-all.py buildship skipgcc > > > >It seems reasonable enough for me. > > > >I unpacked the REL distribution, checked everything out, etc. > > > >However something is very wrong. The newer back-end does not seem > >compatible with the old m3front (not even backward-compatible), and it's > >getting called at the wrong time: > > > >ignoring ../src/m3overrides > > > >rm .M3SHIP > >rm .M3OVERRIDES > >rm .generic.tmp > >rm .generic.tmp > >inhale libsysutils.m3x > >inhale /usr/local/cm3/pkg/m3core/AMD64_LINUX/libm3core.a > >inhale /usr/local/cm3/pkg/libm3/AMD64_LINUX/libm3.a > > > >new source -> compiling EnvUtils.i3 > >m3front ../src/EnvUtils.i3 -w1 > >/home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-alig > >n-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is > >cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 > >compilation terminated. > > m3_backend => 1 > >m3cc (aka cm3cg) failed compiling: EnvUtils.ic > >rm EnvUtils.is > > > >Why is it calling the new back end? This is the very first thing it does (it > >hasn't cleaned m3cc for some reason). > > > >The new back end doesn't like the output of the old compiler: > > > >(119)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>/home/mika/cm3/cm3/m3-sys > >/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables > >-quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is > >cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 > >compilation terminated. > > > >But the old one in /usr/local/cm3/bin likes it just fine... > > > >(120)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>cm3cg -gstabs+ -m64 -fPIC > > -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o E > >nvUtils.is > >(121)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX> > > > >Is something going wrong with the bootstrapping process? I'm going to > >try to delete the newer back end and see if that fixes it. > > > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Mon Feb 14 03:55:41 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 18:55:41 -0800 Subject: [M3devel] More on threading In-Reply-To: <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> Message-ID: <20110214025541.76AC51A2078@async.async.caltech.edu> All right, I have bootstrapped from the current CVS head with the PTHREAD thread implementation and obtained the following behavior. I'm on AMD64_LINUX here, but can repeat on AMD64_FREEBSD, LINUXLIBC6, I386_DARWIN if either of those is of any special interest. The AMD64_LINUX system reports as follows: Linux ukdsl204 2.6.26-2-amd64 #1 SMP Thu Sep 16 15:56:38 UTC 2010 x86_64 GNU/Linux Thread test results: all : 1. assert failed, ThreadPThread.m3, line 966 2. SEGV pc = 0x430f40 = Move + 0x6b in ../src/runtime/common/RTCollector.m3 3. ThreadPThreadC.c:372: ThreadPThread__pthread_mutex_delete: Assertion `e == 0' failed. read : OK nxread [read w/o try-except] : OK tryexcept : OK fork : OK forktoomuch [Process.Create w/o Process.Wait] : OK alloc : deadlock creat [ make new threads ] : deadlock lock : OK Now we can also try combinations, but I won't go tediously through all possibilites, just a few: alloc,nxread: SEGV pc = 0x430f40 = Move + 0x6b in ../src/runtime/common/RTCollector.m3 The "simplest" test that shows problem is "-tests alloc -n 2" (just two instances of alloc). Mika Tony Hosking writes: >I know it worked properly at one point in time, when I stressed it very = >hard for some GC-related work I was doing. >I need to see instances of failure (deadlock, crashes) captured in the = >debugger, running with the latest head code. >I don't want to try and diagnose old versions of the system if at all = >possible. >I may have some time to try your thread test program in the next couple = >of weeks. Please let me know workloads that reliably cause failure. > > >On Feb 13, 2011, at 1:17 PM, Mika Nystrom wrote: > >> I don't think pthreads works properly anywhere at >> the moment. My guess is that it never has, if you stress it hard = >enough. From rodney_bates at lcwb.coop Mon Feb 14 16:53:19 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 14 Feb 2011 09:53:19 -0600 Subject: [M3devel] threading on Windows? In-Reply-To: <20110209220057.790671A2078@async.async.caltech.edu> References: <20110209220057.790671A2078@async.async.caltech.edu> Message-ID: <4D594FEF.1030309@lcwb.coop> On 02/09/2011 04:00 PM, Mika Nystrom wrote: > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... Hmm, I'll try to look at that. I have been using the release branch for work on other projects. Even there, I have been seeing some problems in m3gdb not handling 64-bit processors. > > Mika > From mika at async.caltech.edu Mon Feb 14 22:18:29 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Mon, 14 Feb 2011 13:18:29 -0800 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110213220804.7A9001A2078@async.async.caltech.edu> References: <20110213220804.7A9001A2078@async.async.caltech.edu> Message-ID: <20110214211829.BA3761A2078@async.async.caltech.edu> Hi again m3devel, Does anyone understand the code in ThreadPosix.m3? After looking through it again I think my attached solution (sent out yesterday afternoon) is right, but I'm still just a little bit fuzzy on how the code works (is supposed to work) but I have some hypotheses. I realized it's been a long, long, loooong time since I coded anything with signals and signal handlers and I've forgotten much of the semantics. After all since Concurrent Pascal came out in '76 signals has always been the "old" way of doing things, right? (And I quit programming C in favor of M3 in 1999...) In ThreadPosixC.c, there's a function called setup_sigvtalrm that registers the Modula-3 procedure switch_thread as a signal handler. My understanding is that in C, signals are masked atomically on the call into the signal handler and unmasked atomically when the handler returns (unless the signal mask is changed, etc., while in the handler). However, switch_thread... doesn't return? Or? Or it does a longjmp somewhere? Which means signals may not be unmasked(?) Which is the reason for the calling "allow_sigvtalrm" as the first statement of the signal handler? Now I can't just turn on ALL signals in the signal handler because then I may get very deeply nested signal handlers, so I only call allow_allsignals (or whatever we call it) from the first-level signal handler. Hmm. Maybe that is not quite right? Do we eventually get back there? Yes I think we must because certain things depend on inCritical being zero... and when inCritical is zero SIGCHLD is again enabled. No I still don't fully understand how the code is supposed to work, especially in the presence of nested signal handling. Does anyone understand better? (I am still going to try to commit my patch later today because I don't think it could possibly make things worse and it does make a significant real-world improvement to the compiler performance under user threads.) Mika Mika Nystrom writes: >Hi Tony, > >I found a problem with the change. I would appreciate it if you and >Jay could think about what I found and my solution (and anybody else >is welcome to as well, of course). > >When fiddling with cm3 to get it to run in parallel, I pushed it up to >fork up to 100 processes at once, just to see what happens. When I did >that (with my SIGCHLD patch) I got a few errors such as illegal instruction >and a bus error or two. I believe what was happening was that I was >taking too many signals at once, because of the following code in >ThreadPosix.m3: > >PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); (* here *) > > INC(inCritical); > (* mark signal as being delivered *) > IF sig >= 0 AND sig < MaxSigs THEN > gotSigs := gotSigs + SET OF Sig { sig } > END; > DEC(inCritical); > > IF inCritical = 0 AND heapState.inCritical = 0 THEN > InternalYield () > END; > END switch_thread; > >I found it a bit odd that the first action of the signal handler is to >enable signals. With VTALRM this is innocuous, but in my case, imagine >what happens if you have a lot of exited child processes queued up. >What happens is that your signal stack gets wound deeper and deeper, and >each time you catch one SIGCHLD you ask for another. Eventually you >run out of stack space. > >My change, which appears to fix this problem, is to split up the "allow" >function as follows: > >PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); (* VTALRM only *) > > INC(inCritical); > IF inCritical = 1 THEN allow_othersigs (); END; (* SIGCHLD *) > > (* mark signal as being delivered *) > IF sig >= 0 AND sig < MaxSigs THEN > gotSigs := gotSigs + SET OF Sig { sig } > END; > DEC(inCritical); > > IF inCritical = 0 AND heapState.inCritical = 0 THEN > InternalYield () > END; > END switch_thread; > >This way, SIGCHLD is not taken in nested signal handlers. But is there a >problem here? Well, the fallback is that it takes 0.1 seconds to wake >up on Process.Wait . > >Is there a chance that SIGCHLD signals are left pending? My argument is >no, since in the "base" environment (i.e., not executing in a signal handler) >SIGCHLD is enabled as a postcondition of allow_othersigs. > >All *other* occurrences of allow_sigvtalrm are of course replaced with > >allow_sigvtalrm(); allow_othersigs(); > >The goal is that SIGCHLD be enabled precisely when SIGVTALRM is except inside >the signal handler itself. > > Mika > > >Tony Hosking writes: >>Sorry, I missed you e-mail describing this change. Looks good. >> >>Antony Hosking | Associate Professor | Computer Science | Purdue = >>University >>305 N. University Street | West Lafayette | IN 47907 | USA >>Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >>On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: >> >>> Tony, >>>=20 >>> The old user threads has an old problem with Process.Wait. It has a=20= >> >>> Thread.Pause(0.1d0) in it. This causes the compiler to run three = >>times >>> slower than it should. The m3build I've been using for years has >>> that Thread.Pause removed (so it uses 100% CPU instead, = >>busy-waiting...) >>>=20 >>> But yeah user threads works the same as it used to (except for=20 >>> now calling pthread_atfork, which I'll return to in another email...) >>>=20 >>> Mika ... From rcolebur at SCIRES.COM Tue Feb 15 00:01:14 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Mon, 14 Feb 2011 18:01:14 -0500 Subject: [M3devel] More on threading In-Reply-To: <20110213181717.4F0E91A2078@async.async.caltech.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> Message-ID: Mika wrote... >But no I don't use Windows. I was asking on here how to get started with >Modula-3 on Windows once but my machine was/is a Windows 2000 system and >apparently CM3 no longer works on win2k..? I use CM3 on Windows 2000/XP/7 and it works on all of these versions. --Randy Coleburn From rcolebur at SCIRES.COM Tue Feb 15 00:54:50 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Mon, 14 Feb 2011 18:54:50 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , , , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, , <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , , , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , , , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu>, , Message-ID: Jay: Up to now, I've been using my Windows CMD scripts to "upgrade", namely: scripts\dev\windows\ RCC_upgradeCM3.CMD But, based on your message, I am trying to do as you suggest, i.e., use your "upgrade.py". I am running Windows 7. I have just installed Python 2.7.1 via http://www.python.org/ftp/python/2.7.1/python-2.7.1.msi Here is error message I get: C:\cm3\Sandbox\scripts\python>upgrade.py Traceback (most recent call last): File "C:\cm3\Sandbox\scripts\python\upgrade.py", line 4, in import pylib File "C:\cm3\Sandbox\scripts\python\pylib.py", line 612, in if Target.startswith("NT386"): AttributeError: 'NoneType' object has no attribute 'startswith' Please advise. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Saturday, February 12, 2011 5:39 AM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? TextUtils.mo : error LNK2001: unresolved external symbol __alloca TextUtils.mo : error LNK2001: unresolved external symbol _Csetjmp__Jumpbuf_size sysutils.dll : fatal error LNK1120: 2 unresolved externals Most likely you didn't build it correctly. Go back to a working cm3/m3core/libm3 and run upgrade.py. There should be no longer any references to alloca of Csetjmp__Jumpbuf_size. Really, Csetjmp__Jumpbuf_size should never have been a problem. That likely also indicates some sort of incorrect bootstrap. > Are you saying you do, or do not, want me to work further on the solving the "problem," regardless of how we describe it ? I still want alloca to work, either via a function call or a specific m3cg method. (I still don't see that RTHooks.Alloca makes better sense; I still don't see a way to avoid hacking the compiler, given the custom calling convention.) It will be difficult to test the fix until/unless m3front is changed back. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Fri, 11 Feb 2011 18:17:15 -0500 Subject: Re: [M3devel] threading on Windows? Jay: Ok, I don't mean to start a new issue or upset anyone. I see you checked in some new changes. I updated to latest HEAD and tried to build, but again I get unresolved symbols beginning with m3-libs\sysutils (see attached sysutils.lst) So, not sure what you mean by "it should be ok now". Are you saying you do, or do not, want me to work further on the solving the "problem," regardless of how we describe it ? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, February 11, 2011 4:52 PM To: Tony; Coleburn, Randy Cc: m3devel Subject: RE: [M3devel] threading on Windows? It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - Jay ________________________________ From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new "heap-based jmpbuf allocation" have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate "alloca" routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I "do" anything, I'll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the "heap based jumpbuf allocation scheme" was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am "in the dark", both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I'm seeing several different symbols that don't resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking > wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 15 02:12:09 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 15 Feb 2011 01:12:09 +0000 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , <20110213181717.4F0E91A2078@async.async.caltech.edu>, Message-ID: Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 15 02:34:59 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 15 Feb 2011 01:34:59 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, ,,, ,,, ,,, ,,, ,,, ,,, ,,, ,,<39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, ,,<9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, ,,, ,,<74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, ,,, , , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu>, , , , , , Message-ID: I'll have to look into it. It is possible I dropped support for "NT386" in favor of "I386_NT". I've definitely used the Python a fair amount on Windows, but less lately. You can probably force it by first setting the environment variable CM3_TARGET to NT386 or I386_NT. Or by giving either of those parameters on the command line. But you shouldn't have to. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Mon, 14 Feb 2011 18:54:50 -0500 Subject: Re: [M3devel] threading on Windows? Jay: Up to now, I?ve been using my Windows CMD scripts to ?upgrade?, namely: scripts\dev\windows\ RCC_upgradeCM3.CMD But, based on your message, I am trying to do as you suggest, i.e., use your ?upgrade.py?. I am running Windows 7. I have just installed Python 2.7.1 via http://www.python.org/ftp/python/2.7.1/python-2.7.1.msi Here is error message I get: C:\cm3\Sandbox\scripts\python>upgrade.py Traceback (most recent call last): File "C:\cm3\Sandbox\scripts\python\upgrade.py", line 4, in import pylib File "C:\cm3\Sandbox\scripts\python\pylib.py", line 612, in if Target.startswith("NT386"): AttributeError: 'NoneType' object has no attribute 'startswith' Please advise. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Saturday, February 12, 2011 5:39 AM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? TextUtils.mo : error LNK2001: unresolved external symbol __alloca TextUtils.mo : error LNK2001: unresolved external symbol _Csetjmp__Jumpbuf_size sysutils.dll : fatal error LNK1120: 2 unresolved externals Most likely you didn't build it correctly. Go back to a working cm3/m3core/libm3 and run upgrade.py. There should be no longer any references to alloca of Csetjmp__Jumpbuf_size. Really, Csetjmp__Jumpbuf_size should never have been a problem. That likely also indicates some sort of incorrect bootstrap. > Are you saying you do, or do not, want me to work further on the solving the ?problem,? regardless of how we describe it ? I still want alloca to work, either via a function call or a specific m3cg method. (I still don't see that RTHooks.Alloca makes better sense; I still don't see a way to avoid hacking the compiler, given the custom calling convention.) It will be difficult to test the fix until/unless m3front is changed back. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Fri, 11 Feb 2011 18:17:15 -0500 Subject: Re: [M3devel] threading on Windows? Jay: Ok, I don?t mean to start a new issue or upset anyone. I see you checked in some new changes. I updated to latest HEAD and tried to build, but again I get unresolved symbols beginning with m3-libs\sysutils (see attached sysutils.lst) So, not sure what you mean by ?it should be ok now?. Are you saying you do, or do not, want me to work further on the solving the ?problem,? regardless of how we describe it ? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, February 11, 2011 4:52 PM To: Tony; Coleburn, Randy Cc: m3devel Subject: RE: [M3devel] threading on Windows? It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - Jay From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate ?alloca? routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Tue Feb 15 02:56:57 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Mon, 14 Feb 2011 17:56:57 -0800 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20110215013018.B4DC12474005@birch.elegosoft.com> Message-ID: <20110215015657.38D541A2078@async.async.caltech.edu> Jay K writes: >--_0b68c717-b1e6-43b7-b824-79e05d0f3a43_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >You might want to throttle the number of outstanding promises. Yeah that's in a different module. QMachine doesn't control what's actually done with the promises (how many in parallel are run etc). The idea is: 1. client of QMachine sets "doRecord" flag 2. QMachine writes all the back-end promises to a sequence. 3. client clears doRecord flag After that it's up to the client to launch the promises as it desires, one at a time or 100 at a time, as the case may be. I'm looking at how to specify this now. You're right it's annoying there's no automatic way of doing it. Default will be serial as always, the impatient can specify a flag or a setting somewhere. I'm leaning towards making it both a config variable and a command-line option... The code is kind of ugly but it sped up my compiles so much I immediately felt I couldn't live without it....I'm taking care that it won't change the behavior for anyone else (unless they ask for it). Mika >If only Posix provided good functions for e.g. finding the number of proces= >sors. >Or if NT or Posix made it easy to discover the number of spindles backing a= > path=2C >or to easily measure system utilization and backing down when it gets near = >100%. >You know=2C an idle machine should be maxed out=2C but no more. >A machine with an interactive user should reserve CPU/memory/I/O for forgro= >und activity.. >I don't I've ever seen this stuff done well though. >=20 > - Jay > >=20 >> Date: Tue=2C 15 Feb 2011 02:30:18 +0000 >> To: m3commit at elegosoft.com >> From: mika at elego.de >> Subject: [M3commit] CVS Update: cm3 >>=20 >> CVSROOT: /usr/cvs >> Changes by: mika at birch. 11/02/15 02:30:18 >>=20 >> Modified files: >> cm3/m3-sys/m3quake/src/: QMachine.i3 QMachine.m3 m3makefile=20 >> Added files: >> cm3/m3-sys/m3quake/src/: QPromise.i3 QPromise.m3=20 >>=20 >> Log message: >> Add support for the QMachine to be able to record (certain) Process.Creat= >es as "promises" rather than executing them on the fly. >>=20 >> The purpose of this modification is to allow parallelization of the CM3 c= >ompiler. The vast majority of the runtime of the compilation of a typical M= >odula-3 program is spent in the part "promised" here (when using the GCC ba= >ckend). >>=20 > = > >--_0b68c717-b1e6-43b7-b824-79e05d0f3a43_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >You might want to throttle the number of outstanding promises.
>If only Posix provided good functions for e.g. finding the number of proces= >sors.
>Or if =3BNT or Posix made it easy to discover the number of spindles ba= >cking a path=2C
>or to easily measure system utilization and backing down when it =3Bget= >s near 100%.
>You know=2C an idle machine should be maxed out=2C but no more.
>A machine with an interactive user should reserve =3BCPU/memory/I/O for= > forground activity..
>I don't I've ever seen this stuff done well though.
> =3B
> =3B- Jay

 =3B
>>=3B Date: Tue=2C 15 Feb 2011 02:30:18 +0000
>=3B To: m3commit at elego= >soft.com
>=3B From: mika at elego.de
>=3B Subject: [M3commit] CVS Up= >date: cm3
>=3B
>=3B CVSROOT: /usr/cvs
>=3B Changes by: mika= >@birch. 11/02/15 02:30:18
>=3B
>=3B Modified files:
>=3B cm= >3/m3-sys/m3quake/src/: QMachine.i3 QMachine.m3 m3makefile
>=3B Added = >files:
>=3B cm3/m3-sys/m3quake/src/: QPromise.i3 QPromise.m3
>= >=3B
>=3B Log message:
>=3B Add support for the QMachine to be ab= >le to record (certain) Process.Creates as "promises" rather than executing = >them on the fly.
>=3B
>=3B The purpose of this modification is t= >o allow parallelization of the CM3 compiler. The vast majority of the runti= >me of the compilation of a typical Modula-3 program is spent in the part "p= >romised" here (when using the GCC backend).
>=3B
> y> >= > >--_0b68c717-b1e6-43b7-b824-79e05d0f3a43_-- From wagner at elegosoft.com Tue Feb 15 08:53:37 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 15 Feb 2011 08:53:37 +0100 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110214211829.BA3761A2078@async.async.caltech.edu> References: <20110213220804.7A9001A2078@async.async.caltech.edu> <20110214211829.BA3761A2078@async.async.caltech.edu> Message-ID: <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> Yes, please commit changes that improve matters, even if they are not perfect. Perhaps Tony will review your latest suggestion again? Your thread test program should be integrated into the Hudson regression test framework for the cm3 current line; I don't think it is already. That way we would get an overview of the thread state for all platforms at least once a day. It would be great if the tests would show up in the cm3-current-test-m3tests-TARGET jobs; it shouldn't be too difficult. Basically, only one script should need to be extended, and some XML result generation added. IIRC it's in cm3/scripts/regression/defs.sh in test_m3tests(), or (in quake) in the m3tests m3makefile. Can you do that? If not, I may have a look myself at the weekend... Olaf PS: We really should get all our code base into a state that you can run all your applications without any problems on current. After all there aren't that many active M3 users... PPS: I need to return to my offline testing of LDAP protocol handlers for online charging servers now :-/ Quoting Mika Nystrom : > Hi again m3devel, > > Does anyone understand the code in ThreadPosix.m3? > > After looking through it again I think my attached solution (sent out > yesterday afternoon) is right, but I'm still just a little bit fuzzy on > how the code works (is supposed to work) but I have some hypotheses. > > I realized it's been a long, long, loooong time since I coded anything > with signals and signal handlers and I've forgotten much of the semantics. > After all since Concurrent Pascal came out in '76 signals has always been > the "old" way of doing things, right? (And I quit programming C in favor > of M3 in 1999...) > > In ThreadPosixC.c, there's a function called setup_sigvtalrm that > registers the Modula-3 procedure switch_thread as a signal handler. > > My understanding is that in C, signals are masked atomically on the > call into the signal handler and unmasked atomically when the handler > returns (unless the signal mask is changed, etc., while in the handler). > > However, switch_thread... doesn't return? Or? Or it does a longjmp > somewhere? Which means signals may not be unmasked(?) Which is the reason > for the calling "allow_sigvtalrm" as the first statement of the signal > handler? > > Now I can't just turn on ALL signals in the signal handler because then > I may get very deeply nested signal handlers, so I only call allow_allsignals > (or whatever we call it) from the first-level signal handler. Hmm. > Maybe that is not quite right? Do we eventually get back there? Yes > I think we must because certain things depend on inCritical being zero... > and when inCritical is zero SIGCHLD is again enabled. > > No I still don't fully understand how the code is supposed to work, > especially in the presence of nested signal handling. Does anyone > understand better? > > (I am still going to try to commit my patch later today because I don't > think it could possibly make things worse and it does make a significant > real-world improvement to the compiler performance under user threads.) > > Mika > > Mika Nystrom writes: >> Hi Tony, >> >> I found a problem with the change. I would appreciate it if you and >> Jay could think about what I found and my solution (and anybody else >> is welcome to as well, of course). >> >> When fiddling with cm3 to get it to run in parallel, I pushed it up to >> fork up to 100 processes at once, just to see what happens. When I did >> that (with my SIGCHLD patch) I got a few errors such as illegal instruction >> and a bus error or two. I believe what was happening was that I was >> taking too many signals at once, because of the following code in >> ThreadPosix.m3: >> >> PROCEDURE switch_thread (sig: int) RAISES {Alerted} = >> BEGIN >> allow_sigvtalrm (); (* here *) >> >> INC(inCritical); >> (* mark signal as being delivered *) >> IF sig >= 0 AND sig < MaxSigs THEN >> gotSigs := gotSigs + SET OF Sig { sig } >> END; >> DEC(inCritical); >> >> IF inCritical = 0 AND heapState.inCritical = 0 THEN >> InternalYield () >> END; >> END switch_thread; >> >> I found it a bit odd that the first action of the signal handler is to >> enable signals. With VTALRM this is innocuous, but in my case, imagine >> what happens if you have a lot of exited child processes queued up. >> What happens is that your signal stack gets wound deeper and deeper, and >> each time you catch one SIGCHLD you ask for another. Eventually you >> run out of stack space. >> >> My change, which appears to fix this problem, is to split up the "allow" >> function as follows: >> >> PROCEDURE switch_thread (sig: int) RAISES {Alerted} = >> BEGIN >> allow_sigvtalrm (); (* VTALRM only *) >> >> INC(inCritical); >> IF inCritical = 1 THEN allow_othersigs (); END; (* SIGCHLD *) >> >> (* mark signal as being delivered *) >> IF sig >= 0 AND sig < MaxSigs THEN >> gotSigs := gotSigs + SET OF Sig { sig } >> END; >> DEC(inCritical); >> >> IF inCritical = 0 AND heapState.inCritical = 0 THEN >> InternalYield () >> END; >> END switch_thread; >> >> This way, SIGCHLD is not taken in nested signal handlers. But is there a >> problem here? Well, the fallback is that it takes 0.1 seconds to wake >> up on Process.Wait . >> >> Is there a chance that SIGCHLD signals are left pending? My argument is >> no, since in the "base" environment (i.e., not executing in a >> signal handler) >> SIGCHLD is enabled as a postcondition of allow_othersigs. >> >> All *other* occurrences of allow_sigvtalrm are of course replaced with >> >> allow_sigvtalrm(); allow_othersigs(); >> >> The goal is that SIGCHLD be enabled precisely when SIGVTALRM is >> except inside >> the signal handler itself. >> >> Mika >> >> >> Tony Hosking writes: >>> Sorry, I missed you e-mail describing this change. Looks good. >>> >>> Antony Hosking | Associate Professor | Computer Science | Purdue = >>> University >>> 305 N. University Street | West Lafayette | IN 47907 | USA >>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>> >>> >>> >>> >>> On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: >>> >>>> Tony, >>>> =20 >>>> The old user threads has an old problem with Process.Wait. It has a=20= >>> >>>> Thread.Pause(0.1d0) in it. This causes the compiler to run three = >>> times >>>> slower than it should. The m3build I've been using for years has >>>> that Thread.Pause removed (so it uses 100% CPU instead, = >>> busy-waiting...) >>>> =20 >>>> But yeah user threads works the same as it used to (except for=20 >>>> now calling pthread_atfork, which I'll return to in another email...) >>>> =20 >>>> Mika > ... > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rcolebur at SCIRES.COM Tue Feb 15 23:05:37 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Tue, 15 Feb 2011 17:05:37 -0500 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , <20110213181717.4F0E91A2078@async.async.caltech.edu>, Message-ID: Jay, since I have a Windows2000 system, I can test the MAC address discovery (or anything else you need). Just let me know. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Monday, February 14, 2011 8:12 PM To: Coleburn, Randy; m3devel Subject: Re: [M3devel] More on threading Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Feb 16 01:56:49 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 16 Feb 2011 00:56:49 +0000 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , , , <20110213181717.4F0E91A2078@async.async.caltech.edu>, , , , Message-ID: It should work. http://dcvs.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/uid/WIN32/MachineID.c?rev=1.14;content-type=text%2Fplain Bing for "MSDN GetIfTable" => http://msdn.microsoft.com/en-us/library/aa365943(VS.85).aspx Mika, what made you think cm3 doesn't work on Windows 2000?? I don't think anything has been remotely suggested. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Tue, 15 Feb 2011 17:05:37 -0500 Subject: Re: [M3devel] More on threading Jay, since I have a Windows2000 system, I can test the MAC address discovery (or anything else you need). Just let me know. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Monday, February 14, 2011 8:12 PM To: Coleburn, Randy; m3devel Subject: Re: [M3devel] More on threading Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Wed Feb 16 05:17:06 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Tue, 15 Feb 2011 20:17:06 -0800 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , , , <20110213181717.4F0E91A2078@async.async.caltech.edu>, , , , Message-ID: <20110216041706.87A2D1A2078@async.async.caltech.edu> Jay K writes: >--_a2f17729-cb11-4f2b-8c70-c3b583cda751_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >It should work. >=20 >=20 >http://dcvs.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/uid/WIN3= >2/MachineID.c?rev=3D1.14=3Bcontent-type=3Dtext%2Fplain >=20 >=20 >Bing for "MSDN GetIfTable" >=3D> http://msdn.microsoft.com/en-us/library/aa365943(VS.85).aspx >=20 >=20 >Mika=2C what made you think cm3 doesn't work on Windows 2000?? >I don't think anything has been remotely suggested. Hi Jay, I searched my email and I do indeed misremember---sort of. I had an email exchange with you on Friday 28 August 2009 where I ran into a bunch of problems that I didn't know how to solve, and I gave up there. And you asked me if I "planned to upgrade to XP or newer anytime soon". I may have taken that as a sign that I shouldn't expect things to work on win2k but you didn't actually say that anywhere. Mika From wagner at elegosoft.com Thu Feb 17 23:47:30 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Thu, 17 Feb 2011 23:47:30 +0100 Subject: [M3devel] cm3-ide In-Reply-To: References: Message-ID: <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com> Quoting ??? ????????? : > Hi, > I am a student of Software Engineering. > I want to learn the language modula-3 and I'm trying to install IDE on my > computer. > I have windows 7 operating system and I can not find any suitable > installation file. > Do you have appropriate file or instructions how to do it on another system? > I need it for exercise in school and I'd love it if you could help me. > Thank you. > zeev blumentzvaig What exactly have you tried already? The archive for Windows listed on http://www.opencm3.net/download.html, http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should work for Windows 7, too, as far as I know. You simply need to unpack it and install a suitable linker, as described for example in http://www.opencm3.net/installation-windows.html. In order to help you, we need to know exactly what you have done and what failed. I'm not an expert on Windows; others on the list may be able to help better, so I CC this to m3devel. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Thu Feb 17 23:56:05 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Thu, 17 Feb 2011 14:56:05 -0800 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> References: <20110213220804.7A9001A2078@async.async.caltech.edu> <20110214211829.BA3761A2078@async.async.caltech.edu> <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> Message-ID: <20110217225605.644AC1A2078@async.async.caltech.edu> Hi Olaf, I have committed the SIGCHLD diff and run quite heavy jobs with it in the code, and so far I have not seen any problems. I am fairly sure there won't be any. Can you show me how to integrate the test program? Note that when it fails it quite often (usually) hangs rather than producing any specific error output. I have also definitely run into pthreads problems on I386_DARWIN now, not running the threadtest program. They show up when I enable the parallel back-end builds (I suspect they will do that on all architectures with pthreads---I've rebuilt all my CM3 installations with user threads). Mika Olaf Wagner writes: >Yes, please commit changes that improve matters, even if they are not >perfect. Perhaps Tony will review your latest suggestion again? > >Your thread test program should be integrated into the Hudson regression >test framework for the cm3 current line; I don't think it is already. >That way we would get an overview of the thread state for all platforms >at least once a day. It would be great if the tests would show up >in the cm3-current-test-m3tests-TARGET jobs; it shouldn't be too >difficult. Basically, only one script should need to be extended, and >some XML result generation added. IIRC it's in cm3/scripts/regression/defs.s= >h >in test_m3tests(), or (in quake) in the m3tests m3makefile. > >Can you do that? If not, I may have a look myself at the weekend... > >Olaf > >PS: We really should get all our code base into a state that you can >run all your applications without any problems on current. After all >there aren't that many active M3 users... > >PPS: I need to return to my offline testing of LDAP protocol handlers >for online charging servers now :-/ > >Quoting Mika Nystrom : > >> Hi again m3devel, >> >> Does anyone understand the code in ThreadPosix.m3? >> >> After looking through it again I think my attached solution (sent out >> yesterday afternoon) is right, but I'm still just a little bit fuzzy on >> how the code works (is supposed to work) but I have some hypotheses. >> >> I realized it's been a long, long, loooong time since I coded anything >> with signals and signal handlers and I've forgotten much of the semantics. >> After all since Concurrent Pascal came out in '76 signals has always been >> the "old" way of doing things, right? (And I quit programming C in favor >> of M3 in 1999...) >> >> In ThreadPosixC.c, there's a function called setup_sigvtalrm that >> registers the Modula-3 procedure switch_thread as a signal handler. >> >> My understanding is that in C, signals are masked atomically on the >> call into the signal handler and unmasked atomically when the handler >> returns (unless the signal mask is changed, etc., while in the handler). >> >> However, switch_thread... doesn't return? Or? Or it does a longjmp >> somewhere? Which means signals may not be unmasked(?) Which is the reaso= >n >> for the calling "allow_sigvtalrm" as the first statement of the signal >> handler? >> >> Now I can't just turn on ALL signals in the signal handler because then >> I may get very deeply nested signal handlers, so I only call allow_allsign= >als >> (or whatever we call it) from the first-level signal handler. Hmm. >> Maybe that is not quite right? Do we eventually get back there? Yes >> I think we must because certain things depend on inCritical being zero... >> and when inCritical is zero SIGCHLD is again enabled. >> >> No I still don't fully understand how the code is supposed to work, >> especially in the presence of nested signal handling. Does anyone >> understand better? >> >> (I am still going to try to commit my patch later today because I don't >> think it could possibly make things worse and it does make a significant >> real-world improvement to the compiler performance under user threads.) >> >> Mika >> >> Mika Nystrom writes: >>> Hi Tony, >>> >>> I found a problem with the change. I would appreciate it if you and >>> Jay could think about what I found and my solution (and anybody else >>> is welcome to as well, of course). >>> >>> When fiddling with cm3 to get it to run in parallel, I pushed it up to >>> fork up to 100 processes at once, just to see what happens. When I did >>> that (with my SIGCHLD patch) I got a few errors such as illegal instructi= >on >>> and a bus error or two. I believe what was happening was that I was >>> taking too many signals at once, because of the following code in >>> ThreadPosix.m3: >>> >>> PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>> BEGIN >>> allow_sigvtalrm (); (* here *) >>> >>> INC(inCritical); >>> (* mark signal as being delivered *) >>> IF sig >=3D 0 AND sig < MaxSigs THEN >>> gotSigs :=3D gotSigs + SET OF Sig { sig } >>> END; >>> DEC(inCritical); >>> >>> IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >>> InternalYield () >>> END; >>> END switch_thread; >>> >>> I found it a bit odd that the first action of the signal handler is to >>> enable signals. With VTALRM this is innocuous, but in my case, imagine >>> what happens if you have a lot of exited child processes queued up. >>> What happens is that your signal stack gets wound deeper and deeper, and >>> each time you catch one SIGCHLD you ask for another. Eventually you >>> run out of stack space. >>> >>> My change, which appears to fix this problem, is to split up the "allow" >>> function as follows: >>> >>> PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>> BEGIN >>> allow_sigvtalrm (); (* VTALRM only *) >>> >>> INC(inCritical); >>> IF inCritical =3D 1 THEN allow_othersigs (); END; (* SIGCHLD *) >>> >>> (* mark signal as being delivered *) >>> IF sig >=3D 0 AND sig < MaxSigs THEN >>> gotSigs :=3D gotSigs + SET OF Sig { sig } >>> END; >>> DEC(inCritical); >>> >>> IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >>> InternalYield () >>> END; >>> END switch_thread; >>> >>> This way, SIGCHLD is not taken in nested signal handlers. But is there a >>> problem here? Well, the fallback is that it takes 0.1 seconds to wake >>> up on Process.Wait . >>> >>> Is there a chance that SIGCHLD signals are left pending? My argument is >>> no, since in the "base" environment (i.e., not executing in a =20 >>> signal handler) >>> SIGCHLD is enabled as a postcondition of allow_othersigs. >>> >>> All *other* occurrences of allow_sigvtalrm are of course replaced with >>> >>> allow_sigvtalrm(); allow_othersigs(); >>> >>> The goal is that SIGCHLD be enabled precisely when SIGVTALRM is =20 >>> except inside >>> the signal handler itself. >>> >>> Mika >>> >>> >>> Tony Hosking writes: >>>> Sorry, I missed you e-mail describing this change. Looks good. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue =3D >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: >>>> >>>>> Tony, >>>>> =3D20 >>>>> The old user threads has an old problem with Process.Wait. It has a=3D= >20=3D >>>> >>>>> Thread.Pause(0.1d0) in it. This causes the compiler to run three =3D >>>> times >>>>> slower than it should. The m3build I've been using for years has >>>>> that Thread.Pause removed (so it uses 100% CPU instead, =3D >>>> busy-waiting...) >>>>> =3D20 >>>>> But yeah user threads works the same as it used to (except for=3D20 >>>>> now calling pthread_atfork, which I'll return to in another email...) >>>>> =3D20 >>>>> Mika >> ... >> > > > >--=20 >Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa= >ny >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz= >: Berlin >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Fri Feb 18 09:32:46 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 18 Feb 2011 09:32:46 +0100 Subject: [M3devel] Fwd: Re: cm3-ide Message-ID: <20110218093246.fzjch5luokkwgokk@mail.elegosoft.com> FYI ----- Forwarded message from zeev.blu at gmail.com ----- Date: Fri, 18 Feb 2011 09:03:38 +0200 From: ??? ????????? Reply-To: ??? ????????? Subject: Re: cm3-ide To: Olaf Wagner Cc: m3devel at elegosoft.com, m3-support at elego.de, moriah.moshe at gmail.com Hi, First, thanks for the quick reply. I have tried to use the file you gave me, and some other files from that site. I have some questions: 1. If I understand correctly, the method of working with this compiler is through the CMD in WINDOWS. Is not this language a IDE to work with? 2. When I tried to compile , I got a linking error: " link is not recognized as an internal or external command, operable program or batch file." And then another error: "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " And finally I received: " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" failed. " Thanks for your help zeev blumentzvaig 2011/2/18 Olaf Wagner > Quoting ??? ????????? : > > Hi, >> I am a student of Software Engineering. >> I want to learn the language modula-3 and I'm trying to install IDE on my >> computer. >> I have windows 7 operating system and I can not find any suitable >> installation file. >> Do you have appropriate file or instructions how to do it on another >> system? >> I need it for exercise in school and I'd love it if you could help me. >> Thank you. >> zeev blumentzvaig >> > > What exactly have you tried already? > > The archive for Windows listed on http://www.opencm3.net/download.html, > http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should > work for Windows 7, too, as far as I know. You simply need to > unpack it and install a suitable linker, as described for example in > http://www.opencm3.net/installation-windows.html. > > In order to help you, we need to know exactly what you have done and > what failed. I'm not an expert on Windows; others on the list may > be able to help better, so I CC this to m3devel. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 > 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: > DE163214194 > > ----- End forwarded message ----- -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 -------------- next part -------------- Hi, First, thanks for the quick reply. I have tried to use the file you gave me, and some other files from that site. I have some questions: 1. If I understand correctly, the method of working with this compiler is through the CMD in WINDOWS. Is not this language a IDE to work with? 2. When I tried to compile , I got a linking error: " link is not recognized as an internal or external command, operable program or batch file." And then another error: "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " And finally I received: " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" failed. " Thanks for your help zeev blumentzvaig 2011/2/18 Olaf Wagner > Quoting ??? ????????? : > > Hi, >> I am a student of Software Engineering. >> I want to learn the language modula-3 and I'm trying to install IDE on my >> computer. >> I have windows 7 operating system and I can not find any suitable >> installation file. >> Do you have appropriate file or instructions how to do it on another >> system? >> I need it for exercise in school and I'd love it if you could help me. >> Thank you. >> zeev blumentzvaig >> > > What exactly have you tried already? > > The archive for Windows listed on http://www.opencm3.net/download.html, > http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should > work for Windows 7, too, as far as I know. You simply need to > unpack it and install a suitable linker, as described for example in > http://www.opencm3.net/installation-windows.html. > > In order to help you, we need to know exactly what you have done and > what failed. I'm not an expert on Windows; others on the list may > be able to help better, so I CC this to m3devel. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 > 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: > DE163214194 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Fri Feb 18 09:40:14 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 18 Feb 2011 09:40:14 +0100 Subject: [M3devel] cm3-ide In-Reply-To: References: <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com> Message-ID: <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> Quoting ??? ????????? : > Hi, > First, thanks for the quick reply. > > I have tried to use the file you gave me, and some other files from that > site. > I have some questions: > 1. If I understand correctly, the method of working with this compiler is > through the CMD in WINDOWS. Is not this language a IDE to work with? A language and its implementation in a compiler does not necessarily require a full-fledged intergrated development environment like Eclipse. My faourite IDE when I was actively using M3 was always XEmacs :-) There _is_ a web-based cm3-ide (which should be contained in that archive) that allows you to build and manage your projects with a standard web browser, but that might be quite different from your expectations of an IDE, too ;-) > 2. When I tried to compile , I got a linking error: > > " link is not recognized as an internal or external command, operable > program or batch file." > > And then another error: > > "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " > > And finally I received: > > " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" > failed. " You either missed the installation of a suitable system linker as described in the "Windows installation notes" referred to below or you didn't set the appropriate variables in your environment or in the cm3 configuration files. CM3 does not include a system linker, and on Windows you have got to install one yourself, while on all other systems it usually is installed by default. You can use the cm3 option "-commands" to actually view the external commands the builder is calling. I hope this helps, Olaf > Thanks for your help > > zeev blumentzvaig > > > 2011/2/18 Olaf Wagner > >> Quoting ??? ????????? : >> >> Hi, >>> I am a student of Software Engineering. >>> I want to learn the language modula-3 and I'm trying to install IDE on my >>> computer. >>> I have windows 7 operating system and I can not find any suitable >>> installation file. >>> Do you have appropriate file or instructions how to do it on another >>> system? >>> I need it for exercise in school and I'd love it if you could help me. >>> Thank you. >>> zeev blumentzvaig >>> >> >> What exactly have you tried already? >> >> The archive for Windows listed on http://www.opencm3.net/download.html, >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should >> work for Windows 7, too, as far as I know. You simply need to >> unpack it and install a suitable linker, as described for example in >> http://www.opencm3.net/installation-windows.html. >> >> In order to help you, we need to know exactly what you have done and >> what failed. I'm not an expert on Windows; others on the list may >> be able to help better, so I CC this to m3devel. >> >> Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Fri Feb 18 10:14:14 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 18 Feb 2011 10:14:14 +0100 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110217225605.644AC1A2078@async.async.caltech.edu> References: <20110213220804.7A9001A2078@async.async.caltech.edu> <20110214211829.BA3761A2078@async.async.caltech.edu> <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> <20110217225605.644AC1A2078@async.async.caltech.edu> Message-ID: <20110218101414.yu4y8iyg0kkw40cg@mail.elegosoft.com> Quoting Mika Nystrom : > Hi Olaf, > > I have committed the SIGCHLD diff and run quite heavy jobs with it > in the code, and so far I have not seen any problems. I am fairly sure > there won't be any. Sounds good. > Can you show me how to integrate the test program? I already gave some hints below. I think what you definitely could do is writing the required XML output, either in M3 or as shell script. There's a simple schema describing what is required; some time ago we've put a copy on our web server: http://www.elegosoft.com/misc/JUnitXSchema.xsd It may be easier to just look at an example, e.g. http://hudson.modula3.com:8080/job/cm3-current-test-m3tests-AMD64_FREEBSD/ws/cm3/m3-sys/m3tests/m3tests-results.xml > Note that when it fails > it quite often (usually) hangs rather than producing any specific error > output. If we cannot avoid non-termination on our own, we must rely on the Hudson job timeout. This would definitely mean that we run the program in their own jobs or at the end of the other tests. Still, I think the whole test job will fail if it doesn't terminate. A runtime supervision for regression testing is something that is missing in our M3 world as has been noted before :-( If you provide the XML output, I can care for the integration into Hudson this or the next weekend. Or alternatively, you can do that on your own if you like and I help as good as I can. If you haven't got an account on Hudson, I can quickly create one. > I have also definitely run into pthreads problems on I386_DARWIN now, > not running the threadtest program. They show up when I enable the > parallel back-end builds (I suspect they will do that on all architectures > with pthreads---I've rebuilt all my CM3 installations with user threads). Hm, sounds like the complete switch to pthreads might have been a little too early, but we haven't had any reliable data to decide on IIRC. All the world uses pthreads, so I assumed they would be OK for us, too :-/ However, we need to fix the problems now; I hope Antony can spare some time. He's been the main architect for that. And it's good to know that user threads still work. Olaf > Mika > > Olaf Wagner writes: >> Yes, please commit changes that improve matters, even if they are not >> perfect. Perhaps Tony will review your latest suggestion again? >> >> Your thread test program should be integrated into the Hudson regression >> test framework for the cm3 current line; I don't think it is already. >> That way we would get an overview of the thread state for all platforms >> at least once a day. It would be great if the tests would show up >> in the cm3-current-test-m3tests-TARGET jobs; it shouldn't be too >> difficult. Basically, only one script should need to be extended, and >> some XML result generation added. IIRC it's in >> cm3/scripts/regression/defs.s= >> h >> in test_m3tests(), or (in quake) in the m3tests m3makefile. >> >> Can you do that? If not, I may have a look myself at the weekend... >> >> Olaf >> >> PS: We really should get all our code base into a state that you can >> run all your applications without any problems on current. After all >> there aren't that many active M3 users... >> >> PPS: I need to return to my offline testing of LDAP protocol handlers >> for online charging servers now :-/ >> >> Quoting Mika Nystrom : >> >>> Hi again m3devel, >>> >>> Does anyone understand the code in ThreadPosix.m3? >>> >>> After looking through it again I think my attached solution (sent out >>> yesterday afternoon) is right, but I'm still just a little bit fuzzy on >>> how the code works (is supposed to work) but I have some hypotheses. >>> >>> I realized it's been a long, long, loooong time since I coded anything >>> with signals and signal handlers and I've forgotten much of the semantics. >>> After all since Concurrent Pascal came out in '76 signals has always been >>> the "old" way of doing things, right? (And I quit programming C in favor >>> of M3 in 1999...) >>> >>> In ThreadPosixC.c, there's a function called setup_sigvtalrm that >>> registers the Modula-3 procedure switch_thread as a signal handler. >>> >>> My understanding is that in C, signals are masked atomically on the >>> call into the signal handler and unmasked atomically when the handler >>> returns (unless the signal mask is changed, etc., while in the handler). >>> >>> However, switch_thread... doesn't return? Or? Or it does a longjmp >>> somewhere? Which means signals may not be unmasked(?) Which is the reaso= >> n >>> for the calling "allow_sigvtalrm" as the first statement of the signal >>> handler? >>> >>> Now I can't just turn on ALL signals in the signal handler because then >>> I may get very deeply nested signal handlers, so I only call allow_allsign= >> als >>> (or whatever we call it) from the first-level signal handler. Hmm. >>> Maybe that is not quite right? Do we eventually get back there? Yes >>> I think we must because certain things depend on inCritical being zero... >>> and when inCritical is zero SIGCHLD is again enabled. >>> >>> No I still don't fully understand how the code is supposed to work, >>> especially in the presence of nested signal handling. Does anyone >>> understand better? >>> >>> (I am still going to try to commit my patch later today because I don't >>> think it could possibly make things worse and it does make a significant >>> real-world improvement to the compiler performance under user threads.) >>> >>> Mika >>> >>> Mika Nystrom writes: >>>> Hi Tony, >>>> >>>> I found a problem with the change. I would appreciate it if you and >>>> Jay could think about what I found and my solution (and anybody else >>>> is welcome to as well, of course). >>>> >>>> When fiddling with cm3 to get it to run in parallel, I pushed it up to >>>> fork up to 100 processes at once, just to see what happens. When I did >>>> that (with my SIGCHLD patch) I got a few errors such as illegal instructi= >> on >>>> and a bus error or two. I believe what was happening was that I was >>>> taking too many signals at once, because of the following code in >>>> ThreadPosix.m3: >>>> >>>> PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>>> BEGIN >>>> allow_sigvtalrm (); (* here *) >>>> >>>> INC(inCritical); >>>> (* mark signal as being delivered *) >>>> IF sig >=3D 0 AND sig < MaxSigs THEN >>>> gotSigs :=3D gotSigs + SET OF Sig { sig } >>>> END; >>>> DEC(inCritical); >>>> >>>> IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >>>> InternalYield () >>>> END; >>>> END switch_thread; >>>> >>>> I found it a bit odd that the first action of the signal handler is to >>>> enable signals. With VTALRM this is innocuous, but in my case, imagine >>>> what happens if you have a lot of exited child processes queued up. >>>> What happens is that your signal stack gets wound deeper and deeper, and >>>> each time you catch one SIGCHLD you ask for another. Eventually you >>>> run out of stack space. >>>> >>>> My change, which appears to fix this problem, is to split up the "allow" >>>> function as follows: >>>> >>>> PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>>> BEGIN >>>> allow_sigvtalrm (); (* VTALRM only *) >>>> >>>> INC(inCritical); >>>> IF inCritical =3D 1 THEN allow_othersigs (); END; (* SIGCHLD *) >>>> >>>> (* mark signal as being delivered *) >>>> IF sig >=3D 0 AND sig < MaxSigs THEN >>>> gotSigs :=3D gotSigs + SET OF Sig { sig } >>>> END; >>>> DEC(inCritical); >>>> >>>> IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >>>> InternalYield () >>>> END; >>>> END switch_thread; >>>> >>>> This way, SIGCHLD is not taken in nested signal handlers. But is there a >>>> problem here? Well, the fallback is that it takes 0.1 seconds to wake >>>> up on Process.Wait . >>>> >>>> Is there a chance that SIGCHLD signals are left pending? My argument is >>>> no, since in the "base" environment (i.e., not executing in a =20 >>>> signal handler) >>>> SIGCHLD is enabled as a postcondition of allow_othersigs. >>>> >>>> All *other* occurrences of allow_sigvtalrm are of course replaced with >>>> >>>> allow_sigvtalrm(); allow_othersigs(); >>>> >>>> The goal is that SIGCHLD be enabled precisely when SIGVTALRM is =20 >>>> except inside >>>> the signal handler itself. >>>> >>>> Mika >>>> >>>> >>>> Tony Hosking writes: >>>>> Sorry, I missed you e-mail describing this change. Looks good. >>>>> >>>>> Antony Hosking | Associate Professor | Computer Science | Purdue =3D >>>>> University >>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>> >>>>> >>>>> >>>>> >>>>> On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: >>>>> >>>>>> Tony, >>>>>> =3D20 >>>>>> The old user threads has an old problem with Process.Wait. It has a=3D= >> 20=3D >>>>> >>>>>> Thread.Pause(0.1d0) in it. This causes the compiler to run three =3D >>>>> times >>>>>> slower than it should. The m3build I've been using for years has >>>>>> that Thread.Pause removed (so it uses 100% CPU instead, =3D >>>>> busy-waiting...) >>>>>> =3D20 >>>>>> But yeah user threads works the same as it used to (except for=3D20 >>>>>> now calling pthread_atfork, which I'll return to in another email...) >>>>>> =3D20 >>>>>> Mika >>> ... >>> >> >> >> >> --=20 >> Olaf Wagner -- elego Software Solutions GmbH >> Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa= >> ny >> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 >> http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz= >> : Berlin >> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Fri Feb 18 13:55:32 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 18 Feb 2011 12:55:32 +0000 Subject: [M3devel] cm3-ide In-Reply-To: <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> References: , <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com>, , <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> Message-ID: > A language and its implementation in a compiler does not necessarily > require a full-fledged intergrated development environment like Eclipse. Programmers today are not used to using plain text editors. They are used to wizards and intellisense. It would take a lot of work I think to make Modula-3 at all usable to any programmer under 30 years old. Even older folks have largely become spoiled. - Jay > Date: Fri, 18 Feb 2011 09:40:14 +0100 > From: wagner at elegosoft.com > To: zeev.blu at gmail.com > CC: m3devel at elegosoft.com; m3-support at elego.de; moriah.moshe at gmail.com > Subject: Re: [M3devel] cm3-ide > > Quoting ??? ????????? : > > > Hi, > > First, thanks for the quick reply. > > > > I have tried to use the file you gave me, and some other files from that > > site. > > I have some questions: > > 1. If I understand correctly, the method of working with this compiler is > > through the CMD in WINDOWS. Is not this language a IDE to work with? > > A language and its implementation in a compiler does not necessarily > require a full-fledged intergrated development environment like Eclipse. > My faourite IDE when I was actively using M3 was always XEmacs :-) > There _is_ a web-based cm3-ide (which should be contained in that > archive) that allows you to build and manage your projects with a > standard web browser, but that might be quite different from your > expectations of an IDE, too ;-) > > > 2. When I tried to compile , I got a linking error: > > > > " link is not recognized as an internal or external command, operable > > program or batch file." > > > > And then another error: > > > > "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " > > > > And finally I received: > > > > " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" > > failed. " > > You either missed the installation of a suitable system linker as > described in the "Windows installation notes" referred to below or > you didn't set the appropriate variables in your environment or in > the cm3 configuration files. CM3 does not include a system linker, > and on Windows you have got to install one yourself, while on all > other systems it usually is installed by default. > > You can use the cm3 option "-commands" to actually view the external > commands the builder is calling. > > I hope this helps, > > Olaf > > > Thanks for your help > > > > zeev blumentzvaig > > > > > > 2011/2/18 Olaf Wagner > > > >> Quoting ??? ????????? : > >> > >> Hi, > >>> I am a student of Software Engineering. > >>> I want to learn the language modula-3 and I'm trying to install IDE on my > >>> computer. > >>> I have windows 7 operating system and I can not find any suitable > >>> installation file. > >>> Do you have appropriate file or instructions how to do it on another > >>> system? > >>> I need it for exercise in school and I'd love it if you could help me. > >>> Thank you. > >>> zeev blumentzvaig > >>> > >> > >> What exactly have you tried already? > >> > >> The archive for Windows listed on http://www.opencm3.net/download.html, > >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should > >> work for Windows 7, too, as far as I know. You simply need to > >> unpack it and install a suitable linker, as described for example in > >> http://www.opencm3.net/installation-windows.html. > >> > >> In order to help you, we need to know exactly what you have done and > >> what failed. I'm not an expert on Windows; others on the list may > >> be able to help better, so I CC this to m3devel. > >> > >> Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Fri Feb 18 14:12:21 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 18 Feb 2011 14:12:21 +0100 Subject: [M3devel] cm3-ide In-Reply-To: References: , <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com>, , <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> Message-ID: <20110218141221.075c2m16gwcw4s4g@mail.elegosoft.com> Quoting Jay K : > > A language and its implementation in a compiler does not necessarily > > require a full-fledged intergrated development environment like Eclipse. > > Programmers today are not used to using plain text editors. > They are used to wizards and intellisense. > It would take a lot of work I think to make Modula-3 > at all usable to any programmer under 30 years old. I don't think that an integration of M3 into Eclispe for example would be very difficult. As I'm not using it myself, someone else will have to provide it though. > Even older folks have largely become spoiled. I'm still fairly content with using the command line and text editors, even though that must seem old-fashioned ;-) Olaf > - Jay >> Date: Fri, 18 Feb 2011 09:40:14 +0100 >> From: wagner at elegosoft.com >> To: zeev.blu at gmail.com >> CC: m3devel at elegosoft.com; m3-support at elego.de; moriah.moshe at gmail.com >> Subject: Re: [M3devel] cm3-ide >> >> Quoting ??? ????????? : >> >> > Hi, >> > First, thanks for the quick reply. >> > >> > I have tried to use the file you gave me, and some other files from that >> > site. >> > I have some questions: >> > 1. If I understand correctly, the method of working with this compiler is >> > through the CMD in WINDOWS. Is not this language a IDE to work with? >> >> A language and its implementation in a compiler does not necessarily >> require a full-fledged intergrated development environment like Eclipse. >> My faourite IDE when I was actively using M3 was always XEmacs :-) >> There _is_ a web-based cm3-ide (which should be contained in that >> archive) that allows you to build and manage your projects with a >> standard web browser, but that might be quite different from your >> expectations of an IDE, too ;-) >> >> > 2. When I tried to compile , I got a linking error: >> > >> > " link is not recognized as an internal or external command, operable >> > program or batch file." >> > >> > And then another error: >> > >> > "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " >> > >> > And finally I received: >> > >> > " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" >> > failed. " >> >> You either missed the installation of a suitable system linker as >> described in the "Windows installation notes" referred to below or >> you didn't set the appropriate variables in your environment or in >> the cm3 configuration files. CM3 does not include a system linker, >> and on Windows you have got to install one yourself, while on all >> other systems it usually is installed by default. >> >> You can use the cm3 option "-commands" to actually view the external >> commands the builder is calling. >> >> I hope this helps, >> >> Olaf >> >> > Thanks for your help >> > >> > zeev blumentzvaig >> > >> > >> > 2011/2/18 Olaf Wagner >> > >> >> Quoting ??? ????????? : >> >> >> >> Hi, >> >>> I am a student of Software Engineering. >> >>> I want to learn the language modula-3 and I'm trying to install >> IDE on my >> >>> computer. >> >>> I have windows 7 operating system and I can not find any suitable >> >>> installation file. >> >>> Do you have appropriate file or instructions how to do it on another >> >>> system? >> >>> I need it for exercise in school and I'd love it if you could help me. >> >>> Thank you. >> >>> zeev blumentzvaig >> >>> >> >> >> >> What exactly have you tried already? >> >> >> >> The archive for Windows listed on http://www.opencm3.net/download.html, >> >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should >> >> work for Windows 7, too, as far as I know. You simply need to >> >> unpack it and install a suitable linker, as described for example in >> >> http://www.opencm3.net/installation-windows.html. >> >> >> >> In order to help you, we need to know exactly what you have done and >> >> what failed. I'm not an expert on Windows; others on the list may >> >> be able to help better, so I CC this to m3devel. >> >> >> >> Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hosking at cs.purdue.edu Fri Feb 18 15:42:58 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 18 Feb 2011 09:42:58 -0500 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110218101414.yu4y8iyg0kkw40cg@mail.elegosoft.com> References: <20110213220804.7A9001A2078@async.async.caltech.edu> <20110214211829.BA3761A2078@async.async.caltech.edu> <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> <20110217225605.644AC1A2078@async.async.caltech.edu> <20110218101414.yu4y8iyg0kkw40cg@mail.elegosoft.com> Message-ID: <947FF939-07BD-42F5-9E0B-DBCA9A51E68B@cs.purdue.edu> There has been a lot of untested churn in the pthreads implementation since I last looked closely at it. I think we have seen some bitrot that needs attention. I hope to have some time soon. On Feb 18, 2011, at 4:14 AM, Olaf Wagner wrote: > Hm, sounds like the complete switch to pthreads might have been a little > too early, but we haven't had any reliable data to decide on IIRC. All > the world uses pthreads, so I assumed they would be OK for us, too :-/ > However, we need to fix the problems now; I hope Antony can spare some > time. He's been the main architect for that. From dabenavidesd at yahoo.es Fri Feb 18 15:37:24 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Fri, 18 Feb 2011 14:37:24 +0000 (GMT) Subject: [M3devel] cm3-ide In-Reply-To: <20110218141221.075c2m16gwcw4s4g@mail.elegosoft.com> Message-ID: <31891.31208.qm@web29708.mail.ird.yahoo.com> Hi all: yes, it's agood idea, in fact there is one still in preparation M3clipse, the syntactic analyzer, parser and abstract tree builder and has not yet its semantic analyzer, all are written in Java, so it helps to this purposes, perhaps you can can have a hand of it, for now it just colorize its syntax and so see it: http://sourceforge.net/projects/m3clipse/files (it doesn't have any files right now, I believe because of recent crashed project repository server, I can try to put it on the web if any interest its license is EPL) You have this feature in another integrated development environment in another good package, cooledit, I think it features what you are suggesting, even it has an integrated keyword highlighting and code completion but still though it has integrated gdb window support, you might want to add m3gdb support to it (ideally I would say): http://freshmeat.net/projects/cooledit If you need cooledit Modula-3 syntax files you can tell me or be put on the repository for everybody if any interested just tell me, this is written by Pino Zollo. Yet another "cool" option is an "intelligent" or simply advanced integrated programming environment, it has a graphical debugger (in an internal debugging language) and so you can take a look at it in its book: http://www.cs.brown.edu/~spr/research/envfield.html And finally about cm3ide, I think it makes sense to have an distributed environment for reading its hyperlinked source code in almost any web browser by free, i.e a browser plugin which has an speech synthesizer for blind or low vision people. Of course any other custom plugin for any type of checking is a wanted feature, i.e. static analysis or runtime failures could be detected and prompted by it, which is sort of good if have any failure is good to be reported about it, and just not only throw a program debug image file don't you think? Hope it helps, thanks in advance. --- El vie, 18/2/11, Olaf Wagner escribi?: > De: Olaf Wagner > Asunto: Re: [M3devel] cm3-ide > Para: "Jay K" > CC: "m3devel" , m3-support at elego.de, moriah.moshe at gmail.com, zeev.blu at gmail.com > Fecha: viernes, 18 de febrero, 2011 08:12 > Quoting Jay K : > > > > A language and its implementation in a > compiler does not necessarily > > > require a full-fledged intergrated > development environment like Eclipse. > > > > Programmers today are not used to using plain text > editors. > > They are used to wizards and intellisense. > > It would take a lot of work I think to make Modula-3 > > at all usable to any programmer under 30 years old. > > I don't think that an integration of M3 into Eclispe for > example > would be very difficult. As I'm not using it myself, > someone else > will have to provide it though. > > > Even older folks have largely become spoiled. > > I'm still fairly content with using the command line and > text > editors, even though that must seem old-fashioned ;-) > > Olaf > > > - Jay > >> Date: Fri, 18 Feb 2011 09:40:14 +0100 > >> From: wagner at elegosoft.com > >> To: zeev.blu at gmail.com > >> CC: m3devel at elegosoft.com; > m3-support at elego.de; > moriah.moshe at gmail.com > >> Subject: Re: [M3devel] cm3-ide > >> > >> Quoting ??? ????????? : > >> > >> > Hi, > >> > First, thanks for the quick reply. > >> > > >> > I have tried to use the file you gave me, and > some other files from that > >> > site. > >> > I have some questions: > >> > 1. If I understand correctly, the method of > working with this compiler is > >> > through the CMD in WINDOWS. Is not this > language a IDE to work with? > >> > >> A language and its implementation in a compiler > does not necessarily > >> require a full-fledged intergrated development > environment like Eclipse. > >> My faourite IDE when I was actively using M3 was > always XEmacs :-) > >> There _is_ a web-based cm3-ide (which should be > contained in that > >> archive) that allows you to build and manage your > projects with a > >> standard web browser, but that might be quite > different from your > >> expectations of an IDE, too ;-) > >> > >> > 2. When I tried to compile , I got a linking > error: > >> > > >> > " link is not recognized as an internal or > external command, operable > >> > program or batch file." > >> > > >> > And then another error: > >> > > >> > "... \cm3 \ bin \ cm3.cfg ", line 455: quake > runtime error: link failed " > >> > > >> > And finally I received: > >> > > >> > " Fatal Error: procedure "m3_link" defined in > "... \ cm3 \ bin \ cm3.cfg" > >> > failed. " > >> > >> You either missed the installation of a suitable > system linker as > >> described in the "Windows installation notes" > referred to below or > >> you didn't set the appropriate variables in your > environment or in > >> the cm3 configuration files. CM3 does not include > a system linker, > >> and on Windows you have got to install one > yourself, while on all > >> other systems it usually is installed by default. > >> > >> You can use the cm3 option "-commands" to actually > view the external > >> commands the builder is calling. > >> > >> I hope this helps, > >> > >> Olaf > >> > >> > Thanks for your help > >> > > >> > zeev blumentzvaig > >> > > >> > > >> > 2011/2/18 Olaf Wagner > >> > > >> >> Quoting ??? ????????? : > >> >> > >> >> Hi, > >> >>> I am a student of Software > Engineering. > >> >>> I want to learn the language modula-3 > and I'm trying to install > >> IDE on my > >> >>> computer. > >> >>> I have windows 7 operating system and > I can not find any suitable > >> >>> installation file. > >> >>> Do you have appropriate file or > instructions how to do it on another > >> >>> system? > >> >>> I need it for exercise in school and > I'd love it if you could help me. > >> >>> Thank you. > >> >>> zeev blumentzvaig > >> >>> > >> >> > >> >> What exactly have you tried already? > >> >> > >> >> The archive for Windows listed on http://www.opencm3.net/download.html, > >> >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, > should > >> >> work for Windows 7, too, as far as I > know. You simply need to > >> >> unpack it and install a suitable linker, > as described for example in > >> >> http://www.opencm3.net/installation-windows.html. > >> >> > >> >> In order to help you, we need to know > exactly what you have done and > >> >> what failed. I'm not an expert on > Windows; others on the list may > >> >> be able to help better, so I CC this to > m3devel. > >> >> > >> >> Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 > 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | > Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | > USt-IdNr: DE163214194 > > From hosking at cs.purdue.edu Fri Feb 18 15:55:40 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 18 Feb 2011 09:55:40 -0500 Subject: [M3devel] cm3-ide In-Reply-To: References: , <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com>, , <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> Message-ID: Ergo: let's get M3 integrated with Eclipse. Unfortunately, adding new languages properly to Eclipse is a painful process. On Feb 18, 2011, at 7:55 AM, Jay K wrote: > > A language and its implementation in a compiler does not necessarily > > require a full-fledged intergrated development environment like Eclipse. > > > Programmers today are not used to using plain text editors. > They are used to wizards and intellisense. > It would take a lot of work I think to make Modula-3 > at all usable to any programmer under 30 years old. > Even older folks have largely become spoiled. > > - Jay > > Date: Fri, 18 Feb 2011 09:40:14 +0100 > > From: wagner at elegosoft.com > > To: zeev.blu at gmail.com > > CC: m3devel at elegosoft.com; m3-support at elego.de; moriah.moshe at gmail.com > > Subject: Re: [M3devel] cm3-ide > > > > Quoting ??? ????????? : > > > > > Hi, > > > First, thanks for the quick reply. > > > > > > I have tried to use the file you gave me, and some other files from that > > > site. > > > I have some questions: > > > 1. If I understand correctly, the method of working with this compiler is > > > through the CMD in WINDOWS. Is not this language a IDE to work with? > > > > A language and its implementation in a compiler does not necessarily > > require a full-fledged intergrated development environment like Eclipse. > > My faourite IDE when I was actively using M3 was always XEmacs :-) > > There _is_ a web-based cm3-ide (which should be contained in that > > archive) that allows you to build and manage your projects with a > > standard web browser, but that might be quite different from your > > expectations of an IDE, too ;-) > > > > > 2. When I tried to compile , I got a linking error: > > > > > > " link is not recognized as an internal or external command, operable > > > program or batch file." > > > > > > And then another error: > > > > > > "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " > > > > > > And finally I received: > > > > > > " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" > > > failed. " > > > > You either missed the installation of a suitable system linker as > > described in the "Windows installation notes" referred to below or > > you didn't set the appropriate variables in your environment or in > > the cm3 configuration files. CM3 does not include a system linker, > > and on Windows you have got to install one yourself, while on all > > other systems it usually is installed by default. > > > > You can use the cm3 option "-commands" to actually view the external > > commands the builder is calling. > > > > I hope this helps, > > > > Olaf > > > > > Thanks for your help > > > > > > zeev blumentzvaig > > > > > > > > > 2011/2/18 Olaf Wagner > > > > > >> Quoting ??? ????????? : > > >> > > >> Hi, > > >>> I am a student of Software Engineering. > > >>> I want to learn the language modula-3 and I'm trying to install IDE on my > > >>> computer. > > >>> I have windows 7 operating system and I can not find any suitable > > >>> installation file. > > >>> Do you have appropriate file or instructions how to do it on another > > >>> system? > > >>> I need it for exercise in school and I'd love it if you could help me. > > >>> Thank you. > > >>> zeev blumentzvaig > > >>> > > >> > > >> What exactly have you tried already? > > >> > > >> The archive for Windows listed on http://www.opencm3.net/download.html, > > >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should > > >> work for Windows 7, too, as far as I know. You simply need to > > >> unpack it and install a suitable linker, as described for example in > > >> http://www.opencm3.net/installation-windows.html. > > >> > > >> In order to help you, we need to know exactly what you have done and > > >> what failed. I'm not an expert on Windows; others on the list may > > >> be able to help better, so I CC this to m3devel. > > >> > > >> Olaf > > -- > > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 19 01:22:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 18 Feb 2011 16:22:50 -0800 Subject: [M3devel] Text.Hash in CM3 vs PM3 Message-ID: <20110219002250.9E9EE1A2078@async.async.caltech.edu> Hi m3devel, Am I reading the code correctly if I say that Text.Hash (e.g., String8.Hash) is approximately four times slower (on i386 and other 32-bit systems) in CM3 than it was in PM3 (eight times on Alpha and amd64) (i.e., UnsafeHash.Hash)? More, I suppose, on Alpha 21164 and Alpha 21064, which don't even have byte operations... Does anyone out there depend on the specific hash values of existing strings? (I'm not sure the SRC/PM3 algorithm can be adapted to CM3, maybe it can...) Mika From rcolebur at SCIRES.COM Sat Feb 19 10:34:38 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sat, 19 Feb 2011 04:34:38 -0500 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , , , <20110213181717.4F0E91A2078@async.async.caltech.edu>, , , , , Message-ID: Jay: For Windows 2000 SP4, using Visual C++ 2005 Express Edition: 1. I tried building the MachineID.c program in the first example link, but it runs into an unresolved symbol at link time. cl MachineID.c Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 Copyright (C) Microsoft Corporation. All rights reserved. MachineID.c Microsoft (R) Incremental Linker Version 8.00.50727.762 Copyright (C) Microsoft Corporation. All rights reserved. /out:MachineID.exe MachineID.obj LIBCMT.lib(crt0.obj) : error LNK2019: unresolved external symbol _main referenced in function ___tmainCRTStartup MachineID.exe : fatal error LNK1120: 1 unresolved externals 2. I was able to build the program in the 2nd link. Here is the output from running this program: Num Entries: 2 Index[0]: 16777219 InterfaceName[0]: osoft Visual Studio 8\Common7\Tools\ Description[0]: EL574ND4 Ethernet Adapter Type[0]: Ethernet Mtu[0]: 1500 Speed[0]: 100000000 Physical Addr: 00-00-86-5E-DF-A0 Admin Status[0]: 1 Oper Status[0]: Operational Index[1]: 1 InterfaceName[1]: Description[1]: MS TCP Loopback interface Type[1]: Software Lookback Mtu[1]: 1500 Speed[1]: 10000000 Physical Addr: Admin Status[1]: 1 Oper Status[1]: Operational Regards, Randy ________________________________ From: jayk123 at hotmail.com [jayk123 at hotmail.com] On Behalf Of Jay K [jay.krell at cornell.edu] Sent: Tuesday, February 15, 2011 7:56 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] More on threading It should work. http://dcvs.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/uid/WIN32/MachineID.c?rev=1.14;content-type=text%2Fplain Bing for "MSDN GetIfTable" => http://msdn.microsoft.com/en-us/library/aa365943(VS.85).aspx Mika, what made you think cm3 doesn't work on Windows 2000?? I don't think anything has been remotely suggested. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Tue, 15 Feb 2011 17:05:37 -0500 Subject: Re: [M3devel] More on threading Jay, since I have a Windows2000 system, I can test the MAC address discovery (or anything else you need). Just let me know. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Monday, February 14, 2011 8:12 PM To: Coleburn, Randy; m3devel Subject: Re: [M3devel] More on threading Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 19 11:03:54 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 19 Feb 2011 10:03:54 +0000 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, ,,, ,,<20110213181717.4F0E91A2078@async.async.caltech.edu>, , , , , , , , , , Message-ID: 1. That is the expected result. Please please look closer. 2. Yeah, the sample I derived from is a bit buggy but I think I fixed its bugs, I vaguely recall. I'm sure there's no problem here, anyway. At least on Windows 2000 and newer. Maybe on NT 3.x, NT 4.0, Win9x, maybe depending on the install of IE. But I doubt anyone cares. Heck, for a very long time, this code didn't really work for MacOSX, FreeBSD, NetBSD, OpenBSD, Solaris, only Windows, Linux, OSF/1, and nobody seemed to notice.. (it falls back to the local IPv4 address, which isn't very unique typically). I think it works everywhere now, through heavy #ifdef. (Given the level of #ifdef, I wonder if we should either combine Windows and Posix into one file, or split up Posix into multiple files, or just leave it alone..) Thanks, - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Sat, 19 Feb 2011 04:34:38 -0500 Subject: Re: [M3devel] More on threading Jay: For Windows 2000 SP4, using Visual C++ 2005 Express Edition: 1. I tried building the MachineID.c program in the first example link, but it runs into an unresolved symbol at link time. cl MachineID.c Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 Copyright (C) Microsoft Corporation. All rights reserved. MachineID.c Microsoft (R) Incremental Linker Version 8.00.50727.762 Copyright (C) Microsoft Corporation. All rights reserved. /out:MachineID.exe MachineID.obj LIBCMT.lib(crt0.obj) : error LNK2019: unresolved external symbol _main referenced in function ___tmainCRTStartup MachineID.exe : fatal error LNK1120: 1 unresolved externals 2. I was able to build the program in the 2nd link. Here is the output from running this program: Num Entries: 2 Index[0]: 16777219 InterfaceName[0]: osoft Visual Studio 8\Common7\Tools\ Description[0]: EL574ND4 Ethernet Adapter Type[0]: Ethernet Mtu[0]: 1500 Speed[0]: 100000000 Physical Addr: 00-00-86-5E-DF-A0 Admin Status[0]: 1 Oper Status[0]: Operational Index[1]: 1 InterfaceName[1]: Description[1]: MS TCP Loopback interface Type[1]: Software Lookback Mtu[1]: 1500 Speed[1]: 10000000 Physical Addr: Admin Status[1]: 1 Oper Status[1]: Operational Regards, Randy From: jayk123 at hotmail.com [jayk123 at hotmail.com] On Behalf Of Jay K [jay.krell at cornell.edu] Sent: Tuesday, February 15, 2011 7:56 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] More on threading It should work. http://dcvs.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/uid/WIN32/MachineID.c?rev=1.14;content-type=text%2Fplain Bing for "MSDN GetIfTable" => http://msdn.microsoft.com/en-us/library/aa365943(VS.85).aspx Mika, what made you think cm3 doesn't work on Windows 2000?? I don't think anything has been remotely suggested. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Tue, 15 Feb 2011 17:05:37 -0500 Subject: Re: [M3devel] More on threading Jay, since I have a Windows2000 system, I can test the MAC address discovery (or anything else you need). Just let me know. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Monday, February 14, 2011 8:12 PM To: Coleburn, Randy; m3devel Subject: Re: [M3devel] More on threading Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 19 19:29:30 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 10:29:30 -0800 Subject: [M3devel] SEGV mapping to RuntimeError Message-ID: <20110219182930.BDCAB1A2078@async.async.caltech.edu> Dear m3devel, For a while it has annoyed me that segmentation violations cause an unconditional program abort. I've changed that now so that (under user threads at least) we instead get a RuntimeError. Here's an example of the mechanism at work in an interactive Scheme environment. Consider the unhelpful interface and module Crash: INTERFACE Crash; PROCEDURE Me(); END Crash. MODULE Crash; PROCEDURE Me() = VAR ptr : REF INTEGER := NIL; BEGIN ptr^ := 0 END Me; BEGIN END Crash. Here's an example of what happens if you now call this from an interactive interpreter that catches the exception RuntimeError.E: M-Scheme Experimental LITHP ITH LITHENING. > (require-modules "m3") #t > (Crash.Me) EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > (+ 3 4) 7 > I just realized I may have broken pthreads, let me go back and double-check it. runtime/POSIX and thread/POSIX don't refer to the same thing do they... Mika From jay.krell at cornell.edu Sat Feb 19 20:53:49 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 19 Feb 2011 19:53:49 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110219182930.BDCAB1A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> Message-ID: Letting any code run after a SIGSEGV is dubious. Imagine the heap is corrupted. And then you run more code. And the code happens to call malloc. Or printf to log something. I suppose there might be an application that maps memory gradually, as pieces of a buffer are hit. Might. - Jay > To: m3devel at elegosoft.com > Date: Sat, 19 Feb 2011 10:29:30 -0800 > From: mika at async.caltech.edu > Subject: [M3devel] SEGV mapping to RuntimeError > > > Dear m3devel, > > For a while it has annoyed me that segmentation violations cause an > unconditional program abort. I've changed that now so that (under user > threads at least) we instead get a RuntimeError. Here's an example of > the mechanism at work in an interactive Scheme environment. Consider > the unhelpful interface and module Crash: > > INTERFACE Crash; PROCEDURE Me(); END Crash. > > MODULE Crash; > > PROCEDURE Me() = > VAR ptr : REF INTEGER := NIL; BEGIN > ptr^ := 0 > END Me; > > BEGIN END Crash. > > Here's an example of what happens if you now call this from an interactive > interpreter that catches the exception RuntimeError.E: > > M-Scheme Experimental > LITHP ITH LITHENING. > > (require-modules "m3") > #t > > (Crash.Me) > EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > > (+ 3 4) > 7 > > > > I just realized I may have broken pthreads, let me go back and double-check it. > runtime/POSIX and thread/POSIX don't refer to the same thing do they... > > Mika > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 19 21:38:21 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 12:38:21 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> Message-ID: <20110219203821.2016B1A2078@async.async.caltech.edu> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, you know! "Corrupting the heap" is something that can only happen as a result of an unchecked runtime error. Unchecked runtime errors cannot happen in modules not marked UNSAFE. SEGV is, however, used by the CM3 implementation (and its predecessors) to signal a certain kind of *checked* runtime error, namely, the dereferencing of a NIL reference. Correct me if I am wrong, but an attempt to dereference NIL is not going to leave the heap corrupted? And if you stick to safe code, the only SEGVs I think you get in the current CM3 are ones from NIL dereferences. Hence, as long as you stick with safe code, the only time the code I checked in earlier gets triggered is for NIL dereferences, which should never corrupt the heap. So SEGV is not sometimes, but in fact always recoverable. :-) Mika P.S. the bit above "if you stick to safe code": if you actually program in Modula-3 you almost never use UNSAFE. I went through my repository and I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, many of the UNSAFE modules are glue code to Fortran routines, which could relatively easily be verified to be safe in the Modula-3 sense. Almost all what remains is glue to some C library, which wouldn't be necessary if the rest of the world would wake up out of the dark ages, but I don't have the time to rewrite every single library from scratch myself. Jay K writes: >--_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Letting any code run after a SIGSEGV is dubious. >Imagine the heap is corrupted. >And then you run more code. >And the code happens to call malloc. >Or printf to log something. >=20 >I suppose there might be an application that maps memory >gradually=2C as pieces of a buffer are hit. Might. >=20 > - Jay >=20 >> To: m3devel at elegosoft.com >> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >> From: mika at async.caltech.edu >> Subject: [M3devel] SEGV mapping to RuntimeError >>=20 >>=20 >> Dear m3devel=2C >>=20 >> For a while it has annoyed me that segmentation violations cause an >> unconditional program abort. I've changed that now so that (under user >> threads at least) we instead get a RuntimeError. Here's an example of >> the mechanism at work in an interactive Scheme environment. Consider >> the unhelpful interface and module Crash: >>=20 >> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>=20 >> MODULE Crash=3B >>=20 >> PROCEDURE Me() =3D >> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >> ptr^ :=3D 0 >> END Me=3B >>=20 >> BEGIN END Crash. >>=20 >> Here's an example of what happens if you now call this from an interactiv= >e >> interpreter that catches the exception RuntimeError.E: >>=20 >> M-Scheme Experimental >> LITHP ITH LITHENING. >> > (require-modules "m3") >> #t >> > (Crash.Me) >> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >> > (+ 3 4)=20 >> 7 >> >=20 >>=20 >> I just realized I may have broken pthreads=2C let me go back and double-c= >heck it.=20 >> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>=20 >> Mika >>=20 > = > >--_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >Letting any code run after a SIGSEGV is dubious.
>Imagine the heap =3Bis corrupted.
>And then you run more code.
>And the code happens to call malloc.
>Or printf to log something.
> =3B
>I suppose there might be an application that maps memory
>gradually=2C as pieces of a buffer are hit. Might.
> =3B
> =3B- Jay
 =3B
>>=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >=2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >olations cause an
>=3B unconditional program abort. I've changed that = >now so that (under user
>=3B threads at least) we instead get a Runtim= >eError. Here's an example of
>=3B the mechanism at work in an interact= >ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>=3B Here's an example of what happens if you now call this from an inter= >active
>=3B interpreter that catches the exception RuntimeError.E:
= >>=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >gt=3B >=3B (require-modules "m3")
>=3B #t
>=3B >=3B (Crash.Me= >)
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > location.
>=3B >=3B (+ 3 4)
>=3B 7
>=3B >=3B
>= >=3B
>=3B I just realized I may have broken pthreads=2C let me go back= > and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >= > >--_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- From rodney_bates at lcwb.coop Sun Feb 20 00:17:55 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sat, 19 Feb 2011 17:17:55 -0600 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110219203821.2016B1A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> Message-ID: <4D604FA3.3060708@lcwb.coop> I know of one other place the compilers rely on hardware memory protection to detect a checked runtime error, and that is stack overflow. This won't corrupt anything, but is hard to distinguish from dereferencing NIL. This could probably be distinguished after the fact by some low-level, target-dependent code. I have found it by looking at assembly code at the point of failure--usually right after a stack pointer push. Detecting this via compiler-generated checks would probably be more extravagant than many other checks, as it is so frequent. I am not aware of any really good solution to this in any implementation of any language. On 02/19/2011 02:38 PM, Mika Nystrom wrote: > Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > you know! > > "Corrupting the heap" is something that can only happen as a result of > an unchecked runtime error. Unchecked runtime errors cannot happen in > modules not marked UNSAFE. > > SEGV is, however, used by the CM3 implementation (and its predecessors) > to signal a certain kind of *checked* runtime error, namely, the > dereferencing of a NIL reference. Correct me if I am wrong, but an > attempt to dereference NIL is not going to leave the heap corrupted? > > And if you stick to safe code, the only SEGVs I think you get in the > current CM3 are ones from NIL dereferences. > > Hence, as long as you stick with safe code, the only time the code I > checked in earlier gets triggered is for NIL dereferences, which should > never corrupt the heap. So SEGV is not sometimes, but in fact always > recoverable. > > :-) > > Mika > > P.S. the bit above "if you stick to safe code": if you actually program in > Modula-3 you almost never use UNSAFE. I went through my repository and > I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > many of the UNSAFE modules are glue code to Fortran routines, which > could relatively easily be verified to be safe in the Modula-3 sense. > Almost all what remains is glue to some C library, which wouldn't be > necessary if the rest of the world would wake up out of the dark ages, but > I don't have the time to rewrite every single library from scratch myself. > > > Jay K writes: >> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >> Content-Type: text/plain; charset="iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> >> Letting any code run after a SIGSEGV is dubious. >> Imagine the heap is corrupted. >> And then you run more code. >> And the code happens to call malloc. >> Or printf to log something. >> =20 >> I suppose there might be an application that maps memory >> gradually=2C as pieces of a buffer are hit. Might. >> =20 >> - Jay >> =20 >>> To: m3devel at elegosoft.com >>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>> From: mika at async.caltech.edu >>> Subject: [M3devel] SEGV mapping to RuntimeError >>> =20 >>> =20 >>> Dear m3devel=2C >>> =20 >>> For a while it has annoyed me that segmentation violations cause an >>> unconditional program abort. I've changed that now so that (under user >>> threads at least) we instead get a RuntimeError. Here's an example of >>> the mechanism at work in an interactive Scheme environment. Consider >>> the unhelpful interface and module Crash: >>> =20 >>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>> =20 >>> MODULE Crash=3B >>> =20 >>> PROCEDURE Me() =3D >>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>> ptr^ :=3D 0 >>> END Me=3B >>> =20 >>> BEGIN END Crash. >>> =20 >>> Here's an example of what happens if you now call this from an interactiv= >> e >>> interpreter that catches the exception RuntimeError.E: >>> =20 >>> M-Scheme Experimental >>> LITHP ITH LITHENING. >>>> (require-modules "m3") >>> #t >>>> (Crash.Me) >>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>> (+ 3 4)=20 >>> 7 >>>> =20 >>> =20 >>> I just realized I may have broken pthreads=2C let me go back and double-c= >> heck it.=20 >>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>> =20 >>> Mika >>> =20 >> = >> >> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >> Content-Type: text/html; charset="iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> >> >> >> >> >> Letting any code run after a SIGSEGV is dubious.
>> Imagine the heap =3Bis corrupted.
>> And then you run more code.
>> And the code happens to call malloc.
>> Or printf to log something.
>>  =3B
>> I suppose there might be an application that maps memory
>> gradually=2C as pieces of a buffer are hit. Might.
>>  =3B
>>  =3B- Jay
 =3B
>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >> olations cause an
>=3B unconditional program abort. I've changed that = >> now so that (under user
>=3B threads at least) we instead get a Runtim= >> eError. Here's an example of
>=3B the mechanism at work in an interact= >> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >> >=3B Here's an example of what happens if you now call this from an inter= >> active
>=3B interpreter that catches the exception RuntimeError.E:
= >> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >> >> = >> >> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > From mika at async.caltech.edu Sun Feb 20 00:27:29 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 15:27:29 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <4D604FA3.3060708@lcwb.coop> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> Message-ID: <20110219232729.53A991A2078@async.async.caltech.edu> Ah, yes, stack protection. Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. Hmm, I get SIGILL on AMD64_FREEBSD as well: time ../AMD64_FREEBSD/stubexample M-Scheme Experimental LITHP ITH LITHENING. > (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) f > (f 0) Illegal instruction 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w What absolutely must not happen, of course, is that the runtime hangs while executing only safe code... Mika "Rodney M. Bates" writes: >I know of one other place the compilers rely on hardware memory protection >to detect a checked runtime error, and that is stack overflow. This won't >corrupt anything, but is hard to distinguish from dereferencing NIL. >This could probably be distinguished after the fact by some low-level, >target-dependent code. I have found it by looking at assembly code at >the point of failure--usually right after a stack pointer push. > >Detecting this via compiler-generated checks would probably be more >extravagant than many other checks, as it is so frequent. I am not >aware of any really good solution to this in any implementation of any >language. > >On 02/19/2011 02:38 PM, Mika Nystrom wrote: >> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >> you know! >> >> "Corrupting the heap" is something that can only happen as a result of >> an unchecked runtime error. Unchecked runtime errors cannot happen in >> modules not marked UNSAFE. >> >> SEGV is, however, used by the CM3 implementation (and its predecessors) >> to signal a certain kind of *checked* runtime error, namely, the >> dereferencing of a NIL reference. Correct me if I am wrong, but an >> attempt to dereference NIL is not going to leave the heap corrupted? >> >> And if you stick to safe code, the only SEGVs I think you get in the >> current CM3 are ones from NIL dereferences. >> >> Hence, as long as you stick with safe code, the only time the code I >> checked in earlier gets triggered is for NIL dereferences, which should >> never corrupt the heap. So SEGV is not sometimes, but in fact always >> recoverable. >> >> :-) >> >> Mika >> >> P.S. the bit above "if you stick to safe code": if you actually program in >> Modula-3 you almost never use UNSAFE. I went through my repository and >> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >> many of the UNSAFE modules are glue code to Fortran routines, which >> could relatively easily be verified to be safe in the Modula-3 sense. >> Almost all what remains is glue to some C library, which wouldn't be >> necessary if the rest of the world would wake up out of the dark ages, but >> I don't have the time to rewrite every single library from scratch myself. >> >> >> Jay K writes: >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>> Content-Type: text/plain; charset="iso-8859-1" >>> Content-Transfer-Encoding: quoted-printable >>> >>> >>> Letting any code run after a SIGSEGV is dubious. >>> Imagine the heap is corrupted. >>> And then you run more code. >>> And the code happens to call malloc. >>> Or printf to log something. >>> =20 >>> I suppose there might be an application that maps memory >>> gradually=2C as pieces of a buffer are hit. Might. >>> =20 >>> - Jay >>> =20 >>>> To: m3devel at elegosoft.com >>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>> From: mika at async.caltech.edu >>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>> =20 >>>> =20 >>>> Dear m3devel=2C >>>> =20 >>>> For a while it has annoyed me that segmentation violations cause an >>>> unconditional program abort. I've changed that now so that (under user >>>> threads at least) we instead get a RuntimeError. Here's an example of >>>> the mechanism at work in an interactive Scheme environment. Consider >>>> the unhelpful interface and module Crash: >>>> =20 >>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>> =20 >>>> MODULE Crash=3B >>>> =20 >>>> PROCEDURE Me() =3D >>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>> ptr^ :=3D 0 >>>> END Me=3B >>>> =20 >>>> BEGIN END Crash. >>>> =20 >>>> Here's an example of what happens if you now call this from an interactiv= >>> e >>>> interpreter that catches the exception RuntimeError.E: >>>> =20 >>>> M-Scheme Experimental >>>> LITHP ITH LITHENING. >>>>> (require-modules "m3") >>>> #t >>>>> (Crash.Me) >>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>> (+ 3 4)=20 >>>> 7 >>>>> =20 >>>> =20 >>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>> heck it.=20 >>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>> =20 >>>> Mika >>>> =20 >>> = >>> >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>> Content-Type: text/html; charset="iso-8859-1" >>> Content-Transfer-Encoding: quoted-printable >>> >>> >>> >>> >>> >>> >>> Letting any code run after a SIGSEGV is dubious.
>>> Imagine the heap =3Bis corrupted.
>>> And then you run more code.
>>> And the code happens to call malloc.
>>> Or printf to log something.
>>>  =3B
>>> I suppose there might be an application that maps memory
>>> gradually=2C as pieces of a buffer are hit. Might.
>>>  =3B
>>>  =3B- Jay
 =3B
>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>> olations cause an
>=3B unconditional program abort. I've changed that = >>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>> >=3B Here's an example of what happens if you now call this from an inter= >>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>> >>> = >>> >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >> From jay.krell at cornell.edu Sun Feb 20 00:36:25 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 19 Feb 2011 23:36:25 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110219232729.53A991A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu> Message-ID: Modula-3 doesn't live in a vacuum. The vast majority of surrounding code is C and C++. There are bugs everywhere. - Jay > To: rodney_bates at lcwb.coop > Date: Sat, 19 Feb 2011 15:27:29 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > Ah, yes, stack protection. > > Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > > Hmm, I get SIGILL on AMD64_FREEBSD as well: > > time ../AMD64_FREEBSD/stubexample > M-Scheme Experimental > LITHP ITH LITHENING. > > (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > f > > (f 0) > Illegal instruction > 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > > What absolutely must not happen, of course, is that the runtime hangs > while executing only safe code... > > Mika > > "Rodney M. Bates" writes: > >I know of one other place the compilers rely on hardware memory protection > >to detect a checked runtime error, and that is stack overflow. This won't > >corrupt anything, but is hard to distinguish from dereferencing NIL. > >This could probably be distinguished after the fact by some low-level, > >target-dependent code. I have found it by looking at assembly code at > >the point of failure--usually right after a stack pointer push. > > > >Detecting this via compiler-generated checks would probably be more > >extravagant than many other checks, as it is so frequent. I am not > >aware of any really good solution to this in any implementation of any > >language. > > > >On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >> you know! > >> > >> "Corrupting the heap" is something that can only happen as a result of > >> an unchecked runtime error. Unchecked runtime errors cannot happen in > >> modules not marked UNSAFE. > >> > >> SEGV is, however, used by the CM3 implementation (and its predecessors) > >> to signal a certain kind of *checked* runtime error, namely, the > >> dereferencing of a NIL reference. Correct me if I am wrong, but an > >> attempt to dereference NIL is not going to leave the heap corrupted? > >> > >> And if you stick to safe code, the only SEGVs I think you get in the > >> current CM3 are ones from NIL dereferences. > >> > >> Hence, as long as you stick with safe code, the only time the code I > >> checked in earlier gets triggered is for NIL dereferences, which should > >> never corrupt the heap. So SEGV is not sometimes, but in fact always > >> recoverable. > >> > >> :-) > >> > >> Mika > >> > >> P.S. the bit above "if you stick to safe code": if you actually program in > >> Modula-3 you almost never use UNSAFE. I went through my repository and > >> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >> many of the UNSAFE modules are glue code to Fortran routines, which > >> could relatively easily be verified to be safe in the Modula-3 sense. > >> Almost all what remains is glue to some C library, which wouldn't be > >> necessary if the rest of the world would wake up out of the dark ages, but > >> I don't have the time to rewrite every single library from scratch myself. > >> > >> > >> Jay K writes: > >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>> Content-Type: text/plain; charset="iso-8859-1" > >>> Content-Transfer-Encoding: quoted-printable > >>> > >>> > >>> Letting any code run after a SIGSEGV is dubious. > >>> Imagine the heap is corrupted. > >>> And then you run more code. > >>> And the code happens to call malloc. > >>> Or printf to log something. > >>> =20 > >>> I suppose there might be an application that maps memory > >>> gradually=2C as pieces of a buffer are hit. Might. > >>> =20 > >>> - Jay > >>> =20 > >>>> To: m3devel at elegosoft.com > >>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>> From: mika at async.caltech.edu > >>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>> =20 > >>>> =20 > >>>> Dear m3devel=2C > >>>> =20 > >>>> For a while it has annoyed me that segmentation violations cause an > >>>> unconditional program abort. I've changed that now so that (under user > >>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>> the mechanism at work in an interactive Scheme environment. Consider > >>>> the unhelpful interface and module Crash: > >>>> =20 > >>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>> =20 > >>>> MODULE Crash=3B > >>>> =20 > >>>> PROCEDURE Me() =3D > >>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>> ptr^ :=3D 0 > >>>> END Me=3B > >>>> =20 > >>>> BEGIN END Crash. > >>>> =20 > >>>> Here's an example of what happens if you now call this from an interactiv= > >>> e > >>>> interpreter that catches the exception RuntimeError.E: > >>>> =20 > >>>> M-Scheme Experimental > >>>> LITHP ITH LITHENING. > >>>>> (require-modules "m3") > >>>> #t > >>>>> (Crash.Me) > >>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>> (+ 3 4)=20 > >>>> 7 > >>>>> =20 > >>>> =20 > >>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>> heck it.=20 > >>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>> =20 > >>>> Mika > >>>> =20 > >>> = > >>> > >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>> Content-Type: text/html; charset="iso-8859-1" > >>> Content-Transfer-Encoding: quoted-printable > >>> > >>> > >>> > >>> > >>> > >>> > >>> Letting any code run after a SIGSEGV is dubious.
> >>> Imagine the heap =3Bis corrupted.
> >>> And then you run more code.
> >>> And the code happens to call malloc.
> >>> Or printf to log something.
> >>>  =3B
> >>> I suppose there might be an application that maps memory
> >>> gradually=2C as pieces of a buffer are hit. Might.
> >>>  =3B
> >>>  =3B- Jay
 =3B
> >>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>> olations cause an
>=3B unconditional program abort. I've changed that = > >>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>> >=3B Here's an example of what happens if you now call this from an inter= > >>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>> > >>> = > >>> > >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 20 01:13:14 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 16:13:14 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu> Message-ID: <20110220001314.7F2951A2078@async.async.caltech.edu> Jay K writes: >--_1802b75d-f353-43c9-bd6c-02a34e2b7ae8_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Modula-3 doesn't live in a vacuum. >The vast majority of surrounding code is C and C++. >There are bugs everywhere. > - Jay It's rather up to the programmer using it, isn't it? If you write your code in pure Modula-3, you should be able to rely on the guarantees of the Green Book (and therefore that SIGSEGV means something relatively specific, not that "the world just ended"). If you insist on linking with broken C and C++... well you get what you deserve, probably best not to try to handle RuntimeError.E in that case. In any case, to an interactive user of my Scheme interpreter it is very useful that a NIL dereference drops him at the read-eval-print loop rather than killing the world. E.g., if as I had yesterday, a 1-gigabyte datastructure parsed in memory, and I lost the world because I happened to try to set-car! on a null list somewhere. Mika From jay.krell at cornell.edu Sun Feb 20 01:26:30 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 20 Feb 2011 00:26:30 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220001314.7F2951A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , <4D604FA3.3060708@lcwb.coop>, , <20110219232729.53A991A2078@async.async.caltech.edu>, , <20110220001314.7F2951A2078@async.async.caltech.edu> Message-ID: NIL might be a special case. But by and large, continuing to do anything after a SIGSEGV is a bad idea. You think (you say) you are continuing on and preserving state, but the state of the world is now arbitrarily altered and unknown and all future computation based on it is suspect. Best to throw it away. And avoid incorrect results when there is a such a strong hint that things have gone awry. In the Windows C programming world, you can catch access violatios (SIGSEGV) very easily with __except. But prevailing wisdom now and for a long time is exactly what I'm saying -- that catching and continuing isn't "robust", it is letting the damage of bugs continue further. I'm undecided as to the level of guarantees of the safety guarantees, given the surrounding unsafe world. I believe safety depends on a lack of bugs in certain places, certain fairly large places, and therefore the safety guarantees should be considered significant and useful, but not absolute. There really are bugs everywhere. In the compiler (Modula-3, C, etc.), in the C runtime, in the kernels, in the CPUs. Recent AMD processors have a bug in integer division even..one of the most surprising things I've seen in a long time.. (really, recent processors, integer division, I'm not talking about the ancient Pentium FDIV bug). There is a published errata for it and it has been fixed. But I managed to hit it in my code.. - Jay > To: jay.krell at cornell.edu > Date: Sat, 19 Feb 2011 16:13:14 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > Jay K writes: > >--_1802b75d-f353-43c9-bd6c-02a34e2b7ae8_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >Modula-3 doesn't live in a vacuum. > >The vast majority of surrounding code is C and C++. > >There are bugs everywhere. > > - Jay > > It's rather up to the programmer using it, isn't it? If you write your > code in pure Modula-3, you should be able to rely on the guarantees of > the Green Book (and therefore that SIGSEGV means something relatively > specific, not that "the world just ended"). If you insist on linking > with broken C and C++... well you get what you deserve, probably best > not to try to handle RuntimeError.E in that case. > > In any case, to an interactive user of my Scheme interpreter it is very > useful that a NIL dereference drops him at the read-eval-print loop > rather than killing the world. E.g., if as I had yesterday, a 1-gigabyte > datastructure parsed in memory, and I lost the world because I happened > to try to set-car! on a null list somewhere. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 20 01:59:04 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 16:59:04 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , <4D604FA3.3060708@lcwb.coop>, , <20110219232729.53A991A2078@async.async.caltech.edu>, , <20110220001314.7F2951A2078@async.async.caltech.edu> Message-ID: <20110220005904.A15F21A2078@async.async.caltech.edu> Jay K writes: >--_e4a53a7c-4058-44e7-bcce-18b6ae35efbe_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >NIL might be a special case. >But by and large=2C continuing to do anything after a SIGSEGV is a bad idea= >. >You think (you say) you are continuing on and preserving state=2C but the s= >tate >of the world is now arbitrarily altered and unknown and all future computat= >ion >based on it is suspect. Best to throw it away. And avoid incorrect results >when there is a such a strong hint that things have gone awry. Let's try this again. The Green Book unequivocally says that non-UNSAFE code can't get the world into a "state arbitrarily altered and unknown." That doesn't depend on the special case NIL, that's inherent in the design of the language and runtime. Yet non-UNSAFE code can (for efficiency reasons of the implementation) cause a SIGSEGV. If the state of the world is somehow arbitrarily altered and unknown I would suggest tracking down the bugs (in UNSAFE code, compiler, C code, wherever) and fixing them rather than repeating the C programmer's mantra that SIGSEGV is or ought to be fatal to the point that you need to start from a clean slate. This sort of thing is precisely what Modula-3 was invented for!!!! Why do you think (crazy) people still use the programming language, fifteen years after it's been declared dead? By the way, every year that I teach a class, I always ask the students what programming languages they use/learn/want to learn. Over the years I've seen a steady progression of C++, Java, Python... until last year. The most popular answer was "C". I must say it felt a bit odd to be, technologically, 20 years ahead of people 20 years younger. Now, I always used to feel a bit bad about forcing students to study a "dead" programming language that would not help them in employment, research, etc. (Modular-3? What's that?) But better than a language that should be dead. Mika From jay.krell at cornell.edu Sun Feb 20 02:30:01 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 20 Feb 2011 01:30:01 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220005904.A15F21A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, ,,, ,,<20110219203821.2016B1A2078@async.async.caltech.edu>, ,,<4D604FA3.3060708@lcwb.coop>, , , <20110219232729.53A991A2078@async.async.caltech.edu>, , , , <20110220001314.7F2951A2078@async.async.caltech.edu>, , <20110220005904.A15F21A2078@async.async.caltech.edu> Message-ID: I just don't think because the Green Book says something, or because it is inherent in the design, that is necessarily true or inherent in the implementation. There are bugs *everywhere*, and they gradually chip away at all aspects of correctness. Much of this, there is nothing you can do about. The integer divide bug is an example that there is precious little you can do about. You can check that the errata is fixed, and not depend on the correctness until you have first check for it, but inevitably tons and tons of code will run before your correctness check (e.g. booting the OS) and so the state is already not guaranteed. I've also had my code fail to page in. That is another thing you can't do anything about, in usermode...except maybe keep your code to under a page in size. :) Simple progressing from one line of code to the next, or pieces within the line, can fail. But SIGSEGV I think is in another realm. It is a sign of problems that should generally be ignored. There is a term "fail fast" -- at a sign of problem, fail. Don't press on. There are arguments either way. The world, really, the larger human enterprise and setting, is not fail fast. We have lots of problems, yet almost everything almost always goes on. - Jay > To: jay.krell at cornell.edu > Date: Sat, 19 Feb 2011 16:59:04 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > Jay K writes: > >--_e4a53a7c-4058-44e7-bcce-18b6ae35efbe_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >NIL might be a special case. > >But by and large=2C continuing to do anything after a SIGSEGV is a bad idea= > >. > >You think (you say) you are continuing on and preserving state=2C but the s= > >tate > >of the world is now arbitrarily altered and unknown and all future computat= > >ion > >based on it is suspect. Best to throw it away. And avoid incorrect results > >when there is a such a strong hint that things have gone awry. > > Let's try this again. > > The Green Book unequivocally says that non-UNSAFE code can't get the > world into a "state arbitrarily altered and unknown." That doesn't > depend on the special case NIL, that's inherent in the design of the > language and runtime. > > Yet non-UNSAFE code can (for efficiency reasons of the implementation) > cause a SIGSEGV. > > If the state of the world is somehow arbitrarily altered and unknown I > would suggest tracking down the bugs (in UNSAFE code, compiler, C code, > wherever) and fixing them rather than repeating the C programmer's mantra > that SIGSEGV is or ought to be fatal to the point that you need to start > from a clean slate. > > This sort of thing is precisely what Modula-3 was invented for!!!! > Why do you think (crazy) people still use the programming language, > fifteen years after it's been declared dead? > > By the way, every year that I teach a class, I always ask the students > what programming languages they use/learn/want to learn. Over the years > I've seen a steady progression of C++, Java, Python... until last year. > The most popular answer was "C". I must say it felt a bit odd to be, > technologically, 20 years ahead of people 20 years younger. Now, > I always used to feel a bit bad about forcing students to study a > "dead" programming language that would not help them in employment, > research, etc. (Modular-3? What's that?) But better than a language > that should be dead. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 20 02:51:07 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 17:51:07 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , , <4D604FA3.3060708@lcwb.coop>, , , <20110219232729.53A991A2078@async.async.caltech.edu>, , , , <20110220001314.7F2951A2078@async.async.caltech.edu>, , <20110220005904.A15F21A2078@async.async.caltech.edu> Message-ID: <20110220015107.110461A2078@async.async.caltech.edu> >But SIGSEGV I think is in another realm. It is a sign of problems that shou= >ld generally >be ignored. >There is a term "fail fast" -- at a sign of problem=2C fail. Don't press on= >. >There are arguments either way. >The world=2C really=2C the larger human enterprise and setting=2C is not fa= >il fast. >We have lots of problems=2C yet almost everything almost always goes on. Well, see, when you're using a Lisp interpreter, SIGSEGV in called code is not in another realm (this is my problem). It is also not in another realm if you are implementing an operating system without hardware memory protection (e.g., SPIN) or if you're running untrusted outside code in a "sandbox" (which you CAN build with Modula-3, again without memory protection). How would you like it if you had to reboot your computer every time you dereferenced a null pointer? Sounds a bit like the 1950s doesn't it? Why should I have to restart my Lisp interpreter if I slip on the keyboard? These are things you would never dream of doing in C but that Modula-3 is eminently suited for doing. For precisely these sorts of purposes, there's a big distinction designed into the language between failures in "safe" code (which mean just that the code failed) and failures in "unsafe" code (which mean that the world ended). If you're not doing any of the special things listed above you're free not to catch RuntimeError.E, in which case your program will crash on an uncaught exception if it receives a SIGSEGV. >I just don't think because the Green Book says something=2C or because it i= >s inherent in the >design=2C that is necessarily true or inherent in the implementation. >There are bugs *everywhere*=2C and they gradually chip away at all aspects = >of correctness. Dijkstra teaches that one should code to the specification. Coding to the implementation is a sure path to losing one's mind. Mika From jay.krell at cornell.edu Sun Feb 20 07:36:12 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 20 Feb 2011 06:36:12 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220015107.110461A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , ,,, , ,,<20110219203821.2016B1A2078@async.async.caltech.edu>, , ,,<4D604FA3.3060708@lcwb.coop>, , ,,<20110219232729.53A991A2078@async.async.caltech.edu>, ,,, , , <20110220001314.7F2951A2078@async.async.caltech.edu>, , , , <20110220005904.A15F21A2078@async.async.caltech.edu>, , <20110220015107.110461A2078@async.async.caltech.edu> Message-ID: - You can't get SIGSEGV w/o memory protection. SIGSEGV is generated by hardware. - Untrusted code should not be run in-process, at least unsafe untrusted code. - I don't have to reboot my computer for SIGSEGV, but individual processes are killed. The damage is contained as soon as it is detected. The nice thing about safe languages is the error is caught just before any damage is caused, i.e. array bounds checking. - Jay > To: jay.krell at cornell.edu > Date: Sat, 19 Feb 2011 17:51:07 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > >But SIGSEGV I think is in another realm. It is a sign of problems that shou= > >ld generally > >be ignored. > >There is a term "fail fast" -- at a sign of problem=2C fail. Don't press on= > >. > >There are arguments either way. > >The world=2C really=2C the larger human enterprise and setting=2C is not fa= > >il fast. > >We have lots of problems=2C yet almost everything almost always goes on. > > Well, see, when you're using a Lisp interpreter, SIGSEGV in called > code is not in another realm (this is my problem). It is also not > in another realm if you are implementing an operating system without > hardware memory protection (e.g., SPIN) or if you're running untrusted > outside code in a "sandbox" (which you CAN build with Modula-3, again > without memory protection). How would you like it if you had to reboot > your computer every time you dereferenced a null pointer? Sounds a bit > like the 1950s doesn't it? Why should I have to restart my Lisp > interpreter if I slip on the keyboard? > > These are things you would never dream of doing in C but that Modula-3 > is eminently suited for doing. For precisely these sorts of purposes, > there's a big distinction designed into the language between failures > in "safe" code (which mean just that the code failed) and failures in > "unsafe" code (which mean that the world ended). > > If you're not doing any of the special things listed above you're free > not to catch RuntimeError.E, in which case your program will crash on > an uncaught exception if it receives a SIGSEGV. > > > >I just don't think because the Green Book says something=2C or because it i= > >s inherent in the > >design=2C that is necessarily true or inherent in the implementation. > >There are bugs *everywhere*=2C and they gradually chip away at all aspects = > >of correctness. > > Dijkstra teaches that one should code to the specification. Coding to > the implementation is a sure path to losing one's mind. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Sun Feb 20 13:28:52 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 20 Feb 2011 13:28:52 +0100 Subject: [M3devel] platform-independent kill process Message-ID: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> I've been thinking a bit about the hanging tests problem in our regression test framework. I'd like to provide a simple utility to start a program with time supervision. But as far as I can see, Modula-3 does not provide a platform-independent way of terminating a process. Of course, on Unix I can send signals; but what would I do on Windows? A quick scan of the win32 interfaces in m3core hasn't brought up anything, but I might have looked for the wrong texts. Could we easily extend the Process interface e.g. by PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException ? Any opinions about that? Different interface suggestions? Rather create ProcessExtras than extending a `standard' interface? We currently only have to consider Unix and Windows, or are there other active flavours of OSs to care for? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Sun Feb 20 17:54:00 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 08:54:00 -0800 Subject: [M3devel] platform-independent kill process In-Reply-To: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> References: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> Message-ID: <20110220165400.384291A2078@async.async.caltech.edu> While poking around the compiler sources I found something called M3Process.Interrupt in m3middle---note the comment about Win32: PROCEDURE Interrupt (t: Process.T) RAISES {OSError.E}; (* Send an interrupt signal to process 't'. *) PROCEDURE Abort (t: Process.T) RAISES {OSError.E}; (* Make a best effort attempt to stop process 't'. Warning: aborting a process may leave it in a broken state or fail to release its OS resources (ie. Win32 is broken!) *) POSIX implementation: PROCEDURE Interrupt (t: Process.T) RAISES {OSError.E} = VAR pid := Process.GetID (t); BEGIN IF Usignal.kill (pid, Usignal.SIGINT) # 0 THEN OSErrorPosix.Raise (); END; END Interrupt; PROCEDURE Abort (t: Process.T) RAISES {OSError.E} = VAR pid := Process.GetID (t); BEGIN IF Usignal.kill (pid, Usignal.SIGKILL) # 0 THEN OSErrorPosix.Raise (); END; END Abort; WIN32: PROCEDURE Interrupt (t: Process.T) RAISES {OSError.E} = BEGIN Abort (t); (* can't do any better yet... *) END Interrupt; PROCEDURE Abort (t: Process.T) RAISES {OSError.E} = VAR hProcess := LOOPHOLE (Process.GetID (t), WinNT.HANDLE); BEGIN IF WinBase.TerminateProcess (hProcess, 1) = 0 THEN OSErrorWin32.Raise (); END; END Abort; I don't know enough about Windows to deciper the warning about releasing resources... Mika Olaf Wagner writes: >I've been thinking a bit about the hanging tests problem in our regression >test framework. I'd like to provide a simple utility to start a program >with time supervision. But as far as I can see, Modula-3 does not provide >a platform-independent way of terminating a process. Of course, on Unix >I can send signals; but what would I do on Windows? A quick scan of the >win32 interfaces in m3core hasn't brought up anything, but I might >have looked for the wrong texts. > >Could we easily extend the Process interface e.g. by > > PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException > >? >Any opinions about that? Different interface suggestions? >Rather create ProcessExtras than extending a `standard' interface? > >We currently only have to consider Unix and Windows, or are there >other active flavours of OSs to care for? > >Olaf >--=20 >Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb=E4ude 12, 13355 Berlin, Germany >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch=E4ftsf=FChrer: Olaf Wagner | Sitz: Berl= >in >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Sun Feb 20 18:12:19 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 09:12:19 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , , , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , , , <4D604FA3.3060708@lcwb.coop>, , , , <20110219232729.53A991A2078@async.async.caltech.edu>, , , , , , <20110220001314.7F2951A2078@async.async.caltech.edu>, , , , <20110220005904.A15F21A2078@async.async.caltech.edu>, , <20110220015107.110461A2078@async.async.caltech.edu> Message-ID: <20110220171219.9CE011A2078@async.async.caltech.edu> Jay K writes: > - You can't get SIGSEGV w/o memory protection. SIGSEGV is generated by har= >dware. >=20 I'm not interested in SIGSEGV per se. The point is to map whatever bad memory reference can be generated by non-UNSAFE code to a RuntimeError.E. If you have the hardware to do that, you use it. A Modula-3 implementation on a system without memory protection would have to generate RuntimeError.E using a different mechanism. > - Untrusted code should not be run in-process=2C at least unsafe untrusted= > code. Why not? If you can prove it can't cause damage, then it can't. Why should Java be able to run untrusted code and not we? >=20 > - I don't have to reboot my computer for SIGSEGV=2C but individual process= >es are killed. It's the same argument. Why do you trust your kernel? It's buggy too! If a program crashes, better to reboot the machine to be sure you didn't infect something else. Heck, unplug it for three minutes just to make sure... > The damage is contained as soon as it is detected. The nice thing about = >safe languages > is the error is caught just before any damage is caused=2C i.e. array b= >ounds checking. My point exactly. No reason to start from a clean slate, if you have some obvious earlier place to jump back to. All the examples I've given of situations where it's useful to catch the RuntimeError.E corresponding to SIGSEGV are systems that can be thought of as pseudo-OSes. You have a "job" or "process" doing something within the context of a larger system, or at least there are levels of trust. You're running "an untrusted module" or "a user command". I won't call them "processes" because the terminology would get confusing, but it's the same idea. If one of these modules crashes, there is really no need to abort the whole world, and it would be extremely annoying to user(s) to have to do so. Mika From rodney_bates at lcwb.coop Sun Feb 20 18:58:27 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 20 Feb 2011 11:58:27 -0600 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110219232729.53A991A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> <20110219232729.53A991A2078@async.async.caltech.edu> Message-ID: <4D615643.7060107@lcwb.coop> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. Probably a fully protected guard page at the end of the stack. This technique always worries me a bit because a procedure with a really big activation record could jump right past it. Probably it would almost always access the first page of the big area before storing anything into later pages. On 02/19/2011 05:27 PM, Mika Nystrom wrote: > Ah, yes, stack protection. > > Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > > Hmm, I get SIGILL on AMD64_FREEBSD as well: > > time ../AMD64_FREEBSD/stubexample > M-Scheme Experimental > LITHP ITH LITHENING. >> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > f >> (f 0) > Illegal instruction > 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > > What absolutely must not happen, of course, is that the runtime hangs > while executing only safe code... > > Mika > > "Rodney M. Bates" writes: >> I know of one other place the compilers rely on hardware memory protection >> to detect a checked runtime error, and that is stack overflow. This won't >> corrupt anything, but is hard to distinguish from dereferencing NIL. >> This could probably be distinguished after the fact by some low-level, >> target-dependent code. I have found it by looking at assembly code at >> the point of failure--usually right after a stack pointer push. >> >> Detecting this via compiler-generated checks would probably be more >> extravagant than many other checks, as it is so frequent. I am not >> aware of any really good solution to this in any implementation of any >> language. >> >> On 02/19/2011 02:38 PM, Mika Nystrom wrote: >>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >>> you know! >>> >>> "Corrupting the heap" is something that can only happen as a result of >>> an unchecked runtime error. Unchecked runtime errors cannot happen in >>> modules not marked UNSAFE. >>> >>> SEGV is, however, used by the CM3 implementation (and its predecessors) >>> to signal a certain kind of *checked* runtime error, namely, the >>> dereferencing of a NIL reference. Correct me if I am wrong, but an >>> attempt to dereference NIL is not going to leave the heap corrupted? >>> >>> And if you stick to safe code, the only SEGVs I think you get in the >>> current CM3 are ones from NIL dereferences. >>> >>> Hence, as long as you stick with safe code, the only time the code I >>> checked in earlier gets triggered is for NIL dereferences, which should >>> never corrupt the heap. So SEGV is not sometimes, but in fact always >>> recoverable. >>> >>> :-) >>> >>> Mika >>> >>> P.S. the bit above "if you stick to safe code": if you actually program in >>> Modula-3 you almost never use UNSAFE. I went through my repository and >>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >>> many of the UNSAFE modules are glue code to Fortran routines, which >>> could relatively easily be verified to be safe in the Modula-3 sense. >>> Almost all what remains is glue to some C library, which wouldn't be >>> necessary if the rest of the world would wake up out of the dark ages, but >>> I don't have the time to rewrite every single library from scratch myself. >>> >>> >>> Jay K writes: >>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>> Content-Type: text/plain; charset="iso-8859-1" >>>> Content-Transfer-Encoding: quoted-printable >>>> >>>> >>>> Letting any code run after a SIGSEGV is dubious. >>>> Imagine the heap is corrupted. >>>> And then you run more code. >>>> And the code happens to call malloc. >>>> Or printf to log something. >>>> =20 >>>> I suppose there might be an application that maps memory >>>> gradually=2C as pieces of a buffer are hit. Might. >>>> =20 >>>> - Jay >>>> =20 >>>>> To: m3devel at elegosoft.com >>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>>> From: mika at async.caltech.edu >>>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>>> =20 >>>>> =20 >>>>> Dear m3devel=2C >>>>> =20 >>>>> For a while it has annoyed me that segmentation violations cause an >>>>> unconditional program abort. I've changed that now so that (under user >>>>> threads at least) we instead get a RuntimeError. Here's an example of >>>>> the mechanism at work in an interactive Scheme environment. Consider >>>>> the unhelpful interface and module Crash: >>>>> =20 >>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>>> =20 >>>>> MODULE Crash=3B >>>>> =20 >>>>> PROCEDURE Me() =3D >>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>>> ptr^ :=3D 0 >>>>> END Me=3B >>>>> =20 >>>>> BEGIN END Crash. >>>>> =20 >>>>> Here's an example of what happens if you now call this from an interactiv= >>>> e >>>>> interpreter that catches the exception RuntimeError.E: >>>>> =20 >>>>> M-Scheme Experimental >>>>> LITHP ITH LITHENING. >>>>>> (require-modules "m3") >>>>> #t >>>>>> (Crash.Me) >>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>>> (+ 3 4)=20 >>>>> 7 >>>>>> =20 >>>>> =20 >>>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>>> heck it.=20 >>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>>> =20 >>>>> Mika >>>>> =20 >>>> = >>>> >>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>> Content-Type: text/html; charset="iso-8859-1" >>>> Content-Transfer-Encoding: quoted-printable >>>> >>>> >>>> >>>> >>>> >>>> >>>> Letting any code run after a SIGSEGV is dubious.
>>>> Imagine the heap =3Bis corrupted.
>>>> And then you run more code.
>>>> And the code happens to call malloc.
>>>> Or printf to log something.
>>>>  =3B
>>>> I suppose there might be an application that maps memory
>>>> gradually=2C as pieces of a buffer are hit. Might.
>>>>  =3B
>>>>  =3B- Jay
 =3B
>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>>> olations cause an
>=3B unconditional program abort. I've changed that = >>>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>>> >=3B Here's an example of what happens if you now call this from an inter= >>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>>> >>>> = >>>> >>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >>> > From mika at async.caltech.edu Sun Feb 20 19:37:46 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 10:37:46 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <4D615643.7060107@lcwb.coop> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> <20110219232729.53A991A2078@async.async.caltech.edu> <4D615643.7060107@lcwb.coop> Message-ID: <20110220183746.195C11A2078@async.async.caltech.edu> On a 64-bit machine, at least, there ought to be enough virtual memory that you could just have a gap between thread stacks big enough to allow for a protection area larger than the largest possible (implementation-defined) activation record, no? I know I've run into trouble with very large activation records in the past (and not because I was running out of stack space, either). Or at least a procedure with a very large activation record (or a procedure calling it) could be required to call some sort of check routine "EnoughStackSpaceRemaining()" before starting to scribble on the activation record? Also the end of the activation record must be written to at least once, or else the memory protection won't be triggered. In any case if this is done properly the same mechanism I proposed for SIGSEGV ought to be able to catch stack overflow, no? Well, as long as signals are delivered on a separate stack. If signals are delivered on the same stack, the signal handler would get nastier, it would have to make space through some manipulations (maybe temporarily unporotecting the redzone page?) for its own purposes... but I don't see why it couldn't be done. Not sure why I'm getting SIGILL... maybe I am getting my signal handler activated inside the redzone page because of a difference in signal handling..? I remember reading something about sigaltstack... I would of course love to be able to recover from stack overflow, too. In some sense, since it's a generally unknown limit, it's even less of a fatal error than a NIL dereference (hence makes even more sense to catch it). Mika "Rodney M. Bates" writes: >I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. >Probably a fully protected guard page at the end of the stack. This technique >always worries me a bit because a procedure with a really big activation record >could jump right past it. Probably it would almost always access the first page >of the big area before storing anything into later pages. > >On 02/19/2011 05:27 PM, Mika Nystrom wrote: >> Ah, yes, stack protection. >> >> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. >> >> Hmm, I get SIGILL on AMD64_FREEBSD as well: >> >> time ../AMD64_FREEBSD/stubexample >> M-Scheme Experimental >> LITHP ITH LITHENING. >>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) >> f >>> (f 0) >> Illegal instruction >> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w >> >> What absolutely must not happen, of course, is that the runtime hangs >> while executing only safe code... >> >> Mika >> >> "Rodney M. Bates" writes: >>> I know of one other place the compilers rely on hardware memory protection >>> to detect a checked runtime error, and that is stack overflow. This won't >>> corrupt anything, but is hard to distinguish from dereferencing NIL. >>> This could probably be distinguished after the fact by some low-level, >>> target-dependent code. I have found it by looking at assembly code at >>> the point of failure--usually right after a stack pointer push. >>> >>> Detecting this via compiler-generated checks would probably be more >>> extravagant than many other checks, as it is so frequent. I am not >>> aware of any really good solution to this in any implementation of any >>> language. >>> >>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: >>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >>>> you know! >>>> >>>> "Corrupting the heap" is something that can only happen as a result of >>>> an unchecked runtime error. Unchecked runtime errors cannot happen in >>>> modules not marked UNSAFE. >>>> >>>> SEGV is, however, used by the CM3 implementation (and its predecessors) >>>> to signal a certain kind of *checked* runtime error, namely, the >>>> dereferencing of a NIL reference. Correct me if I am wrong, but an >>>> attempt to dereference NIL is not going to leave the heap corrupted? >>>> >>>> And if you stick to safe code, the only SEGVs I think you get in the >>>> current CM3 are ones from NIL dereferences. >>>> >>>> Hence, as long as you stick with safe code, the only time the code I >>>> checked in earlier gets triggered is for NIL dereferences, which should >>>> never corrupt the heap. So SEGV is not sometimes, but in fact always >>>> recoverable. >>>> >>>> :-) >>>> >>>> Mika >>>> >>>> P.S. the bit above "if you stick to safe code": if you actually program in >>>> Modula-3 you almost never use UNSAFE. I went through my repository and >>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >>>> many of the UNSAFE modules are glue code to Fortran routines, which >>>> could relatively easily be verified to be safe in the Modula-3 sense. >>>> Almost all what remains is glue to some C library, which wouldn't be >>>> necessary if the rest of the world would wake up out of the dark ages, but >>>> I don't have the time to rewrite every single library from scratch myself. >>>> >>>> >>>> Jay K writes: >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>> Content-Type: text/plain; charset="iso-8859-1" >>>>> Content-Transfer-Encoding: quoted-printable >>>>> >>>>> >>>>> Letting any code run after a SIGSEGV is dubious. >>>>> Imagine the heap is corrupted. >>>>> And then you run more code. >>>>> And the code happens to call malloc. >>>>> Or printf to log something. >>>>> =20 >>>>> I suppose there might be an application that maps memory >>>>> gradually=2C as pieces of a buffer are hit. Might. >>>>> =20 >>>>> - Jay >>>>> =20 >>>>>> To: m3devel at elegosoft.com >>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>>>> From: mika at async.caltech.edu >>>>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>>>> =20 >>>>>> =20 >>>>>> Dear m3devel=2C >>>>>> =20 >>>>>> For a while it has annoyed me that segmentation violations cause an >>>>>> unconditional program abort. I've changed that now so that (under user >>>>>> threads at least) we instead get a RuntimeError. Here's an example of >>>>>> the mechanism at work in an interactive Scheme environment. Consider >>>>>> the unhelpful interface and module Crash: >>>>>> =20 >>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>>>> =20 >>>>>> MODULE Crash=3B >>>>>> =20 >>>>>> PROCEDURE Me() =3D >>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>>>> ptr^ :=3D 0 >>>>>> END Me=3B >>>>>> =20 >>>>>> BEGIN END Crash. >>>>>> =20 >>>>>> Here's an example of what happens if you now call this from an interactiv= >>>>> e >>>>>> interpreter that catches the exception RuntimeError.E: >>>>>> =20 >>>>>> M-Scheme Experimental >>>>>> LITHP ITH LITHENING. >>>>>>> (require-modules "m3") >>>>>> #t >>>>>>> (Crash.Me) >>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>>>> (+ 3 4)=20 >>>>>> 7 >>>>>>> =20 >>>>>> =20 >>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>>>> heck it.=20 >>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>>>> =20 >>>>>> Mika >>>>>> =20 >>>>> = >>>>> >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>> Content-Type: text/html; charset="iso-8859-1" >>>>> Content-Transfer-Encoding: quoted-printable >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Letting any code run after a SIGSEGV is dubious.
>>>>> Imagine the heap =3Bis corrupted.
>>>>> And then you run more code.
>>>>> And the code happens to call malloc.
>>>>> Or printf to log something.
>>>>>  =3B
>>>>> I suppose there might be an application that maps memory
>>>>> gradually=2C as pieces of a buffer are hit. Might.
>>>>>  =3B
>>>>>  =3B- Jay
 =3B
>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>>>> olations cause an
>=3B unconditional program abort. I've changed that = >>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>>>> >=3B Here's an example of what happens if you now call this from an inter= >>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>>>> >>>>> = >>>>> >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >>>> >> From jay.krell at cornell.edu Sun Feb 20 23:38:19 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 20 Feb 2011 22:38:19 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220183746.195C11A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu>, <4D615643.7060107@lcwb.coop>, <20110220183746.195C11A2078@async.async.caltech.edu> Message-ID: Probably for this reason: There is a requirement on NT that stack pages be touched in order. Functions with locals totally more than 4K call _chkstk (aka _alloca) to allocate their stack, instead of the usual register subtraction. It contains a loop that touches a byte every 4K. Otherwise, if you have lots of functions with small frames, the stack is touched by virtue of the call function pushing the return address. Modula-3 has long failed to uphold this contract, and still does. I've never seen it clearly documented, but you can see it is what the C compiler does. I had thought there were other reasons for this behavior. I thought you actually get an exception of the stack is touched out of order ("the first time"). But I think I did an experiment long ago with Modula-3 and there was no exception. I don't know about other platforms. I've also seen evidence in gcc and/or Target.i3 that compiler writers know well about such mechanisms -- there being a constant to set as to what size locals trigger special behavior. But m3back doesn't do anything here. Probably we could use _chkstk unconditonally as well -- need to see what it does for numbers smaller than 4K. But that'd be a deoptimization in the common case, and deoptimizing only as necessary should be easy enough. - Jay > To: rodney_bates at lcwb.coop > Date: Sun, 20 Feb 2011 10:37:46 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > On a 64-bit machine, at least, there ought to be enough virtual > memory that you could just have a gap between thread stacks big > enough to allow for a protection area larger than the largest possible > (implementation-defined) activation record, no? I know I've run into > trouble with very large activation records in the past (and not because > I was running out of stack space, either). > > Or at least a procedure with a very large activation record (or > a procedure calling it) could be required to call some sort of check > routine "EnoughStackSpaceRemaining()" before starting to scribble > on the activation record? > > Also the end of the activation record must be written to at least once, > or else the memory protection won't be triggered. > > In any case if this is done properly the same mechanism I proposed for > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > signals are delivered on a separate stack. If signals are delivered on > the same stack, the signal handler would get nastier, it would have to > make space through some manipulations (maybe temporarily unporotecting > the redzone page?) for its own purposes... but I don't see why it > couldn't be done. > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > activated inside the redzone page because of a difference in signal > handling..? I remember reading something about sigaltstack... > > I would of course love to be able to recover from stack overflow, too. > In some sense, since it's a generally unknown limit, it's even less of > a fatal error than a NIL dereference (hence makes even more sense to > catch it). > > Mika > > "Rodney M. Bates" writes: > >I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >Probably a fully protected guard page at the end of the stack. This technique > >always worries me a bit because a procedure with a really big activation record > >could jump right past it. Probably it would almost always access the first page > >of the big area before storing anything into later pages. > > > >On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >> Ah, yes, stack protection. > >> > >> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >> > >> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >> > >> time ../AMD64_FREEBSD/stubexample > >> M-Scheme Experimental > >> LITHP ITH LITHENING. > >>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >> f > >>> (f 0) > >> Illegal instruction > >> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >> > >> What absolutely must not happen, of course, is that the runtime hangs > >> while executing only safe code... > >> > >> Mika > >> > >> "Rodney M. Bates" writes: > >>> I know of one other place the compilers rely on hardware memory protection > >>> to detect a checked runtime error, and that is stack overflow. This won't > >>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>> This could probably be distinguished after the fact by some low-level, > >>> target-dependent code. I have found it by looking at assembly code at > >>> the point of failure--usually right after a stack pointer push. > >>> > >>> Detecting this via compiler-generated checks would probably be more > >>> extravagant than many other checks, as it is so frequent. I am not > >>> aware of any really good solution to this in any implementation of any > >>> language. > >>> > >>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>> you know! > >>>> > >>>> "Corrupting the heap" is something that can only happen as a result of > >>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>> modules not marked UNSAFE. > >>>> > >>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>> to signal a certain kind of *checked* runtime error, namely, the > >>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>> > >>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>> current CM3 are ones from NIL dereferences. > >>>> > >>>> Hence, as long as you stick with safe code, the only time the code I > >>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>> recoverable. > >>>> > >>>> :-) > >>>> > >>>> Mika > >>>> > >>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>> Almost all what remains is glue to some C library, which wouldn't be > >>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>> I don't have the time to rewrite every single library from scratch myself. > >>>> > >>>> > >>>> Jay K writes: > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious. > >>>>> Imagine the heap is corrupted. > >>>>> And then you run more code. > >>>>> And the code happens to call malloc. > >>>>> Or printf to log something. > >>>>> =20 > >>>>> I suppose there might be an application that maps memory > >>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>> =20 > >>>>> - Jay > >>>>> =20 > >>>>>> To: m3devel at elegosoft.com > >>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>> From: mika at async.caltech.edu > >>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>> =20 > >>>>>> =20 > >>>>>> Dear m3devel=2C > >>>>>> =20 > >>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>> the unhelpful interface and module Crash: > >>>>>> =20 > >>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>> =20 > >>>>>> MODULE Crash=3B > >>>>>> =20 > >>>>>> PROCEDURE Me() =3D > >>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>> ptr^ :=3D 0 > >>>>>> END Me=3B > >>>>>> =20 > >>>>>> BEGIN END Crash. > >>>>>> =20 > >>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>> e > >>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>> =20 > >>>>>> M-Scheme Experimental > >>>>>> LITHP ITH LITHENING. > >>>>>>> (require-modules "m3") > >>>>>> #t > >>>>>>> (Crash.Me) > >>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>> (+ 3 4)=20 > >>>>>> 7 > >>>>>>> =20 > >>>>>> =20 > >>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>> heck it.=20 > >>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>> =20 > >>>>>> Mika > >>>>>> =20 > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>> Imagine the heap =3Bis corrupted.
> >>>>> And then you run more code.
> >>>>> And the code happens to call malloc.
> >>>>> Or printf to log something.
> >>>>>  =3B
> >>>>> I suppose there might be an application that maps memory
> >>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>  =3B
> >>>>>  =3B- Jay
 =3B
> >>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>> > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>> > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 01:41:28 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 00:41:28 +0000 Subject: [M3devel] libm3/Process Message-ID: Process.Wait frees up resources. Is that the right way to go? Shouldn't it register a cleanup function for the garbage collector to call, like we do for mutexes? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Mon Feb 21 02:15:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 17:15:50 -0800 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20110221003248.11C9C2474005@birch.elegosoft.com>, Message-ID: <20110221011550.0D2791A2078@async.async.caltech.edu> Jay I didn't quite follow your previous messages. What is the correct solution, and what did you mean when you were talking about the garbage collector? It does seem like---if a Process handle goes out of scope and gets collected then yes the runtime should do the equivalent of "EVAL Process.Wait" at some future time. But that seems unrelated/orthogonal to the issues with the Windows process killing... The process killing in m3middle/M3Process I think is only called in a few places in the compiler and then under what already are dubious circumstances. For instance I see it done where Process.Create has failed. But if Process.Create has failed, then there's no process to kill, so it all seems rather pointless. It is also called as one of the last actions before exiting the parent process, so there's another reason for its being pointless. In other words I think it's safe to say that if you have an idea for how to implement process-killing correctly on Windows you can do it however you like without fear of breaking any existing code... Mika But I can Jay K writes: >--_d36c70d3-e71b-433d-abdc-06703a929f89_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >The code here confuses hProcess and dwProcessId. >Can we introduce a public function here in libm3 instead of m3middle? >ie: to avoid answering the question as to how to expose hProcess? >And to avoid the loophole as well? > >Thanks=2C > - Jay > >From: jay.krell at cornell.edu >To: jkrell at elego.de=3B m3commit at elegosoft.com >Subject: RE: [M3commit] CVS Update: cm3 >Date: Mon=2C 21 Feb 2011 00:35:38 +0000 > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= >rect here. > > - Jay > >> Date: Mon=2C 21 Feb 2011 01:32:47 +0000 >> To: m3commit at elegosoft.com >> From: jkrell at elego.de >> Subject: [M3commit] CVS Update: cm3 >>=20 >> CVSROOT: /usr/cvs >> Changes by: jkrell at birch. 11/02/21 01:32:47 >>=20 >> Modified files: >> cm3/m3-sys/m3middle/src/: M3Process.i3=20 >>=20 >> Log message: >> remove incorrect comments=3B Win32 is not broken >>=20 > = > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >The code here confuses hProcess and dwProcessId.
Can we introduce a publ= >ic function here in libm3 instead of m3middle?
ie: to avoid answering th= >e question as to how to expose hProcess?
And to avoid the loophole as we= >ll?

Thanks=2C
 =3B- Jay

From: = >jay.krell at cornell.edu
To: jkrell at elego.de=3B m3commit at elegosoft.com
S= >ubject: RE: [M3commit] CVS Update: cm3
Date: Mon=2C 21 Feb 2011 00:35:38= > +0000

> >"> > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= >rect here.

 =3B- Jay

>=3B Date: Mon=2C 21 Feb 2011 01:3= >2:47 +0000
>=3B To: m3commit at elegosoft.com
>=3B From: jkrell at eleg= >o.de
>=3B Subject: [M3commit] CVS Update: cm3
>=3B
>=3B CVS= >ROOT: /usr/cvs
>=3B Changes by: jkrell at birch. 11/02/21 01:32:47
>= >=3B
>=3B Modified files:
>=3B cm3/m3-sys/m3middle/src/: M3Proce= >ss.i3
>=3B
>=3B Log message:
>=3B remove incorrect commen= >ts=3B Win32 is not broken
>=3B
>= > >--_d36c70d3-e71b-433d-abdc-06703a929f89_-- From rodney_bates at lcwb.coop Mon Feb 21 03:21:27 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 20 Feb 2011 20:21:27 -0600 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220183746.195C11A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> <20110219232729.53A991A2078@async.async.caltech.edu> <4D615643.7060107@lcwb.coop> <20110220183746.195C11A2078@async.async.caltech.edu> Message-ID: <4D61CC27.7070704@lcwb.coop> On 02/20/2011 12:37 PM, Mika Nystrom wrote: > On a 64-bit machine, at least, there ought to be enough virtual > memory that you could just have a gap between thread stacks big > enough to allow for a protection area larger than the largest possible > (implementation-defined) activation record, no? I know I've run into > trouble with very large activation records in the past (and not because > I was running out of stack space, either). > > Or at least a procedure with a very large activation record (or > a procedure calling it) could be required to call some sort of check > routine "EnoughStackSpaceRemaining()" before starting to scribble > on the activation record? Hmm, I like this idea. It would introduce normal-case runtime overhead only for such procedures, and these are likely rare. Also, assuming the procedure actually uses very much of its large AR, it should also have enough computation time to wash out the stack check overhead. > > Also the end of the activation record must be written to at least once, > or else the memory protection won't be triggered. > I was thinking (as an alternative mechanism) of having the compiler intentionally add enough artificial write(s) as necessary to ensure storing within the red zone, and not just beyond it. This seems trickier to get right and harder to distinguish after the fact from a NIL dereference. > In any case if this is done properly the same mechanism I proposed for > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > signals are delivered on a separate stack. If signals are delivered on > the same stack, the signal handler would get nastier, it would have to > make space through some manipulations (maybe temporarily unporotecting > the redzone page?) for its own purposes... but I don't see why it > couldn't be done. > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > activated inside the redzone page because of a difference in signal > handling..? I remember reading something about sigaltstack... > > I would of course love to be able to recover from stack overflow, too. > In some sense, since it's a generally unknown limit, it's even less of > a fatal error than a NIL dereference (hence makes even more sense to > catch it). I think this would be a nice mechanism to have available. It would have to be used with some care. In any case, it would be really nice and more frequently so, to at least have runtime error messages that distinguished stack overflow from NIL deref. > > Mika > > "Rodney M. Bates" writes: >> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. >> Probably a fully protected guard page at the end of the stack. This technique >> always worries me a bit because a procedure with a really big activation record >> could jump right past it. Probably it would almost always access the first page >> of the big area before storing anything into later pages. >> >> On 02/19/2011 05:27 PM, Mika Nystrom wrote: >>> Ah, yes, stack protection. >>> >>> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. >>> >>> Hmm, I get SIGILL on AMD64_FREEBSD as well: >>> >>> time ../AMD64_FREEBSD/stubexample >>> M-Scheme Experimental >>> LITHP ITH LITHENING. >>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) >>> f >>>> (f 0) >>> Illegal instruction >>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w >>> >>> What absolutely must not happen, of course, is that the runtime hangs >>> while executing only safe code... >>> >>> Mika >>> >>> "Rodney M. Bates" writes: >>>> I know of one other place the compilers rely on hardware memory protection >>>> to detect a checked runtime error, and that is stack overflow. This won't >>>> corrupt anything, but is hard to distinguish from dereferencing NIL. >>>> This could probably be distinguished after the fact by some low-level, >>>> target-dependent code. I have found it by looking at assembly code at >>>> the point of failure--usually right after a stack pointer push. >>>> >>>> Detecting this via compiler-generated checks would probably be more >>>> extravagant than many other checks, as it is so frequent. I am not >>>> aware of any really good solution to this in any implementation of any >>>> language. >>>> >>>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: >>>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >>>>> you know! >>>>> >>>>> "Corrupting the heap" is something that can only happen as a result of >>>>> an unchecked runtime error. Unchecked runtime errors cannot happen in >>>>> modules not marked UNSAFE. >>>>> >>>>> SEGV is, however, used by the CM3 implementation (and its predecessors) >>>>> to signal a certain kind of *checked* runtime error, namely, the >>>>> dereferencing of a NIL reference. Correct me if I am wrong, but an >>>>> attempt to dereference NIL is not going to leave the heap corrupted? >>>>> >>>>> And if you stick to safe code, the only SEGVs I think you get in the >>>>> current CM3 are ones from NIL dereferences. >>>>> >>>>> Hence, as long as you stick with safe code, the only time the code I >>>>> checked in earlier gets triggered is for NIL dereferences, which should >>>>> never corrupt the heap. So SEGV is not sometimes, but in fact always >>>>> recoverable. >>>>> >>>>> :-) >>>>> >>>>> Mika >>>>> >>>>> P.S. the bit above "if you stick to safe code": if you actually program in >>>>> Modula-3 you almost never use UNSAFE. I went through my repository and >>>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >>>>> many of the UNSAFE modules are glue code to Fortran routines, which >>>>> could relatively easily be verified to be safe in the Modula-3 sense. >>>>> Almost all what remains is glue to some C library, which wouldn't be >>>>> necessary if the rest of the world would wake up out of the dark ages, but >>>>> I don't have the time to rewrite every single library from scratch myself. >>>>> >>>>> >>>>> Jay K writes: >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>>> Content-Type: text/plain; charset="iso-8859-1" >>>>>> Content-Transfer-Encoding: quoted-printable >>>>>> >>>>>> >>>>>> Letting any code run after a SIGSEGV is dubious. >>>>>> Imagine the heap is corrupted. >>>>>> And then you run more code. >>>>>> And the code happens to call malloc. >>>>>> Or printf to log something. >>>>>> =20 >>>>>> I suppose there might be an application that maps memory >>>>>> gradually=2C as pieces of a buffer are hit. Might. >>>>>> =20 >>>>>> - Jay >>>>>> =20 >>>>>>> To: m3devel at elegosoft.com >>>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>>>>> From: mika at async.caltech.edu >>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>>>>> =20 >>>>>>> =20 >>>>>>> Dear m3devel=2C >>>>>>> =20 >>>>>>> For a while it has annoyed me that segmentation violations cause an >>>>>>> unconditional program abort. I've changed that now so that (under user >>>>>>> threads at least) we instead get a RuntimeError. Here's an example of >>>>>>> the mechanism at work in an interactive Scheme environment. Consider >>>>>>> the unhelpful interface and module Crash: >>>>>>> =20 >>>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>>>>> =20 >>>>>>> MODULE Crash=3B >>>>>>> =20 >>>>>>> PROCEDURE Me() =3D >>>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>>>>> ptr^ :=3D 0 >>>>>>> END Me=3B >>>>>>> =20 >>>>>>> BEGIN END Crash. >>>>>>> =20 >>>>>>> Here's an example of what happens if you now call this from an interactiv= >>>>>> e >>>>>>> interpreter that catches the exception RuntimeError.E: >>>>>>> =20 >>>>>>> M-Scheme Experimental >>>>>>> LITHP ITH LITHENING. >>>>>>>> (require-modules "m3") >>>>>>> #t >>>>>>>> (Crash.Me) >>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>>>>> (+ 3 4)=20 >>>>>>> 7 >>>>>>>> =20 >>>>>>> =20 >>>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>>>>> heck it.=20 >>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>>>>> =20 >>>>>>> Mika >>>>>>> =20 >>>>>> = >>>>>> >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>>> Content-Type: text/html; charset="iso-8859-1" >>>>>> Content-Transfer-Encoding: quoted-printable >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Letting any code run after a SIGSEGV is dubious.
>>>>>> Imagine the heap =3Bis corrupted.
>>>>>> And then you run more code.
>>>>>> And the code happens to call malloc.
>>>>>> Or printf to log something.
>>>>>>  =3B
>>>>>> I suppose there might be an application that maps memory
>>>>>> gradually=2C as pieces of a buffer are hit. Might.
>>>>>>  =3B
>>>>>>  =3B- Jay
 =3B
>>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>>>>> olations cause an
>=3B unconditional program abort. I've changed that = >>>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>>>>> >=3B Here's an example of what happens if you now call this from an inter= >>>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>>>>> >>>>>> = >>>>>> >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >>>>> >>> > From jay.krell at cornell.edu Mon Feb 21 04:24:53 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 03:24:53 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <4D61CC27.7070704@lcwb.coop> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu>, <4D615643.7060107@lcwb.coop>, <20110220183746.195C11A2078@async.async.caltech.edu>, <4D61CC27.7070704@lcwb.coop> Message-ID: We should look at what C does on the various targets. NT already has a specific easy simple mechanism/policy here, that Modula-3 should follow but doesn't. Specifically, C code in NT can overflow its stack, but it is caught "right away" and not via random corruption of neighboring memory. Perhaps, hopefully, C code in other targets is this good. If not, well, the NT mechanism makes a lot of sense for all targets: if function has less than a page of locals, do nothing special. if function has more than a page of locals, touch each page, in order, at the start of the function. "page" is hardware specific, however I believe all of our targets have 4K or 8K pages, and if you just hardcode 4K, that works, with a slight slight pessimization on targets with 8K pages (e.g. I think Sparc and IA64, but I'd have to dig around). Actually NT reserves like two pages at the end of the stack. One to trigger stack overflow exception, one to give an exception handler a little bit of room to deal with it, such as by capturing a dump of some sort, and exiting, and then if the exception handler uses more than its page, the process is I think terminated. FURTHERMORE, this is a detail that SHOULD be handled by the existing gcc backend. I don't know if it does, but it ought to. But the NT backend does not, and should. It's possible this problem has regressed with the 1K jmpbuf I put in. (I'm very very inclined to shrink that down either to precise levels or approximate but usually smaller levels. m3core did have an assert at startup that the size was large enough. Still, we can't get the 128 bit alignment some targets prefer (powerpc) or require (hppa64). Still, the alloca solution is better, if done right.) - Jay > Date: Sun, 20 Feb 2011 20:21:27 -0600 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > > On 02/20/2011 12:37 PM, Mika Nystrom wrote: > > On a 64-bit machine, at least, there ought to be enough virtual > > memory that you could just have a gap between thread stacks big > > enough to allow for a protection area larger than the largest possible > > (implementation-defined) activation record, no? I know I've run into > > trouble with very large activation records in the past (and not because > > I was running out of stack space, either). > > > > Or at least a procedure with a very large activation record (or > > a procedure calling it) could be required to call some sort of check > > routine "EnoughStackSpaceRemaining()" before starting to scribble > > on the activation record? > > Hmm, I like this idea. It would introduce normal-case runtime overhead > only for such procedures, and these are likely rare. Also, assuming the procedure > actually uses very much of its large AR, it should also have enough computation > time to wash out the stack check overhead. > > > > > Also the end of the activation record must be written to at least once, > > or else the memory protection won't be triggered. > > > > I was thinking (as an alternative mechanism) of having the compiler intentionally > add enough artificial write(s) as necessary to ensure storing within the > red zone, and not just beyond it. This seems trickier to get right and > harder to distinguish after the fact from a NIL dereference. > > > In any case if this is done properly the same mechanism I proposed for > > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > > signals are delivered on a separate stack. If signals are delivered on > > the same stack, the signal handler would get nastier, it would have to > > make space through some manipulations (maybe temporarily unporotecting > > the redzone page?) for its own purposes... but I don't see why it > > couldn't be done. > > > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > > activated inside the redzone page because of a difference in signal > > handling..? I remember reading something about sigaltstack... > > > > I would of course love to be able to recover from stack overflow, too. > > In some sense, since it's a generally unknown limit, it's even less of > > a fatal error than a NIL dereference (hence makes even more sense to > > catch it). > > I think this would be a nice mechanism to have available. It would have to > be used with some care. In any case, it would be really nice and more > frequently so, to at least have runtime error messages that distinguished > stack overflow from NIL deref. > > > > > Mika > > > > "Rodney M. Bates" writes: > >> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >> Probably a fully protected guard page at the end of the stack. This technique > >> always worries me a bit because a procedure with a really big activation record > >> could jump right past it. Probably it would almost always access the first page > >> of the big area before storing anything into later pages. > >> > >> On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >>> Ah, yes, stack protection. > >>> > >>> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >>> > >>> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >>> > >>> time ../AMD64_FREEBSD/stubexample > >>> M-Scheme Experimental > >>> LITHP ITH LITHENING. > >>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >>> f > >>>> (f 0) > >>> Illegal instruction > >>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >>> > >>> What absolutely must not happen, of course, is that the runtime hangs > >>> while executing only safe code... > >>> > >>> Mika > >>> > >>> "Rodney M. Bates" writes: > >>>> I know of one other place the compilers rely on hardware memory protection > >>>> to detect a checked runtime error, and that is stack overflow. This won't > >>>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>>> This could probably be distinguished after the fact by some low-level, > >>>> target-dependent code. I have found it by looking at assembly code at > >>>> the point of failure--usually right after a stack pointer push. > >>>> > >>>> Detecting this via compiler-generated checks would probably be more > >>>> extravagant than many other checks, as it is so frequent. I am not > >>>> aware of any really good solution to this in any implementation of any > >>>> language. > >>>> > >>>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>>> you know! > >>>>> > >>>>> "Corrupting the heap" is something that can only happen as a result of > >>>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>>> modules not marked UNSAFE. > >>>>> > >>>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>>> to signal a certain kind of *checked* runtime error, namely, the > >>>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>>> > >>>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>>> current CM3 are ones from NIL dereferences. > >>>>> > >>>>> Hence, as long as you stick with safe code, the only time the code I > >>>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>>> recoverable. > >>>>> > >>>>> :-) > >>>>> > >>>>> Mika > >>>>> > >>>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>>> Almost all what remains is glue to some C library, which wouldn't be > >>>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>>> I don't have the time to rewrite every single library from scratch myself. > >>>>> > >>>>> > >>>>> Jay K writes: > >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>>> Content-Transfer-Encoding: quoted-printable > >>>>>> > >>>>>> > >>>>>> Letting any code run after a SIGSEGV is dubious. > >>>>>> Imagine the heap is corrupted. > >>>>>> And then you run more code. > >>>>>> And the code happens to call malloc. > >>>>>> Or printf to log something. > >>>>>> =20 > >>>>>> I suppose there might be an application that maps memory > >>>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>>> =20 > >>>>>> - Jay > >>>>>> =20 > >>>>>>> To: m3devel at elegosoft.com > >>>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>>> From: mika at async.caltech.edu > >>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>>> =20 > >>>>>>> =20 > >>>>>>> Dear m3devel=2C > >>>>>>> =20 > >>>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>>> the unhelpful interface and module Crash: > >>>>>>> =20 > >>>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>>> =20 > >>>>>>> MODULE Crash=3B > >>>>>>> =20 > >>>>>>> PROCEDURE Me() =3D > >>>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>>> ptr^ :=3D 0 > >>>>>>> END Me=3B > >>>>>>> =20 > >>>>>>> BEGIN END Crash. > >>>>>>> =20 > >>>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>>> e > >>>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>>> =20 > >>>>>>> M-Scheme Experimental > >>>>>>> LITHP ITH LITHENING. > >>>>>>>> (require-modules "m3") > >>>>>>> #t > >>>>>>>> (Crash.Me) > >>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>>> (+ 3 4)=20 > >>>>>>> 7 > >>>>>>>> =20 > >>>>>>> =20 > >>>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>>> heck it.=20 > >>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>>> =20 > >>>>>>> Mika > >>>>>>> =20 > >>>>>> = > >>>>>> > >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>>> Content-Transfer-Encoding: quoted-printable > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>>> Imagine the heap =3Bis corrupted.
> >>>>>> And then you run more code.
> >>>>>> And the code happens to call malloc.
> >>>>>> Or printf to log something.
> >>>>>>  =3B
> >>>>>> I suppose there might be an application that maps memory
> >>>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>>  =3B
> >>>>>>  =3B- Jay
 =3B
> >>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>>> > >>>>>> = > >>>>>> > >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>>> > >>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 04:28:10 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 03:28:10 +0000 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: <20110221011550.0D2791A2078@async.async.caltech.edu> References: <20110221003248.11C9C2474005@birch.elegosoft.com>, , , , <20110221011550.0D2791A2078@async.async.caltech.edu> Message-ID: Wait? No, that is probably just bogus posix thinking. You can just close your handle to the process. You don't have to wait for it to exit. It is somewhat orthogonal. The thing is, the request for Process.Kill, got me looking at when does libm3 close the Process handle and when does it use the handle correctly (i.e. not m3middle's Process.Abort!) It does the CloseHandle in Process.Wait, so if that is never called, there is a leak. Anyway, this should be easy to make correct. (So easy, though, it boggles the mind as to why it was done this way in the first place. I do have to check somethig -- it is SLIGHTLY possible I broke it. I know I fiddled around with the definition of process ids, made them match the underlying platform. I'll check the history. m3middle uses a LOOPHOLE on some "id" to get the "handle". Anyway, I'll check and make it correct.) - Jay > To: jay.krell at cornell.edu > Date: Sun, 20 Feb 2011 17:15:50 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] [M3commit] CVS Update: cm3 > > > Jay I didn't quite follow your previous messages. What is the correct > solution, and what did you mean when you were talking about the garbage > collector? It does seem like---if a Process handle goes out of scope > and gets collected then yes the runtime should do the equivalent of "EVAL > Process.Wait" at some future time. But that seems unrelated/orthogonal > to the issues with the Windows process killing... > > The process killing in m3middle/M3Process I think is only called in > a few places in the compiler and then under what already are dubious > circumstances. For instance I see it done where Process.Create has > failed. But if Process.Create has failed, then there's no process to > kill, so it all seems rather pointless. It is also called as one of > the last actions before exiting the parent process, so there's another > reason for its being pointless. > > In other words I think it's safe to say that if you have an idea for > how to implement process-killing correctly on Windows you can do it > however you like without fear of breaking any existing code... > > Mika > > But I can > > Jay K writes: > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >The code here confuses hProcess and dwProcessId. > >Can we introduce a public function here in libm3 instead of m3middle? > >ie: to avoid answering the question as to how to expose hProcess? > >And to avoid the loophole as well? > > > >Thanks=2C > > - Jay > > > >From: jay.krell at cornell.edu > >To: jkrell at elego.de=3B m3commit at elegosoft.com > >Subject: RE: [M3commit] CVS Update: cm3 > >Date: Mon=2C 21 Feb 2011 00:35:38 +0000 > > > > > > > > > > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= > >rect here. > > > > - Jay > > > >> Date: Mon=2C 21 Feb 2011 01:32:47 +0000 > >> To: m3commit at elegosoft.com > >> From: jkrell at elego.de > >> Subject: [M3commit] CVS Update: cm3 > >>=20 > >> CVSROOT: /usr/cvs > >> Changes by: jkrell at birch. 11/02/21 01:32:47 > >>=20 > >> Modified files: > >> cm3/m3-sys/m3middle/src/: M3Process.i3=20 > >>=20 > >> Log message: > >> remove incorrect comments=3B Win32 is not broken > >>=20 > > = > > > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ > >Content-Type: text/html; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > > > > > > > >The code here confuses hProcess and dwProcessId.
Can we introduce a publ= > >ic function here in libm3 instead of m3middle?
ie: to avoid answering th= > >e question as to how to expose hProcess?
And to avoid the loophole as we= > >ll?

Thanks=2C
 =3B- Jay

From: = > >jay.krell at cornell.edu
To: jkrell at elego.de=3B m3commit at elegosoft.com
S= > >ubject: RE: [M3commit] CVS Update: cm3
Date: Mon=2C 21 Feb 2011 00:35:38= > > +0000

> > > > >"> > > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= > >rect here.

 =3B- Jay

>=3B Date: Mon=2C 21 Feb 2011 01:3= > >2:47 +0000
>=3B To: m3commit at elegosoft.com
>=3B From: jkrell at eleg= > >o.de
>=3B Subject: [M3commit] CVS Update: cm3
>=3B
>=3B CVS= > >ROOT: /usr/cvs
>=3B Changes by: jkrell at birch. 11/02/21 01:32:47
>= > >=3B
>=3B Modified files:
>=3B cm3/m3-sys/m3middle/src/: M3Proce= > >ss.i3
>=3B
>=3B Log message:
>=3B remove incorrect commen= > >ts=3B Win32 is not broken
>=3B
> >= > > > >--_d36c70d3-e71b-433d-abdc-06703a929f89_-- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 04:46:04 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 03:46:04 +0000 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20110221003248.11C9C2474005@birch.elegosoft.com>, , , , , , , <20110221011550.0D2791A2078@async.async.caltech.edu>, Message-ID: Ok, it looks like I did break it. http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/os/WIN32/ProcessWin32.m3.diff?r1=1.6;r2=1.7 GetID used to return the handle. I changed it to return the id. m3middle/Process.Abort LOOPHOLEs GetID to a handle. I think the GetID change should probably stay. But certainly one or the other must change. The handle is not unique across the system, just the process, whereas the id is unique across the system. If Process.i3 only mandates within the process, then maybe the old way. Maybe. - Jay From: jay.krell at cornell.edu To: mika at async.caltech.edu Date: Mon, 21 Feb 2011 03:28:10 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] [M3commit] CVS Update: cm3 Wait? No, that is probably just bogus posix thinking. You can just close your handle to the process. You don't have to wait for it to exit. It is somewhat orthogonal. The thing is, the request for Process.Kill, got me looking at when does libm3 close the Process handle and when does it use the handle correctly (i.e. not m3middle's Process.Abort!) It does the CloseHandle in Process.Wait, so if that is never called, there is a leak. Anyway, this should be easy to make correct. (So easy, though, it boggles the mind as to why it was done this way in the first place. I do have to check somethig -- it is SLIGHTLY possible I broke it. I know I fiddled around with the definition of process ids, made them match the underlying platform. I'll check the history. m3middle uses a LOOPHOLE on some "id" to get the "handle". Anyway, I'll check and make it correct.) - Jay > To: jay.krell at cornell.edu > Date: Sun, 20 Feb 2011 17:15:50 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] [M3commit] CVS Update: cm3 > > > Jay I didn't quite follow your previous messages. What is the correct > solution, and what did you mean when you were talking about the garbage > collector? It does seem like---if a Process handle goes out of scope > and gets collected then yes the runtime should do the equivalent of "EVAL > Process.Wait" at some future time. But that seems unrelated/orthogonal > to the issues with the Windows process killing... > > The process killing in m3middle/M3Process I think is only called in > a few places in the compiler and then under what already are dubious > circumstances. For instance I see it done where Process.Create has > failed. But if Process.Create has failed, then there's no process to > kill, so it all seems rather pointless. It is also called as one of > the last actions before exiting the parent process, so there's another > reason for its being pointless. > > In other words I think it's safe to say that if you have an idea for > how to implement process-killing correctly on Windows you can do it > however you like without fear of breaking any existing code... > > Mika > > But I can > > Jay K writes: > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >The code here confuses hProcess and dwProcessId. > >Can we introduce a public function here in libm3 instead of m3middle? > >ie: to avoid answering the question as to how to expose hProcess? > >And to avoid the loophole as well? > > > >Thanks=2C > > - Jay > > > >From: jay.krell at cornell.edu > >To: jkrell at elego.de=3B m3commit at elegosoft.com > >Subject: RE: [M3commit] CVS Update: cm3 > >Date: Mon=2C 21 Feb 2011 00:35:38 +0000 > > > > > > > > > > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= > >rect here. > > > > - Jay > > > >> Date: Mon=2C 21 Feb 2011 01:32:47 +0000 > >> To: m3commit at elegosoft.com > >> From: jkrell at elego.de > >> Subject: [M3commit] CVS Update: cm3 > >>=20 > >> CVSROOT: /usr/cvs > >> Changes by: jkrell at birch. 11/02/21 01:32:47 > >>=20 > >> Modified files: > >> cm3/m3-sys/m3middle/src/: M3Process.i3=20 > >>=20 > >> Log message: > >> remove incorrect comments=3B Win32 is not broken > >>=20 > > = > > > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ > >Content-Type: text/html; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > > > > > > > >The code here confuses hProcess and dwProcessId.
Can we introduce a publ= > >ic function here in libm3 instead of m3middle?
ie: to avoid answering th= > >e question as to how to expose hProcess?
And to avoid the loophole as we= > >ll?

Thanks=2C
 =3B- Jay

From: = > >jay.krell at cornell.edu
To: jkrell at elego.de=3B m3commit at elegosoft.com
S= > >ubject: RE: [M3commit] CVS Update: cm3
Date: Mon=2C 21 Feb 2011 00:35:38= > > +0000

> > > > >"> > > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= > >rect here.

 =3B- Jay

>=3B Date: Mon=2C 21 Feb 2011 01:3= > >2:47 +0000
>=3B To: m3commit at elegosoft.com
>=3B From: jkrell at eleg= > >o.de
>=3B Subject: [M3commit] CVS Update: cm3
>=3B
>=3B CVS= > >ROOT: /usr/cvs
>=3B Changes by: jkrell at birch. 11/02/21 01:32:47
>= > >=3B
>=3B Modified files:
>=3B cm3/m3-sys/m3middle/src/: M3Proce= > >ss.i3
>=3B
>=3B Log message:
>=3B remove incorrect commen= > >ts=3B Win32 is not broken
>=3B
> >= > > > >--_d36c70d3-e71b-433d-abdc-06703a929f89_-- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 04:49:34 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 03:49:34 +0000 Subject: [M3devel] platform-independent kill process In-Reply-To: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> References: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> Message-ID: Yes, please. I'm not sure Terminate should return anything though. In Win32 you would use TerminateProcess(). If you must have the exit code, that would be GetExitCodeProcess. Though the process might still be running and not have an orderly exit code. Oh, and TerminateProcess does have to set the exit code. In m3middle they set it to 1. > We currently only have to consider Unix and Windows, or are there > other active flavours of OSs to care for? I don't think so. There was OS/2, BeOS, Amiga, MacOS < 10.... - Jay > Date: Sun, 20 Feb 2011 13:28:52 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: [M3devel] platform-independent kill process > > I've been thinking a bit about the hanging tests problem in our regression > test framework. I'd like to provide a simple utility to start a program > with time supervision. But as far as I can see, Modula-3 does not provide > a platform-independent way of terminating a process. Of course, on Unix > I can send signals; but what would I do on Windows? A quick scan of the > win32 interfaces in m3core hasn't brought up anything, but I might > have looked for the wrong texts. > > Could we easily extend the Process interface e.g. by > > PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException > > ? > Any opinions about that? Different interface suggestions? > Rather create ProcessExtras than extending a `standard' interface? > > We currently only have to consider Unix and Windows, or are there > other active flavours of OSs to care for? > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Mon Feb 21 05:13:54 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sun, 20 Feb 2011 23:13:54 -0500 Subject: [M3devel] platform-independent kill process In-Reply-To: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> References: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> Message-ID: Olaf: If the regression tests are written in Modula-3, why not use Network Objects to provide an interface that can be called to tell the program to stop? This would be a "graceful" shutdown mechanism, rather than just killing the process at the OS level. I've used this technique before with other types of programs. Regards, Randy -----Original Message----- From: Olaf Wagner [mailto:wagner at elegosoft.com] Sent: Sunday, February 20, 2011 7:29 AM To: m3devel at elegosoft.com Subject: [M3devel] platform-independent kill process I've been thinking a bit about the hanging tests problem in our regression test framework. I'd like to provide a simple utility to start a program with time supervision. But as far as I can see, Modula-3 does not provide a platform-independent way of terminating a process. Of course, on Unix I can send signals; but what would I do on Windows? A quick scan of the win32 interfaces in m3core hasn't brought up anything, but I might have looked for the wrong texts. Could we easily extend the Process interface e.g. by PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException ? Any opinions about that? Different interface suggestions? Rather create ProcessExtras than extending a `standard' interface? We currently only have to consider Unix and Windows, or are there other active flavours of OSs to care for? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Mon Feb 21 05:50:31 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 20:50:31 -0800 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20110221003248.11C9C2474005@birch.elegosoft.com>, , , , <20110221011550.0D2791A2078@async.async.caltech.edu> Message-ID: <20110221045031.1D5D81A2078@async.async.caltech.edu> Jay K writes: >--_0b0e48b4-5c54-4fbb-8adf-2eb48374df09_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Wait? No=2C that is probably just bogus posix thinking. >You can just close your handle to the process. >You don't have to wait for it to exit. Well on Unix I don't think you can do that, can you? You can't tell the system that you're not going to be calling "wait4" on a particular PID... or can you? If not, the portable thing to do is to do what I said... register a GC cleanup such that when a Process.T goes out of scope the runtime does a Process.Wait in the background. It'd work on both Unix and Windows and use existing code rather than introducing a new abstraction. One issue is that with my SIGCHLD modification you will get zombies if you don't call wait4 on Unix. Process.i3 does mention this possibility so I don't think I "broke" anything. (Coding to the specification cuts both ways...) Mika From wagner at elegosoft.com Mon Feb 21 07:49:08 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 21 Feb 2011 07:49:08 +0100 Subject: [M3devel] platform-independent kill process In-Reply-To: References: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> Message-ID: <20110221074908.jndwjzx2bqcoo880@mail.elegosoft.com> Quoting "Coleburn, Randy" : > Olaf: > > If the regression tests are written in Modula-3, why not use Network > Objects to provide an interface that can be called to tell the > program to stop? Baucse that involves too many other sub-systems and components. I'd like to keep the requirements for tests limited. When we test such basic things like threading, we shouldn't have to assume that the network subsystem and RPC are working well, too. > This would be a "graceful" shutdown mechanism, rather than just > killing the process at the OS level. I'm not sure that it would work in the presence of threading problems, but I may be mistaken there. > I've used this technique before with other types of programs. Nothing speaking against that, if your application needs all the functionality anyway. However, I think M3 should support a basic mechanism on process level. Olaf > Regards, > Randy > > -----Original Message----- > From: Olaf Wagner [mailto:wagner at elegosoft.com] > Sent: Sunday, February 20, 2011 7:29 AM > To: m3devel at elegosoft.com > Subject: [M3devel] platform-independent kill process > > I've been thinking a bit about the hanging tests problem in our regression > test framework. I'd like to provide a simple utility to start a program > with time supervision. But as far as I can see, Modula-3 does not provide > a platform-independent way of terminating a process. Of course, on Unix > I can send signals; but what would I do on Windows? A quick scan of the > win32 interfaces in m3core hasn't brought up anything, but I might > have looked for the wrong texts. > > Could we easily extend the Process interface e.g. by > > PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException > > ? > Any opinions about that? Different interface suggestions? > Rather create ProcessExtras than extending a `standard' interface? > > We currently only have to consider Unix and Windows, or are there > other active flavours of OSs to care for? > > Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Mon Feb 21 12:12:10 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 11:12:10 +0000 Subject: [M3devel] libm3/Process.ExitCode? Message-ID: TYPE ExitCode = [0 .. 16_7FFFFFFF]; (* An exit code (or status) of zero normally means successful termination, and a non-zero value normally indicates an error, but the exact conventions vary between systems and programs. *) Is there a good reason this isn't a full INTEGER? Or at least a full Cstdint.int32_t? In the Win32 case, we mask the actual exit code with this, which will lead to the non-zero 0x80000000 becoming zero. I'm inclined to change it like: before: RETURN Word.And(status, LAST(ExitCode)) after: (* for some reason, cut off the high bit, but * if that turns non-zero into zero, then still return non-zero *) maskedStatus := Word.And(status, LAST(ExitCode)); IF status <> 0 and maskedStatus = 0 THEN maskedStatus := 1; END; RETURN maskedStatus Or change ExitCode to plain lossless INTEGER. But maybe negative values were avoided for some reason? I understand that on Posix, I think, you just get an 8 bit unsigned value. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 13:00:14 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 12:00:14 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , <4D604FA3.3060708@lcwb.coop>, , <20110219232729.53A991A2078@async.async.caltech.edu>, , <4D615643.7060107@lcwb.coop>, , <20110220183746.195C11A2078@async.async.caltech.edu>, Message-ID: Ok, I'm disappointed to report that I did some quick checking on Darwin/x86, Linux/x86, Solaris/sparc32/x86 and none of them seem to be smart about this. Er, less they were being really smart. int F1(void) { volatile char a[4000]; return a[3999]; } int F2(void) { volatile char a[6000]; return a[5999]; } int F4(void) { volatile char a[60000]; return a[59999]; } I guess I should redo all the tests but accessing [0] also/instead. So, yes, we should definitely consider doing something. However...however, the frontend doesn't know which locals wiil be optimized away. The backend is setup to do this...but it is only used for NT systems, darn. jbook2:gcc jay$ grep CHECK_STACK_LIMIT */*/*/* gcc/config/i386/cygming.h:#define CHECK_STACK_LIMIT 4000 gcc/config/i386/cygwin.asm: than CHECK_STACK_LIMIT bytes in one go. Touching the stack at 4K gcc/config/i386/i386-interix.h:#define CHECK_STACK_LIMIT 0x1000 gcc/config/i386/i386.c:#ifndef CHECK_STACK_LIMIT gcc/config/i386/i386.c:#define CHECK_STACK_LIMIT (-1) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT)) gcc/config/i386/i386.c: else if (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT))) gcc/config/i386/i386.md:#ifndef CHECK_STACK_LIMIT gcc/config/i386/i386.md:#define CHECK_STACK_LIMIT 0 gcc/config/i386/i386.md: if (CHECK_STACK_LIMIT && CONST_INT_P (operands[1]) gcc/config/i386/i386.md: && INTVAL (operands[1]) < CHECK_STACK_LIMIT) jbook2:gcc jay$ grep CHECK_STACK_LIMIT */*/*/*/* jbook2:gcc jay$ grep TARGET_STACK_PROBE * jbook2:gcc jay$ grep TARGET_STACK_PROBE */* gcc/ChangeLog: * config/i386/i386.c (ix86_expand_prologue) [TARGET_STACK_PROBE]: gcc/ChangeLog-2005: (TARGET_TLS_DIRECT_SEG_REFS, TARGET_STACK_PROBE) jbook2:gcc jay$ grep TARGET_STACK_PROBE */*/* jbook2:gcc jay$ grep TARGET_STACK_PROBE */*/*/* gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT)) gcc/config/i386/i386.c: else if (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT))) gcc/config/i386/i386.md: "!TARGET_64BIT && TARGET_STACK_PROBE" gcc/config/i386/i386.md: "TARGET_64BIT && TARGET_STACK_PROBE" gcc/config/i386/i386.md: "TARGET_STACK_PROBE" - Jay From: jay.krell at cornell.edu To: mika at async.caltech.edu; rodney_bates at lcwb.coop Date: Sun, 20 Feb 2011 22:38:19 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] SEGV mapping to RuntimeError Probably for this reason: There is a requirement on NT that stack pages be touched in order. Functions with locals totally more than 4K call _chkstk (aka _alloca) to allocate their stack, instead of the usual register subtraction. It contains a loop that touches a byte every 4K. Otherwise, if you have lots of functions with small frames, the stack is touched by virtue of the call function pushing the return address. Modula-3 has long failed to uphold this contract, and still does. I've never seen it clearly documented, but you can see it is what the C compiler does. I had thought there were other reasons for this behavior. I thought you actually get an exception of the stack is touched out of order ("the first time"). But I think I did an experiment long ago with Modula-3 and there was no exception. I don't know about other platforms. I've also seen evidence in gcc and/or Target.i3 that compiler writers know well about such mechanisms -- there being a constant to set as to what size locals trigger special behavior. But m3back doesn't do anything here. Probably we could use _chkstk unconditonally as well -- need to see what it does for numbers smaller than 4K. But that'd be a deoptimization in the common case, and deoptimizing only as necessary should be easy enough. - Jay > To: rodney_bates at lcwb.coop > Date: Sun, 20 Feb 2011 10:37:46 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > On a 64-bit machine, at least, there ought to be enough virtual > memory that you could just have a gap between thread stacks big > enough to allow for a protection area larger than the largest possible > (implementation-defined) activation record, no? I know I've run into > trouble with very large activation records in the past (and not because > I was running out of stack space, either). > > Or at least a procedure with a very large activation record (or > a procedure calling it) could be required to call some sort of check > routine "EnoughStackSpaceRemaining()" before starting to scribble > on the activation record? > > Also the end of the activation record must be written to at least once, > or else the memory protection won't be triggered. > > In any case if this is done properly the same mechanism I proposed for > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > signals are delivered on a separate stack. If signals are delivered on > the same stack, the signal handler would get nastier, it would have to > make space through some manipulations (maybe temporarily unporotecting > the redzone page?) for its own purposes... but I don't see why it > couldn't be done. > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > activated inside the redzone page because of a difference in signal > handling..? I remember reading something about sigaltstack... > > I would of course love to be able to recover from stack overflow, too. > In some sense, since it's a generally unknown limit, it's even less of > a fatal error than a NIL dereference (hence makes even more sense to > catch it). > > Mika > > "Rodney M. Bates" writes: > >I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >Probably a fully protected guard page at the end of the stack. This technique > >always worries me a bit because a procedure with a really big activation record > >could jump right past it. Probably it would almost always access the first page > >of the big area before storing anything into later pages. > > > >On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >> Ah, yes, stack protection. > >> > >> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >> > >> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >> > >> time ../AMD64_FREEBSD/stubexample > >> M-Scheme Experimental > >> LITHP ITH LITHENING. > >>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >> f > >>> (f 0) > >> Illegal instruction > >> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >> > >> What absolutely must not happen, of course, is that the runtime hangs > >> while executing only safe code... > >> > >> Mika > >> > >> "Rodney M. Bates" writes: > >>> I know of one other place the compilers rely on hardware memory protection > >>> to detect a checked runtime error, and that is stack overflow. This won't > >>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>> This could probably be distinguished after the fact by some low-level, > >>> target-dependent code. I have found it by looking at assembly code at > >>> the point of failure--usually right after a stack pointer push. > >>> > >>> Detecting this via compiler-generated checks would probably be more > >>> extravagant than many other checks, as it is so frequent. I am not > >>> aware of any really good solution to this in any implementation of any > >>> language. > >>> > >>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>> you know! > >>>> > >>>> "Corrupting the heap" is something that can only happen as a result of > >>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>> modules not marked UNSAFE. > >>>> > >>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>> to signal a certain kind of *checked* runtime error, namely, the > >>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>> > >>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>> current CM3 are ones from NIL dereferences. > >>>> > >>>> Hence, as long as you stick with safe code, the only time the code I > >>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>> recoverable. > >>>> > >>>> :-) > >>>> > >>>> Mika > >>>> > >>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>> Almost all what remains is glue to some C library, which wouldn't be > >>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>> I don't have the time to rewrite every single library from scratch myself. > >>>> > >>>> > >>>> Jay K writes: > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious. > >>>>> Imagine the heap is corrupted. > >>>>> And then you run more code. > >>>>> And the code happens to call malloc. > >>>>> Or printf to log something. > >>>>> =20 > >>>>> I suppose there might be an application that maps memory > >>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>> =20 > >>>>> - Jay > >>>>> =20 > >>>>>> To: m3devel at elegosoft.com > >>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>> From: mika at async.caltech.edu > >>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>> =20 > >>>>>> =20 > >>>>>> Dear m3devel=2C > >>>>>> =20 > >>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>> the unhelpful interface and module Crash: > >>>>>> =20 > >>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>> =20 > >>>>>> MODULE Crash=3B > >>>>>> =20 > >>>>>> PROCEDURE Me() =3D > >>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>> ptr^ :=3D 0 > >>>>>> END Me=3B > >>>>>> =20 > >>>>>> BEGIN END Crash. > >>>>>> =20 > >>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>> e > >>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>> =20 > >>>>>> M-Scheme Experimental > >>>>>> LITHP ITH LITHENING. > >>>>>>> (require-modules "m3") > >>>>>> #t > >>>>>>> (Crash.Me) > >>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>> (+ 3 4)=20 > >>>>>> 7 > >>>>>>> =20 > >>>>>> =20 > >>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>> heck it.=20 > >>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>> =20 > >>>>>> Mika > >>>>>> =20 > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>> Imagine the heap =3Bis corrupted.
> >>>>> And then you run more code.
> >>>>> And the code happens to call malloc.
> >>>>> Or printf to log something.
> >>>>>  =3B
> >>>>> I suppose there might be an application that maps memory
> >>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>  =3B
> >>>>>  =3B- Jay
 =3B
> >>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>> > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>> > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 13:05:33 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 12:05:33 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, ,,, ,,<20110219203821.2016B1A2078@async.async.caltech.edu>, ,,<4D604FA3.3060708@lcwb.coop>, ,,<20110219232729.53A991A2078@async.async.caltech.edu>, ,,<4D615643.7060107@lcwb.coop>, , , <20110220183746.195C11A2078@async.async.caltech.edu>, , , Message-ID: Hm. Maybe we just need to pass -fstack-check to the backend. Needs experimentation... - Jay From: jay.krell at cornell.edu To: mika at async.caltech.edu; rodney_bates at lcwb.coop Date: Mon, 21 Feb 2011 12:00:14 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] SEGV mapping to RuntimeError Ok, I'm disappointed to report that I did some quick checking on Darwin/x86, Linux/x86, Solaris/sparc32/x86 and none of them seem to be smart about this. Er, less they were being really smart. int F1(void) { volatile char a[4000]; return a[3999]; } int F2(void) { volatile char a[6000]; return a[5999]; } int F4(void) { volatile char a[60000]; return a[59999]; } I guess I should redo all the tests but accessing [0] also/instead. So, yes, we should definitely consider doing something. However...however, the frontend doesn't know which locals wiil be optimized away. The backend is setup to do this...but it is only used for NT systems, darn. jbook2:gcc jay$ grep CHECK_STACK_LIMIT */*/*/* gcc/config/i386/cygming.h:#define CHECK_STACK_LIMIT 4000 gcc/config/i386/cygwin.asm: than CHECK_STACK_LIMIT bytes in one go. Touching the stack at 4K gcc/config/i386/i386-interix.h:#define CHECK_STACK_LIMIT 0x1000 gcc/config/i386/i386.c:#ifndef CHECK_STACK_LIMIT gcc/config/i386/i386.c:#define CHECK_STACK_LIMIT (-1) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT)) gcc/config/i386/i386.c: else if (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT))) gcc/config/i386/i386.md:#ifndef CHECK_STACK_LIMIT gcc/config/i386/i386.md:#define CHECK_STACK_LIMIT 0 gcc/config/i386/i386.md: if (CHECK_STACK_LIMIT && CONST_INT_P (operands[1]) gcc/config/i386/i386.md: && INTVAL (operands[1]) < CHECK_STACK_LIMIT) jbook2:gcc jay$ grep CHECK_STACK_LIMIT */*/*/*/* jbook2:gcc jay$ grep TARGET_STACK_PROBE * jbook2:gcc jay$ grep TARGET_STACK_PROBE */* gcc/ChangeLog: * config/i386/i386.c (ix86_expand_prologue) [TARGET_STACK_PROBE]: gcc/ChangeLog-2005: (TARGET_TLS_DIRECT_SEG_REFS, TARGET_STACK_PROBE) jbook2:gcc jay$ grep TARGET_STACK_PROBE */*/* jbook2:gcc jay$ grep TARGET_STACK_PROBE */*/*/* gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT)) gcc/config/i386/i386.c: else if (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT))) gcc/config/i386/i386.md: "!TARGET_64BIT && TARGET_STACK_PROBE" gcc/config/i386/i386.md: "TARGET_64BIT && TARGET_STACK_PROBE" gcc/config/i386/i386.md: "TARGET_STACK_PROBE" - Jay From: jay.krell at cornell.edu To: mika at async.caltech.edu; rodney_bates at lcwb.coop Date: Sun, 20 Feb 2011 22:38:19 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] SEGV mapping to RuntimeError Probably for this reason: There is a requirement on NT that stack pages be touched in order. Functions with locals totally more than 4K call _chkstk (aka _alloca) to allocate their stack, instead of the usual register subtraction. It contains a loop that touches a byte every 4K. Otherwise, if you have lots of functions with small frames, the stack is touched by virtue of the call function pushing the return address. Modula-3 has long failed to uphold this contract, and still does. I've never seen it clearly documented, but you can see it is what the C compiler does. I had thought there were other reasons for this behavior. I thought you actually get an exception of the stack is touched out of order ("the first time"). But I think I did an experiment long ago with Modula-3 and there was no exception. I don't know about other platforms. I've also seen evidence in gcc and/or Target.i3 that compiler writers know well about such mechanisms -- there being a constant to set as to what size locals trigger special behavior. But m3back doesn't do anything here. Probably we could use _chkstk unconditonally as well -- need to see what it does for numbers smaller than 4K. But that'd be a deoptimization in the common case, and deoptimizing only as necessary should be easy enough. - Jay > To: rodney_bates at lcwb.coop > Date: Sun, 20 Feb 2011 10:37:46 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > On a 64-bit machine, at least, there ought to be enough virtual > memory that you could just have a gap between thread stacks big > enough to allow for a protection area larger than the largest possible > (implementation-defined) activation record, no? I know I've run into > trouble with very large activation records in the past (and not because > I was running out of stack space, either). > > Or at least a procedure with a very large activation record (or > a procedure calling it) could be required to call some sort of check > routine "EnoughStackSpaceRemaining()" before starting to scribble > on the activation record? > > Also the end of the activation record must be written to at least once, > or else the memory protection won't be triggered. > > In any case if this is done properly the same mechanism I proposed for > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > signals are delivered on a separate stack. If signals are delivered on > the same stack, the signal handler would get nastier, it would have to > make space through some manipulations (maybe temporarily unporotecting > the redzone page?) for its own purposes... but I don't see why it > couldn't be done. > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > activated inside the redzone page because of a difference in signal > handling..? I remember reading something about sigaltstack... > > I would of course love to be able to recover from stack overflow, too. > In some sense, since it's a generally unknown limit, it's even less of > a fatal error than a NIL dereference (hence makes even more sense to > catch it). > > Mika > > "Rodney M. Bates" writes: > >I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >Probably a fully protected guard page at the end of the stack. This technique > >always worries me a bit because a procedure with a really big activation record > >could jump right past it. Probably it would almost always access the first page > >of the big area before storing anything into later pages. > > > >On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >> Ah, yes, stack protection. > >> > >> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >> > >> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >> > >> time ../AMD64_FREEBSD/stubexample > >> M-Scheme Experimental > >> LITHP ITH LITHENING. > >>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >> f > >>> (f 0) > >> Illegal instruction > >> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >> > >> What absolutely must not happen, of course, is that the runtime hangs > >> while executing only safe code... > >> > >> Mika > >> > >> "Rodney M. Bates" writes: > >>> I know of one other place the compilers rely on hardware memory protection > >>> to detect a checked runtime error, and that is stack overflow. This won't > >>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>> This could probably be distinguished after the fact by some low-level, > >>> target-dependent code. I have found it by looking at assembly code at > >>> the point of failure--usually right after a stack pointer push. > >>> > >>> Detecting this via compiler-generated checks would probably be more > >>> extravagant than many other checks, as it is so frequent. I am not > >>> aware of any really good solution to this in any implementation of any > >>> language. > >>> > >>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>> you know! > >>>> > >>>> "Corrupting the heap" is something that can only happen as a result of > >>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>> modules not marked UNSAFE. > >>>> > >>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>> to signal a certain kind of *checked* runtime error, namely, the > >>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>> > >>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>> current CM3 are ones from NIL dereferences. > >>>> > >>>> Hence, as long as you stick with safe code, the only time the code I > >>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>> recoverable. > >>>> > >>>> :-) > >>>> > >>>> Mika > >>>> > >>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>> Almost all what remains is glue to some C library, which wouldn't be > >>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>> I don't have the time to rewrite every single library from scratch myself. > >>>> > >>>> > >>>> Jay K writes: > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious. > >>>>> Imagine the heap is corrupted. > >>>>> And then you run more code. > >>>>> And the code happens to call malloc. > >>>>> Or printf to log something. > >>>>> =20 > >>>>> I suppose there might be an application that maps memory > >>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>> =20 > >>>>> - Jay > >>>>> =20 > >>>>>> To: m3devel at elegosoft.com > >>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>> From: mika at async.caltech.edu > >>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>> =20 > >>>>>> =20 > >>>>>> Dear m3devel=2C > >>>>>> =20 > >>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>> the unhelpful interface and module Crash: > >>>>>> =20 > >>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>> =20 > >>>>>> MODULE Crash=3B > >>>>>> =20 > >>>>>> PROCEDURE Me() =3D > >>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>> ptr^ :=3D 0 > >>>>>> END Me=3B > >>>>>> =20 > >>>>>> BEGIN END Crash. > >>>>>> =20 > >>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>> e > >>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>> =20 > >>>>>> M-Scheme Experimental > >>>>>> LITHP ITH LITHENING. > >>>>>>> (require-modules "m3") > >>>>>> #t > >>>>>>> (Crash.Me) > >>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>> (+ 3 4)=20 > >>>>>> 7 > >>>>>>> =20 > >>>>>> =20 > >>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>> heck it.=20 > >>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>> =20 > >>>>>> Mika > >>>>>> =20 > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>> Imagine the heap =3Bis corrupted.
> >>>>> And then you run more code.
> >>>>> And the code happens to call malloc.
> >>>>> Or printf to log something.
> >>>>>  =3B
> >>>>> I suppose there might be an application that maps memory
> >>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>  =3B
> >>>>>  =3B- Jay
 =3B
> >>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>> > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>> > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Tue Feb 22 01:01:12 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Mon, 21 Feb 2011 16:01:12 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> Message-ID: <20110222000112.5A7131A2078@async.async.caltech.edu> There is in fact a small problem with my change to the way SIGSEGV is handled. It now appears that bad memory references are all happening in ThreadPosix.m3 (i.e., we've lost some of the debugging help of the error message). I'm not quite sure how to fix this although I'm sure all the bits are around. My guess is that instead of performing RAISE RuntimeError.E in the thread switcher, the code ought to call RTHooks.Raise with an argument derived from the faulting address as reported to the signal handler. This is probably quite easy, but does anyone know the right routines to call? One would need to track down the routine that faulted as well as the RT0 record corresponding to the exception RuntimeError.E. Mika Jay K writes: >Letting any code run after a SIGSEGV is dubious. >Imagine the heap is corrupted. >And then you run more code. >And the code happens to call malloc. >Or printf to log something. >=20 >I suppose there might be an application that maps memory >gradually=2C as pieces of a buffer are hit. Might. From dabenavidesd at yahoo.es Tue Feb 22 02:07:45 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Tue, 22 Feb 2011 01:07:45 +0000 (GMT) Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222000112.5A7131A2078@async.async.caltech.edu> Message-ID: <533385.35225.qm@web29704.mail.ird.yahoo.com> Hi all: although this is not of my full understanding, CM3 JVM was supposed to handle in such a way the exceptions of jvm ?-op fault script programs, given that: there was a runtime above the CM3 runtime and besides having its own spaces for stack, thread stacks and runtime machinery, loader, etc. Please just to advert this is and I guess was the case for CM3 JVM, if so, let the system handle it in that way, why do you need to fully manage to handle that as in Modula-3 runtime per se, my believe is that you could do that for the language environment routines separately, that's why pragmas are really useful. I don't believe it allowed to signal like in Modula-3 terms threads but you could for instance lock a thread until you have time in the scheduler from Modula-3, still this is more flexible nowadays I believe with multiprocessors, etc. Similarly SPIN used or performed protection mechanism, like domain level, and at language level the heap freezes when handling untrusted code (even extra checked array bounds violation faults in non-Modula-3 code), so you have same runtime I believe for doing that too in Modula-3, again, is just another time of problem arises, perhaps this can be tried for thread stacks, etc and in kernel threads well as something alike, I really think could be done. Thanks in advance --- El lun, 21/2/11, Mika Nystrom escribi?: > De: Mika Nystrom > Asunto: Re: [M3devel] SEGV mapping to RuntimeError > Para: "Jay K" , m3devel at elegosoft.com > Fecha: lunes, 21 de febrero, 2011 19:01 > > There is in fact a small problem with my change to the way > SIGSEGV is > handled. It now appears that bad memory references > are all happening > in ThreadPosix.m3 (i.e., we've lost some of the debugging > help of the > error message). > > I'm not quite sure how to fix this although I'm sure all > the bits are > around. My guess is that instead of performing RAISE > RuntimeError.E > in the thread switcher, the code ought to call > RTHooks.Raise with an > argument derived from the faulting address as reported to > the signal > handler. This is probably quite easy, but does anyone > know the > right routines to call? One would need to track down > the routine that > faulted as well as the RT0 record corresponding to the > exception > RuntimeError.E. > > Mika > > Jay K writes: > >Letting any code run after a SIGSEGV is dubious. > >Imagine the heap is corrupted. > >And then you run more code. > >And the code happens to call malloc. > >Or printf to log something. > >=20 > >I suppose there might be an application that maps > memory > >gradually=2C as pieces of a buffer are hit. Might. > From mika at async.caltech.edu Tue Feb 22 03:12:00 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Mon, 21 Feb 2011 18:12:00 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <533385.35225.qm@web29704.mail.ird.yahoo.com> References: <533385.35225.qm@web29704.mail.ird.yahoo.com> Message-ID: <20110222021200.9F0EE1A2078@async.async.caltech.edu> There are several issues here and yes this the RuntimeError.E exception is taken either from SPIN or the CM3 JVM, I don't know which. I only understand the user threads code so I have made that raise the RuntimeError.E instead of dumping core on a segfault. I think this is what the CM JVM was supposed to do as well---but honestly I can't tell if that work was ever completed. They do support mapping various other runtime errors (subrange mismatch, e.g.) into exceptions, and that works well. However the segmentation fault mapping was not functional: there was no other way of handling it than dumping core on the whole M3 environment and runtime before my changes last week. My last email was about a bug in my changes---not in the actual functionality but the fact that if you *don't* catch the RuntimeError.E exception that results from a bad memory reference, the runtime quits with a slightly confusing error message, apparently implicating the threading system when the bug is actually in normal, non-UNSAFE code. Mika "Daniel Alejandro Benavides D." writes: >Hi all: >although this is not of my full understanding, CM3 JVM was supposed to hand= >le in such a way the exceptions of jvm =B5-op fault script programs, given = >that: there was a runtime above the CM3 runtime and besides having its own = >spaces for stack, thread stacks and runtime machinery, loader, etc. >Please just to advert this is and I guess was the case for CM3 JVM, if so, = >let the system handle it in that way, why do you need to fully manage to ha= >ndle that as in Modula-3 runtime per se, my believe is that you could do th= >at for the language environment routines separately, that's why pragmas are= > really useful. I don't believe it allowed to signal like in Modula-3 terms= > threads but you could for instance lock a thread until you have time in th= >e scheduler from Modula-3, still this is more flexible nowadays I believe w= >ith multiprocessors, etc. >Similarly SPIN used or performed protection mechanism, like domain level, a= >nd at language level the heap freezes when handling untrusted code (even ex= >tra checked array bounds violation faults in non-Modula-3 code), so you hav= >e same runtime I believe for doing that too in Modula-3, again, is just ano= >ther time of problem arises, perhaps this can be tried for thread stacks, e= >tc and in kernel threads well as something alike, I really think could be d= >one. >Thanks in advance > >--- El lun, 21/2/11, Mika Nystrom escribi=F3: > >> De: Mika Nystrom >> Asunto: Re: [M3devel] SEGV mapping to RuntimeError >> Para: "Jay K" , m3devel at elegosoft.com >> Fecha: lunes, 21 de febrero, 2011 19:01 >>=20 >> There is in fact a small problem with my change to the way >> SIGSEGV is >> handled. It now appears that bad memory references >> are all happening >> in ThreadPosix.m3 (i.e., we've lost some of the debugging >> help of the >> error message). >>=20 >> I'm not quite sure how to fix this although I'm sure all >> the bits are >> around. My guess is that instead of performing RAISE >> RuntimeError.E >> in the thread switcher, the code ought to call >> RTHooks.Raise with an >> argument derived from the faulting address as reported to >> the signal >> handler. This is probably quite easy, but does anyone >> know the=20 >> right routines to call? One would need to track down >> the routine that >> faulted as well as the RT0 record corresponding to the >> exception >> RuntimeError.E. >>=20 >> Mika >>=20 >> Jay K writes: >> >Letting any code run after a SIGSEGV is dubious. >> >Imagine the heap is corrupted. >> >And then you run more code. >> >And the code happens to call malloc. >> >Or printf to log something. >> >=3D20 >> >I suppose there might be an application that maps >> memory >> >gradually=3D2C as pieces of a buffer are hit. Might. >> =0A=0A=0A From dabenavidesd at yahoo.es Tue Feb 22 05:31:58 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Tue, 22 Feb 2011 04:31:58 +0000 (GMT) Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222021200.9F0EE1A2078@async.async.caltech.edu> Message-ID: <806751.73807.qm@web29712.mail.ird.yahoo.com> Hi all: yes I see, things are harder than I thought here but still seems the problem is not directly in the runtime environment system, perhaps this is more just another time of System native restrictions: according to this guys: http://grsecurity.net/confighelp.php/ (see for Automatically emulate sigreturn trampolines) that's why perhaps the SPIN could do that without requiring the process to terminate which is otherwise is very odd I believe for such restrictions, still need to check more details: Interesting this emails go into the details of more advanced programs to analyze ine ach platform the runtime behavior many years ago, look for Hp-UX implementation "bug". Again grsecurity OS doers talk again even of (see CVSup): http://pax.grsecurity.net/docs/randexec.txt I hope it helps a bit, thanks in advance --- El lun, 21/2/11, Mika Nystrom escribi?: > De: Mika Nystrom > Asunto: Re: [M3devel] SEGV mapping to RuntimeError > Para: "Daniel Alejandro Benavides D." > CC: m3devel at elegosoft.com > Fecha: lunes, 21 de febrero, 2011 21:12 > > There are several issues here and yes this the > RuntimeError.E exception > is taken either from SPIN or the CM3 JVM, I don't know > which. > > I only understand the user threads code so I have made that > raise the > RuntimeError.E instead of dumping core on a segfault. > I think this is what > the CM JVM was supposed to do as well---but honestly I > can't tell if that > work was ever completed. They do support mapping > various other runtime > errors (subrange mismatch, e.g.) into exceptions, and that > works well. > However the segmentation fault mapping was not functional: > there was no > other way of handling it than dumping core on the whole M3 > environment > and runtime before my changes last week. > > My last email was about a bug in my changes---not in the > actual functionality > but the fact that if you *don't* catch the RuntimeError.E > exception that > results from a bad memory reference, the runtime quits with > a slightly > confusing error message, apparently implicating the > threading system when > the bug is actually in normal, non-UNSAFE code. > > Mika > > "Daniel Alejandro Benavides D." writes: > >Hi all: > >although this is not of my full understanding, CM3 JVM > was supposed to hand= > >le in such a way the exceptions of jvm =B5-op fault > script programs, given = > >that: there was a runtime above the CM3 runtime and > besides having its own = > >spaces for stack, thread stacks and runtime machinery, > loader, etc. > >Please just to advert this is and I guess was the case > for CM3 JVM, if so, = > >let the system handle it in that way, why do you need > to fully manage to ha= > >ndle that as in Modula-3 runtime per se, my believe is > that you could do th= > >at for the language environment routines separately, > that's why pragmas are= > > really useful. I don't believe it allowed to signal > like in Modula-3 terms= > > threads but you could for instance lock a thread until > you have time in th= > >e scheduler from Modula-3, still this is more flexible > nowadays I believe w= > >ith multiprocessors, etc. > >Similarly SPIN used or performed protection mechanism, > like domain level, a= > >nd at language level the heap freezes when handling > untrusted code (even ex= > >tra checked array bounds violation faults in > non-Modula-3 code), so you hav= > >e same runtime I believe for doing that too in > Modula-3, again, is just ano= > >ther time of problem arises, perhaps this can be tried > for thread stacks, e= > >tc and in kernel threads well as something alike, I > really think could be d= > >one. > >Thanks in advance > > > >--- El lun, 21/2/11, Mika Nystrom > escribi=F3: > > > >> De: Mika Nystrom > >> Asunto: Re: [M3devel] SEGV mapping to > RuntimeError > >> Para: "Jay K" , > m3devel at elegosoft.com > >> Fecha: lunes, 21 de febrero, 2011 19:01 > >>=20 > >> There is in fact a small problem with my change to > the way > >> SIGSEGV is > >> handled. It now appears that bad memory > references > >> are all happening > >> in ThreadPosix.m3 (i.e., we've lost some of the > debugging > >> help of the > >> error message). > >>=20 > >> I'm not quite sure how to fix this although I'm > sure all > >> the bits are > >> around. My guess is that instead of > performing RAISE > >> RuntimeError.E > >> in the thread switcher, the code ought to call > >> RTHooks.Raise with an > >> argument derived from the faulting address as > reported to > >> the signal > >> handler. This is probably quite easy, but > does anyone > >> know the=20 > >> right routines to call? One would need to > track down > >> the routine that > >> faulted as well as the RT0 record corresponding to > the > >> exception > >> RuntimeError.E. > >>=20 > >> Mika > >>=20 > >> Jay K writes: > >> >Letting any code run after a SIGSEGV is > dubious. > >> >Imagine the heap is corrupted. > >> >And then you run more code. > >> >And the code happens to call malloc. > >> >Or printf to log something. > >> >=3D20 > >> >I suppose there might be an application that > maps > >> memory > >> >gradually=3D2C as pieces of a buffer are hit. > Might. > >> =0A=0A=0A > From dabenavidesd at yahoo.es Tue Feb 22 05:40:34 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Tue, 22 Feb 2011 04:40:34 +0000 (GMT) Subject: [M3devel] SEGV mapping to RuntimeError Message-ID: <748970.47663.qm@web29719.mail.ird.yahoo.com> Hi all: sorry missed the second link in the comments about HP--UX and Nil dereferences: http://www.usenix.org/publications/library/proceedings/sd96/full_papers/small.txt Still they seemed to be some option in the compiler to check where platform needed to catch the RuntimeError either from compiler inserted checks for NIL and from trapped OS call or just ignore it in some circunstances like HP-UX case, see for that it. Thanks in advance --- El lun, 21/2/11, Daniel Alejandro Benavides D. escribi?: > De: Daniel Alejandro Benavides D. > Asunto: Re: [M3devel] SEGV mapping to RuntimeError > Para: "Mika Nystrom" > CC: m3devel at elegosoft.com > Fecha: lunes, 21 de febrero, 2011 23:31 > Hi all: > yes I see, things are harder than I thought here but still > seems the problem is not directly in the runtime environment > system, perhaps this is more just another time of System > native restrictions: > according to this guys: > http://grsecurity.net/confighelp.php/ > (see for Automatically emulate sigreturn trampolines) > that's why perhaps the SPIN could do that without requiring > the process to terminate which is otherwise is very odd I > believe for such restrictions, still need to check more > details: > Interesting this emails go into the details of more > advanced programs to analyze ine ach platform the runtime > behavior many years ago, look for Hp-UX implementation > "bug". > Again grsecurity OS doers talk again even of (see CVSup): > http://pax.grsecurity.net/docs/randexec.txt > I hope it helps a bit, thanks in advance > > --- El lun, 21/2/11, Mika Nystrom > escribi?: > > > De: Mika Nystrom > > Asunto: Re: [M3devel] SEGV mapping to RuntimeError > > Para: "Daniel Alejandro Benavides D." > > CC: m3devel at elegosoft.com > > Fecha: lunes, 21 de febrero, 2011 21:12 > > > > There are several issues here and yes this the > > RuntimeError.E exception > > is taken either from SPIN or the CM3 JVM, I don't > know > > which. > > > > I only understand the user threads code so I have made > that > > raise the > > RuntimeError.E instead of dumping core on a segfault. > > > I think this is what > > the CM JVM was supposed to do as well---but honestly > I > > can't tell if that > > work was ever completed. They do support > mapping > > various other runtime > > errors (subrange mismatch, e.g.) into exceptions, and > that > > works well. > > However the segmentation fault mapping was not > functional: > > there was no > > other way of handling it than dumping core on the > whole M3 > > environment > > and runtime before my changes last week. > > > > My last email was about a bug in my changes---not in > the > > actual functionality > > but the fact that if you *don't* catch the > RuntimeError.E > > exception that > > results from a bad memory reference, the runtime quits > with > > a slightly > > confusing error message, apparently implicating the > > threading system when > > the bug is actually in normal, non-UNSAFE code. > > > > Mika > > > > "Daniel Alejandro Benavides D." writes: > > >Hi all: > > >although this is not of my full understanding, CM3 > JVM > > was supposed to hand= > > >le in such a way the exceptions of jvm =B5-op > fault > > script programs, given = > > >that: there was a runtime above the CM3 runtime > and > > besides having its own = > > >spaces for stack, thread stacks and runtime > machinery, > > loader, etc. > > >Please just to advert this is and I guess was the > case > > for CM3 JVM, if so, = > > >let the system handle it in that way, why do you > need > > to fully manage to ha= > > >ndle that as in Modula-3 runtime per se, my > believe is > > that you could do th= > > >at for the language environment routines > separately, > > that's why pragmas are= > > > really useful. I don't believe it allowed to > signal > > like in Modula-3 terms= > > > threads but you could for instance lock a thread > until > > you have time in th= > > >e scheduler from Modula-3, still this is more > flexible > > nowadays I believe w= > > >ith multiprocessors, etc. > > >Similarly SPIN used or performed protection > mechanism, > > like domain level, a= > > >nd at language level the heap freezes when > handling > > untrusted code (even ex= > > >tra checked array bounds violation faults in > > non-Modula-3 code), so you hav= > > >e same runtime I believe for doing that too in > > Modula-3, again, is just ano= > > >ther time of problem arises, perhaps this can be > tried > > for thread stacks, e= > > >tc and in kernel threads well as something alike, > I > > really think could be d= > > >one. > > >Thanks in advance > > > > > >--- El lun, 21/2/11, Mika Nystrom > > escribi=F3: > > > > > >> De: Mika Nystrom > > >> Asunto: Re: [M3devel] SEGV mapping to > > RuntimeError > > >> Para: "Jay K" , > > m3devel at elegosoft.com > > >> Fecha: lunes, 21 de febrero, 2011 19:01 > > >>=20 > > >> There is in fact a small problem with my > change to > > the way > > >> SIGSEGV is > > >> handled. It now appears that bad > memory > > references > > >> are all happening > > >> in ThreadPosix.m3 (i.e., we've lost some of > the > > debugging > > >> help of the > > >> error message). > > >>=20 > > >> I'm not quite sure how to fix this although > I'm > > sure all > > >> the bits are > > >> around. My guess is that instead of > > performing RAISE > > >> RuntimeError.E > > >> in the thread switcher, the code ought to > call > > >> RTHooks.Raise with an > > >> argument derived from the faulting address > as > > reported to > > >> the signal > > >> handler. This is probably quite easy, > but > > does anyone > > >> know the=20 > > >> right routines to call? One would need > to > > track down > > >> the routine that > > >> faulted as well as the RT0 record > corresponding to > > the > > >> exception > > >> RuntimeError.E. > > >>=20 > > >> Mika > > >>=20 > > >> Jay K writes: > > >> >Letting any code run after a SIGSEGV is > > dubious. > > >> >Imagine the heap is corrupted. > > >> >And then you run more code. > > >> >And the code happens to call malloc. > > >> >Or printf to log something. > > >> >=3D20 > > >> >I suppose there might be an application > that > > maps > > >> memory > > >> >gradually=3D2C as pieces of a buffer are > hit. > > Might. > > >> =0A=0A=0A > > > > > > From mika at async.caltech.edu Tue Feb 22 18:44:05 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Tue, 22 Feb 2011 09:44:05 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <4D61CC27.7070704@lcwb.coop> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> <20110219232729.53A991A2078@async.async.caltech.edu> <4D615643.7060107@lcwb.coop> <20110220183746.195C11A2078@async.async.caltech.edu> <4D61CC27.7070704@lcwb.coop> Message-ID: <20110222174405.E15481A2078@async.async.caltech.edu> Ok so I was thinking about this. Why on earth is stack overflow even a problem? Consider the following procedure call (in my code, stack grows upwards): (* sp at x, pc at y *) y: P(args) z: next_statement decompose as follows: (* sp at x, pc at y *) y: Push(args etc. and ret. address z) Jump(P) z: next_statement Now, we say: y: ok := check_stack(size of frame) IF NOT ok THEN abort() END; Push(args etc. and ret. address z) Jump(P) z: next_statement (note check_stack and the following IF can be implemented by hardware, need not actually be an instruction) Let me change the code a tad: y: ok := check_stack(size of frame) y':IF NOT ok THEN WITH new_stack_bottom = malloc(stack_size) huge_amount = new_stack_bottom - sp DO create_redzone_at(new_stack_bottom+stack_size-redzone_size) EVAL alloca(huge_amount) END END; Push(args etc. and ret. address z) Jump(P) z: IF NOT ok THEN destroy_redzone(...); free(new_stack_bottom) END Note 1. cleanup of redzone could be postponed to return of caller....when alloca in any case has to be cleaned up. Note 2. the test IF NOT ok at z is more expensive to implement than the one at y because you can't really use hardware for it. A hardware callback can be arranged though: VAR ptr := sp; y: ok := check_stack(size of frame) y':IF NOT ok THEN ptr := 0; (* illegal address *) fault_address := z; WITH new_stack_bottom = malloc(stack_size) huge_amount = new_stack_bottom - sp DO create_redzone_at(new_stack_bottom+stack_size-redzone_size) EVAL alloca(huge_amount) END END; Push(args etc. and ret. address z) Jump(P) z: EVAL ptr^ (* [ NOT ok -> hardware callback to SEGV: ] *) SEGV(signalpc): IF NOT ok AND signalpc = fault_address THEN destroy_redzone(...); free(new_stack_bottom) END Mika "Rodney M. Bates" writes: > > >On 02/20/2011 12:37 PM, Mika Nystrom wrote: >> On a 64-bit machine, at least, there ought to be enough virtual >> memory that you could just have a gap between thread stacks big >> enough to allow for a protection area larger than the largest possible >> (implementation-defined) activation record, no? I know I've run into >> trouble with very large activation records in the past (and not because >> I was running out of stack space, either). >> >> Or at least a procedure with a very large activation record (or >> a procedure calling it) could be required to call some sort of check >> routine "EnoughStackSpaceRemaining()" before starting to scribble >> on the activation record? > >Hmm, I like this idea. It would introduce normal-case runtime overhead >only for such procedures, and these are likely rare. Also, assuming the procedure >actually uses very much of its large AR, it should also have enough computation >time to wash out the stack check overhead. > >> >> Also the end of the activation record must be written to at least once, >> or else the memory protection won't be triggered. >> > >I was thinking (as an alternative mechanism) of having the compiler intentionally >add enough artificial write(s) as necessary to ensure storing within the >red zone, and not just beyond it. This seems trickier to get right and >harder to distinguish after the fact from a NIL dereference. > >> In any case if this is done properly the same mechanism I proposed for >> SIGSEGV ought to be able to catch stack overflow, no? Well, as long as >> signals are delivered on a separate stack. If signals are delivered on >> the same stack, the signal handler would get nastier, it would have to >> make space through some manipulations (maybe temporarily unporotecting >> the redzone page?) for its own purposes... but I don't see why it >> couldn't be done. >> >> Not sure why I'm getting SIGILL... maybe I am getting my signal handler >> activated inside the redzone page because of a difference in signal >> handling..? I remember reading something about sigaltstack... >> >> I would of course love to be able to recover from stack overflow, too. >> In some sense, since it's a generally unknown limit, it's even less of >> a fatal error than a NIL dereference (hence makes even more sense to >> catch it). > >I think this would be a nice mechanism to have available. It would have to >be used with some care. In any case, it would be really nice and more >frequently so, to at least have runtime error messages that distinguished >stack overflow from NIL deref. > >> >> Mika >> >> "Rodney M. Bates" writes: >>> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. >>> Probably a fully protected guard page at the end of the stack. This technique >>> always worries me a bit because a procedure with a really big activation record >>> could jump right past it. Probably it would almost always access the first page >>> of the big area before storing anything into later pages. >>> >>> On 02/19/2011 05:27 PM, Mika Nystrom wrote: >>>> Ah, yes, stack protection. >>>> >>>> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. >>>> >>>> Hmm, I get SIGILL on AMD64_FREEBSD as well: >>>> >>>> time ../AMD64_FREEBSD/stubexample >>>> M-Scheme Experimental >>>> LITHP ITH LITHENING. >>>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) >>>> f >>>>> (f 0) >>>> Illegal instruction >>>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w >>>> >>>> What absolutely must not happen, of course, is that the runtime hangs >>>> while executing only safe code... >>>> >>>> Mika >>>> >>>> "Rodney M. Bates" writes: >>>>> I know of one other place the compilers rely on hardware memory protection >>>>> to detect a checked runtime error, and that is stack overflow. This won't >>>>> corrupt anything, but is hard to distinguish from dereferencing NIL. >>>>> This could probably be distinguished after the fact by some low-level, >>>>> target-dependent code. I have found it by looking at assembly code at >>>>> the point of failure--usually right after a stack pointer push. >>>>> >>>>> Detecting this via compiler-generated checks would probably be more >>>>> extravagant than many other checks, as it is so frequent. I am not >>>>> aware of any really good solution to this in any implementation of any >>>>> language. >>>>> >>>>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: >>>>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >>>>>> you know! >>>>>> >>>>>> "Corrupting the heap" is something that can only happen as a result of >>>>>> an unchecked runtime error. Unchecked runtime errors cannot happen in >>>>>> modules not marked UNSAFE. >>>>>> >>>>>> SEGV is, however, used by the CM3 implementation (and its predecessors) >>>>>> to signal a certain kind of *checked* runtime error, namely, the >>>>>> dereferencing of a NIL reference. Correct me if I am wrong, but an >>>>>> attempt to dereference NIL is not going to leave the heap corrupted? >>>>>> >>>>>> And if you stick to safe code, the only SEGVs I think you get in the >>>>>> current CM3 are ones from NIL dereferences. >>>>>> >>>>>> Hence, as long as you stick with safe code, the only time the code I >>>>>> checked in earlier gets triggered is for NIL dereferences, which should >>>>>> never corrupt the heap. So SEGV is not sometimes, but in fact always >>>>>> recoverable. >>>>>> >>>>>> :-) >>>>>> >>>>>> Mika >>>>>> >>>>>> P.S. the bit above "if you stick to safe code": if you actually program in >>>>>> Modula-3 you almost never use UNSAFE. I went through my repository and >>>>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >>>>>> many of the UNSAFE modules are glue code to Fortran routines, which >>>>>> could relatively easily be verified to be safe in the Modula-3 sense. >>>>>> Almost all what remains is glue to some C library, which wouldn't be >>>>>> necessary if the rest of the world would wake up out of the dark ages, but >>>>>> I don't have the time to rewrite every single library from scratch myself. >>>>>> >>>>>> >>>>>> Jay K writes: >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>>>> Content-Type: text/plain; charset="iso-8859-1" >>>>>>> Content-Transfer-Encoding: quoted-printable >>>>>>> >>>>>>> >>>>>>> Letting any code run after a SIGSEGV is dubious. >>>>>>> Imagine the heap is corrupted. >>>>>>> And then you run more code. >>>>>>> And the code happens to call malloc. >>>>>>> Or printf to log something. >>>>>>> =20 >>>>>>> I suppose there might be an application that maps memory >>>>>>> gradually=2C as pieces of a buffer are hit. Might. >>>>>>> =20 >>>>>>> - Jay >>>>>>> =20 >>>>>>>> To: m3devel at elegosoft.com >>>>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>>>>>> From: mika at async.caltech.edu >>>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>>>>>> =20 >>>>>>>> =20 >>>>>>>> Dear m3devel=2C >>>>>>>> =20 >>>>>>>> For a while it has annoyed me that segmentation violations cause an >>>>>>>> unconditional program abort. I've changed that now so that (under user >>>>>>>> threads at least) we instead get a RuntimeError. Here's an example of >>>>>>>> the mechanism at work in an interactive Scheme environment. Consider >>>>>>>> the unhelpful interface and module Crash: >>>>>>>> =20 >>>>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>>>>>> =20 >>>>>>>> MODULE Crash=3B >>>>>>>> =20 >>>>>>>> PROCEDURE Me() =3D >>>>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>>>>>> ptr^ :=3D 0 >>>>>>>> END Me=3B >>>>>>>> =20 >>>>>>>> BEGIN END Crash. >>>>>>>> =20 >>>>>>>> Here's an example of what happens if you now call this from an interactiv= >>>>>>> e >>>>>>>> interpreter that catches the exception RuntimeError.E: >>>>>>>> =20 >>>>>>>> M-Scheme Experimental >>>>>>>> LITHP ITH LITHENING. >>>>>>>>> (require-modules "m3") >>>>>>>> #t >>>>>>>>> (Crash.Me) >>>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>>>>>> (+ 3 4)=20 >>>>>>>> 7 >>>>>>>>> =20 >>>>>>>> =20 >>>>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>>>>>> heck it.=20 >>>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>>>>>> =20 >>>>>>>> Mika >>>>>>>> =20 >>>>>>> = >>>>>>> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>>>> Content-Type: text/html; charset="iso-8859-1" >>>>>>> Content-Transfer-Encoding: quoted-printable >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Letting any code run after a SIGSEGV is dubious.
>>>>>>> Imagine the heap =3Bis corrupted.
>>>>>>> And then you run more code.
>>>>>>> And the code happens to call malloc.
>>>>>>> Or printf to log something.
>>>>>>>  =3B
>>>>>>> I suppose there might be an application that maps memory
>>>>>>> gradually=2C as pieces of a buffer are hit. Might.
>>>>>>>  =3B
>>>>>>>  =3B- Jay
 =3B
>>>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>>>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>>>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>>>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>>>>>> olations cause an
>=3B unconditional program abort. I've changed that = >>>>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>>>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>>>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>>>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>>>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>>>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>>>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>>>>>> >=3B Here's an example of what happens if you now call this from an inter= >>>>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>>>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>>>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>>>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>>>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>>>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>>>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>>>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>>>>>> >>>>>>> = >>>>>>> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >>>>>> >>>> >> From rodney_bates at lcwb.coop Tue Feb 22 18:59:08 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Tue, 22 Feb 2011 11:59:08 -0600 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222021200.9F0EE1A2078@async.async.caltech.edu> References: <533385.35225.qm@web29704.mail.ird.yahoo.com> <20110222021200.9F0EE1A2078@async.async.caltech.edu> Message-ID: <4D63F96C.7020500@lcwb.coop> I don't know how relevant this is, but FWIW, I played with some patches in an interactive program I have that catches its own assertion failures and recovers from some of them. I was already using RTProcess.RegisterExitor with my own procedure. I made the exitor just call RTException.Raise. Then I deliberately did a NIL deref inside a try block catching RTException.E and ignoring it. It did prevent termination of my program. But there were two other problems: 1) The SIGSEGV messages came out on the command line before the exitor was invoked. 2) After the exception was caught, the program was catatonic, without using CPU. This is with pthreads. I haven't figured out why the wait. m3gdb seems to be useless in looking at other threads than the one it happens to get into, when using pthreads. On 02/21/2011 08:12 PM, Mika Nystrom wrote: > There are several issues here and yes this the RuntimeError.E exception > is taken either from SPIN or the CM3 JVM, I don't know which. > > I only understand the user threads code so I have made that raise the > RuntimeError.E instead of dumping core on a segfault. I think this is what > the CM JVM was supposed to do as well---but honestly I can't tell if that > work was ever completed. They do support mapping various other runtime > errors (subrange mismatch, e.g.) into exceptions, and that works well. > However the segmentation fault mapping was not functional: there was no > other way of handling it than dumping core on the whole M3 environment > and runtime before my changes last week. > > My last email was about a bug in my changes---not in the actual functionality > but the fact that if you *don't* catch the RuntimeError.E exception that > results from a bad memory reference, the runtime quits with a slightly > confusing error message, apparently implicating the threading system when > the bug is actually in normal, non-UNSAFE code. > > Mika > From jay.krell at cornell.edu Tue Feb 22 19:06:42 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 22 Feb 2011 18:06:42 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222174405.E15481A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu>, <4D615643.7060107@lcwb.coop>, <20110220183746.195C11A2078@async.async.caltech.edu>, <4D61CC27.7070704@lcwb.coop>, <20110222174405.E15481A2078@async.async.caltech.edu> Message-ID: (aside, and possible agreement: right -- an interpreter should consider NOT recursing on the machine stack whenever code it is interpreting recurses, but definitely some do) - Jay > To: rodney_bates at lcwb.coop > Date: Tue, 22 Feb 2011 09:44:05 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > Ok so I was thinking about this. > > Why on earth is stack overflow even a problem? > > Consider the following procedure call (in my code, stack grows upwards): > > (* sp at x, pc at y *) > y: P(args) > z: next_statement > > decompose as follows: > > (* sp at x, pc at y *) > y: Push(args etc. and ret. address z) > Jump(P) > z: next_statement > > Now, we say: > > y: ok := check_stack(size of frame) > IF NOT ok THEN abort() END; > Push(args etc. and ret. address z) > Jump(P) > z: next_statement > > (note check_stack and the following IF can be implemented by hardware, > need not actually be an instruction) > > Let me change the code a tad: > > y: ok := check_stack(size of frame) > y':IF NOT ok THEN > WITH new_stack_bottom = malloc(stack_size) > huge_amount = new_stack_bottom - sp DO > create_redzone_at(new_stack_bottom+stack_size-redzone_size) > EVAL alloca(huge_amount) > END > END; > Push(args etc. and ret. address z) > Jump(P) > z: IF NOT ok THEN destroy_redzone(...); free(new_stack_bottom) END > > Note 1. cleanup of redzone could be postponed to return of caller....when > alloca in any case has to be cleaned up. > > Note 2. the test IF NOT ok at z is more expensive to implement than the > one at y because you can't really use hardware for it. A hardware callback > can be arranged though: > > VAR ptr := sp; > y: ok := check_stack(size of frame) > y':IF NOT ok THEN > ptr := 0; (* illegal address *) > fault_address := z; > WITH new_stack_bottom = malloc(stack_size) > huge_amount = new_stack_bottom - sp DO > create_redzone_at(new_stack_bottom+stack_size-redzone_size) > EVAL alloca(huge_amount) > END > END; > Push(args etc. and ret. address z) > Jump(P) > z: EVAL ptr^ (* [ NOT ok -> hardware callback to SEGV: ] *) > > SEGV(signalpc): IF NOT ok AND signalpc = fault_address THEN destroy_redzone(...); free(new_stack_bottom) END > > Mika > > > > > > "Rodney M. Bates" writes: > > > > > >On 02/20/2011 12:37 PM, Mika Nystrom wrote: > >> On a 64-bit machine, at least, there ought to be enough virtual > >> memory that you could just have a gap between thread stacks big > >> enough to allow for a protection area larger than the largest possible > >> (implementation-defined) activation record, no? I know I've run into > >> trouble with very large activation records in the past (and not because > >> I was running out of stack space, either). > >> > >> Or at least a procedure with a very large activation record (or > >> a procedure calling it) could be required to call some sort of check > >> routine "EnoughStackSpaceRemaining()" before starting to scribble > >> on the activation record? > > > >Hmm, I like this idea. It would introduce normal-case runtime overhead > >only for such procedures, and these are likely rare. Also, assuming the procedure > >actually uses very much of its large AR, it should also have enough computation > >time to wash out the stack check overhead. > > > >> > >> Also the end of the activation record must be written to at least once, > >> or else the memory protection won't be triggered. > >> > > > >I was thinking (as an alternative mechanism) of having the compiler intentionally > >add enough artificial write(s) as necessary to ensure storing within the > >red zone, and not just beyond it. This seems trickier to get right and > >harder to distinguish after the fact from a NIL dereference. > > > >> In any case if this is done properly the same mechanism I proposed for > >> SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > >> signals are delivered on a separate stack. If signals are delivered on > >> the same stack, the signal handler would get nastier, it would have to > >> make space through some manipulations (maybe temporarily unporotecting > >> the redzone page?) for its own purposes... but I don't see why it > >> couldn't be done. > >> > >> Not sure why I'm getting SIGILL... maybe I am getting my signal handler > >> activated inside the redzone page because of a difference in signal > >> handling..? I remember reading something about sigaltstack... > >> > >> I would of course love to be able to recover from stack overflow, too. > >> In some sense, since it's a generally unknown limit, it's even less of > >> a fatal error than a NIL dereference (hence makes even more sense to > >> catch it). > > > >I think this would be a nice mechanism to have available. It would have to > >be used with some care. In any case, it would be really nice and more > >frequently so, to at least have runtime error messages that distinguished > >stack overflow from NIL deref. > > > >> > >> Mika > >> > >> "Rodney M. Bates" writes: > >>> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >>> Probably a fully protected guard page at the end of the stack. This technique > >>> always worries me a bit because a procedure with a really big activation record > >>> could jump right past it. Probably it would almost always access the first page > >>> of the big area before storing anything into later pages. > >>> > >>> On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >>>> Ah, yes, stack protection. > >>>> > >>>> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >>>> > >>>> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >>>> > >>>> time ../AMD64_FREEBSD/stubexample > >>>> M-Scheme Experimental > >>>> LITHP ITH LITHENING. > >>>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >>>> f > >>>>> (f 0) > >>>> Illegal instruction > >>>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >>>> > >>>> What absolutely must not happen, of course, is that the runtime hangs > >>>> while executing only safe code... > >>>> > >>>> Mika > >>>> > >>>> "Rodney M. Bates" writes: > >>>>> I know of one other place the compilers rely on hardware memory protection > >>>>> to detect a checked runtime error, and that is stack overflow. This won't > >>>>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>>>> This could probably be distinguished after the fact by some low-level, > >>>>> target-dependent code. I have found it by looking at assembly code at > >>>>> the point of failure--usually right after a stack pointer push. > >>>>> > >>>>> Detecting this via compiler-generated checks would probably be more > >>>>> extravagant than many other checks, as it is so frequent. I am not > >>>>> aware of any really good solution to this in any implementation of any > >>>>> language. > >>>>> > >>>>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>>>> you know! > >>>>>> > >>>>>> "Corrupting the heap" is something that can only happen as a result of > >>>>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>>>> modules not marked UNSAFE. > >>>>>> > >>>>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>>>> to signal a certain kind of *checked* runtime error, namely, the > >>>>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>>>> > >>>>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>>>> current CM3 are ones from NIL dereferences. > >>>>>> > >>>>>> Hence, as long as you stick with safe code, the only time the code I > >>>>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>>>> recoverable. > >>>>>> > >>>>>> :-) > >>>>>> > >>>>>> Mika > >>>>>> > >>>>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>>>> Almost all what remains is glue to some C library, which wouldn't be > >>>>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>>>> I don't have the time to rewrite every single library from scratch myself. > >>>>>> > >>>>>> > >>>>>> Jay K writes: > >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>>>> Content-Transfer-Encoding: quoted-printable > >>>>>>> > >>>>>>> > >>>>>>> Letting any code run after a SIGSEGV is dubious. > >>>>>>> Imagine the heap is corrupted. > >>>>>>> And then you run more code. > >>>>>>> And the code happens to call malloc. > >>>>>>> Or printf to log something. > >>>>>>> =20 > >>>>>>> I suppose there might be an application that maps memory > >>>>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>>>> =20 > >>>>>>> - Jay > >>>>>>> =20 > >>>>>>>> To: m3devel at elegosoft.com > >>>>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>>>> From: mika at async.caltech.edu > >>>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>>>> =20 > >>>>>>>> =20 > >>>>>>>> Dear m3devel=2C > >>>>>>>> =20 > >>>>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>>>> the unhelpful interface and module Crash: > >>>>>>>> =20 > >>>>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>>>> =20 > >>>>>>>> MODULE Crash=3B > >>>>>>>> =20 > >>>>>>>> PROCEDURE Me() =3D > >>>>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>>>> ptr^ :=3D 0 > >>>>>>>> END Me=3B > >>>>>>>> =20 > >>>>>>>> BEGIN END Crash. > >>>>>>>> =20 > >>>>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>>>> e > >>>>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>>>> =20 > >>>>>>>> M-Scheme Experimental > >>>>>>>> LITHP ITH LITHENING. > >>>>>>>>> (require-modules "m3") > >>>>>>>> #t > >>>>>>>>> (Crash.Me) > >>>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>>>> (+ 3 4)=20 > >>>>>>>> 7 > >>>>>>>>> =20 > >>>>>>>> =20 > >>>>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>>>> heck it.=20 > >>>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>>>> =20 > >>>>>>>> Mika > >>>>>>>> =20 > >>>>>>> = > >>>>>>> > >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>>>> Content-Transfer-Encoding: quoted-printable > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>>>> Imagine the heap =3Bis corrupted.
> >>>>>>> And then you run more code.
> >>>>>>> And the code happens to call malloc.
> >>>>>>> Or printf to log something.
> >>>>>>>  =3B
> >>>>>>> I suppose there might be an application that maps memory
> >>>>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>>>  =3B
> >>>>>>>  =3B- Jay
 =3B
> >>>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>>>> > >>>>>>> = > >>>>>>> > >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>>>> > >>>> > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Tue Feb 22 19:45:02 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Tue, 22 Feb 2011 10:45:02 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu>, <4D615643.7060107@lcwb.coop>, <20110220183746.195C11A2078@async.async.caltech.edu>, <4D61CC27.7070704@lcwb.coop>, <20110222174405.E15481A2078@async.async.caltech.edu> Message-ID: <20110222184502.88B921A2078@async.async.caltech.edu> Jay K writes: >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >(aside=2C and possible agreement: right -- an interpreter should consider N= >OT recursing on the machine >stack whenever code it is interpreting recurses=2C but definitely some do) >=20 > - Jay Jay, yes I agree---but: 1. of course it's easier to code the interpreter to recurse on the machine stack 2. code called out from the interpreter will in any case recurse on the (a) machine stack 3. again if one can "fix" the stack mechanism so that running out of stack space is non-fatal, there's no reason not to use the machine stack I've found that Modula-3's garbage collector is brutally slow and that converting code to using stack-allocated memory whenever possible is a huge performance win under both PM3 and CM3. Perhaps this is a problem with the garbage collector. I also have a partial design for a Scheme compiler (emitting Modula-3) that would use the machine stack whenever possible rather than heap allocation and do a "tricky thing" with local procedures to copy the stack whenever a reference to the stack escapes. (Most of the time you can implement Scheme on a stack but sometimes you need to keep part of it live after you return; hence implementations (including mine) heap-allocate local variables...) I realize my code is perhaps difficult to read. The gist of it is this: when you run out of stack space, allocate a new stack anywhere else in memory. Make it appear that you have alloca'd all the space between the old and new stacks (into a variable that you of course never touch), then continue. An issue is what to do when you return, you have to make sure that you restore your old redzone.. perhaps you don't need to free the stack eagerly: you could instead remember you have that page allocated and re-use it next time you hit the redzone, or put it in a pool to allow it to be used by another thread. The arbitrary choice of stack sizes has always bothered me greatly. Some of my programs have thousands of threads and they use stack space irregularly... Mika >> To: rodney_bates at lcwb.coop >> Date: Tue=2C 22 Feb 2011 09:44:05 -0800 >> From: mika at async.caltech.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] SEGV mapping to RuntimeError >>=20 >>=20 >> Ok so I was thinking about this. >>=20 >> Why on earth is stack overflow even a problem? >>=20 >> Consider the following procedure call (in my code=2C stack grows upwards)= >: >>=20 >> (* sp at x=2C pc at y *) >> y: P(args) >> z: next_statement >>=20 >> decompose as follows: >>=20 >> (* sp at x=2C pc at y *) >> y: Push(args etc. and ret. address z) >> Jump(P) >> z: next_statement >>=20 >> Now=2C we say: >>=20 >> y: ok :=3D check_stack(size of frame) >> IF NOT ok THEN abort() END=3B >> Push(args etc. and ret. address z) >> Jump(P) >> z: next_statement >>=20 >> (note check_stack and the following IF can be implemented by hardware=2C >> need not actually be an instruction) >>=20 >> Let me change the code a tad: >>=20 >> y: ok :=3D check_stack(size of frame) >> y':IF NOT ok THEN=20 >> WITH new_stack_bottom =3D malloc(stack_size) >> huge_amount =3D new_stack_bottom - sp DO >> create_redzone_at(new_stack_bottom+stack_size-redzone_size) >> EVAL alloca(huge_amount)=20 >> END >> END=3B >> Push(args etc. and ret. address z) >> Jump(P) >> z: IF NOT ok THEN destroy_redzone(...)=3B free(new_stack_bottom) END >>=20 >> Note 1. cleanup of redzone could be postponed to return of caller....when >> alloca in any case has to be cleaned up. >>=20 >> Note 2. the test IF NOT ok at z is more expensive to implement than the >> one at y because you can't really use hardware for it. A hardware callbac= >k >> can be arranged though: >>=20 >> VAR ptr :=3D sp=3B >> y: ok :=3D check_stack(size of frame) >> y':IF NOT ok THEN=20 >> ptr :=3D 0=3B (* illegal address *) >> fault_address :=3D z=3B >> WITH new_stack_bottom =3D malloc(stack_size) >> huge_amount =3D new_stack_bottom - sp DO >> create_redzone_at(new_stack_bottom+stack_size-redzone_size) >> EVAL alloca(huge_amount)=20 >> END >> END=3B >> Push(args etc. and ret. address z) >> Jump(P) >> z: EVAL ptr^ (* [ NOT ok -> hardware callback to SEGV: ] *) >>=20 >> SEGV(signalpc): IF NOT ok AND signalpc =3D fault_address THEN destroy_red= >zone(...)=3B free(new_stack_bottom) END >>=20 >> Mika >>=20 >>=20 >>=20 >>=20 >>=20 >> "Rodney M. Bates" writes: >> > >> > >> >On 02/20/2011 12:37 PM=2C Mika Nystrom wrote: >> >> On a 64-bit machine=2C at least=2C there ought to be enough virtual >> >> memory that you could just have a gap between thread stacks big >> >> enough to allow for a protection area larger than the largest possible >> >> (implementation-defined) activation record=2C no? I know I've run into >> >> trouble with very large activation records in the past (and not becaus= >e >> >> I was running out of stack space=2C either). >> >> >> >> Or at least a procedure with a very large activation record (or >> >> a procedure calling it) could be required to call some sort of check >> >> routine "EnoughStackSpaceRemaining()" before starting to scribble >> >> on the activation record? >> > >> >Hmm=2C I like this idea. It would introduce normal-case runtime overhead >> >only for such procedures=2C and these are likely rare. Also=2C assuming = >the procedure >> >actually uses very much of its large AR=2C it should also have enough co= >mputation >> >time to wash out the stack check overhead. >> > >> >> >> >> Also the end of the activation record must be written to at least once= >=2C >> >> or else the memory protection won't be triggered. >> >> >> > >> >I was thinking (as an alternative mechanism) of having the compiler inte= >ntionally >> >add enough artificial write(s) as necessary to ensure storing within the >> >red zone=2C and not just beyond it. This seems trickier to get right and >> >harder to distinguish after the fact from a NIL dereference. >> > >> >> In any case if this is done properly the same mechanism I proposed for >> >> SIGSEGV ought to be able to catch stack overflow=2C no? Well=2C as lon= >g as >> >> signals are delivered on a separate stack. If signals are delivered on >> >> the same stack=2C the signal handler would get nastier=2C it would hav= >e to >> >> make space through some manipulations (maybe temporarily unporotecting >> >> the redzone page?) for its own purposes... but I don't see why it >> >> couldn't be done. >> >> >> >> Not sure why I'm getting SIGILL... maybe I am getting my signal handle= >r >> >> activated inside the redzone page because of a difference in signal >> >> handling..? I remember reading something about sigaltstack... >> >> >> >> I would of course love to be able to recover from stack overflow=2C to= >o. >> >> In some sense=2C since it's a generally unknown limit=2C it's even les= >s of >> >> a fatal error than a NIL dereference (hence makes even more sense to >> >> catch it). >> > >> >I think this would be a nice mechanism to have available. It would have = >to >> >be used with some care. In any case=2C it would be really nice and more >> >frequently so=2C to at least have runtime error messages that distinguis= >hed >> >stack overflow from NIL deref. >> > >> >> >> >> Mika >> >> >> >> "Rodney M. Bates" writes: >> >>> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AM= >D64_LINUX. >> >>> Probably a fully protected guard page at the end of the stack. This t= >echnique >> >>> always worries me a bit because a procedure with a really big activat= >ion record >> >>> could jump right past it. Probably it would almost always access the = >first page >> >>> of the big area before storing anything into later pages. >> >>> >> >>> On 02/19/2011 05:27 PM=2C Mika Nystrom wrote: >> >>>> Ah=2C yes=2C stack protection. >> >>>> >> >>>> Do you know if it's a SIGSEGV=2C not a SIGBUS? I know I have seen SI= >GILL on Macs. >> >>>> >> >>>> Hmm=2C I get SIGILL on AMD64_FREEBSD as well: >> >>>> >> >>>> time ../AMD64_FREEBSD/stubexample >> >>>> M-Scheme Experimental >> >>>> LITHP ITH LITHENING. >> >>>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) >> >>>> f >> >>>>> (f 0) >> >>>> Illegal instruction >> >>>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w >> >>>> >> >>>> What absolutely must not happen=2C of course=2C is that the runtime = >hangs >> >>>> while executing only safe code... >> >>>> >> >>>> Mika >> >>>> >> >>>> "Rodney M. Bates" writes: >> >>>>> I know of one other place the compilers rely on hardware memory pro= >tection >> >>>>> to detect a checked runtime error=2C and that is stack overflow. Th= >is won't >> >>>>> corrupt anything=2C but is hard to distinguish from dereferencing N= >IL. >> >>>>> This could probably be distinguished after the fact by some low-lev= >el=2C >> >>>>> target-dependent code. I have found it by looking at assembly code = >at >> >>>>> the point of failure--usually right after a stack pointer push. >> >>>>> >> >>>>> Detecting this via compiler-generated checks would probably be more >> >>>>> extravagant than many other checks=2C as it is so frequent. I am no= >t >> >>>>> aware of any really good solution to this in any implementation of = >any >> >>>>> language. >> >>>>> >> >>>>> On 02/19/2011 02:38 PM=2C Mika Nystrom wrote: >> >>>>>> Jay=2C sometimes I wonder about you: this is a Modula-3 mailing li= >st=2C >> >>>>>> you know! >> >>>>>> >> >>>>>> "Corrupting the heap" is something that can only happen as a resul= >t of >> >>>>>> an unchecked runtime error. Unchecked runtime errors cannot happen= > in >> >>>>>> modules not marked UNSAFE. >> >>>>>> >> >>>>>> SEGV is=2C however=2C used by the CM3 implementation (and its pred= >ecessors) >> >>>>>> to signal a certain kind of *checked* runtime error=2C namely=2C t= >he >> >>>>>> dereferencing of a NIL reference. Correct me if I am wrong=2C but = >an >> >>>>>> attempt to dereference NIL is not going to leave the heap corrupte= >d? >> >>>>>> >> >>>>>> And if you stick to safe code=2C the only SEGVs I think you get in= > the >> >>>>>> current CM3 are ones from NIL dereferences. >> >>>>>> >> >>>>>> Hence=2C as long as you stick with safe code=2C the only time the = >code I >> >>>>>> checked in earlier gets triggered is for NIL dereferences=2C which= > should >> >>>>>> never corrupt the heap. So SEGV is not sometimes=2C but in fact al= >ways >> >>>>>> recoverable. >> >>>>>> >> >>>>>> :-) >> >>>>>> >> >>>>>> Mika >> >>>>>> >> >>>>>> P.S. the bit above "if you stick to safe code": if you actually pr= >ogram in >> >>>>>> Modula-3 you almost never use UNSAFE. I went through my repository= > and >> >>>>>> I have 40 modules using UNSAFE out of a total of 4=2C559. Furtherm= >ore=2C >> >>>>>> many of the UNSAFE modules are glue code to Fortran routines=2C wh= >ich >> >>>>>> could relatively easily be verified to be safe in the Modula-3 sen= >se. >> >>>>>> Almost all what remains is glue to some C library=2C which wouldn'= >t be >> >>>>>> necessary if the rest of the world would wake up out of the dark a= >ges=2C but >> >>>>>> I don't have the time to rewrite every single library from scratch= > myself. >> >>>>>> >> >>>>>> >> >>>>>> Jay K writes: >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >> >>>>>>> Content-Type: text/plain=3B charset=3D"iso-8859-1" >> >>>>>>> Content-Transfer-Encoding: quoted-printable >> >>>>>>> >> >>>>>>> >> >>>>>>> Letting any code run after a SIGSEGV is dubious. >> >>>>>>> Imagine the heap is corrupted. >> >>>>>>> And then you run more code. >> >>>>>>> And the code happens to call malloc. >> >>>>>>> Or printf to log something. >> >>>>>>> =3D20 >> >>>>>>> I suppose there might be an application that maps memory >> >>>>>>> gradually=3D2C as pieces of a buffer are hit. Might. >> >>>>>>> =3D20 >> >>>>>>> - Jay >> >>>>>>> =3D20 >> >>>>>>>> To: m3devel at elegosoft.com >> >>>>>>>> Date: Sat=3D2C 19 Feb 2011 10:29:30 -0800 >> >>>>>>>> From: mika at async.caltech.edu >> >>>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError >> >>>>>>>> =3D20 >> >>>>>>>> =3D20 >> >>>>>>>> Dear m3devel=3D2C >> >>>>>>>> =3D20 >> >>>>>>>> For a while it has annoyed me that segmentation violations cause= > an >> >>>>>>>> unconditional program abort. I've changed that now so that (unde= >r user >> >>>>>>>> threads at least) we instead get a RuntimeError. Here's an examp= >le of >> >>>>>>>> the mechanism at work in an interactive Scheme environment. Cons= >ider >> >>>>>>>> the unhelpful interface and module Crash: >> >>>>>>>> =3D20 >> >>>>>>>> INTERFACE Crash=3D3B PROCEDURE Me()=3D3B END Crash. >> >>>>>>>> =3D20 >> >>>>>>>> MODULE Crash=3D3B >> >>>>>>>> =3D20 >> >>>>>>>> PROCEDURE Me() =3D3D >> >>>>>>>> VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN >> >>>>>>>> ptr^ :=3D3D 0 >> >>>>>>>> END Me=3D3B >> >>>>>>>> =3D20 >> >>>>>>>> BEGIN END Crash. >> >>>>>>>> =3D20 >> >>>>>>>> Here's an example of what happens if you now call this from an i= >nteractiv=3D >> >>>>>>> e >> >>>>>>>> interpreter that catches the exception RuntimeError.E: >> >>>>>>>> =3D20 >> >>>>>>>> M-Scheme Experimental >> >>>>>>>> LITHP ITH LITHENING. >> >>>>>>>>> (require-modules "m3") >> >>>>>>>> #t >> >>>>>>>>> (Crash.Me) >> >>>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory = >location. >> >>>>>>>>> (+ 3 4)=3D20 >> >>>>>>>> 7 >> >>>>>>>>> =3D20 >> >>>>>>>> =3D20 >> >>>>>>>> I just realized I may have broken pthreads=3D2C let me go back a= >nd double-c=3D >> >>>>>>> heck it.=3D20 >> >>>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do = >they... >> >>>>>>>> =3D20 >> >>>>>>>> Mika >> >>>>>>>> =3D20 >> >>>>>>> =3D >> >>>>>>> >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >> >>>>>>> Content-Type: text/html=3B charset=3D"iso-8859-1" >> >>>>>>> Content-Transfer-Encoding: quoted-printable >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> Letting any code run after a SIGSEGV is dubious.
>> >>>>>>> Imagine the heap =3D3Bis corrupted.
>> >>>>>>> And then you run more code.
>> >>>>>>> And the code happens to call malloc.
>> >>>>>>> Or printf to log something.
>> >>>>>>>  =3D3B
>> >>>>>>> I suppose there might be an application that maps memory
>> >>>>>>> gradually=3D2C as pieces of a buffer are hit. Might.
>> >>>>>>>  =3D3B
>> >>>>>>>  =3D3B- Jay
 =3D3B
>> >>>>>>> >=3D3B To: m3devel at elegosoft.com
>=3D3B Date: Sat=3D2C 19 = >Feb 2011 10:29:3=3D >> >>>>>>> 0 -0800
>=3D3B From: mika at async.caltech.edu
>=3D3B Subj= >ect: [M3devel]=3D >> >>>>>>> SEGV mapping to RuntimeError
>=3D3B
>=3D3B
>=3D3B = >Dear m3devel=3D >> >>>>>>> =3D2C
>=3D3B
>=3D3B For a while it has annoyed me that = >segmentation vi=3D >> >>>>>>> olations cause an
>=3D3B unconditional program abort. I've c= >hanged that =3D >> >>>>>>> now so that (under user
>=3D3B threads at least) we instead = >get a Runtim=3D >> >>>>>>> eError. Here's an example of
>=3D3B the mechanism at work in= > an interact=3D >> >>>>>>> ive Scheme environment. Consider
>=3D3B the unhelpful interf= >ace and modu=3D >> >>>>>>> le Crash:
>=3D3B
>=3D3B INTERFACE Crash=3D3B PROCEDURE = >Me()=3D3B END Cra=3D >> >>>>>>> sh.
>=3D3B
>=3D3B MODULE Crash=3D3B
>=3D3B
>= >=3D3B PROCEDURE Me(=3D >> >>>>>>> ) =3D3D
>=3D3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGINR>>=3D3B ptr^ :=3D3D=3D >> >>>>>>> 0
>=3D3B END Me=3D3B
>=3D3B
>=3D3B BEGIN END Crash= >.
>=3D3B
=3D >> >>>>>>> >=3D3B Here's an example of what happens if you now call this f= >rom an inter=3D >> >>>>>>> active
>=3D3B interpreter that catches the exception Runtime= >Error.E:
=3D >> >>>>>>> >=3D3B
>=3D3B M-Scheme Experimental
>=3D3B LITHP ITH = >LITHENING.
&=3D >> >>>>>>> gt=3D3B>=3D3B (require-modules "m3")
>=3D3B #t
>=3D3B= >>=3D3B (Crash.Me=3D >> >>>>>>> )
>=3D3B EXCEPTION! RuntimeError! Attempt to reference an il= >legal memory=3D >> >>>>>>> location.
>=3D3B>=3D3B (+ 3 4)
>=3D3B 7
>=3D3B&g= >t=3D3B
>=3D >> >>>>>>> =3D3B
>=3D3B I just realized I may have broken pthreads=3D2C= > let me go back=3D >> >>>>>>> and double-check it.
>=3D3B runtime/POSIX and thread/POSIX d= >on't refer=3D >> >>>>>>> to the same thing do they...
>=3D3B
>=3D3B Mika
>= >=3D3B
=3D >> >>>>>>> >> >>>>>>> =3D >> >>>>>>> >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >> >>>>>> >> >>>> >> >> > = > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >(aside=2C and possible agreement: right -- an interpreter should consider N= >OT recursing on the machine
>stack whenever code it is interpreting recurses=2C but definitely some do)<= >BR> > =3B
> =3B- Jay
 =3B
>>=3B To: rodney_bates at lcwb.coop
>=3B Date: Tue=2C 22 Feb 2011 09:44:= >05 -0800
>=3B From: mika at async.caltech.edu
>=3B CC: m3devel at elego= >soft.com
>=3B Subject: Re: [M3devel] SEGV mapping to RuntimeError
&= >gt=3B
>=3B
>=3B Ok so I was thinking about this.
>=3B
= >>=3B Why on earth is stack overflow even a problem?
>=3B
>=3B = >Consider the following procedure call (in my code=2C stack grows upwards):<= >BR>>=3B
>=3B (* sp at x=2C pc at y *)
>=3B y: P(args)
>= >=3B z: next_statement
>=3B
>=3B decompose as follows:
>=3B = >
>=3B (* sp at x=2C pc at y *)
>=3B y: Push(args etc. and ret. ad= >dress z)
>=3B Jump(P)
>=3B z: next_statement
>=3B
>=3B= > Now=2C we say:
>=3B
>=3B y: ok :=3D check_stack(size of frame)<= >BR>>=3B IF NOT ok THEN abort() END=3B
>=3B Push(args etc. and ret. a= >ddress z)
>=3B Jump(P)
>=3B z: next_statement
>=3B
>= >=3B (note check_stack and the following IF can be implemented by hardware= >=2C
>=3B need not actually be an instruction)
>=3B
>=3B Let= > me change the code a tad:
>=3B
>=3B y: ok :=3D check_stack(size= > of frame)
>=3B y':IF NOT ok THEN
>=3B WITH new_stack_bottom =3D= > malloc(stack_size)
>=3B huge_amount =3D new_stack_bottom - sp DO
&= >gt=3B create_redzone_at(new_stack_bottom+stack_size-redzone_size)
>=3B= > EVAL alloca(huge_amount)
>=3B END
>=3B END=3B
>=3B Push(ar= >gs etc. and ret. address z)
>=3B Jump(P)
>=3B z: IF NOT ok THEN d= >estroy_redzone(...)=3B free(new_stack_bottom) END
>=3B
>=3B Note= > 1. cleanup of redzone could be postponed to return of caller....when
&g= >t=3B alloca in any case has to be cleaned up.
>=3B
>=3B Note 2. = >the test IF NOT ok at z is more expensive to implement than the
>=3B o= >ne at y because you can't really use hardware for it. A hardware callbackR>>=3B can be arranged though:
>=3B
>=3B VAR ptr :=3D sp=3B>>=3B y: ok :=3D check_stack(size of frame)
>=3B y':IF NOT ok THEN <= >BR>>=3B ptr :=3D 0=3B (* illegal address *)
>=3B fault_address :=3D = >z=3B
>=3B WITH new_stack_bottom =3D malloc(stack_size)
>=3B huge_= >amount =3D new_stack_bottom - sp DO
>=3B create_redzone_at(new_stack_b= >ottom+stack_size-redzone_size)
>=3B EVAL alloca(huge_amount)
>= >=3B END
>=3B END=3B
>=3B Push(args etc. and ret. address z)
&g= >t=3B Jump(P)
>=3B z: EVAL ptr^ (* [ NOT ok ->=3B hardware callback t= >o SEGV: ] *)
>=3B
>=3B SEGV(signalpc): IF NOT ok AND signalpc = >=3D fault_address THEN destroy_redzone(...)=3B free(new_stack_bottom) ENDR>>=3B
>=3B Mika
>=3B
>=3B
>=3B
>=3B
>= >=3B
>=3B "Rodney M. Bates" writes:
>=3B >=3B
>=3B >=3B<= >BR>>=3B >=3BOn 02/20/2011 12:37 PM=2C Mika Nystrom wrote:
>=3B >= >=3B>=3B On a 64-bit machine=2C at least=2C there ought to be enough virtu= >al
>=3B >=3B>=3B memory that you could just have a gap between thr= >ead stacks big
>=3B >=3B>=3B enough to allow for a protection area= > larger than the largest possible
>=3B >=3B>=3B (implementation-de= >fined) activation record=2C no? I know I've run into
>=3B >=3B>=3B= > trouble with very large activation records in the past (and not because>>=3B >=3B>=3B I was running out of stack space=2C either).
>=3B= > >=3B>=3B
>=3B >=3B>=3B Or at least a procedure with a very la= >rge activation record (or
>=3B >=3B>=3B a procedure calling it) co= >uld be required to call some sort of check
>=3B >=3B>=3B routine "= >EnoughStackSpaceRemaining()" before starting to scribble
>=3B >=3B&g= >t=3B on the activation record?
>=3B >=3B
>=3B >=3BHmm=2C I li= >ke this idea. It would introduce normal-case runtime overhead
>=3B >= >=3Bonly for such procedures=2C and these are likely rare. Also=2C assuming = >the procedure
>=3B >=3Bactually uses very much of its large AR=2C it= > should also have enough computation
>=3B >=3Btime to wash out the s= >tack check overhead.
>=3B >=3B
>=3B >=3B>=3B
>=3B >= >=3B>=3B Also the end of the activation record must be written to at least= > once=2C
>=3B >=3B>=3B or else the memory protection won't be trig= >gered.
>=3B >=3B>=3B
>=3B >=3B
>=3B >=3BI was thinki= >ng (as an alternative mechanism) of having the compiler intentionally
&g= >t=3B >=3Badd enough artificial write(s) as necessary to ensure storing wi= >thin the
>=3B >=3Bred zone=2C and not just beyond it. This seems tri= >ckier to get right and
>=3B >=3Bharder to distinguish after the fact= > from a NIL dereference.
>=3B >=3B
>=3B >=3B>=3B In any cas= >e if this is done properly the same mechanism I proposed for
>=3B >= >=3B>=3B SIGSEGV ought to be able to catch stack overflow=2C no? Well=2C a= >s long as
>=3B >=3B>=3B signals are delivered on a separate stack.= > If signals are delivered on
>=3B >=3B>=3B the same stack=2C the s= >ignal handler would get nastier=2C it would have to
>=3B >=3B>=3B = >make space through some manipulations (maybe temporarily unporotecting
&= >gt=3B >=3B>=3B the redzone page?) for its own purposes... but I don't s= >ee why it
>=3B >=3B>=3B couldn't be done.
>=3B >=3B>=3BR>>=3B >=3B>=3B Not sure why I'm getting SIGILL... maybe I am getting= > my signal handler
>=3B >=3B>=3B activated inside the redzone page= > because of a difference in signal
>=3B >=3B>=3B handling..? I rem= >ember reading something about sigaltstack...
>=3B >=3B>=3B
>= >=3B >=3B>=3B I would of course love to be able to recover from stack ov= >erflow=2C too.
>=3B >=3B>=3B In some sense=2C since it's a general= >ly unknown limit=2C it's even less of
>=3B >=3B>=3B a fatal error = >than a NIL dereference (hence makes even more sense to
>=3B >=3B>= >=3B catch it).
>=3B >=3B
>=3B >=3BI think this would be a nic= >e mechanism to have available. It would have to
>=3B >=3Bbe used wit= >h some care. In any case=2C it would be really nice and more
>=3B >= >=3Bfrequently so=2C to at least have runtime error messages that distinguis= >hed
>=3B >=3Bstack overflow from NIL deref.
>=3B >=3B
>= >=3B >=3B>=3B
>=3B >=3B>=3B Mika
>=3B >=3B>=3B
>= >=3B >=3B>=3B "Rodney M. Bates" writes:
>=3B >=3B>=3B>=3B I a= >m pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX= >.
>=3B >=3B>=3B>=3B Probably a fully protected guard page at the= > end of the stack. This technique
>=3B >=3B>=3B>=3B always worri= >es me a bit because a procedure with a really big activation record
>= >=3B >=3B>=3B>=3B could jump right past it. Probably it would almost a= >lways access the first page
>=3B >=3B>=3B>=3B of the big area be= >fore storing anything into later pages.
>=3B >=3B>=3B>=3B
>= >=3B >=3B>=3B>=3B On 02/19/2011 05:27 PM=2C Mika Nystrom wrote:
>= >=3B >=3B>=3B>=3B>=3B Ah=2C yes=2C stack protection.
>=3B >= >=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B Do you know if it'= >s a SIGSEGV=2C not a SIGBUS? I know I have seen SIGILL on Macs.
>=3B &= >gt=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B Hmm=2C I get SIG= >ILL on AMD64_FREEBSD as well:
>=3B >=3B>=3B>=3B>=3B
>=3B = >>=3B>=3B>=3B>=3B time ../AMD64_FREEBSD/stubexample
>=3B >=3B= >>=3B>=3B>=3B M-Scheme Experimental
>=3B >=3B>=3B>=3B>=3B= > LITHP ITH LITHENING.
>=3B >=3B>=3B>=3B>=3B>=3B (define (f a= >) (+ (f (+ a 1)) (f (+ a 2))))
>=3B >=3B>=3B>=3B>=3B f
>= >=3B >=3B>=3B>=3B>=3B>=3B (f 0)
>=3B >=3B>=3B>=3B>=3B= > Illegal instruction
>=3B >=3B>=3B>=3B>=3B 3.847u 0.368s 0:13.= >32 31.5% 2160+284478k 0+0io 0pf+0w
>=3B >=3B>=3B>=3B>=3B
&g= >t=3B >=3B>=3B>=3B>=3B What absolutely must not happen=2C of course= >=2C is that the runtime hangs
>=3B >=3B>=3B>=3B>=3B while exec= >uting only safe code...
>=3B >=3B>=3B>=3B>=3B
>=3B >=3B= >>=3B>=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B
>=3B >=3B= >>=3B>=3B>=3B "Rodney M. Bates" writes:
>=3B >=3B>=3B>=3B&g= >t=3B>=3B I know of one other place the compilers rely on hardware memory = >protection
>=3B >=3B>=3B>=3B>=3B>=3B to detect a checked run= >time error=2C and that is stack overflow. This won't
>=3B >=3B>=3B= >>=3B>=3B>=3B corrupt anything=2C but is hard to distinguish from dere= >ferencing NIL.
>=3B >=3B>=3B>=3B>=3B>=3B This could probably= > be distinguished after the fact by some low-level=2C
>=3B >=3B>= >=3B>=3B>=3B>=3B target-dependent code. I have found it by looking at = >assembly code at
>=3B >=3B>=3B>=3B>=3B>=3B the point of fail= >ure--usually right after a stack pointer push.
>=3B >=3B>=3B>=3B= >>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B Detecting this via co= >mpiler-generated checks would probably be more
>=3B >=3B>=3B>=3B= >>=3B>=3B extravagant than many other checks=2C as it is so frequent. I = >am not
>=3B >=3B>=3B>=3B>=3B>=3B aware of any really good so= >lution to this in any implementation of any
>=3B >=3B>=3B>=3B>= >=3B>=3B language.
>=3B >=3B>=3B>=3B>=3B>=3B
>=3B >= >=3B>=3B>=3B>=3B>=3B On 02/19/2011 02:38 PM=2C Mika Nystrom wrote:R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B Jay=2C sometimes I wonder abo= >ut you: this is a Modula-3 mailing list=2C
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B you know!
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>>=3B >=3B>=3B>=3B>=3B>=3B>=3B "Corrupting the heap" is somet= hing that can only happen as a result of
>=3B >=3B>=3B>=3B>=3B= >>=3B>=3B an unchecked runtime error. Unchecked runtime errors cannot ha= >ppen in
>=3B >=3B>=3B>=3B>=3B>=3B>=3B modules not marked U= >NSAFE.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B SEGV is=2C however=2C used by the CM3 implement= >ation (and its predecessors)
>=3B >=3B>=3B>=3B>=3B>=3B>=3B= > to signal a certain kind of *checked* runtime error=2C namely=2C the
&g= >t=3B >=3B>=3B>=3B>=3B>=3B>=3B dereferencing of a NIL reference.= > Correct me if I am wrong=2C but an
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B attempt to dereference NIL is not going to leave the heap corrupt= >ed?
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B&g= >t=3B>=3B>=3B>=3B And if you stick to safe code=2C the only SEGVs I th= >ink you get in the
>=3B >=3B>=3B>=3B>=3B>=3B>=3B current C= >M3 are ones from NIL dereferences.
>=3B >=3B>=3B>=3B>=3B>=3B= >>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B Hence=2C as long as y= >ou stick with safe code=2C the only time the code I
>=3B >=3B>=3B&= >gt=3B>=3B>=3B>=3B checked in earlier gets triggered is for NIL derefe= >rences=2C which should
>=3B >=3B>=3B>=3B>=3B>=3B>=3B never= > corrupt the heap. So SEGV is not sometimes=2C but in fact always
>=3B= > >=3B>=3B>=3B>=3B>=3B>=3B recoverable.
>=3B >=3B>=3B&g= >t=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B :-)R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B&= >gt=3B>=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
&g= >t=3B >=3B>=3B>=3B>=3B>=3B>=3B P.S. the bit above "if you stick = >to safe code": if you actually program in
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B Modula-3 you almost never use UNSAFE. I went through my rep= >ository and
>=3B >=3B>=3B>=3B>=3B>=3B>=3B I have 40 module= >s using UNSAFE out of a total of 4=2C559. Furthermore=2C
>=3B >=3B&g= >t=3B>=3B>=3B>=3B>=3B many of the UNSAFE modules are glue code to Fo= >rtran routines=2C which
>=3B >=3B>=3B>=3B>=3B>=3B>=3B coul= >d relatively easily be verified to be safe in the Modula-3 sense.
>=3B= > >=3B>=3B>=3B>=3B>=3B>=3B Almost all what remains is glue to so= >me C library=2C which wouldn't be
>=3B >=3B>=3B>=3B>=3B>=3B&= >gt=3B necessary if the rest of the world would wake up out of the dark ages= >=2C but
>=3B >=3B>=3B>=3B>=3B>=3B>=3B I don't have the tim= >e to rewrite every single library from scratch myself.
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B<= >BR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B Jay K writes:
>=3B >= >=3B>=3B>=3B>=3B>=3B>=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912= >854f_
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Content-Type: te= >xt/plain=3B charset=3D"iso-8859-1"
>=3B >=3B>=3B>=3B>=3B>=3B= >>=3B>=3B Content-Transfer-Encoding: quoted-printable
>=3B >=3B&g= >t=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B= >>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Letting a= >ny code run after a SIGSEGV is dubious.
>=3B >=3B>=3B>=3B>=3B&= >gt=3B>=3B>=3B Imagine the heap is corrupted.
>=3B >=3B>=3B>= >=3B>=3B>=3B>=3B>=3B And then you run more code.
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B And the code happens to call malloc.
&= >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Or printf to log something= >.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>=3B >= >=3B>=3B>=3B>=3B>=3B>=3B>=3B I suppose there might be an applica= >tion that maps memory
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B = >gradually=3D2C as pieces of a buffer are hit. Might.
>=3B >=3B>=3B= >>=3B>=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B - Jay
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B = >=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B To: m3deve= >l at elegosoft.com
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = >Date: Sat=3D2C 19 Feb 2011 10:29:30 -0800
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B>=3B>=3B From: mika at async.caltech.edu
>=3B >=3B&g= >t=3B>=3B>=3B>=3B>=3B>=3B>=3B Subject: [M3devel] SEGV mapping to= > RuntimeError
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = >=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
&= >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B Dear m3devel=3D2C>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>=3B &g= >t=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B For a while it has annoyed m= >e that segmentation violations cause an
>=3B >=3B>=3B>=3B>=3B&= >gt=3B>=3B>=3B>=3B unconditional program abort. I've changed that now = >so that (under user
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>= >=3B threads at least) we instead get a RuntimeError. Here's an example ofR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B the mechanism at = >work in an interactive Scheme environment. Consider
>=3B >=3B>=3B&= >gt=3B>=3B>=3B>=3B>=3B>=3B the unhelpful interface and module Cras= >h:
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>= >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B INTERFACE Crash=3D3B P= >ROCEDURE Me()=3D3B END Crash.
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B MODULE Crash=3D3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= >=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = >PROCEDURE Me() =3D3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&g= >t=3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN
>=3B >=3B>=3B>= >=3B>=3B>=3B>=3B>=3B>=3B ptr^ :=3D3D 0
>=3B >=3B>=3B>= >=3B>=3B>=3B>=3B>=3B>=3B END Me=3D3B
>=3B >=3B>=3B>=3B&= >gt=3B>=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B>=3B BEGIN END Crash.
>=3B >=3B>=3B>=3B>=3B&g= >t=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B= >>=3B>=3B Here's an example of what happens if you now call this from an= > interactiv=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B e
&g= >t=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B interpreter that catc= >hes the exception RuntimeError.E:
>=3B >=3B>=3B>=3B>=3B>=3B&= >gt=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= >=3B>=3B M-Scheme Experimental
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B>=3B LITHP ITH LITHENING.
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B>=3B>=3B (require-modules "m3")
>=3B >=3B>=3B&g= >t=3B>=3B>=3B>=3B>=3B>=3B #t
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B>=3B>=3B (Crash.Me)
>=3B >=3B>=3B>=3B>=3B&g= >t=3B>=3B>=3B>=3B EXCEPTION! RuntimeError! Attempt to reference an ill= >egal memory location.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B>=3B (+ 3 4)=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= >=3B>=3B 7
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B>= >=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20<= >BR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B I just realized = >I may have broken pthreads=3D2C let me go back and double-c=3D
>=3B &g= >t=3B>=3B>=3B>=3B>=3B>=3B>=3B heck it.=3D20
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B>=3B runtime/POSIX and thread/POSIX don'= >t refer to the same thing do they...
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B= > =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D
>=3B &= >gt=3B>=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_
>=3B &g= >t=3B>=3B>=3B>=3B>=3B>=3B>=3B Content-Type: text/html=3B charset= >=3D"iso-8859-1"
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Conten= >t-Transfer-Encoding: quoted-printable
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3B= >html>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3Bhead&= >gt=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3Bstyle>= >=3B<=3B!--
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B .hmmessag= >e P
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B {
>=3B >=3B= >>=3B>=3B>=3B>=3B>=3B>=3B margin:0px=3D3B
>=3B >=3B>=3B= >>=3B>=3B>=3B>=3B>=3B padding:0px
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B>=3B }
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= >=3B body.hmmessage
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B {R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B font-size: 10pt=3D3B>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B font-family:Tahoma
&g= >t=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B }
>=3B >=3B>=3B>= >=3B>=3B>=3B>=3B>=3B -->=3B<=3B/style>=3B
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B <=3B/head>=3B
>=3B >=3B>=3B&= >gt=3B>=3B>=3B>=3B>=3B <=3Bbody class=3D3D'hmmessage'>=3B
>= >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Letting any code run after a= > SIGSEGV is dubious.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B= >>=3B>=3B Imagine the heap&=3Bnbsp=3D3Bis corrupted.<=3BBR>=3B>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B And then you run more co= >de.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B And = >the code happens to call malloc.<=3BBR>=3B
>=3B >=3B>=3B>=3B= >>=3B>=3B>=3B>=3B Or printf to log something.<=3BBR>=3B
>= >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B &=3Bnbsp=3D3B<=3BBR>= >=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B I suppose there mi= >ght be an application that maps memory<=3BBR>=3B
>=3B >=3B>=3B= >>=3B>=3B>=3B>=3B>=3B gradually=3D2C as pieces of a buffer are hit= >. Might.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B= > &=3Bnbsp=3D3B<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B &=3Bnbsp=3D3B- Jay<=3BBR>=3B&=3Bnbsp=3D3B<=3BBR>=3B= >
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B &=3Bgt=3D3B To: m3= >devel at elegosoft.com<=3BBR>=3B&=3Bgt=3D3B Date: Sat=3D2C 19 Feb 2011 = >10:29:3=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B 0 -0800<= >=3BBR>=3B&=3Bgt=3D3B From: mika at async.caltech.edu<=3BBR>=3B&=3B= >gt=3D3B Subject: [M3devel]=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B SEGV mapping to RuntimeError<=3BBR>=3B&=3Bgt=3D3B<=3BBR&= >gt=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B Dear m3devel=3D
>=3B &= >gt=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D2C<=3BBR>=3B&=3Bgt=3D3B= ><=3BBR>=3B&=3Bgt=3D3B For a while it has annoyed me that segmentatio= >n vi=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B olations cause= > an<=3BBR>=3B&=3Bgt=3D3B unconditional program abort. I've changed t= >hat =3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B now so that (u= >nder user<=3BBR>=3B&=3Bgt=3D3B threads at least) we instead get a Ru= >ntim=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B eError. Here's= > an example of<=3BBR>=3B&=3Bgt=3D3B the mechanism at work in an inte= >ract=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B ive Scheme env= >ironment. Consider<=3BBR>=3B&=3Bgt=3D3B the unhelpful interface and = >modu=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B le Crash:<= >=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B INTERFACE Crash=3D3B = >PROCEDURE Me()=3D3B END Cra=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B sh.<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B MODUL= >E Crash=3D3B<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B PROCED= >URE Me(=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B ) =3D3D<= >=3BBR>=3B&=3Bgt=3D3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN<=3B= >BR>=3B&=3Bgt=3D3B ptr^ :=3D3D=3D
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B 0<=3BBR>=3B&=3Bgt=3D3B END Me=3D3B<=3BBR>=3B&= >=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B BEGIN END Crash.<=3BBR>=3B&= >=3Bgt=3D3B<=3BBR>=3B=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B &=3Bgt=3D3B Here's an example of what happens if you now call this= > from an inter=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B acti= >ve<=3BBR>=3B&=3Bgt=3D3B interpreter that catches the exception Runti= >meError.E:<=3BBR>=3B=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B &=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B M-Scheme Experimental<= >=3BBR>=3B&=3Bgt=3D3B LITHP ITH LITHENING.<=3BBR>=3B&=3B=3D
&= >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B gt=3D3B&=3Bgt=3D3B (req= >uire-modules "m3")<=3BBR>=3B&=3Bgt=3D3B #t<=3BBR>=3B&=3Bgt=3D= >3B&=3Bgt=3D3B (Crash.Me=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B )<=3BBR>=3B&=3Bgt=3D3B EXCEPTION! RuntimeError! Attempt to= > reference an illegal memory=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B location.<=3BBR>=3B&=3Bgt=3D3B&=3Bgt=3D3B (+ 3 4)<=3B= >BR>=3B&=3Bgt=3D3B 7<=3BBR>=3B&=3Bgt=3D3B&=3Bgt=3D3B<=3BBR&= >gt=3B&=3Bgt=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D3= >B<=3BBR>=3B&=3Bgt=3D3B I just realized I may have broken pthreads=3D= >2C let me go back=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B a= >nd double-check it.<=3BBR>=3B&=3Bgt=3D3B runtime/POSIX and thread/PO= >SIX don't refer=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B to = >the same thing do they...<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bg= >t=3D3B Mika<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B =3D
>=3B >=3B&= >gt=3B>=3B>=3B>=3B>=3B>=3B <=3B/body>=3B
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B <=3B/html>=3B=3D
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B&= >gt=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_--
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B
>=3B &g= >t=3B>=3B
>= > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_-- From jay.krell at cornell.edu Tue Feb 22 23:08:58 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 22 Feb 2011 22:08:58 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222184502.88B921A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , <4D604FA3.3060708@lcwb.coop>, , <20110219232729.53A991A2078@async.async.caltech.edu>, , <4D615643.7060107@lcwb.coop>, , <20110220183746.195C11A2078@async.async.caltech.edu>, , <4D61CC27.7070704@lcwb.coop>, , <20110222174405.E15481A2078@async.async.caltech.edu>, , <20110222184502.88B921A2078@async.async.caltech.edu> Message-ID: Heap vs. stack is indeed an unsolvable dilemna. Stack is much faster, but some arbitrary size must be chosen, that bothers me too, and detection and recovery from out of stack is tricky and not portable. Heap is slow, but portable, and generally limited to address space (but subject to fragmentation), easy to detect exhaustion. GC heap is usually fast to allocate, competitive with stack allocation, but the more you allocate, the more work for the GC to do. There is no free lunch -- not even as cheap as it might seem. :) Some systems..which I have never worked on..statically allocate everything. They need to know they will never run out of memory. > I realize my code is perhaps difficult to read. Not necessarily. I didn't even try yet. :) For many programmers, it is difficult to get them to read any code. :) - Jay > To: jay.krell at cornell.edu > Date: Tue, 22 Feb 2011 10:45:02 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > Jay K writes: > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >(aside=2C and possible agreement: right -- an interpreter should consider N= > >OT recursing on the machine > >stack whenever code it is interpreting recurses=2C but definitely some do) > >=20 > > - Jay > > > Jay, yes I agree---but: > > 1. of course it's easier to code the interpreter to recurse on the machine > stack > > 2. code called out from the interpreter will in any case recurse on the (a) > machine stack > > 3. again if one can "fix" the stack mechanism so that running out of > stack space is non-fatal, there's no reason not to use the machine stack > > I've found that Modula-3's garbage collector is brutally slow and that > converting code to using stack-allocated memory whenever possible is a > huge performance win under both PM3 and CM3. Perhaps this is a problem > with the garbage collector. > > I also have a partial design for a Scheme compiler (emitting Modula-3) > that would use the machine stack whenever possible rather than heap > allocation and do a "tricky thing" with local procedures to copy the > stack whenever a reference to the stack escapes. (Most of the time you > can implement Scheme on a stack but sometimes you need to keep part > of it live after you return; hence implementations (including mine) > heap-allocate local variables...) > > I realize my code is perhaps difficult to read. The gist of it is this: > when you run out of stack space, allocate a new stack anywhere else > in memory. Make it appear that you have alloca'd all the space between > the old and new stacks (into a variable that you of course never touch), > then continue. An issue is what to do when you return, you have to make > sure that you restore your old redzone.. perhaps you don't need to free > the stack eagerly: you could instead remember you have that page allocated > and re-use it next time you hit the redzone, or put it in a pool to allow > it to be used by another thread. > > The arbitrary choice of stack sizes has always bothered me greatly. > Some of my programs have thousands of threads and they use stack space > irregularly... > > Mika > > > >> To: rodney_bates at lcwb.coop > >> Date: Tue=2C 22 Feb 2011 09:44:05 -0800 > >> From: mika at async.caltech.edu > >> CC: m3devel at elegosoft.com > >> Subject: Re: [M3devel] SEGV mapping to RuntimeError > >>=20 > >>=20 > >> Ok so I was thinking about this. > >>=20 > >> Why on earth is stack overflow even a problem? > >>=20 > >> Consider the following procedure call (in my code=2C stack grows upwards)= > >: > >>=20 > >> (* sp at x=2C pc at y *) > >> y: P(args) > >> z: next_statement > >>=20 > >> decompose as follows: > >>=20 > >> (* sp at x=2C pc at y *) > >> y: Push(args etc. and ret. address z) > >> Jump(P) > >> z: next_statement > >>=20 > >> Now=2C we say: > >>=20 > >> y: ok :=3D check_stack(size of frame) > >> IF NOT ok THEN abort() END=3B > >> Push(args etc. and ret. address z) > >> Jump(P) > >> z: next_statement > >>=20 > >> (note check_stack and the following IF can be implemented by hardware=2C > >> need not actually be an instruction) > >>=20 > >> Let me change the code a tad: > >>=20 > >> y: ok :=3D check_stack(size of frame) > >> y':IF NOT ok THEN=20 > >> WITH new_stack_bottom =3D malloc(stack_size) > >> huge_amount =3D new_stack_bottom - sp DO > >> create_redzone_at(new_stack_bottom+stack_size-redzone_size) > >> EVAL alloca(huge_amount)=20 > >> END > >> END=3B > >> Push(args etc. and ret. address z) > >> Jump(P) > >> z: IF NOT ok THEN destroy_redzone(...)=3B free(new_stack_bottom) END > >>=20 > >> Note 1. cleanup of redzone could be postponed to return of caller....when > >> alloca in any case has to be cleaned up. > >>=20 > >> Note 2. the test IF NOT ok at z is more expensive to implement than the > >> one at y because you can't really use hardware for it. A hardware callbac= > >k > >> can be arranged though: > >>=20 > >> VAR ptr :=3D sp=3B > >> y: ok :=3D check_stack(size of frame) > >> y':IF NOT ok THEN=20 > >> ptr :=3D 0=3B (* illegal address *) > >> fault_address :=3D z=3B > >> WITH new_stack_bottom =3D malloc(stack_size) > >> huge_amount =3D new_stack_bottom - sp DO > >> create_redzone_at(new_stack_bottom+stack_size-redzone_size) > >> EVAL alloca(huge_amount)=20 > >> END > >> END=3B > >> Push(args etc. and ret. address z) > >> Jump(P) > >> z: EVAL ptr^ (* [ NOT ok -> hardware callback to SEGV: ] *) > >>=20 > >> SEGV(signalpc): IF NOT ok AND signalpc =3D fault_address THEN destroy_red= > >zone(...)=3B free(new_stack_bottom) END > >>=20 > >> Mika > >>=20 > >>=20 > >>=20 > >>=20 > >>=20 > >> "Rodney M. Bates" writes: > >> > > >> > > >> >On 02/20/2011 12:37 PM=2C Mika Nystrom wrote: > >> >> On a 64-bit machine=2C at least=2C there ought to be enough virtual > >> >> memory that you could just have a gap between thread stacks big > >> >> enough to allow for a protection area larger than the largest possible > >> >> (implementation-defined) activation record=2C no? I know I've run into > >> >> trouble with very large activation records in the past (and not becaus= > >e > >> >> I was running out of stack space=2C either). > >> >> > >> >> Or at least a procedure with a very large activation record (or > >> >> a procedure calling it) could be required to call some sort of check > >> >> routine "EnoughStackSpaceRemaining()" before starting to scribble > >> >> on the activation record? > >> > > >> >Hmm=2C I like this idea. It would introduce normal-case runtime overhead > >> >only for such procedures=2C and these are likely rare. Also=2C assuming = > >the procedure > >> >actually uses very much of its large AR=2C it should also have enough co= > >mputation > >> >time to wash out the stack check overhead. > >> > > >> >> > >> >> Also the end of the activation record must be written to at least once= > >=2C > >> >> or else the memory protection won't be triggered. > >> >> > >> > > >> >I was thinking (as an alternative mechanism) of having the compiler inte= > >ntionally > >> >add enough artificial write(s) as necessary to ensure storing within the > >> >red zone=2C and not just beyond it. This seems trickier to get right and > >> >harder to distinguish after the fact from a NIL dereference. > >> > > >> >> In any case if this is done properly the same mechanism I proposed for > >> >> SIGSEGV ought to be able to catch stack overflow=2C no? Well=2C as lon= > >g as > >> >> signals are delivered on a separate stack. If signals are delivered on > >> >> the same stack=2C the signal handler would get nastier=2C it would hav= > >e to > >> >> make space through some manipulations (maybe temporarily unporotecting > >> >> the redzone page?) for its own purposes... but I don't see why it > >> >> couldn't be done. > >> >> > >> >> Not sure why I'm getting SIGILL... maybe I am getting my signal handle= > >r > >> >> activated inside the redzone page because of a difference in signal > >> >> handling..? I remember reading something about sigaltstack... > >> >> > >> >> I would of course love to be able to recover from stack overflow=2C to= > >o. > >> >> In some sense=2C since it's a generally unknown limit=2C it's even les= > >s of > >> >> a fatal error than a NIL dereference (hence makes even more sense to > >> >> catch it). > >> > > >> >I think this would be a nice mechanism to have available. It would have = > >to > >> >be used with some care. In any case=2C it would be really nice and more > >> >frequently so=2C to at least have runtime error messages that distinguis= > >hed > >> >stack overflow from NIL deref. > >> > > >> >> > >> >> Mika > >> >> > >> >> "Rodney M. Bates" writes: > >> >>> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AM= > >D64_LINUX. > >> >>> Probably a fully protected guard page at the end of the stack. This t= > >echnique > >> >>> always worries me a bit because a procedure with a really big activat= > >ion record > >> >>> could jump right past it. Probably it would almost always access the = > >first page > >> >>> of the big area before storing anything into later pages. > >> >>> > >> >>> On 02/19/2011 05:27 PM=2C Mika Nystrom wrote: > >> >>>> Ah=2C yes=2C stack protection. > >> >>>> > >> >>>> Do you know if it's a SIGSEGV=2C not a SIGBUS? I know I have seen SI= > >GILL on Macs. > >> >>>> > >> >>>> Hmm=2C I get SIGILL on AMD64_FREEBSD as well: > >> >>>> > >> >>>> time ../AMD64_FREEBSD/stubexample > >> >>>> M-Scheme Experimental > >> >>>> LITHP ITH LITHENING. > >> >>>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >> >>>> f > >> >>>>> (f 0) > >> >>>> Illegal instruction > >> >>>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >> >>>> > >> >>>> What absolutely must not happen=2C of course=2C is that the runtime = > >hangs > >> >>>> while executing only safe code... > >> >>>> > >> >>>> Mika > >> >>>> > >> >>>> "Rodney M. Bates" writes: > >> >>>>> I know of one other place the compilers rely on hardware memory pro= > >tection > >> >>>>> to detect a checked runtime error=2C and that is stack overflow. Th= > >is won't > >> >>>>> corrupt anything=2C but is hard to distinguish from dereferencing N= > >IL. > >> >>>>> This could probably be distinguished after the fact by some low-lev= > >el=2C > >> >>>>> target-dependent code. I have found it by looking at assembly code = > >at > >> >>>>> the point of failure--usually right after a stack pointer push. > >> >>>>> > >> >>>>> Detecting this via compiler-generated checks would probably be more > >> >>>>> extravagant than many other checks=2C as it is so frequent. I am no= > >t > >> >>>>> aware of any really good solution to this in any implementation of = > >any > >> >>>>> language. > >> >>>>> > >> >>>>> On 02/19/2011 02:38 PM=2C Mika Nystrom wrote: > >> >>>>>> Jay=2C sometimes I wonder about you: this is a Modula-3 mailing li= > >st=2C > >> >>>>>> you know! > >> >>>>>> > >> >>>>>> "Corrupting the heap" is something that can only happen as a resul= > >t of > >> >>>>>> an unchecked runtime error. Unchecked runtime errors cannot happen= > > in > >> >>>>>> modules not marked UNSAFE. > >> >>>>>> > >> >>>>>> SEGV is=2C however=2C used by the CM3 implementation (and its pred= > >ecessors) > >> >>>>>> to signal a certain kind of *checked* runtime error=2C namely=2C t= > >he > >> >>>>>> dereferencing of a NIL reference. Correct me if I am wrong=2C but = > >an > >> >>>>>> attempt to dereference NIL is not going to leave the heap corrupte= > >d? > >> >>>>>> > >> >>>>>> And if you stick to safe code=2C the only SEGVs I think you get in= > > the > >> >>>>>> current CM3 are ones from NIL dereferences. > >> >>>>>> > >> >>>>>> Hence=2C as long as you stick with safe code=2C the only time the = > >code I > >> >>>>>> checked in earlier gets triggered is for NIL dereferences=2C which= > > should > >> >>>>>> never corrupt the heap. So SEGV is not sometimes=2C but in fact al= > >ways > >> >>>>>> recoverable. > >> >>>>>> > >> >>>>>> :-) > >> >>>>>> > >> >>>>>> Mika > >> >>>>>> > >> >>>>>> P.S. the bit above "if you stick to safe code": if you actually pr= > >ogram in > >> >>>>>> Modula-3 you almost never use UNSAFE. I went through my repository= > > and > >> >>>>>> I have 40 modules using UNSAFE out of a total of 4=2C559. Furtherm= > >ore=2C > >> >>>>>> many of the UNSAFE modules are glue code to Fortran routines=2C wh= > >ich > >> >>>>>> could relatively easily be verified to be safe in the Modula-3 sen= > >se. > >> >>>>>> Almost all what remains is glue to some C library=2C which wouldn'= > >t be > >> >>>>>> necessary if the rest of the world would wake up out of the dark a= > >ges=2C but > >> >>>>>> I don't have the time to rewrite every single library from scratch= > > myself. > >> >>>>>> > >> >>>>>> > >> >>>>>> Jay K writes: > >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >> >>>>>>> Content-Type: text/plain=3B charset=3D"iso-8859-1" > >> >>>>>>> Content-Transfer-Encoding: quoted-printable > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> Letting any code run after a SIGSEGV is dubious. > >> >>>>>>> Imagine the heap is corrupted. > >> >>>>>>> And then you run more code. > >> >>>>>>> And the code happens to call malloc. > >> >>>>>>> Or printf to log something. > >> >>>>>>> =3D20 > >> >>>>>>> I suppose there might be an application that maps memory > >> >>>>>>> gradually=3D2C as pieces of a buffer are hit. Might. > >> >>>>>>> =3D20 > >> >>>>>>> - Jay > >> >>>>>>> =3D20 > >> >>>>>>>> To: m3devel at elegosoft.com > >> >>>>>>>> Date: Sat=3D2C 19 Feb 2011 10:29:30 -0800 > >> >>>>>>>> From: mika at async.caltech.edu > >> >>>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >> >>>>>>>> =3D20 > >> >>>>>>>> =3D20 > >> >>>>>>>> Dear m3devel=3D2C > >> >>>>>>>> =3D20 > >> >>>>>>>> For a while it has annoyed me that segmentation violations cause= > > an > >> >>>>>>>> unconditional program abort. I've changed that now so that (unde= > >r user > >> >>>>>>>> threads at least) we instead get a RuntimeError. Here's an examp= > >le of > >> >>>>>>>> the mechanism at work in an interactive Scheme environment. Cons= > >ider > >> >>>>>>>> the unhelpful interface and module Crash: > >> >>>>>>>> =3D20 > >> >>>>>>>> INTERFACE Crash=3D3B PROCEDURE Me()=3D3B END Crash. > >> >>>>>>>> =3D20 > >> >>>>>>>> MODULE Crash=3D3B > >> >>>>>>>> =3D20 > >> >>>>>>>> PROCEDURE Me() =3D3D > >> >>>>>>>> VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN > >> >>>>>>>> ptr^ :=3D3D 0 > >> >>>>>>>> END Me=3D3B > >> >>>>>>>> =3D20 > >> >>>>>>>> BEGIN END Crash. > >> >>>>>>>> =3D20 > >> >>>>>>>> Here's an example of what happens if you now call this from an i= > >nteractiv=3D > >> >>>>>>> e > >> >>>>>>>> interpreter that catches the exception RuntimeError.E: > >> >>>>>>>> =3D20 > >> >>>>>>>> M-Scheme Experimental > >> >>>>>>>> LITHP ITH LITHENING. > >> >>>>>>>>> (require-modules "m3") > >> >>>>>>>> #t > >> >>>>>>>>> (Crash.Me) > >> >>>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory = > >location. > >> >>>>>>>>> (+ 3 4)=3D20 > >> >>>>>>>> 7 > >> >>>>>>>>> =3D20 > >> >>>>>>>> =3D20 > >> >>>>>>>> I just realized I may have broken pthreads=3D2C let me go back a= > >nd double-c=3D > >> >>>>>>> heck it.=3D20 > >> >>>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do = > >they... > >> >>>>>>>> =3D20 > >> >>>>>>>> Mika > >> >>>>>>>> =3D20 > >> >>>>>>> =3D > >> >>>>>>> > >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >> >>>>>>> Content-Type: text/html=3B charset=3D"iso-8859-1" > >> >>>>>>> Content-Transfer-Encoding: quoted-printable > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> Letting any code run after a SIGSEGV is dubious.
> >> >>>>>>> Imagine the heap =3D3Bis corrupted.
> >> >>>>>>> And then you run more code.
> >> >>>>>>> And the code happens to call malloc.
> >> >>>>>>> Or printf to log something.
> >> >>>>>>>  =3D3B
> >> >>>>>>> I suppose there might be an application that maps memory
> >> >>>>>>> gradually=3D2C as pieces of a buffer are hit. Might.
> >> >>>>>>>  =3D3B
> >> >>>>>>>  =3D3B- Jay
 =3D3B
> >> >>>>>>> >=3D3B To: m3devel at elegosoft.com
>=3D3B Date: Sat=3D2C 19 = > >Feb 2011 10:29:3=3D > >> >>>>>>> 0 -0800
>=3D3B From: mika at async.caltech.edu
>=3D3B Subj= > >ect: [M3devel]=3D > >> >>>>>>> SEGV mapping to RuntimeError
>=3D3B
>=3D3B
>=3D3B = > >Dear m3devel=3D > >> >>>>>>> =3D2C
>=3D3B
>=3D3B For a while it has annoyed me that = > >segmentation vi=3D > >> >>>>>>> olations cause an
>=3D3B unconditional program abort. I've c= > >hanged that =3D > >> >>>>>>> now so that (under user
>=3D3B threads at least) we instead = > >get a Runtim=3D > >> >>>>>>> eError. Here's an example of
>=3D3B the mechanism at work in= > > an interact=3D > >> >>>>>>> ive Scheme environment. Consider
>=3D3B the unhelpful interf= > >ace and modu=3D > >> >>>>>>> le Crash:
>=3D3B
>=3D3B INTERFACE Crash=3D3B PROCEDURE = > >Me()=3D3B END Cra=3D > >> >>>>>>> sh.
>=3D3B
>=3D3B MODULE Crash=3D3B
>=3D3B
>= > >=3D3B PROCEDURE Me(=3D > >> >>>>>>> ) =3D3D
>=3D3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN >R>>=3D3B ptr^ :=3D3D=3D > >> >>>>>>> 0
>=3D3B END Me=3D3B
>=3D3B
>=3D3B BEGIN END Crash= > >.
>=3D3B
=3D > >> >>>>>>> >=3D3B Here's an example of what happens if you now call this f= > >rom an inter=3D > >> >>>>>>> active
>=3D3B interpreter that catches the exception Runtime= > >Error.E:
=3D > >> >>>>>>> >=3D3B
>=3D3B M-Scheme Experimental
>=3D3B LITHP ITH = > >LITHENING.
&=3D > >> >>>>>>> gt=3D3B>=3D3B (require-modules "m3")
>=3D3B #t
>=3D3B= > >>=3D3B (Crash.Me=3D > >> >>>>>>> )
>=3D3B EXCEPTION! RuntimeError! Attempt to reference an il= > >legal memory=3D > >> >>>>>>> location.
>=3D3B>=3D3B (+ 3 4)
>=3D3B 7
>=3D3B&g= > >t=3D3B
>=3D > >> >>>>>>> =3D3B
>=3D3B I just realized I may have broken pthreads=3D2C= > > let me go back=3D > >> >>>>>>> and double-check it.
>=3D3B runtime/POSIX and thread/POSIX d= > >on't refer=3D > >> >>>>>>> to the same thing do they...
>=3D3B
>=3D3B Mika
>= > >=3D3B
=3D > >> >>>>>>> > >> >>>>>>> =3D > >> >>>>>>> > >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >> >>>>>> > >> >>>> > >> >> > > = > > > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_ > >Content-Type: text/html; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > > > > > > > >(aside=2C and possible agreement: right -- an interpreter should consider N= > >OT recursing on the machine
> >stack whenever code it is interpreting recurses=2C but definitely some do)<= > >BR> > > =3B
> > =3B- Jay
 =3B
> >>=3B To: rodney_bates at lcwb.coop
>=3B Date: Tue=2C 22 Feb 2011 09:44:= > >05 -0800
>=3B From: mika at async.caltech.edu
>=3B CC: m3devel at elego= > >soft.com
>=3B Subject: Re: [M3devel] SEGV mapping to RuntimeError
&= > >gt=3B
>=3B
>=3B Ok so I was thinking about this.
>=3B
= > >>=3B Why on earth is stack overflow even a problem?
>=3B
>=3B = > >Consider the following procedure call (in my code=2C stack grows upwards):<= > >BR>>=3B
>=3B (* sp at x=2C pc at y *)
>=3B y: P(args)
>= > >=3B z: next_statement
>=3B
>=3B decompose as follows:
>=3B = > >
>=3B (* sp at x=2C pc at y *)
>=3B y: Push(args etc. and ret. ad= > >dress z)
>=3B Jump(P)
>=3B z: next_statement
>=3B
>=3B= > > Now=2C we say:
>=3B
>=3B y: ok :=3D check_stack(size of frame)<= > >BR>>=3B IF NOT ok THEN abort() END=3B
>=3B Push(args etc. and ret. a= > >ddress z)
>=3B Jump(P)
>=3B z: next_statement
>=3B
>= > >=3B (note check_stack and the following IF can be implemented by hardware= > >=2C
>=3B need not actually be an instruction)
>=3B
>=3B Let= > > me change the code a tad:
>=3B
>=3B y: ok :=3D check_stack(size= > > of frame)
>=3B y':IF NOT ok THEN
>=3B WITH new_stack_bottom =3D= > > malloc(stack_size)
>=3B huge_amount =3D new_stack_bottom - sp DO
&= > >gt=3B create_redzone_at(new_stack_bottom+stack_size-redzone_size)
>=3B= > > EVAL alloca(huge_amount)
>=3B END
>=3B END=3B
>=3B Push(ar= > >gs etc. and ret. address z)
>=3B Jump(P)
>=3B z: IF NOT ok THEN d= > >estroy_redzone(...)=3B free(new_stack_bottom) END
>=3B
>=3B Note= > > 1. cleanup of redzone could be postponed to return of caller....when
&g= > >t=3B alloca in any case has to be cleaned up.
>=3B
>=3B Note 2. = > >the test IF NOT ok at z is more expensive to implement than the
>=3B o= > >ne at y because you can't really use hardware for it. A hardware callback >R>>=3B can be arranged though:
>=3B
>=3B VAR ptr :=3D sp=3B >>>=3B y: ok :=3D check_stack(size of frame)
>=3B y':IF NOT ok THEN <= > >BR>>=3B ptr :=3D 0=3B (* illegal address *)
>=3B fault_address :=3D = > >z=3B
>=3B WITH new_stack_bottom =3D malloc(stack_size)
>=3B huge_= > >amount =3D new_stack_bottom - sp DO
>=3B create_redzone_at(new_stack_b= > >ottom+stack_size-redzone_size)
>=3B EVAL alloca(huge_amount)
>= > >=3B END
>=3B END=3B
>=3B Push(args etc. and ret. address z)
&g= > >t=3B Jump(P)
>=3B z: EVAL ptr^ (* [ NOT ok ->=3B hardware callback t= > >o SEGV: ] *)
>=3B
>=3B SEGV(signalpc): IF NOT ok AND signalpc = > >=3D fault_address THEN destroy_redzone(...)=3B free(new_stack_bottom) END >R>>=3B
>=3B Mika
>=3B
>=3B
>=3B
>=3B
>= > >=3B
>=3B "Rodney M. Bates" writes:
>=3B >=3B
>=3B >=3B<= > >BR>>=3B >=3BOn 02/20/2011 12:37 PM=2C Mika Nystrom wrote:
>=3B >= > >=3B>=3B On a 64-bit machine=2C at least=2C there ought to be enough virtu= > >al
>=3B >=3B>=3B memory that you could just have a gap between thr= > >ead stacks big
>=3B >=3B>=3B enough to allow for a protection area= > > larger than the largest possible
>=3B >=3B>=3B (implementation-de= > >fined) activation record=2C no? I know I've run into
>=3B >=3B>=3B= > > trouble with very large activation records in the past (and not because >>>=3B >=3B>=3B I was running out of stack space=2C either).
>=3B= > > >=3B>=3B
>=3B >=3B>=3B Or at least a procedure with a very la= > >rge activation record (or
>=3B >=3B>=3B a procedure calling it) co= > >uld be required to call some sort of check
>=3B >=3B>=3B routine "= > >EnoughStackSpaceRemaining()" before starting to scribble
>=3B >=3B&g= > >t=3B on the activation record?
>=3B >=3B
>=3B >=3BHmm=2C I li= > >ke this idea. It would introduce normal-case runtime overhead
>=3B >= > >=3Bonly for such procedures=2C and these are likely rare. Also=2C assuming = > >the procedure
>=3B >=3Bactually uses very much of its large AR=2C it= > > should also have enough computation
>=3B >=3Btime to wash out the s= > >tack check overhead.
>=3B >=3B
>=3B >=3B>=3B
>=3B >= > >=3B>=3B Also the end of the activation record must be written to at least= > > once=2C
>=3B >=3B>=3B or else the memory protection won't be trig= > >gered.
>=3B >=3B>=3B
>=3B >=3B
>=3B >=3BI was thinki= > >ng (as an alternative mechanism) of having the compiler intentionally
&g= > >t=3B >=3Badd enough artificial write(s) as necessary to ensure storing wi= > >thin the
>=3B >=3Bred zone=2C and not just beyond it. This seems tri= > >ckier to get right and
>=3B >=3Bharder to distinguish after the fact= > > from a NIL dereference.
>=3B >=3B
>=3B >=3B>=3B In any cas= > >e if this is done properly the same mechanism I proposed for
>=3B >= > >=3B>=3B SIGSEGV ought to be able to catch stack overflow=2C no? Well=2C a= > >s long as
>=3B >=3B>=3B signals are delivered on a separate stack.= > > If signals are delivered on
>=3B >=3B>=3B the same stack=2C the s= > >ignal handler would get nastier=2C it would have to
>=3B >=3B>=3B = > >make space through some manipulations (maybe temporarily unporotecting
&= > >gt=3B >=3B>=3B the redzone page?) for its own purposes... but I don't s= > >ee why it
>=3B >=3B>=3B couldn't be done.
>=3B >=3B>=3B >R>>=3B >=3B>=3B Not sure why I'm getting SIGILL... maybe I am getting= > > my signal handler
>=3B >=3B>=3B activated inside the redzone page= > > because of a difference in signal
>=3B >=3B>=3B handling..? I rem= > >ember reading something about sigaltstack...
>=3B >=3B>=3B
>= > >=3B >=3B>=3B I would of course love to be able to recover from stack ov= > >erflow=2C too.
>=3B >=3B>=3B In some sense=2C since it's a general= > >ly unknown limit=2C it's even less of
>=3B >=3B>=3B a fatal error = > >than a NIL dereference (hence makes even more sense to
>=3B >=3B>= > >=3B catch it).
>=3B >=3B
>=3B >=3BI think this would be a nic= > >e mechanism to have available. It would have to
>=3B >=3Bbe used wit= > >h some care. In any case=2C it would be really nice and more
>=3B >= > >=3Bfrequently so=2C to at least have runtime error messages that distinguis= > >hed
>=3B >=3Bstack overflow from NIL deref.
>=3B >=3B
>= > >=3B >=3B>=3B
>=3B >=3B>=3B Mika
>=3B >=3B>=3B
>= > >=3B >=3B>=3B "Rodney M. Bates" writes:
>=3B >=3B>=3B>=3B I a= > >m pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX= > >.
>=3B >=3B>=3B>=3B Probably a fully protected guard page at the= > > end of the stack. This technique
>=3B >=3B>=3B>=3B always worri= > >es me a bit because a procedure with a really big activation record
>= > >=3B >=3B>=3B>=3B could jump right past it. Probably it would almost a= > >lways access the first page
>=3B >=3B>=3B>=3B of the big area be= > >fore storing anything into later pages.
>=3B >=3B>=3B>=3B
>= > >=3B >=3B>=3B>=3B On 02/19/2011 05:27 PM=2C Mika Nystrom wrote:
>= > >=3B >=3B>=3B>=3B>=3B Ah=2C yes=2C stack protection.
>=3B >= > >=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B Do you know if it'= > >s a SIGSEGV=2C not a SIGBUS? I know I have seen SIGILL on Macs.
>=3B &= > >gt=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B Hmm=2C I get SIG= > >ILL on AMD64_FREEBSD as well:
>=3B >=3B>=3B>=3B>=3B
>=3B = > >>=3B>=3B>=3B>=3B time ../AMD64_FREEBSD/stubexample
>=3B >=3B= > >>=3B>=3B>=3B M-Scheme Experimental
>=3B >=3B>=3B>=3B>=3B= > > LITHP ITH LITHENING.
>=3B >=3B>=3B>=3B>=3B>=3B (define (f a= > >) (+ (f (+ a 1)) (f (+ a 2))))
>=3B >=3B>=3B>=3B>=3B f
>= > >=3B >=3B>=3B>=3B>=3B>=3B (f 0)
>=3B >=3B>=3B>=3B>=3B= > > Illegal instruction
>=3B >=3B>=3B>=3B>=3B 3.847u 0.368s 0:13.= > >32 31.5% 2160+284478k 0+0io 0pf+0w
>=3B >=3B>=3B>=3B>=3B
&g= > >t=3B >=3B>=3B>=3B>=3B What absolutely must not happen=2C of course= > >=2C is that the runtime hangs
>=3B >=3B>=3B>=3B>=3B while exec= > >uting only safe code...
>=3B >=3B>=3B>=3B>=3B
>=3B >=3B= > >>=3B>=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B
>=3B >=3B= > >>=3B>=3B>=3B "Rodney M. Bates" writes:
>=3B >=3B>=3B>=3B&g= > >t=3B>=3B I know of one other place the compilers rely on hardware memory = > >protection
>=3B >=3B>=3B>=3B>=3B>=3B to detect a checked run= > >time error=2C and that is stack overflow. This won't
>=3B >=3B>=3B= > >>=3B>=3B>=3B corrupt anything=2C but is hard to distinguish from dere= > >ferencing NIL.
>=3B >=3B>=3B>=3B>=3B>=3B This could probably= > > be distinguished after the fact by some low-level=2C
>=3B >=3B>= > >=3B>=3B>=3B>=3B target-dependent code. I have found it by looking at = > >assembly code at
>=3B >=3B>=3B>=3B>=3B>=3B the point of fail= > >ure--usually right after a stack pointer push.
>=3B >=3B>=3B>=3B= > >>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B Detecting this via co= > >mpiler-generated checks would probably be more
>=3B >=3B>=3B>=3B= > >>=3B>=3B extravagant than many other checks=2C as it is so frequent. I = > >am not
>=3B >=3B>=3B>=3B>=3B>=3B aware of any really good so= > >lution to this in any implementation of any
>=3B >=3B>=3B>=3B>= > >=3B>=3B language.
>=3B >=3B>=3B>=3B>=3B>=3B
>=3B >= > >=3B>=3B>=3B>=3B>=3B On 02/19/2011 02:38 PM=2C Mika Nystrom wrote: >R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B Jay=2C sometimes I wonder abo= > >ut you: this is a Modula-3 mailing list=2C
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B you know!
>=3B >=3B>=3B>=3B>=3B>=3B>=3B >>>=3B >=3B>=3B>=3B>=3B>=3B>=3B "Corrupting the heap" is somet= > hing that can only happen as a result of
>=3B >=3B>=3B>=3B>=3B= > >>=3B>=3B an unchecked runtime error. Unchecked runtime errors cannot ha= > >ppen in
>=3B >=3B>=3B>=3B>=3B>=3B>=3B modules not marked U= > >NSAFE.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B SEGV is=2C however=2C used by the CM3 implement= > >ation (and its predecessors)
>=3B >=3B>=3B>=3B>=3B>=3B>=3B= > > to signal a certain kind of *checked* runtime error=2C namely=2C the
&g= > >t=3B >=3B>=3B>=3B>=3B>=3B>=3B dereferencing of a NIL reference.= > > Correct me if I am wrong=2C but an
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B attempt to dereference NIL is not going to leave the heap corrupt= > >ed?
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B&g= > >t=3B>=3B>=3B>=3B And if you stick to safe code=2C the only SEGVs I th= > >ink you get in the
>=3B >=3B>=3B>=3B>=3B>=3B>=3B current C= > >M3 are ones from NIL dereferences.
>=3B >=3B>=3B>=3B>=3B>=3B= > >>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B Hence=2C as long as y= > >ou stick with safe code=2C the only time the code I
>=3B >=3B>=3B&= > >gt=3B>=3B>=3B>=3B checked in earlier gets triggered is for NIL derefe= > >rences=2C which should
>=3B >=3B>=3B>=3B>=3B>=3B>=3B never= > > corrupt the heap. So SEGV is not sometimes=2C but in fact always
>=3B= > > >=3B>=3B>=3B>=3B>=3B>=3B recoverable.
>=3B >=3B>=3B&g= > >t=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B :-) >R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B&= > >gt=3B>=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
&g= > >t=3B >=3B>=3B>=3B>=3B>=3B>=3B P.S. the bit above "if you stick = > >to safe code": if you actually program in
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B Modula-3 you almost never use UNSAFE. I went through my rep= > >ository and
>=3B >=3B>=3B>=3B>=3B>=3B>=3B I have 40 module= > >s using UNSAFE out of a total of 4=2C559. Furthermore=2C
>=3B >=3B&g= > >t=3B>=3B>=3B>=3B>=3B many of the UNSAFE modules are glue code to Fo= > >rtran routines=2C which
>=3B >=3B>=3B>=3B>=3B>=3B>=3B coul= > >d relatively easily be verified to be safe in the Modula-3 sense.
>=3B= > > >=3B>=3B>=3B>=3B>=3B>=3B Almost all what remains is glue to so= > >me C library=2C which wouldn't be
>=3B >=3B>=3B>=3B>=3B>=3B&= > >gt=3B necessary if the rest of the world would wake up out of the dark ages= > >=2C but
>=3B >=3B>=3B>=3B>=3B>=3B>=3B I don't have the tim= > >e to rewrite every single library from scratch myself.
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B<= > >BR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B Jay K writes:
>=3B >= > >=3B>=3B>=3B>=3B>=3B>=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912= > >854f_
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Content-Type: te= > >xt/plain=3B charset=3D"iso-8859-1"
>=3B >=3B>=3B>=3B>=3B>=3B= > >>=3B>=3B Content-Transfer-Encoding: quoted-printable
>=3B >=3B&g= > >t=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B= > >>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Letting a= > >ny code run after a SIGSEGV is dubious.
>=3B >=3B>=3B>=3B>=3B&= > >gt=3B>=3B>=3B Imagine the heap is corrupted.
>=3B >=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B And then you run more code.
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B And the code happens to call malloc.
&= > >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Or printf to log something= > >.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>=3B >= > >=3B>=3B>=3B>=3B>=3B>=3B>=3B I suppose there might be an applica= > >tion that maps memory
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >gradually=3D2C as pieces of a buffer are hit. Might.
>=3B >=3B>=3B= > >>=3B>=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B - Jay
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B To: m3deve= > >l at elegosoft.com
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >Date: Sat=3D2C 19 Feb 2011 10:29:30 -0800
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B From: mika at async.caltech.edu
>=3B >=3B&g= > >t=3B>=3B>=3B>=3B>=3B>=3B>=3B Subject: [M3devel] SEGV mapping to= > > RuntimeError
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
&= > >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B Dear m3devel=3D2C >>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>=3B &g= > >t=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B For a while it has annoyed m= > >e that segmentation violations cause an
>=3B >=3B>=3B>=3B>=3B&= > >gt=3B>=3B>=3B>=3B unconditional program abort. I've changed that now = > >so that (under user
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B threads at least) we instead get a RuntimeError. Here's an example of >R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B the mechanism at = > >work in an interactive Scheme environment. Consider
>=3B >=3B>=3B&= > >gt=3B>=3B>=3B>=3B>=3B>=3B the unhelpful interface and module Cras= > >h:
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>= > >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B INTERFACE Crash=3D3B P= > >ROCEDURE Me()=3D3B END Crash.
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B MODULE Crash=3D3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >PROCEDURE Me() =3D3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&g= > >t=3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN
>=3B >=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B ptr^ :=3D3D 0
>=3B >=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B END Me=3D3B
>=3B >=3B>=3B>=3B&= > >gt=3B>=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B BEGIN END Crash.
>=3B >=3B>=3B>=3B>=3B&g= > >t=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B= > >>=3B>=3B Here's an example of what happens if you now call this from an= > > interactiv=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B e
&g= > >t=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B interpreter that catc= > >hes the exception RuntimeError.E:
>=3B >=3B>=3B>=3B>=3B>=3B&= > >gt=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B M-Scheme Experimental
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B LITHP ITH LITHENING.
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B (require-modules "m3")
>=3B >=3B>=3B&g= > >t=3B>=3B>=3B>=3B>=3B>=3B #t
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B (Crash.Me)
>=3B >=3B>=3B>=3B>=3B&g= > >t=3B>=3B>=3B>=3B EXCEPTION! RuntimeError! Attempt to reference an ill= > >egal memory location.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B>=3B (+ 3 4)=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B 7
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20<= > >BR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B I just realized = > >I may have broken pthreads=3D2C let me go back and double-c=3D
>=3B &g= > >t=3B>=3B>=3B>=3B>=3B>=3B>=3B heck it.=3D20
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B>=3B runtime/POSIX and thread/POSIX don'= > >t refer to the same thing do they...
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B= > > =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D
>=3B &= > >gt=3B>=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_
>=3B &g= > >t=3B>=3B>=3B>=3B>=3B>=3B>=3B Content-Type: text/html=3B charset= > >=3D"iso-8859-1"
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Conten= > >t-Transfer-Encoding: quoted-printable
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3B= > >html>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3Bhead&= > >gt=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3Bstyle>= > >=3B<=3B!--
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B .hmmessag= > >e P
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B {
>=3B >=3B= > >>=3B>=3B>=3B>=3B>=3B>=3B margin:0px=3D3B
>=3B >=3B>=3B= > >>=3B>=3B>=3B>=3B>=3B padding:0px
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B }
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B body.hmmessage
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B { >R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B font-size: 10pt=3D3B >>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B font-family:Tahoma
&g= > >t=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B }
>=3B >=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B -->=3B<=3B/style>=3B
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B <=3B/head>=3B
>=3B >=3B>=3B&= > >gt=3B>=3B>=3B>=3B>=3B <=3Bbody class=3D3D'hmmessage'>=3B
>= > >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Letting any code run after a= > > SIGSEGV is dubious.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B= > >>=3B>=3B Imagine the heap&=3Bnbsp=3D3Bis corrupted.<=3BBR>=3B >>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B And then you run more co= > >de.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B And = > >the code happens to call malloc.<=3BBR>=3B
>=3B >=3B>=3B>=3B= > >>=3B>=3B>=3B>=3B Or printf to log something.<=3BBR>=3B
>= > >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B &=3Bnbsp=3D3B<=3BBR>= > >=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B I suppose there mi= > >ght be an application that maps memory<=3BBR>=3B
>=3B >=3B>=3B= > >>=3B>=3B>=3B>=3B>=3B gradually=3D2C as pieces of a buffer are hit= > >. Might.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B= > > &=3Bnbsp=3D3B<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B &=3Bnbsp=3D3B- Jay<=3BBR>=3B&=3Bnbsp=3D3B<=3BBR>=3B= > >
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B &=3Bgt=3D3B To: m3= > >devel at elegosoft.com<=3BBR>=3B&=3Bgt=3D3B Date: Sat=3D2C 19 Feb 2011 = > >10:29:3=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B 0 -0800<= > >=3BBR>=3B&=3Bgt=3D3B From: mika at async.caltech.edu<=3BBR>=3B&=3B= > >gt=3D3B Subject: [M3devel]=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B SEGV mapping to RuntimeError<=3BBR>=3B&=3Bgt=3D3B<=3BBR&= > >gt=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B Dear m3devel=3D
>=3B &= > >gt=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D2C<=3BBR>=3B&=3Bgt=3D3B= > ><=3BBR>=3B&=3Bgt=3D3B For a while it has annoyed me that segmentatio= > >n vi=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B olations cause= > > an<=3BBR>=3B&=3Bgt=3D3B unconditional program abort. I've changed t= > >hat =3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B now so that (u= > >nder user<=3BBR>=3B&=3Bgt=3D3B threads at least) we instead get a Ru= > >ntim=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B eError. Here's= > > an example of<=3BBR>=3B&=3Bgt=3D3B the mechanism at work in an inte= > >ract=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B ive Scheme env= > >ironment. Consider<=3BBR>=3B&=3Bgt=3D3B the unhelpful interface and = > >modu=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B le Crash:<= > >=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B INTERFACE Crash=3D3B = > >PROCEDURE Me()=3D3B END Cra=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B sh.<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B MODUL= > >E Crash=3D3B<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B PROCED= > >URE Me(=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B ) =3D3D<= > >=3BBR>=3B&=3Bgt=3D3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN<=3B= > >BR>=3B&=3Bgt=3D3B ptr^ :=3D3D=3D
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B 0<=3BBR>=3B&=3Bgt=3D3B END Me=3D3B<=3BBR>=3B&= > >=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B BEGIN END Crash.<=3BBR>=3B&= > >=3Bgt=3D3B<=3BBR>=3B=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B &=3Bgt=3D3B Here's an example of what happens if you now call this= > > from an inter=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B acti= > >ve<=3BBR>=3B&=3Bgt=3D3B interpreter that catches the exception Runti= > >meError.E:<=3BBR>=3B=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B &=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B M-Scheme Experimental<= > >=3BBR>=3B&=3Bgt=3D3B LITHP ITH LITHENING.<=3BBR>=3B&=3B=3D
&= > >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B gt=3D3B&=3Bgt=3D3B (req= > >uire-modules "m3")<=3BBR>=3B&=3Bgt=3D3B #t<=3BBR>=3B&=3Bgt=3D= > >3B&=3Bgt=3D3B (Crash.Me=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B )<=3BBR>=3B&=3Bgt=3D3B EXCEPTION! RuntimeError! Attempt to= > > reference an illegal memory=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B location.<=3BBR>=3B&=3Bgt=3D3B&=3Bgt=3D3B (+ 3 4)<=3B= > >BR>=3B&=3Bgt=3D3B 7<=3BBR>=3B&=3Bgt=3D3B&=3Bgt=3D3B<=3BBR&= > >gt=3B&=3Bgt=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D3= > >B<=3BBR>=3B&=3Bgt=3D3B I just realized I may have broken pthreads=3D= > >2C let me go back=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B a= > >nd double-check it.<=3BBR>=3B&=3Bgt=3D3B runtime/POSIX and thread/PO= > >SIX don't refer=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B to = > >the same thing do they...<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bg= > >t=3D3B Mika<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B =3D
>=3B >=3B&= > >gt=3B>=3B>=3B>=3B>=3B>=3B <=3B/body>=3B
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B <=3B/html>=3B=3D
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B&= > >gt=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_--
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B
>=3B &g= > >t=3B>=3B
> >= > > > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_-- -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Wed Feb 23 17:42:06 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Wed, 23 Feb 2011 08:42:06 -0800 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" Message-ID: <20110223164206.BE6931A2078@async.async.caltech.edu> Notes: 1. the Modula-3 garbage collector gets confused (shoulda thought of that), so I have to turn it off 2. however if the GC were fixed, there's no need to "fix up" the stack after the return---you can just allocate the new stacks as GC-traced: even though the collector is compacting they are "pinned" because they are referenced from the thread stack! 3. alloca is built into gcc nowadays, so I had to track down an assembly version: I suspect that the compiler front-end could be modified to emit a call to alloca and thus get gcc to generate the same code, obviating the need for any assembly code at all. (* Unsafe interface ***************************************************) INTERFACE Unsafe; FROM Ctypes IMPORT int; PROCEDURE DoIt(); <*EXTERNAL alloca*> PROCEDURE alloca(size : int) : INTEGER; END Unsafe. (* Unsafe implementation **********************************************) UNSAFE MODULE Unsafe; IMPORT Thread, IO, Fmt; IMPORT RTCollector, Random; EXCEPTION Quit; VAR rand := NEW(Random.Default).init(); PROCEDURE P(cl : Closure; arg : ARRAY[0..100] OF INTEGER) RAISES { Quit } = CONST Fudge = 1024; VAR stack : REF ARRAY OF INTEGER := NIL; BEGIN IO.Put(Fmt.F("Thread %s GetSP()=0x%s\n", Fmt.Int(cl.id), Fmt.Int(GetSP(), base := 16)) ); (*Thread.Pause(0.1d0);*) IF GetSP() < cl.lo + Fudge THEN IF cl.stacks >= rand.integer(3,10) THEN IO.Put(Fmt.F("Thread %s quitting after %s stacks\n", Fmt.Int(cl.id), Fmt.Int(cl.stacks))); RAISE Quit END; stack := NEW(REF ARRAY OF INTEGER, 8*1024); WITH lo = LOOPHOLE(ADR(stack[0]),INTEGER), hi = LOOPHOLE(ADR(stack[LAST(stack^)]),INTEGER), sp = GetSP() DO IO.Put(Fmt.F("Thread %s sp %s new stack lo %s hi %s\n", Fmt.Int(cl.id), Fmt.Int(sp, base := 16), Fmt.Int(lo, base := 16), Fmt.Int(hi, base:= 16))); (* new stack starts sp - hi away *) EVAL alloca(sp-hi); IO.Put(Fmt.F("After alloca, sp %s\n", Fmt.Int(GetSP(),base := 16))); cl.lo := lo; INC(cl.stacks) END END; P(cl, arg) END P; TYPE Closure = Thread.Closure OBJECT id : CARDINAL; lo : INTEGER; stacks : CARDINAL; OVERRIDES apply := Apply END; PROCEDURE Apply(cl : Closure) : REFANY = BEGIN Thread.Pause(1.0d0); LOOP TRY cl.lo := GetSP() - 1024*8; (* pretend we have 8k left *) cl.stacks := 0; P(cl, ARRAY [0..100] OF INTEGER { 0, .. }); EXCEPT Quit => (* start over *) END END END Apply; PROCEDURE DoIt() = BEGIN RTCollector.Disable(); FOR i := 1 TO 2 DO EVAL Thread.Fork(NEW(Closure, id := i)) END; LOOP Thread.Pause(1.0d0) END END DoIt; PROCEDURE GetSP() : INTEGER = BEGIN RETURN alloca(1) END GetSP; BEGIN END Unsafe. (* Main can't be UNSAFE in Modula-3 ***********************************) MODULE Main; IMPORT Unsafe; BEGIN Unsafe.DoIt() END Main. (* alloca.s from FreeBSD (old code from BSD/386) **********************) /*- * Copyright (c) 1990 The Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ .asciz "@(#)alloca.s 5.2 (Berkeley) 5/14/90" #include /* like alloc, but automatic automatic free in return */ .globl alloca .type alloca, @function alloca: popl %edx /* pop return addr */ popl %eax /* pop amount to allocate */ movl %esp,%ecx addl $3,%eax /* round up to next word */ andl $0xfffffffc,%eax subl %eax,%esp movl %esp,%eax /* base of newly allocated space */ pushl 8(%ecx) /* copy possible saved registers */ pushl 4(%ecx) pushl 0(%ecx) pushl %eax /* dummy to pop at callsite */ jmp *%edx /* "return" */ From jay.krell at cornell.edu Wed Feb 23 20:53:42 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 23 Feb 2011 19:53:42 +0000 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: <20110223164206.BE6931A2078@async.async.caltech.edu> References: <20110223164206.BE6931A2078@async.async.caltech.edu> Message-ID: Stack must be contiguous on NT. Some platforms -- NT -- actually know how big the current stack is -- and won't let you alloca past it. Best way to let the target handle making stacks is with Posix makecontext, for the many-but-not-all platforms that have it, or CreateFiber on NT. (OpenBSD and I think Darwin/amd64 lack makecontext. It is in Posix, but deprecated; heck Darwin doesn't even have Posix semaphores seemingly..) To avoid the assembly you could use a hack like: void* Unsafe__alloca(size_t n) { return alloca(n); } Beware. Dangerous platform/compiler-specific stuff here. - Jay > To: m3devel at elegosoft.com > Date: Wed, 23 Feb 2011 08:42:06 -0800 > From: mika at async.caltech.edu > Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" > > Notes: > > 1. the Modula-3 garbage collector gets confused (shoulda thought of that), > so I have to turn it off > > 2. however if the GC were fixed, there's no need to "fix up" the stack > after the return---you can just allocate the new stacks as GC-traced: > even though the collector is compacting they are "pinned" because > they are referenced from the thread stack! > > 3. alloca is built into gcc nowadays, so I had to track down an assembly > version: I suspect that the compiler front-end could be modified to > emit a call to alloca and thus get gcc to generate the same code, > obviating the need for any assembly code at all. > > > (* Unsafe interface ***************************************************) > > INTERFACE Unsafe; > FROM Ctypes IMPORT int; > > PROCEDURE DoIt(); > > <*EXTERNAL alloca*> > PROCEDURE alloca(size : int) : INTEGER; > > END Unsafe. > > (* Unsafe implementation **********************************************) > > UNSAFE MODULE Unsafe; > > IMPORT Thread, IO, Fmt; > IMPORT RTCollector, Random; > > EXCEPTION Quit; > > VAR rand := NEW(Random.Default).init(); > > PROCEDURE P(cl : Closure; arg : ARRAY[0..100] OF INTEGER) RAISES { Quit } = > CONST Fudge = 1024; > VAR stack : REF ARRAY OF INTEGER := NIL; > BEGIN > IO.Put(Fmt.F("Thread %s GetSP()=0x%s\n", > Fmt.Int(cl.id), > Fmt.Int(GetSP(), base := 16)) > ); > (*Thread.Pause(0.1d0);*) > > IF GetSP() < cl.lo + Fudge THEN > IF cl.stacks >= rand.integer(3,10) THEN > IO.Put(Fmt.F("Thread %s quitting after %s stacks\n", > Fmt.Int(cl.id), Fmt.Int(cl.stacks))); > RAISE Quit > END; > > stack := NEW(REF ARRAY OF INTEGER, 8*1024); > WITH lo = LOOPHOLE(ADR(stack[0]),INTEGER), > hi = LOOPHOLE(ADR(stack[LAST(stack^)]),INTEGER), > sp = GetSP() DO > IO.Put(Fmt.F("Thread %s sp %s new stack lo %s hi %s\n", > Fmt.Int(cl.id), > Fmt.Int(sp, base := 16), > Fmt.Int(lo, base := 16), > Fmt.Int(hi, base:= 16))); > > (* new stack starts sp - hi away *) > EVAL alloca(sp-hi); > IO.Put(Fmt.F("After alloca, sp %s\n", > Fmt.Int(GetSP(),base := 16))); > cl.lo := lo; > INC(cl.stacks) > END > END; > > P(cl, arg) > END P; > > TYPE > Closure = Thread.Closure OBJECT > id : CARDINAL; > lo : INTEGER; > stacks : CARDINAL; > OVERRIDES > apply := Apply > END; > > PROCEDURE Apply(cl : Closure) : REFANY = > BEGIN > Thread.Pause(1.0d0); > > LOOP > TRY > cl.lo := GetSP() - 1024*8; (* pretend we have 8k left *) > cl.stacks := 0; > P(cl, ARRAY [0..100] OF INTEGER { 0, .. }); > EXCEPT > Quit => (* start over *) > END > END > END Apply; > > PROCEDURE DoIt() = > BEGIN > RTCollector.Disable(); > > FOR i := 1 TO 2 DO > EVAL Thread.Fork(NEW(Closure, id := i)) > END; > > LOOP Thread.Pause(1.0d0) END > END DoIt; > > PROCEDURE GetSP() : INTEGER = BEGIN RETURN alloca(1) END GetSP; > > BEGIN END Unsafe. > > (* Main can't be UNSAFE in Modula-3 ***********************************) > > MODULE Main; > IMPORT Unsafe; > > BEGIN Unsafe.DoIt() END Main. > > (* alloca.s from FreeBSD (old code from BSD/386) **********************) > > /*- > * Copyright (c) 1990 The Regents of the University of California. > * All rights reserved. > * > * This code is derived from software contributed to Berkeley by > * William Jolitz. > * > * Redistribution and use in source and binary forms, with or without > * modification, are permitted provided that the following conditions > * are met: > * 1. Redistributions of source code must retain the above copyright > * notice, this list of conditions and the following disclaimer. > * 2. Redistributions in binary form must reproduce the above copyright > * notice, this list of conditions and the following disclaimer in the > * documentation and/or other materials provided with the distribution. > * 3. All advertising materials mentioning features or use of this software > * must display the following acknowledgement: > * This product includes software developed by the University of > * California, Berkeley and its contributors. > * 4. Neither the name of the University nor the names of its contributors > * may be used to endorse or promote products derived from this software > * without specific prior written permission. > * > * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND > * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE > * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE > * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL > * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS > * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) > * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT > * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY > * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > * SUCH DAMAGE. > */ > > .asciz "@(#)alloca.s 5.2 (Berkeley) 5/14/90" > #include > > > /* like alloc, but automatic automatic free in return */ > > .globl alloca > .type alloca, @function > alloca: > popl %edx /* pop return addr */ > popl %eax /* pop amount to allocate */ > movl %esp,%ecx > addl $3,%eax /* round up to next word */ > andl $0xfffffffc,%eax > subl %eax,%esp > movl %esp,%eax /* base of newly allocated space */ > pushl 8(%ecx) /* copy possible saved registers */ > pushl 4(%ecx) > pushl 0(%ecx) > pushl %eax /* dummy to pop at callsite */ > jmp *%edx /* "return" */ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Wed Feb 23 21:39:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Wed, 23 Feb 2011 12:39:50 -0800 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: References: <20110223164206.BE6931A2078@async.async.caltech.edu> Message-ID: <20110223203950.30ABF1A2078@async.async.caltech.edu> Jay K writes: >--_60ddb4ae-bef3-432c-a5fd-a46c878ce951_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Stack must be contiguous on NT. >Some platforms -- NT -- actually know how big the current stack is -- and w= >on't let you >alloca past it. Best way to let the target handle making stacks is with Pos= >ix makecontext=2C >for the many-but-not-all platforms that have it=2C or CreateFiber on NT. >(OpenBSD and I think Darwin/amd64 lack makecontext. It is in Posix=2C but d= >eprecated=3B >heck Darwin doesn't even have Posix semaphores seemingly..) The problem with making stacks with makecontext, etc., to me seems to be: what do you do when you want to return to the original stack. Can it be made to "just work"? Note that in my design, the stacks are all contiguous---but overlapping! There are no "new" stacks and no switching of stacks. You just pretend you have a huge data structure sitting where the other guys' stacks (and data, etc.) are. Cleanup is also easy, it's handled by the ordinary GC. > >To avoid the assembly you could use a hack like: >void* Unsafe__alloca(size_t n) { return alloca(n)=3B } Won't work. alloca releases memory at the }. What would avoid it is generating the call to alloca in the front end. > >Beware. >Dangerous platform/compiler-specific stuff here. Not really that specific. It relies on precisely one special property of the underlying system: that alloca doesn't do any error checking. The only changes that need to be made to CM3 are coming up with a mechanism for detecting the need for expanding the stack (could be done by bumping the redzone before winding the stack and handling this in a SEGV handler) and modifying the garbage collector to understand the scheme. I suggest adding a back pointer at the beginning of the newly allocated area that would let the GC go back to the previous one while scanning (or however it works). Users/platforms that don't need or can't support "leapfrog stacks" would not change except that their garbage collectors would have some code that's never executed... I've discussed the above with people who've been doing concurrent programming since before the glaciers came thru (implementations of Concurrent Pascal in the 1970s) and they said that they used linked (noncontiguous) stacks in those days. The state of the art of 2011 seems rather unimpressive... My feeling is that the stack overflow issue is one of the things hindering the acceptance of truly multi-threaded programming (thousands, tens of thousands, millions of threads). Mika From jay.krell at cornell.edu Wed Feb 23 21:51:35 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 23 Feb 2011 20:51:35 +0000 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: <20110223203950.30ABF1A2078@async.async.caltech.edu> References: <20110223164206.BE6931A2078@async.async.caltech.edu>, , <20110223203950.30ABF1A2078@async.async.caltech.edu> Message-ID: > that alloca doesn't do any error checking No. There is also native exception handling to consider. NT essentially does error checking in both. > thousands, tens of thousands, millions of threads - 64bit address spaces - multiple machines ("cloud") > Won't work. alloca releases memory at the }. Only sort of. You can still return that point and still use the memory. You do risk, e.g. if you make any function calls, then the alloca memory and the function parameters might overlap. You could..platform-dependent..probably skip the first "few" bytes of the alloca memory. > Note that in my design, the stacks are all contiguous---but overlapping! Are your stacks also traversing heap and what-not, or are you just chopping up the current stack into lots of smaller stacks? i.e. alloca stack size times n and then create n threads? That's less bad. I had the impression you were skipping over other memory. You know, like, malloc your stack, and then alloca to get the stack pointer to hit the malloced memory. > What would avoid it is generating the call to alloca in the front end. We could w/o much difficulty provide alloca in the frontend. Probably limit it to unsafe modules. At least to support your experimenting and somewhat platform-dependent code. > in the 1970s.. I think the right place to experiment with this stuff is at a lower level. Or with more complete control of the system. ie. in the kernel, or on only one specific kernel, that you build/modify/write-from-scratch yourself. Gotta go, - Jay > To: jay.krell at cornell.edu > Date: Wed, 23 Feb 2011 12:39:50 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" > > Jay K writes: > >--_60ddb4ae-bef3-432c-a5fd-a46c878ce951_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >Stack must be contiguous on NT. > >Some platforms -- NT -- actually know how big the current stack is -- and w= > >on't let you > >alloca past it. Best way to let the target handle making stacks is with Pos= > >ix makecontext=2C > >for the many-but-not-all platforms that have it=2C or CreateFiber on NT. > >(OpenBSD and I think Darwin/amd64 lack makecontext. It is in Posix=2C but d= > >eprecated=3B > >heck Darwin doesn't even have Posix semaphores seemingly..) > > The problem with making stacks with makecontext, etc., to me seems > to be: what do you do when you want to return to the original stack. > Can it be made to "just work"? > > Note that in my design, the stacks are all contiguous---but overlapping! > There are no "new" stacks and no switching of stacks. You just pretend > you have a huge data structure sitting where the other guys' stacks > (and data, etc.) are. > > Cleanup is also easy, it's handled by the ordinary GC. > > > > >To avoid the assembly you could use a hack like: > >void* Unsafe__alloca(size_t n) { return alloca(n)=3B } > > Won't work. alloca releases memory at the }. > > What would avoid it is generating the call to alloca in the front end. > > > > >Beware. > >Dangerous platform/compiler-specific stuff here. > > Not really that specific. It relies on precisely one special property > of the underlying system: that alloca doesn't do any error checking. > > The only changes that need to be made to CM3 are coming up with a > mechanism for detecting the need for expanding the stack (could be done by > bumping the redzone before winding the stack and handling this in a SEGV > handler) and modifying the garbage collector to understand the scheme. > I suggest adding a back pointer at the beginning of the newly allocated > area that would let the GC go back to the previous one while scanning > (or however it works). > > Users/platforms that don't need or can't support "leapfrog stacks" > would not change except that their garbage collectors would have some > code that's never executed... > > I've discussed the above with people who've been doing concurrent > programming since before the glaciers came thru (implementations of > Concurrent Pascal in the 1970s) and they said that they used linked > (noncontiguous) stacks in those days. The state of the art of 2011 > seems rather unimpressive... > > My feeling is that the stack overflow issue is one of the things hindering > the acceptance of truly multi-threaded programming (thousands, tens of > thousands, millions of threads). > > Mika > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Thu Feb 24 03:20:29 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Wed, 23 Feb 2011 18:20:29 -0800 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: References: <20110223164206.BE6931A2078@async.async.caltech.edu>, , <20110223203950.30ABF1A2078@async.async.caltech.edu> Message-ID: <20110224022029.B016E1A2078@async.async.caltech.edu> Jay K writes: >--_6d651072-194c-4975-9e55-045a636fc1ed_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > that alloca doesn't do any error checking > > >No. There is also native exception handling to consider. >NT essentially does error checking in both. Native exception handling? What do you mean? > > >> thousands=2C tens of thousands=2C millions of threads > > - 64bit address spaces > - multiple machines ("cloud") >=20 > > > Won't work. alloca releases memory at the }. >=20 >=20 > Only sort of. > You can still return that point and still use the memory. > You do risk=2C e.g. if you make any function calls=2C then the > alloca memory and the function parameters might overlap. > You could..platform-dependent..probably skip the first > "few" bytes of the alloca memory. Well that's the whole point of the alloca, that it moves the stack pointer beyond the danger region...(and keeps it there till you return) if you let it come back ... it defeats the purpose. > > > > Note that in my design=2C the stacks are all contiguous---but overlappin= >g! >=20 > Are your stacks also traversing heap and what-not=2C > or are you just chopping up the current stack into lots > of smaller stacks? Yes traverses heap, protected pages, I/O buffers, text segments,... whatever junk is in between the old stack and the newly allocated area. >=20 > i.e. alloca stack size times n and then create n threads? > That's less bad. >=20 > I had the impression you were skipping over other memory. >=20 > You know=2C like=2C malloc your stack=2C and then alloca to get > the stack pointer to hit the malloced memory. That's exactly right. malloc a new stack and use alloca to fiddle the stack pointer so it lands at the start of the new stack. Now, I don't know a darn thing about Windows, but I've been looking at Unix code and I see the following: * all alloca does is it modifies the stack pointer. It does this from user code without calling any sort of operating system or other runtime routines. * nothing special is done to "stacks" in Modula-3's "POSIX" user threading library---threads are allocated with a NEW(UNTRACED REF...) using the normal Modula-3 memory management routines. * the concept of "stack" (another one of those EWD inventions, incidentally) itself is a bit of a misnomer. There is only the stack pointer + conventions on how you use it. The "stack" is simply the area between where the thread began execution and the current value of the stack pointer. It's a matter of definition, simply. > > > What would avoid it is generating the call to alloca in the front end. >=20 >We could w/o much difficulty provide alloca in the frontend. >Probably limit it to unsafe modules. >At least to support your experimenting and somewhat platform-dependent code= >. Sure that would be fun :) > > > in the 1970s.. >=20 >=20 >I think the right place to experiment with this stuff is at a lower level. >Or with more complete control of the system. >ie. in the kernel=2C or on only one specific kernel=2C that you >build/modify/write-from-scratch yourself. Let me put it like this: if Modula-3's "user threads" work on architecture/OS combination X, I think you can just about (perhaps you can actually) prove that "leapfrog stacks" would work on the same architecture/OS combination, just based on an analysis of how the user threads allocate and use their stacks. But if user threads don't work (say Windows?) then it probably won't work. We do know, however, that user threads work just about everywhere... Mika > > >Gotta go=2C > - Jay > > > > > > From jay.krell at cornell.edu Thu Feb 24 11:09:33 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 24 Feb 2011 10:09:33 +0000 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: <20110224022029.B016E1A2078@async.async.caltech.edu> References: <20110223164206.BE6931A2078@async.async.caltech.edu>, , , , <20110223203950.30ABF1A2078@async.async.caltech.edu>, , <20110224022029.B016E1A2078@async.async.caltech.edu> Message-ID: > But if user threads don't work (say Windows?) then it probably won't > work. We do know, however, that user threads work just about everywhere. They don't. One could use fibers if one really wanted to do one's own scheduling, and locking. (user threads implies writing your own locks). On NT, alloca touches every page between the current and the result. (not every byte, that'd be wasteful, just every page) Touching the end of the stack generates stack overflow exceptions. So you can't go past it. That is, the very problem people were worried about, is solved on NT. "Native exception handling" is what e.g. C++ uses. I believe it validates the stack to be in range. In particular, on NT, the stack range is available somewhere, and occasionally used. - Jay > To: jay.krell at cornell.edu > Date: Wed, 23 Feb 2011 18:20:29 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" > > Jay K writes: > >--_6d651072-194c-4975-9e55-045a636fc1ed_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > that alloca doesn't do any error checking > > > > > >No. There is also native exception handling to consider. > >NT essentially does error checking in both. > > Native exception handling? What do you mean? > > > > > > >> thousands=2C tens of thousands=2C millions of threads > > > > - 64bit address spaces > > - multiple machines ("cloud") > >=20 > > > > > Won't work. alloca releases memory at the }. > >=20 > >=20 > > Only sort of. > > You can still return that point and still use the memory. > > You do risk=2C e.g. if you make any function calls=2C then the > > alloca memory and the function parameters might overlap. > > You could..platform-dependent..probably skip the first > > "few" bytes of the alloca memory. > > Well that's the whole point of the alloca, that it moves the stack > pointer beyond the danger region...(and keeps it there till you return) > if you let it come back ... it defeats the purpose. > > > > > > > > Note that in my design=2C the stacks are all contiguous---but overlappin= > >g! > >=20 > > Are your stacks also traversing heap and what-not=2C > > or are you just chopping up the current stack into lots > > of smaller stacks? > > > Yes traverses heap, protected pages, I/O buffers, text segments,... whatever > junk is in between the old stack and the newly allocated area. > > > >=20 > > i.e. alloca stack size times n and then create n threads? > > That's less bad. > >=20 > > I had the impression you were skipping over other memory. > >=20 > > You know=2C like=2C malloc your stack=2C and then alloca to get > > the stack pointer to hit the malloced memory. > > That's exactly right. malloc a new stack and use alloca to fiddle > the stack pointer so it lands at the start of the new stack. > > Now, I don't know a darn thing about Windows, but I've been looking at > Unix code and I see the following: > > * all alloca does is it modifies the stack pointer. It does this from > user code without calling any sort of operating system or other runtime > routines. > > * nothing special is done to "stacks" in Modula-3's "POSIX" user threading > library---threads are allocated with a NEW(UNTRACED REF...) using the > normal Modula-3 memory management routines. > > * the concept of "stack" (another one of those EWD inventions, > incidentally) itself is a bit of a misnomer. There is only the > stack pointer + conventions on how you use it. The "stack" is simply > the area between where the thread began execution and the current > value of the stack pointer. It's a matter of definition, simply. > > > > > > What would avoid it is generating the call to alloca in the front end. > >=20 > >We could w/o much difficulty provide alloca in the frontend. > >Probably limit it to unsafe modules. > >At least to support your experimenting and somewhat platform-dependent code= > >. > > Sure that would be fun :) > > > > > > in the 1970s.. > >=20 > >=20 > >I think the right place to experiment with this stuff is at a lower level. > >Or with more complete control of the system. > >ie. in the kernel=2C or on only one specific kernel=2C that you > >build/modify/write-from-scratch yourself. > > Let me put it like this: if Modula-3's "user threads" work on > architecture/OS combination X, I think you can just about (perhaps > you can actually) prove that "leapfrog stacks" would work on the same > architecture/OS combination, just based on an analysis of how the user > threads allocate and use their stacks. But if user threads don't work > (say Windows?) then it probably won't work. We do know, however, that > user threads work just about everywhere... > > Mika > > > > > > >Gotta go=2C > > - Jay > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Feb 24 20:16:58 2011 From: dragisha at m3w.org (=?utf-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 24 Feb 2011 20:16:58 +0100 Subject: [M3devel] new target? :) Message-ID: http://code.google.com/chrome/nativeclient/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Thu Feb 24 22:46:17 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Thu, 24 Feb 2011 21:46:17 +0000 (GMT) Subject: [M3devel] new target? :) In-Reply-To: Message-ID: <758561.47161.qm@web29710.mail.ird.yahoo.com> Hi all: is interesting anyway but to write the web apps in c and good friends, hum ..., let me think if I can write a web server in less than 100 loc of Obliq and indeed one is written as an os extension, why would need any web page that renders c code, I think I would gain much more time writing myself rather than porting C (of course with its flavors) in Modula-3 or better if so in Obliq which is a mobile computing environment indeed, it makes me think, though the idea is good it makes little sense to make everything a client of a web browser rather than an instance of a structured platform, if they need only C why it has to be an instance of a web browser, it must be the browser makes better work for quick dynamic pages with C, or maybe another way of keeping it alive artificially, I mean what is the gain with that, if better runtime but, hum ...? but? it needs javascript, hum ... another loss of time (much time in inefficient checking), Nokia had the same idea with Qt and JS in WRT, (very little RT information), but still Obliq scripts can be type inferenced in much more smoothly way. I think this battle of JavaScript and Obliq and Modula-3 vs others could be fight yet one time again but needs more time for tools to make that a reality, still the others think they can do it, it's very interesting. Thanks in advance --- El jue, 24/2/11, Dragi?a Duri? escribi?: De: Dragi?a Duri? Asunto: [M3devel] new target? :) Para: "m3devel" Fecha: jueves, 24 de febrero, 2011 14:16 http://code.google.com/chrome/nativeclient/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 24 22:58:17 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 24 Feb 2011 21:58:17 +0000 Subject: [M3devel] new target? :) In-Reply-To: <758561.47161.qm@web29710.mail.ird.yahoo.com> References: , <758561.47161.qm@web29710.mail.ird.yahoo.com> Message-ID: I believe nativeclient depends on a fork of gcc. And I'm not sure if it always 32bit x86 or not. Part of what they do is limit the output such as to be easily disassembled and analyzed at load/runtime. If that is all, we could probably just take their patches. There is a paper about it.. I don't know what you link to though. Ultimately..you can tell what I'm going to say..a good solution is Modula-3-to-C, and then use whatever C compiler you want. Including the nativeclient gcc. An iPhone/iPad/iPod cross compiler. Older stuff like Irix/AIX/HPUX that we have no Hudson nodes for. NT/AMD64, NT/IA64. Android. Newer stuff that hasn't come along yet. Basically, generating C or C++ is a great portability mechanism, to old, existing, and new platforms. A bit better than depending on gcc. On Daniel's point, there's an ongoing race to provide the best new development environments that do or do not allow much code reuse. That are or are not "safe". not safe: iPhone, iPad, iPod, native for Android safe: Java on Android, Silverlight in web browser and Windows Phone 7, Go, JavaScript Those platforms have way more mindshare, support, tooling, libraries, than Modula-3 and I'm sure they always will. > let me think if I can write a web server in less than 100 loc It takes 5 lines in Python, as it is already builtin to the libraries -- confusing language and libraries, granted. - Jay Date: Thu, 24 Feb 2011 21:46:17 +0000 From: dabenavidesd at yahoo.es To: m3devel at elegosoft.com; dragisha at m3w.org Subject: Re: [M3devel] new target? :) Hi all: is interesting anyway but to write the web apps in c and good friends, hum ..., let me think if I can write a web server in less than 100 loc of Obliq and indeed one is written as an os extension, why would need any web page that renders c code, I think I would gain much more time writing myself rather than porting C (of course with its flavors) in Modula-3 or better if so in Obliq which is a mobile computing environment indeed, it makes me think, though the idea is good it makes little sense to make everything a client of a web browser rather than an instance of a structured platform, if they need only C why it has to be an instance of a web browser, it must be the browser makes better work for quick dynamic pages with C, or maybe another way of keeping it alive artificially, I mean what is the gain with that, if better runtime but, hum ...? but it needs javascript, hum ... another loss of time (much time in inefficient checking), Nokia had the same idea with Qt and JS in WRT, (very little RT information), but still Obliq scripts can be type inferenced in much more smoothly way. I think this battle of JavaScript and Obliq and Modula-3 vs others could be fight yet one time again but needs more time for tools to make that a reality, still the others think they can do it, it's very interesting. Thanks in advance --- El jue, 24/2/11, Dragi?a Duri? escribi?: De: Dragi?a Duri? Asunto: [M3devel] new target? :) Para: "m3devel" Fecha: jueves, 24 de febrero, 2011 14:16 http://code.google.com/chrome/nativeclient/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Thu Feb 24 23:15:51 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Thu, 24 Feb 2011 14:15:51 -0800 Subject: [M3devel] new target? :) In-Reply-To: References: , <758561.47161.qm@web29710.mail.ird.yahoo.com> Message-ID: <20110224221551.8B9271A2078@async.async.caltech.edu> Jay K writes: ... >Ultimately..you can tell what I'm going to say..a good solution is Modula-3= >-to-C=2C and then use whatever C compiler you want. >Including the nativeclient gcc. An iPhone/iPad/iPod cross compiler. Older s= I'm still curious why they abandoned this path. The first M3 compilers all generated C code. I read something about a Modula-2 compiler that was generating C code and therefore had problems with.. something. Also isn't the back-end m3cg actually quite an interesting program in its own right? You could imagine targeting other language front-ends to that back-end. I understand it is a good enough idea that RMS was very annoyed that m3cg even exists. Mika From jay.krell at cornell.edu Fri Feb 25 00:09:39 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 24 Feb 2011 23:09:39 +0000 Subject: [M3devel] new target? :) In-Reply-To: <20110224221551.8B9271A2078@async.async.caltech.edu> References: , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , <20110224221551.8B9271A2078@async.async.caltech.edu> Message-ID: I never used the system, but one can imagine and it has been said here: slower to compile less good debugging -- and also better debugging! However I still think the large gain in portability, and the actual *gain* in debuggability, would be very worthwhile. (There are pluses and minuses for debugging; you'd get to use gdb instead of m3gdb, and have C syntax). You'd gain portability and ports to various real, old, existing, new systems. e.g. NT/IA64, NT/AMD64, Android, NativeClient. You'd gain a "standard" distribution format of just source.tar.gz + possibly ./configure + make + make install. The that's a little more work. I think there's some issue with the metadata for garbage collection. I think we maybe store frame offsets for locals? We'd maybe have to put all locals in a struct or such? Impeding optimization? Similarly, nested functions would need some handling. You would also gain tremendously better exeption handling implementation. You'd generate C on NT/VMS/OSF1, and use __try/__except/__finally there. You'd generate C++ for all others and use try/catch/finally. You'd get stack walking on essentially all platforms. Except for the small number of platforms that use setjmp/longjmp for C++ exception handling. In some cases you'd get better codegen, as gcc is often not the best. RMS's problem is that normally the gcc frontends and backends are statically linked. frontends: C, C++, Java, Ada Therefore the frontends get infected by the GPL. Good in RMS opinion. m3cg is, technically, a gcc frontend, and therefore GPL. However its input is not human authored text. Its input is the output of a "real" front end -- m3front/cm3. By making the "call mechanism" be write to a file and run a process, the GPL doesn't cross the barrier, leaving cm3/m3front, and more importantly, m3core/libm3, not GPL. I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. - Jay > To: jay.krell at cornell.edu > Date: Thu, 24 Feb 2011 14:15:51 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] new target? :) > > Jay K writes: > ... > >Ultimately..you can tell what I'm going to say..a good solution is Modula-3= > >-to-C=2C and then use whatever C compiler you want. > >Including the nativeclient gcc. An iPhone/iPad/iPod cross compiler. Older s= > > I'm still curious why they abandoned this path. The first M3 compilers > all generated C code. I read something about a Modula-2 compiler that > was generating C code and therefore had problems with.. something. > > Also isn't the back-end m3cg actually quite an interesting program in > its own right? You could imagine targeting other language front-ends > to that back-end. I understand it is a good enough idea that RMS was > very annoyed that m3cg even exists. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 25 00:12:21 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Thu, 24 Feb 2011 23:12:21 +0000 (GMT) Subject: [M3devel] new target? :) In-Reply-To: <20110224221551.8B9271A2078@async.async.caltech.edu> Message-ID: <281816.90569.qm@web29709.mail.ird.yahoo.com> Hi all: yeah, but others had very distinct opinions in tis own right with two experimental projects: http://sandbox.parc.com/parctab/csl9501/node6.html and same house: http://www.cl.cam.ac.uk/research/origami/Origami1997c/ and prior another one which had experience with Trestle http://www.usenix.org/publications/library/proceedings/mobile93/kantarjiev.html So even with the space limitations the systems run in Modula-3 technically better than any other implementation for its computing environment. Perhaps that is the point he computing paradigm and specially performance is vastly superior than those days to run something like Modula-3, but maybe more easy for Obliq nowadays. Thanks in advance --- El jue, 24/2/11, Mika Nystrom escribi?: > De: Mika Nystrom > Asunto: Re: [M3devel] new target? :) > Para: "Jay K" > CC: m3devel at elegosoft.com > Fecha: jueves, 24 de febrero, 2011 17:15 > Jay K writes: > ... > >Ultimately..you can tell what I'm going to say..a good > solution is Modula-3= > >-to-C=2C and then use whatever C compiler you want. > >Including the nativeclient gcc. An iPhone/iPad/iPod > cross compiler. Older s= > > I'm still curious why they abandoned this path. The > first M3 compilers > all generated C code. I read something about a > Modula-2 compiler that > was generating C code and therefore had problems with.. > something. > > Also isn't the back-end m3cg actually quite an interesting > program in > its own right? You could imagine targeting other > language front-ends > to that back-end. I understand it is a good enough > idea that RMS was > very annoyed that m3cg even exists. > > Mika > From mika at async.caltech.edu Fri Feb 25 02:31:00 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Thu, 24 Feb 2011 17:31:00 -0800 Subject: [M3devel] new target? :) In-Reply-To: References: , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , <20110224221551.8B9271A2078@async.async.caltech.edu> Message-ID: <20110225013100.619CD1A2078@async.async.caltech.edu> Jay K writes: ... >m3cg is=2C technically=2C a gcc frontend=2C and therefore GPL. >However its input is not human authored text. >Its input is the output of a "real" front end -- m3front/cm3. >By making the "call mechanism" be write to a file and run a process=2C the = >GPL doesn't cross the barrier=2C >leaving cm3/m3front=2C and more importantly=2C m3core/libm3=2C not GPL. >=20 >=20 >I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. >=20 >=20 Good points all about C but I wonder about what is really lost. Also ...... Modula-3 is supposed to be portable to systems without C. You're supposed to be able to write an entire system in Modula-3. Ok generating C obviously doesn't break that feature but you might have to worry a bit about having too much C sneak into the runtime libraries. Well given that CM3 is released under what is more or less an MIT/BSD license, I think the problem is rather that Some Evil Software Company could write a completely non-free front-end and use m3cg to generate code... Mika From jay.krell at cornell.edu Fri Feb 25 02:55:52 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 25 Feb 2011 01:55:52 +0000 Subject: [M3devel] new target? :) In-Reply-To: <20110225013100.619CD1A2078@async.async.caltech.edu> References: , , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , <20110225013100.619CD1A2078@async.async.caltech.edu> Message-ID: When you write that system that doesn't have the support of any C compiler, sure, go ahead and write a backend just for it, and maybe a linker, and rewrite the small parts that depend on C and Posix in some sort of native way -- including writing all the underlying stuff. Getting Modula-3 working on that system will be a relatively small task, compared to whatever else is needed to build the system. Note for example, we can't e.g. open files in Modula-3, w/o underlying Win32 or Posix functionality. Everywhere that varies for Win32 vs. Posix would have to be revisited in this hypothetical system. This has little/no bearing on the vast vast vast vast majority of real world systems. The current Modula-3 system is NOT implemented entirely in Modula-3. It MUST depend on SOMETHING. So far, for the vast vast vast vast vast majority of systems, that something is written in C, and has a C runtime, compiler, and linker. Yes, I've heard of SPIN. You know..how are you actually going to network I/O or keyboard I/O or power management or video I/O or mouse I/O or sound I/O or preemptive multi-processor scheduling? You need device drivers. You need a processor-specific context switcher. i.e. you need a kernel and you need C AND/OR you need to do a lot of work. If you consider that our system is already forked along Win32 and Posix lines, you can consider these hypothetical systems a third (fourth, fifth, etc.) fork. A third (fourth) implementation of threading. A third implementation of file I/O. A third implementation of Trestle. etc. But this has little/no bearing how we should target existing real systems. You can say, not that "Modula-3 depends on C (runtime, compiler, linker)", but that "Modula-3 on Linux, NT, *BSD, Solaris, Darwin depends on C (runtime, compiler, linker)", and there's absolutely nothing wrong with that, as C and C++ are well supported on those systems and always will be. Increasing that dependency, on those systems, is fine. It would be nice to port the integrated backend to lots more systems, and to write our own linker. But not easy. And notice how the integrated backend also has the worst debugging story of the two/three options, currently. (I say three options because m3cg kind of has two modes, one is incomplete and currently disabled, the other requires m3gdb). One problem I see here, is that, you kind of sort of maybe forseeable want a backend that outputs Java or C#. C# particularly for Windows Phone 7/Nokia. Java kind of for Android, where Java is preferred over C/C++. i.e. there are systems that sort of lack C and C++. Not that they replace it with Modula-3 though. However this isn't super easy, like if you are to include the existing m3core/garbage collector. C# is optionally safe. I don't know if it would serve as a good target. (Optional safety imho makes C# among one of the closest languages to Modula-3. It is missing "compile to native", but that maybe is a good thing.) In addition, C# and Java do each compile down to one well documented "object" file format, so maybe skipping the text is probably sensible. Esp. since with C# there are in-proc libraries for codegen. More likely, if a Java/C# target were really needed, it'd include forking within m3core and replace the garbage collector. This is all highly specuative stuff. - Jay > To: jay.krell at cornell.edu > Date: Thu, 24 Feb 2011 17:31:00 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] new target? :) > > Jay K writes: > ... > >m3cg is=2C technically=2C a gcc frontend=2C and therefore GPL. > >However its input is not human authored text. > >Its input is the output of a "real" front end -- m3front/cm3. > >By making the "call mechanism" be write to a file and run a process=2C the = > >GPL doesn't cross the barrier=2C > >leaving cm3/m3front=2C and more importantly=2C m3core/libm3=2C not GPL. > >=20 > >=20 > >I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. > >=20 > >=20 > > Good points all about C but I wonder about what is really lost. Also ...... > Modula-3 is supposed to be portable to systems without C. You're supposed > to be able to write an entire system in Modula-3. Ok generating C obviously > doesn't break that feature but you might have to worry a bit about having > too much C sneak into the runtime libraries. > > Well given that CM3 is released under what is more or less an MIT/BSD license, > I think the problem is rather that Some Evil Software Company could write > a completely non-free front-end and use m3cg to generate code... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Fri Feb 25 03:35:28 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Thu, 24 Feb 2011 18:35:28 -0800 Subject: [M3devel] new target? :) In-Reply-To: References: , , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , <20110225013100.619CD1A2078@async.async.caltech.edu> Message-ID: <20110225023528.A615D1A2078@async.async.caltech.edu> In general I don't disagree with anything you say. On Linux, BSD, et al., we obviously use C for low-level system-specific stuff because the system call interfaces on those systems are documented for C and not for other languages. As for OSes in M3 I wasn't thinking as much about SPIN as about Topaz. Ok so it was Modula-2+, but it was the direct inspiration for the Modula-3 project: you're supposed to be able to write device drivers and such things in Modula-3. I agree though that targeting C doesn't directly weaken the case for M3 as a systems programming language. There are more similarities between C# and M3 than "unsafe". Look at the way methods are overridden in C#... (compared to Java). I also think almost everyone who had anything to do with M3 at DEC is now at Microsoft. Mika Jay K writes: >--_4e7f3be7-1b6e-44e9-92a2-cc8c4f6b0b8c_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >When you write that system that doesn't have the support of any C compiler= >=2C sure=2C go ahead and >write a backend just for it=2C and maybe a linker=2C and rewrite the small = >parts that depend on C and Posix in some >sort of native way -- including writing all the underlying stuff. >=20 >=20 >Getting Modula-3 working on that system will be a relatively small task=2C = >compared to whatever else > is needed to build the system. >=20 >Note for example=2C we can't e.g. open files in Modula-3=2C w/o underlying = >Win32 or Posix functionality. > Everywhere that varies for Win32 vs. Posix would have to be revisited in = >this hypothetical system. >=20 >=20 >This has little/no bearing on the vast vast vast vast majority of real worl= >d systems. >The current Modula-3 system is NOT implemented entirely in Modula-3. >It MUST depend on SOMETHING. >So far=2C for the vast vast vast vast vast majority of systems=2C that some= >thing is written in C=2C >and has a C runtime=2C compiler=2C and linker. >=20 >=20 >Yes=2C I've heard of SPIN. >=20 >=20 >You know..how are you actually going to network I/O or keyboard I/O or powe= >r management >or video I/O or mouse I/O or sound I/O or preemptive multi-processor schedu= >ling? >You need device drivers. You need a processor-specific context switcher. >i.e. you need a kernel and you need C AND/OR you need to do a lot of work. >=20 >=20 >If you consider that our system is already forked along Win32 and Posix lin= >es=2C you can consider >these hypothetical systems a third (fourth=2C fifth=2C etc.) fork. A third = >(fourth) implementation >of threading. A third implementation of file I/O. A third implementation of= > Trestle. etc. >=20 >=20 >But this has little/no bearing how we should target existing real systems. > You can say=2C not that "Modula-3 depends on C (runtime=2C compiler=2C lin= >ker)"=2C but that "Modula-3 on Linux=2C NT=2C *BSD=2C Solaris=2C Darwin >depends on C (runtime=2C compiler=2C linker)"=2C and there's absolutely not= >hing wrong with that=2C as C and C++ >are well supported on those systems and always will be. Increasing that dep= >endency=2C on those systems=2C is fine. >=20 >=20 >It would be nice to port the integrated backend to lots more systems=2C and= > to write our own linker. >But not easy. >And notice how the integrated backend also has the worst debugging story of= > the two/three options=2C currently. > (I say three options because m3cg kind of has two modes=2C one is incomple= >te and currently disabled=2C > the other requires m3gdb). >=20 >=20 >One problem I see here=2C is that=2C you kind of sort of maybe forseeable w= >ant a backend that outputs Java or C#. > C# particularly for Windows Phone 7/Nokia. > Java kind of for Android=2C where Java is preferred over C/C++. >=20 >=20 >i.e. there are systems that sort of lack C and C++. Not that they replace i= >t with Modula-3 though. >=20 >=20 >However this isn't super easy=2C like if you are to include the existing m3= >core/garbage collector. >C# is optionally safe. I don't know if it would serve as a good target. > (Optional safety imho makes C# among one of the closest languages to Modu= >la-3. It is > missing "compile to native"=2C but that maybe is a good thing.) >In addition=2C C# and Java do each compile down to one well documented "obj= >ect" file format=2C >so maybe skipping the text is probably sensible. Esp. since with C# there a= >re in-proc libraries for codegen. >=20 >=20 >More likely=2C if a Java/C# target were really needed=2C it'd include forki= >ng within m3core=20 >and replace the garbage collector. This is all highly specuative stuff. >=20 >=20 > - Jay > From jay.krell at cornell.edu Fri Feb 25 04:09:01 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 25 Feb 2011 03:09:01 +0000 Subject: [M3devel] new target? :) In-Reply-To: <20110225023528.A615D1A2078@async.async.caltech.edu> References: , , ,,<758561.47161.qm@web29710.mail.ird.yahoo.com>, ,,, , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , , , <20110225013100.619CD1A2078@async.async.caltech.edu>, , <20110225023528.A615D1A2078@async.async.caltech.edu> Message-ID: > I also think almost everyone who had anything to do with M3 at DEC is now at Microsoft Not just M3. M3 wasn't many people. Lots of people at Microsoft were at Digital. > Modula-3 project: you're supposed to be able to write device drivers > and such things in Modula-3. I agree though that targeting C doesn't Understood, agreed -- you can hypothetically write anything in Modula-3. The optional safety is fairly unique and important. Except for C# and Modula-3, all languages seem to be strictly safe or unsafe. I think, you can imagine, either an entire system in Modula-3, with m3core/libm3 coded "directly" in Modula-3 and simply statically or dynamically linked using "standard mechanism", or if there is still a user/kernel boundary, you either have to hack the compiler a bit, or resort to some assembly (possibly embedded in .c files). That is, calling the kernel is inevitably a special call sequence, and cm3 doesn't currently understand anything like that. About device drivers -- a critical piece is the runtime. Imagine writing an NT or Linux driver. Well, if you separate language from library...and limit language features to accomodate missing runtime...and retarget small parts of the runtime perhaps.. you know, there's a certain isomorphism? You can write almost anything in C or Modula-3 and most things will look very similar and map one-to-one. I think I read that Modula-3 can describe anything C can, except unions. You know, don't expect to have garbage collection in the kernel. Maybe not exception handling. Not necessarily the file I/O you are used to. But something. Not the threading you are used to. But something (at least on NT). But certainly you can interface Modula-3 to the NTkernel/Linux/*BSD runtimes, just like we interface to Win32/Posix. I'm not sure it'd buy much. I guess if you really like the syntax... It's another area where compiling to C would help. esp. for the exception handling on NT! __try/__finally are available in C on NT, usermode and kernel, and the implementation is much more efficient than what Modula-3 does currently. I really need to stop saying all this stuff though, either be quiet or do it... - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 25 15:20:48 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Fri, 25 Feb 2011 14:20:48 +0000 (GMT) Subject: [M3devel] new target? :) In-Reply-To: Message-ID: <669680.12232.qm@web29711.mail.ird.yahoo.com> Hi all: It was that there is some OS in Modula-2+ which run enterely in user space and you didn't want to get into the kernel but just for any option you would need to be unsafe (like machine starting, perhaps the OS managed software faults isolated from kernel space) that had the Runtime to time-out protect deadlocks, and it did it pretty much in a microkernel, just that company didn't feel sure o fsuch sophisticated system. Nowadays would be good if one could cooperate it in ARM VM for web platform developing, without the need for? GUI kernel, which indeed could crash the system if exploted, instead feels safer to run from "microkernel" traceable user space threads but making calls to a sort of video controller see below. Someone has the sources of it, just don't how complete, they did develop a UIMS and a window toolkit, plenty of tools to check: http://lists.cloud9.co.uk/pipermail/bbc-micro/2010-January/007795.html Apparently it had an Acorn C and Modula runtime: http://homepage.mac.com/dr2chase/resume.html Also about the machine and video chip: http://www.khantazi.org/Archives/MachineLst.html#A500 And its window toolkit: http://pages.sbcglobal.net/btlewis/resume-formatted.html Thanks in advance --- El jue, 24/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] new target? :) Para: "Mika Nystrom" CC: "m3devel" Fecha: jueves, 24 de febrero, 2011 22:09 ?> ?I also think almost everyone who had anything to do with M3 at DEC is now at Microsoft ? ? Not just M3. M3 wasn't many people. Lots of?people at Microsoft were?at Digital. ? ? > Modula-3 project: you're supposed to be able to write device drivers > and such things in Modula-3. I agree though that targeting C doesn't ? Understood, agreed -- you can hypothetically write anything in Modula-3. The optional safety is fairly unique and important. Except for C# and Modula-3, all languages seem to be strictly safe or unsafe. ? ? I think, you can imagine, either an entire system in Modula-3, with m3core/libm3 coded "directly" in Modula-3 and simply statically or dynamically linked using "standard mechanism", or if there is still a user/kernel boundary, you either have to hack the compiler a bit, or resort to some assembly (possibly embedded in .c files). That is, calling the kernel is inevitably a special call sequence, and cm3 doesn't currently understand anything like that. ? ? About device drivers -- a critical piece is the runtime. Imagine writing an NT or Linux driver. Well, if you separate language from library...and limit language features to accomodate?missing runtime...and retarget small parts of the runtime perhaps.. you know, there's a certain isomorphism? You can write almost anything in C or Modula-3 and most things will look very?similar and map one-to-one. I think I read that Modula-3 can describe anything C can, except unions. ? ? You know, don't expect to have garbage collection in the kernel. Maybe not exception handling. Not necessarily the file I/O you are used to. But something. Not the threading you are used to. But something (at least on NT). But certainly you can interface Modula-3 to the NTkernel/Linux/*BSD runtimes, just like we? interface to Win32/Posix. ? ? I'm not sure it'd buy much. I guess if you really like the syntax... ? ? It's another area where compiling to C would help. esp. for the exception handling on NT! __try/__finally are available in C on NT, usermode and kernel, and the implementation is much more efficient than what Modula-3 does currently. I really need to stop saying all this stuff though, either be quiet or do it... ? ? ?- Jay ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Feb 25 16:08:15 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 25 Feb 2011 10:08:15 -0500 Subject: [M3devel] new target? :) In-Reply-To: References: , , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , <20110225013100.619CD1A2078@async.async.caltech.edu> Message-ID: <04F51416-6EAC-4BBD-8C4B-623245437ECD@cs.purdue.edu> Just to follow up briefly as I have lots of other things on my plate. One of the reasons for targeting the gcc-based backend was to support easy generation of *source*-level debugging information. It is important that the debugger see types as Modula-3 sees them, rather than as C types generated form the Modula-3. Now, I agree that debugging the C-translated types is probably better than nothing. On Feb 24, 2011, at 8:55 PM, Jay K wrote: > When you write that system that doesn't have the support of any C compiler, sure, go ahead and > write a backend just for it, and maybe a linker, and rewrite the small parts that depend on C and Posix in some > sort of native way -- including writing all the underlying stuff. > > > Getting Modula-3 working on that system will be a relatively small task, compared to whatever else > is needed to build the system. > > Note for example, we can't e.g. open files in Modula-3, w/o underlying Win32 or Posix functionality. > Everywhere that varies for Win32 vs. Posix would have to be revisited in this hypothetical system. > > > This has little/no bearing on the vast vast vast vast majority of real world systems. > The current Modula-3 system is NOT implemented entirely in Modula-3. > It MUST depend on SOMETHING. > So far, for the vast vast vast vast vast majority of systems, that something is written in C, > and has a C runtime, compiler, and linker. > > > Yes, I've heard of SPIN. > > > You know..how are you actually going to network I/O or keyboard I/O or power management > or video I/O or mouse I/O or sound I/O or preemptive multi-processor scheduling? > You need device drivers. You need a processor-specific context switcher. > i.e. you need a kernel and you need C AND/OR you need to do a lot of work. > > > If you consider that our system is already forked along Win32 and Posix lines, you can consider > these hypothetical systems a third (fourth, fifth, etc.) fork. A third (fourth) implementation > of threading. A third implementation of file I/O. A third implementation of Trestle. etc. > > > But this has little/no bearing how we should target existing real systems. > You can say, not that "Modula-3 depends on C (runtime, compiler, linker)", but that "Modula-3 on Linux, NT, *BSD, Solaris, Darwin > depends on C (runtime, compiler, linker)", and there's absolutely nothing wrong with that, as C and C++ > are well supported on those systems and always will be. Increasing that dependency, on those systems, is fine. > > > It would be nice to port the integrated backend to lots more systems, and to write our own linker. > But not easy. > And notice how the integrated backend also has the worst debugging story of the two/three options, currently. > (I say three options because m3cg kind of has two modes, one is incomplete and currently disabled, > the other requires m3gdb). > > > One problem I see here, is that, you kind of sort of maybe forseeable want a backend that outputs Java or C#. > C# particularly for Windows Phone 7/Nokia. > Java kind of for Android, where Java is preferred over C/C++. > > > i.e. there are systems that sort of lack C and C++. Not that they replace it with Modula-3 though. > > > However this isn't super easy, like if you are to include the existing m3core/garbage collector. > C# is optionally safe. I don't know if it would serve as a good target. > (Optional safety imho makes C# among one of the closest languages to Modula-3. It is > missing "compile to native", but that maybe is a good thing.) > In addition, C# and Java do each compile down to one well documented "object" file format, > so maybe skipping the text is probably sensible. Esp. since with C# there are in-proc libraries for codegen. > > > More likely, if a Java/C# target were really needed, it'd include forking within m3core > and replace the garbage collector. This is all highly specuative stuff. > > > - Jay > > > > To: jay.krell at cornell.edu > > Date: Thu, 24 Feb 2011 17:31:00 -0800 > > From: mika at async.caltech.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] new target? :) > > > > Jay K writes: > > ... > > >m3cg is=2C technically=2C a gcc frontend=2C and therefore GPL. > > >However its input is not human authored text. > > >Its input is the output of a "real" front end -- m3front/cm3. > > >By making the "call mechanism" be write to a file and run a process=2C the = > > >GPL doesn't cross the barrier=2C > > >leaving cm3/m3front=2C and more importantly=2C m3core/libm3=2C not GPL. > > >=20 > > >=20 > > >I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. > > >=20 > > >=20 > > > > Good points all about C but I wonder about what is really lost. Also ...... > > Modula-3 is supposed to be portable to systems without C. You're supposed > > to be able to write an entire system in Modula-3. Ok generating C obviously > > doesn't break that feature but you might have to worry a bit about having > > too much C sneak into the runtime libraries. > > > > Well given that CM3 is released under what is more or less an MIT/BSD license, > > I think the problem is rather that Some Evil Software Company could write > > a completely non-free front-end and use m3cg to generate code... > > > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Feb 25 19:50:12 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 25 Feb 2011 18:50:12 +0000 Subject: [M3devel] new target? :) In-Reply-To: <04F51416-6EAC-4BBD-8C4B-623245437ECD@cs.purdue.edu> References: , , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , <20110225013100.619CD1A2078@async.async.caltech.edu> , <04F51416-6EAC-4BBD-8C4B-623245437ECD@cs.purdue.edu> Message-ID: #line directives should help source debugging. I agree the types would be a bit off. For example arrays, objects, texts, would have internals showing, and texts might take extra steps to see what you really want. - Jay Subject: Re: [M3devel] new target? :) From: hosking at cs.purdue.edu Date: Fri, 25 Feb 2011 10:08:15 -0500 CC: mika at async.caltech.edu; m3devel at elegosoft.com To: jay.krell at cornell.edu Just to follow up briefly as I have lots of other things on my plate.One of the reasons for targeting the gcc-based backend was to support easy generation of *source*-level debugging information. It is important that the debugger see types as Modula-3 sees them, rather than as C types generated form the Modula-3. Now, I agree that debugging the C-translated types is probably better than nothing. On Feb 24, 2011, at 8:55 PM, Jay K wrote:When you write that system that doesn't have the support of any C compiler, sure, go ahead and write a backend just for it, and maybe a linker, and rewrite the small parts that depend on C and Posix in some sort of native way -- including writing all the underlying stuff. Getting Modula-3 working on that system will be a relatively small task, compared to whatever else is needed to build the system. Note for example, we can't e.g. open files in Modula-3, w/o underlying Win32 or Posix functionality. Everywhere that varies for Win32 vs. Posix would have to be revisited in this hypothetical system. This has little/no bearing on the vast vast vast vast majority of real world systems. The current Modula-3 system is NOT implemented entirely in Modula-3. It MUST depend on SOMETHING. So far, for the vast vast vast vast vast majority of systems, that something is written in C, and has a C runtime, compiler, and linker. Yes, I've heard of SPIN. You know..how are you actually going to network I/O or keyboard I/O or power management or video I/O or mouse I/O or sound I/O or preemptive multi-processor scheduling? You need device drivers. You need a processor-specific context switcher. i.e. you need a kernel and you need C AND/OR you need to do a lot of work. If you consider that our system is already forked along Win32 and Posix lines, you can consider these hypothetical systems a third (fourth, fifth, etc.) fork. A third (fourth) implementation of threading. A third implementation of file I/O. A third implementation of Trestle. etc. But this has little/no bearing how we should target existing real systems. You can say, not that "Modula-3 depends on C (runtime, compiler, linker)", but that "Modula-3 on Linux, NT, *BSD, Solaris, Darwin depends on C (runtime, compiler, linker)", and there's absolutely nothing wrong with that, as C and C++ are well supported on those systems and always will be. Increasing that dependency, on those systems, is fine. It would be nice to port the integrated backend to lots more systems, and to write our own linker. But not easy. And notice how the integrated backend also has the worst debugging story of the two/three options, currently. (I say three options because m3cg kind of has two modes, one is incomplete and currently disabled, the other requires m3gdb). One problem I see here, is that, you kind of sort of maybe forseeable want a backend that outputs Java or C#. C# particularly for Windows Phone 7/Nokia. Java kind of for Android, where Java is preferred over C/C++. i.e. there are systems that sort of lack C and C++. Not that they replace it with Modula-3 though. However this isn't super easy, like if you are to include the existing m3core/garbage collector. C# is optionally safe. I don't know if it would serve as a good target. (Optional safety imho makes C# among one of the closest languages to Modula-3. It is missing "compile to native", but that maybe is a good thing.) In addition, C# and Java do each compile down to one well documented "object" file format, so maybe skipping the text is probably sensible. Esp. since with C# there are in-proc libraries for codegen. More likely, if a Java/C# target were really needed, it'd include forking within m3core and replace the garbage collector. This is all highly specuative stuff. - Jay > To: jay.krell at cornell.edu > Date: Thu, 24 Feb 2011 17:31:00 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] new target? :) > > Jay K writes: > ... > >m3cg is=2C technically=2C a gcc frontend=2C and therefore GPL. > >However its input is not human authored text. > >Its input is the output of a "real" front end -- m3front/cm3. > >By making the "call mechanism" be write to a file and run a process=2C the = > >GPL doesn't cross the barrier=2C > >leaving cm3/m3front=2C and more importantly=2C m3core/libm3=2C not GPL. > >=20 > >=20 > >I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. > >=20 > >=20 > > Good points all about C but I wonder about what is really lost. Also ...... > Modula-3 is supposed to be portable to systems without C. You're supposed > to be able to write an entire system in Modula-3. Ok generating C obviously > doesn't break that feature but you might have to worry a bit about having > too much C sneak into the runtime libraries. > > Well given that CM3 is released under what is more or less an MIT/BSD license, > I think the problem is rather that Some Evil Software Company could write > a completely non-free front-end and use m3cg to generate code... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Fri Feb 25 20:08:49 2011 From: dragisha at m3w.org (dragisha at m3w.org) Date: Fri, 25 Feb 2011 20:08:49 +0100 Subject: [M3devel] new target? :) In-Reply-To: References: Message-ID: <775012e5aaa51d171471059dac6ab8be@rstel.net> I am on the road so this is easiest way to amend this... As I see, some people steered this discussion to web dev... One of things I made some time ago is verry complete FastCGI support for Modula-3, and I will contribute it one of these days, I just need more time to go through everything and ensure it works on all platforms I have available now (OSX, Linux, Windows...)... FastCGI is server side, and there is no question about Modula-3 perfectness for anything server. This link I gave is a way for us to get our perfect platform client side. No oblets (IIRC), no nothing... Pure Modula-3. On Thu, 24 Feb 2011 20:16:58 +0100, Dragi?a Duri? wrote: > http://code.google.com/chrome/nativeclient/ [1] > > Links: > ------ > [1] http://code.google.com/chrome/nativeclient/ From rcolebur at SCIRES.COM Sat Feb 26 10:47:02 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sat, 26 Feb 2011 04:47:02 -0500 Subject: [M3devel] results of threadtest program on Windows7 Message-ID: Mika: I've finally managed to get cm3 rebuilt on Windows 7 again. So, I ran your threadtest program. Here are the results. Note the "..." is where I cut out a bunch of the repeating "ERROR FApply" messages. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe Writing file...done Creating read threads...done Creating fork threads...done Creating alloc threads...done Creating lock threads...done running...printing oldest/median age/newest ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. . . . *** *** runtime error: *** An enumeration or subrange value was out of range. *** file "..\src\Main.m3", line 340 *** laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. . . . laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. . . . ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. Stack trace: FP PC Procedure --------- --------- ------------------------------- 0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 Regards, Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 26 11:25:51 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 26 Feb 2011 10:25:51 +0000 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: Message-ID: The FApply problems are because the wants to run "sleep". I've checked in a sleep subset including a Win32 port (untested, but it is very simple). We should get that available to the test and have it run m3sleep instead of sleep. Or you can manually copy this m3sleep.exe to %windir%\system32\sleep.exe or such. Or put Cygwin sleep.exe in %PATH%, but that is overkill. Probably the thread tester could implement a -sleep flag and just call itself with that.. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Sat, 26 Feb 2011 04:47:02 -0500 Subject: [M3devel] results of threadtest program on Windows7 Mika: I?ve finally managed to get cm3 rebuilt on Windows 7 again. So, I ran your threadtest program. Here are the results. Note the ??? is where I cut out a bunch of the repeating ?ERROR FApply? messages. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exeWriting file...doneCreating read threads...doneCreating fork threads...doneCreating alloc threads...doneCreating lock threads...donerunning...printing oldest/median age/newestERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....****** runtime error:*** An enumeration or subrange value was out of range.*** file "..\src\Main.m3", line 340*** laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.Stack trace: FP PC Procedure--------- --------- ------------------------------- 0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 Regards,Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 26 11:42:47 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 26 Feb 2011 10:42:47 +0000 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: , Message-ID: On Darwin, we often get back EBUSY from pthread_mutex_delete. And often: *** *** runtime error: *** An array subscript was out of range. *** file "../src/runtime/common/RTCollector.m3", line 418 *** Abort trap I'll try to look into this more/sooner... - Jay From: jay.krell at cornell.edu To: rcolebur at scires.com; m3devel at elegosoft.com Date: Sat, 26 Feb 2011 10:25:51 +0000 Subject: Re: [M3devel] results of threadtest program on Windows7 The FApply problems are because the wants to run "sleep". I've checked in a sleep subset including a Win32 port (untested, but it is very simple). We should get that available to the test and have it run m3sleep instead of sleep. Or you can manually copy this m3sleep.exe to %windir%\system32\sleep.exe or such. Or put Cygwin sleep.exe in %PATH%, but that is overkill. Probably the thread tester could implement a -sleep flag and just call itself with that.. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Sat, 26 Feb 2011 04:47:02 -0500 Subject: [M3devel] results of threadtest program on Windows7 Mika: I?ve finally managed to get cm3 rebuilt on Windows 7 again. So, I ran your threadtest program. Here are the results. Note the ??? is where I cut out a bunch of the repeating ?ERROR FApply? messages. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exeWriting file...doneCreating read threads...doneCreating fork threads...doneCreating alloc threads...doneCreating lock threads...donerunning...printing oldest/median age/newestERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....****** runtime error:*** An enumeration or subrange value was out of range.*** file "..\src\Main.m3", line 340*** laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.Stack trace: FP PC Procedure--------- --------- ------------------------------- 0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 Regards,Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 26 18:55:11 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 26 Feb 2011 09:55:11 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: Message-ID: <20110226175511.3F7D21A2078@async.async.caltech.edu> Hi Randy, Hm yes it looks like my Windows programming skills leave something to be desired. You can run the thread tester while skipping a test as follows threadtest -tests ALL,-fork (for instance) if you just run threadtest -sadfassdaf it'll print the tests that are available. As it happens, I just had to upgrade my windows 2000 system to windows 7. Can you give me a very brief description of what you did to install Modula-3 on this system? Mika "Coleburn, Randy" writes: >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/plain; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > >Mika: > >I've finally managed to get cm3 rebuilt on Windows 7 again. > >So, I ran your threadtest program. > >Here are the results. Note the "..." is where I cut out a bunch of the rep= >eating "ERROR FApply" messages. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >*** >*** runtime error: >*** An enumeration or subrange value was out of range. >*** file "..\src\Main.m3", line 340 >*** > >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >Stack trace: > FP PC Procedure >--------- --------- ------------------------------- >0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 > >Regards, >Randy Coleburn > >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/html; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > From mika at async.caltech.edu Sat Feb 26 19:10:44 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 26 Feb 2011 10:10:44 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: Message-ID: <20110226181044.A44DD1A2078@async.async.caltech.edu> Hi Randy, I just implemented Jay's suggestion for removing the dependency on the external program "sleep". You can CVS update and try again now. Mika From jay.krell at cornell.edu Sat Feb 26 22:13:27 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 26 Feb 2011 21:13:27 +0000 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: <20110226181044.A44DD1A2078@async.async.caltech.edu> References: , <20110226181044.A44DD1A2078@async.async.caltech.edu> Message-ID: Thanks. jbook2:thread jay$ ./I386_DARWIN/threadtest Writing file...done Creating read threads...done Creating fork threads...done Creating alloc threads...done Creating lock threads...done running...printing oldest/median age/newest ..........laziest thread is 1/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 1/0/0 lock 1/1/1) ..........laziest thread is 1/0/0 (tests: read 1/0/0 fork 1/1/0 alloc 0/0/0 lock 0/0/0) those aren't errors, right? I can reproduce problems on Darwin -- assertion failures, errors in ThreadPThreadC.c like EBUSY and I think EINVAL. EBUSY is from pthread_mutex_destroy, which means we are freeing a mutex while it is locked, which means there is a problem. Such as the garbage collector collecting non-garbage. It ccould be something else though. I'm trying with user threads. User threads do print the above. But so far has a longer runtime than with pthreads, hasn't hit any SIGSEGV or assertion failure or anything yet. I do wonder .. oh, and then w/ user threads: All tests complete. Congratulations. I do wonder about scheduling order and when the preemptions occur. For example user threads does offer Disable/EnableScheduling. Maybe the timeslices vary a lot and userthreads preempt a lot less? Maybe we can try w/ tuning the user thread timeslice to match whatever pthread uses on whatever platform ( there likely no guarantee or documentation, but for experimental purposes we can likely figure it out experimentally). It will be interesting to see the results on Windows too. I'll probably try that right away. - Jay > To: rcolebur at SCIRES.COM > Date: Sat, 26 Feb 2011 10:10:44 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] results of threadtest program on Windows7 > > > Hi Randy, > > I just implemented Jay's suggestion for removing the dependency on the > external program "sleep". You can CVS update and try again now. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 27 00:10:55 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 26 Feb 2011 15:10:55 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: , <20110226181044.A44DD1A2078@async.async.caltech.edu> Message-ID: <20110226231055.932CA1A2078@async.async.caltech.edu> Jay K writes: >--_fe2b6a60-67dc-46f1-b844-2023fbc40258_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Thanks. > >jbook2:thread jay$ ./I386_DARWIN/threadtest=20 >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >..........laziest thread is 1/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 1/0/0= > lock 1/1/1) >..........laziest thread is 1/0/0 (tests: read 1/0/0 fork 1/1/0 alloc 0/0/0= > lock 0/0/0) > >those aren't errors=2C right? No those are just the noral diagnostics. The numbers are how many seconds the various threads are lagging. User threads do have a "more unfair" scheduler than you can expect with Darwin (see comments in the code)... You can change the number of threads etc but I find that the defaults seem to catch stuff. Lowering the number of threads per test from default 3 to "-n 2" simplifies debugging, perhaps. (More comments in the code.) Mika > >I can reproduce problems on Darwin -- assertion failures=2C errors in Threa= >dPThreadC.c like EBUSY and I think EINVAL. > EBUSY is from pthread_mutex_destroy=2C which means we are freeing a mutex= > while it is locked=2C which means > there is a problem. Such as the garbage collector collecting non-garbage= >. It ccould be something else though. > >I'm trying with user threads. User threads do print the above. But so far h= >as a longer runtime than with pthreads=2C > hasn't hit any SIGSEGV or assertion failure or anything yet. > >I do wonder .. oh=2C and then w/ user threads: > >All tests complete. Congratulations. > >I do wonder about scheduling order and when the preemptions occur. >For example user threads does offer Disable/EnableScheduling. >Maybe the timeslices vary a lot and userthreads preempt a lot less? >Maybe we can try w/ tuning the user thread timeslice to match whatever pthr= >ead uses on whatever platform ( >there likely no guarantee or documentation=2C but for experimental purposes= > we can likely figure it out >experimentally). > >It will be interesting to see the results on Windows too. I'll probably try= > that right away. > > - Jay From rcolebur at SCIRES.COM Sun Feb 27 20:09:08 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sun, 27 Feb 2011 14:09:08 -0500 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: <20110226175511.3F7D21A2078@async.async.caltech.edu> References: <20110226175511.3F7D21A2078@async.async.caltech.edu> Message-ID: Mika: Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. Here is what happens now when I run your threadtest program on Windows 7. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-fork Writing file...done Creating read threads...done Creating nxread threads...done Creating tryexcept threads...done Creating forktoomuch threads...done Creating alloc threads...done Creating creat threads...done Creating lock threads...done running...printing oldest/median age/newest *** *** runtime error: *** An array subscript was out of range. *** file "..\src\runtime\common\RTCollector.m3", line 418 *** *** *** runtime error: *** <*ASSERT*> failed. *** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 *** The last error repeats ad infinitum until I press CNTRL-C to abort. I'll send more info on the Windows install of Modula3 in a subsequent post. Regards, Randy Coleburn -----Original Message----- From: Mika Nystrom [mailto:mika at async.caltech.edu] Sent: Saturday, February 26, 2011 12:55 PM To: Coleburn, Randy Cc: m3devel at elegosoft.com Subject: Re: [M3devel] results of threadtest program on Windows7 Hi Randy, Hm yes it looks like my Windows programming skills leave something to be desired. You can run the thread tester while skipping a test as follows threadtest -tests ALL,-fork (for instance) if you just run threadtest -sadfassdaf it'll print the tests that are available. As it happens, I just had to upgrade my windows 2000 system to windows 7. Can you give me a very brief description of what you did to install Modula-3 on this system? Mika "Coleburn, Randy" writes: >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/plain; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > >Mika: > >I've finally managed to get cm3 rebuilt on Windows 7 again. > >So, I ran your threadtest program. > >Here are the results. Note the "..." is where I cut out a bunch of the rep= >eating "ERROR FApply" messages. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >*** >*** runtime error: >*** An enumeration or subrange value was out of range. >*** file "..\src\Main.m3", line 340 >*** > >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >Stack trace: > FP PC Procedure >--------- --------- ------------------------------- >0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 > >Regards, >Randy Coleburn > >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/html; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > From rcolebur at SCIRES.COM Sun Feb 27 20:45:29 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sun, 27 Feb 2011 14:45:29 -0500 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: <20110226175511.3F7D21A2078@async.async.caltech.edu> Message-ID: Mika et al: Thought I would try something else. I took the sources of your thread test program to an older XP machine that has CM3 circa August 2008. This is the machine and implementation I used when building a major project I did a couple years back. The thread test program does indeed build on this old system, but when I run it, I get different results than with the latest HEAD branch code. After it prints "running...printing oldest/median age/newest", on the next line I get two periods ".." and now the program seems hung. I'll let it "run" for a few more minutes to see if anything else happens before killing it. At least we don't get the subscript and assertion failures on this older CM3 platform. Regards, Randy Coleburn -----Original Message----- From: Coleburn, Randy Sent: Sunday, February 27, 2011 2:09 PM To: m3devel at elegosoft.com Subject: Re: [M3devel] results of threadtest program on Windows7 Mika: Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. Here is what happens now when I run your threadtest program on Windows 7. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-fork Writing file...done Creating read threads...done Creating nxread threads...done Creating tryexcept threads...done Creating forktoomuch threads...done Creating alloc threads...done Creating creat threads...done Creating lock threads...done running...printing oldest/median age/newest *** *** runtime error: *** An array subscript was out of range. *** file "..\src\runtime\common\RTCollector.m3", line 418 *** *** *** runtime error: *** <*ASSERT*> failed. *** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 *** The last error repeats ad infinitum until I press CNTRL-C to abort. I'll send more info on the Windows install of Modula3 in a subsequent post. Regards, Randy Coleburn -----Original Message----- From: Mika Nystrom [mailto:mika at async.caltech.edu] Sent: Saturday, February 26, 2011 12:55 PM To: Coleburn, Randy Cc: m3devel at elegosoft.com Subject: Re: [M3devel] results of threadtest program on Windows7 Hi Randy, Hm yes it looks like my Windows programming skills leave something to be desired. You can run the thread tester while skipping a test as follows threadtest -tests ALL,-fork (for instance) if you just run threadtest -sadfassdaf it'll print the tests that are available. As it happens, I just had to upgrade my windows 2000 system to windows 7. Can you give me a very brief description of what you did to install Modula-3 on this system? Mika "Coleburn, Randy" writes: >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/plain; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > >Mika: > >I've finally managed to get cm3 rebuilt on Windows 7 again. > >So, I ran your threadtest program. > >Here are the results. Note the "..." is where I cut out a bunch of the rep= >eating "ERROR FApply" messages. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >*** >*** runtime error: >*** An enumeration or subrange value was out of range. >*** file "..\src\Main.m3", line 340 >*** > >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >Stack trace: > FP PC Procedure >--------- --------- ------------------------------- >0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 > >Regards, >Randy Coleburn > >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/html; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > From mika at async.caltech.edu Sun Feb 27 21:29:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 27 Feb 2011 12:29:50 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: <20110226175511.3F7D21A2078@async.async.caltech.edu> Message-ID: <20110227202950.46F3B1A2078@async.async.caltech.edu> Hi Randy, You can try it with -tests POSIX as well. I find on user threads it runs very slowly (but it does run) because of how unfair the thread scheduler is. Next step might be to whittle down the tests and see if you can get a failure with a single test running and -n 2. That would likely be the simplest scenario to start further debugging from. Mika "Coleburn, Randy" writes: >Mika et al: > >Thought I would try something else. > >I took the sources of your thread test program to an older XP machine that = >has CM3 circa August 2008. This is the machine and implementation I used w= >hen building a major project I did a couple years back. > >The thread test program does indeed build on this old system, but when I ru= >n it, I get different results than with the latest HEAD branch code. =20 > >After it prints "running...printing oldest/median age/newest", on the next = >line I get two periods ".." and now the program seems hung. I'll let it "r= >un" for a few more minutes to see if anything else happens before killing i= >t. > >At least we don't get the subscript and assertion failures on this older CM= >3 platform. > >Regards, >Randy Coleburn > > >-----Original Message----- >From: Coleburn, Randy=20 >Sent: Sunday, February 27, 2011 2:09 PM >To: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7 > >Mika: > >Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. > >Here is what happens now when I run your threadtest program on Windows 7. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-for= >k >Writing file...done >Creating read threads...done >Creating nxread threads...done >Creating tryexcept threads...done >Creating forktoomuch threads...done >Creating alloc threads...done >Creating creat threads...done >Creating lock threads...done >running...printing oldest/median age/newest > > >*** >*** runtime error: >*** An array subscript was out of range. >*** file "..\src\runtime\common\RTCollector.m3", line 418 >*** > > > >*** >*** runtime error: >*** <*ASSERT*> failed. >*** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 >*** > >The last error repeats ad infinitum until I press CNTRL-C to abort. > >I'll send more info on the Windows install of Modula3 in a subsequent post. > >Regards, >Randy Coleburn > >-----Original Message----- >From: Mika Nystrom [mailto:mika at async.caltech.edu]=20 >Sent: Saturday, February 26, 2011 12:55 PM >To: Coleburn, Randy >Cc: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7=20 > >Hi Randy, > >Hm yes it looks like my Windows programming skills leave something >to be desired. > >You can run the thread tester while skipping a test as follows > > threadtest -tests ALL,-fork > >(for instance) > >if you just run=20 > > threadtest -sadfassdaf > >it'll print the tests that are available. > >As it happens, I just had to upgrade my windows 2000 system to windows 7. >Can you give me a very brief description of what you did to install Modula-= >3 >on this system? > > Mika > >"Coleburn, Randy" writes: >>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>Content-Type: text/plain; charset=3D"us-ascii" >>Content-Transfer-Encoding: quoted-printable >> >>Mika: >> >>I've finally managed to get cm3 rebuilt on Windows 7 again. >> >>So, I ran your threadtest program. >> >>Here are the results. Note the "..." is where I cut out a bunch of the re= >p=3D >>eating "ERROR FApply" messages. >> >>C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >>Writing file...done >>Creating read threads...done >>Creating fork threads...done >>Creating alloc threads...done >>Creating lock threads...done >>running...printing oldest/median age/newest >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>*** >>*** runtime error: >>*** An enumeration or subrange value was out of range. >>*** file "..\src\Main.m3", line 340 >>*** >> >>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D2: The syste= >m c=3D >>annot find the file specified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D2: The syste= >m c=3D >>annot find the file specified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>Stack trace: >> FP PC Procedure >>--------- --------- ------------------------------- >>0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >>0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 >> >>Regards, >>Randy Coleburn >> >>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>Content-Type: text/html; charset=3D"us-ascii" >>Content-Transfer-Encoding: quoted-printable >> From rcolebur at SCIRES.COM Sun Feb 27 22:03:13 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sun, 27 Feb 2011 16:03:13 -0500 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: <20110227202950.46F3B1A2078@async.async.caltech.edu> References: <20110226175511.3F7D21A2078@async.async.caltech.edu> <20110227202950.46F3B1A2078@async.async.caltech.edu> Message-ID: Mika: No change with "-tests POSIX". Interesting twist: On Windows 7, I thought I'd see what the command line options are, and I typed "threadtest -help" rather than reading the code. First time, it produced what appears to be a NIL deref crash. Then, I tried it again and it ran to completion. Something seems non-deterministic here. See below. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe -help Writing file...done Creating read threads...done Creating fork threads...done Creating alloc threads...done Creating lock threads...done running...printing oldest/median age/newest . *** *** runtime error: *** Attempt to reference an illegal memory location. *** pc = 0x77762262 *** Stack trace: FP PC Procedure --------- --------- ------------------------------- 0xcdf998 0x130351b SystemError + 0x64 in ..\src\runtime\NT386\RTSignal.m3 0xcdf9c0 0x77762262 0xcdf9d8 0x12e83b7 LockMutex + 0x4f in ..\src\thread\WIN32\ThreadWin32.m3 0xcdfa00 0x12c7b08 GetChar + 0x28 in ..\src\rw\Rd.m3 0xcdfb38 0x12c12e3 RApply + 0xd3 in ..\src\Main.m3 0xcdfb74 0x12e971f ThreadBase + 0x254 in ..\src\thread\WIN32\ThreadWin32.m3 0xcdfb80 0x76543677 0xcdfbc0 0x77779f02 ......... ......... ... more frames ... C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe -help Writing file...done Creating read threads...done Creating fork threads...done Creating alloc threads...done Creating lock threads...done running...printing oldest/median age/newest ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) All tests complete. Congratulations. Regards, Randy Coleburn -----Original Message----- From: Mika Nystrom [mailto:mika at async.caltech.edu] Sent: Sunday, February 27, 2011 3:30 PM To: Coleburn, Randy Cc: m3devel at elegosoft.com Subject: Re: [M3devel] results of threadtest program on Windows7 Hi Randy, You can try it with -tests POSIX as well. I find on user threads it runs very slowly (but it does run) because of how unfair the thread scheduler is. Next step might be to whittle down the tests and see if you can get a failure with a single test running and -n 2. That would likely be the simplest scenario to start further debugging from. Mika "Coleburn, Randy" writes: >Mika et al: > >Thought I would try something else. > >I took the sources of your thread test program to an older XP machine that = >has CM3 circa August 2008. This is the machine and implementation I used w= >hen building a major project I did a couple years back. > >The thread test program does indeed build on this old system, but when I ru= >n it, I get different results than with the latest HEAD branch code. =20 > >After it prints "running...printing oldest/median age/newest", on the next = >line I get two periods ".." and now the program seems hung. I'll let it "r= >un" for a few more minutes to see if anything else happens before killing i= >t. > >At least we don't get the subscript and assertion failures on this older CM= >3 platform. > >Regards, >Randy Coleburn > > >-----Original Message----- >From: Coleburn, Randy=20 >Sent: Sunday, February 27, 2011 2:09 PM >To: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7 > >Mika: > >Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. > >Here is what happens now when I run your threadtest program on Windows 7. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-for= >k >Writing file...done >Creating read threads...done >Creating nxread threads...done >Creating tryexcept threads...done >Creating forktoomuch threads...done >Creating alloc threads...done >Creating creat threads...done >Creating lock threads...done >running...printing oldest/median age/newest > > >*** >*** runtime error: >*** An array subscript was out of range. >*** file "..\src\runtime\common\RTCollector.m3", line 418 >*** > > > >*** >*** runtime error: >*** <*ASSERT*> failed. >*** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 >*** > >The last error repeats ad infinitum until I press CNTRL-C to abort. > >I'll send more info on the Windows install of Modula3 in a subsequent post. > >Regards, >Randy Coleburn > >-----Original Message----- >From: Mika Nystrom [mailto:mika at async.caltech.edu]=20 >Sent: Saturday, February 26, 2011 12:55 PM >To: Coleburn, Randy >Cc: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7=20 > >Hi Randy, > >Hm yes it looks like my Windows programming skills leave something >to be desired. > >You can run the thread tester while skipping a test as follows > > threadtest -tests ALL,-fork > >(for instance) > >if you just run=20 > > threadtest -sadfassdaf > >it'll print the tests that are available. > >As it happens, I just had to upgrade my windows 2000 system to windows 7. >Can you give me a very brief description of what you did to install Modula-= >3 >on this system? > > Mika > >"Coleburn, Randy" writes: >>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>Content-Type: text/plain; charset=3D"us-ascii" >>Content-Transfer-Encoding: quoted-printable >> >>Mika: >> >>I've finally managed to get cm3 rebuilt on Windows 7 again. >> >>So, I ran your threadtest program. >> >>Here are the results. Note the "..." is where I cut out a bunch of the re= >p=3D >>eating "ERROR FApply" messages. >> >>C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >>Writing file...done >>Creating read threads...done >>Creating fork threads...done >>Creating alloc threads...done >>Creating lock threads...done >>running...printing oldest/median age/newest >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>*** >>*** runtime error: >>*** An enumeration or subrange value was out of range. >>*** file "..\src\Main.m3", line 340 >>*** >> >>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D2: The syste= >m c=3D >>annot find the file specified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D2: The syste= >m c=3D >>annot find the file specified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>Stack trace: >> FP PC Procedure >>--------- --------- ------------------------------- >>0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >>0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 >> >>Regards, >>Randy Coleburn >> >>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>Content-Type: text/html; charset=3D"us-ascii" >>Content-Transfer-Encoding: quoted-printable >> From mika at async.caltech.edu Mon Feb 28 00:11:25 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 27 Feb 2011 15:11:25 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: <20110226175511.3F7D21A2078@async.async.caltech.edu> <20110227202950.46F3B1A2078@async.async.caltech.edu> Message-ID: <20110227231125.3DC611A2078@async.async.caltech.edu> Ah, it just doesn't check command-line arguments that carefully. I think what you did is equivalent to "-tests STD". Mika "Coleburn, Randy" writes: >Mika: > >No change with "-tests POSIX". > >Interesting twist: On Windows 7, I thought I'd see what the command line o= >ptions are, and I typed "threadtest -help" rather than reading the code. > >First time, it produced what appears to be a NIL deref crash. Then, I trie= >d it again and it ran to completion. Something seems non-deterministic her= >e. See below. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe -help >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >. > >*** >*** runtime error: >*** Attempt to reference an illegal memory location. >*** pc =3D 0x77762262 >*** > >Stack trace: > FP PC Procedure >--------- --------- ------------------------------- > 0xcdf998 0x130351b SystemError + 0x64 in ..\src\runtime\NT386\RTSignal.m= >3 > 0xcdf9c0 0x77762262 > 0xcdf9d8 0x12e83b7 LockMutex + 0x4f in ..\src\thread\WIN32\ThreadWin32.m= >3 > 0xcdfa00 0x12c7b08 GetChar + 0x28 in ..\src\rw\Rd.m3 > 0xcdfb38 0x12c12e3 RApply + 0xd3 in ..\src\Main.m3 > 0xcdfb74 0x12e971f ThreadBase + 0x254 in ..\src\thread\WIN32\ThreadWin32= >.m3 > 0xcdfb80 0x76543677 > 0xcdfbc0 0x77779f02 >......... ......... ... more frames ... > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe -help >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >All tests complete. Congratulations. > >Regards, >Randy Coleburn > >-----Original Message----- >From: Mika Nystrom [mailto:mika at async.caltech.edu]=20 >Sent: Sunday, February 27, 2011 3:30 PM >To: Coleburn, Randy >Cc: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7=20 > >Hi Randy, > >You can try it with -tests POSIX as well. > >I find on user threads it runs very slowly (but it does run) because of how= > unfair >the thread scheduler is. > >Next step might be to whittle down the tests and see if you can get a failu= >re with >a single test running and -n 2. That would likely be the simplest scenario= > to >start further debugging from. > > Mika > >"Coleburn, Randy" writes: >>Mika et al: >> >>Thought I would try something else. >> >>I took the sources of your thread test program to an older XP machine that= > =3D >>has CM3 circa August 2008. This is the machine and implementation I used = >w=3D >>hen building a major project I did a couple years back. >> >>The thread test program does indeed build on this old system, but when I r= >u=3D >>n it, I get different results than with the latest HEAD branch code. =3D20 >> >>After it prints "running...printing oldest/median age/newest", on the next= > =3D >>line I get two periods ".." and now the program seems hung. I'll let it "= >r=3D >>un" for a few more minutes to see if anything else happens before killing = >i=3D >>t. >> >>At least we don't get the subscript and assertion failures on this older C= >M=3D >>3 platform. >> >>Regards, >>Randy Coleburn >> >> >>-----Original Message----- >>From: Coleburn, Randy=3D20 >>Sent: Sunday, February 27, 2011 2:09 PM >>To: m3devel at elegosoft.com >>Subject: Re: [M3devel] results of threadtest program on Windows7 >> >>Mika: >> >>Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. >> >>Here is what happens now when I run your threadtest program on Windows 7. >> >>C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-fo= >r=3D >>k >>Writing file...done >>Creating read threads...done >>Creating nxread threads...done >>Creating tryexcept threads...done >>Creating forktoomuch threads...done >>Creating alloc threads...done >>Creating creat threads...done >>Creating lock threads...done >>running...printing oldest/median age/newest >> >> >>*** >>*** runtime error: >>*** An array subscript was out of range. >>*** file "..\src\runtime\common\RTCollector.m3", line 418 >>*** >> >> >> >>*** >>*** runtime error: >>*** <*ASSERT*> failed. >>*** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 >>*** >> >>The last error repeats ad infinitum until I press CNTRL-C to abort. >> >>I'll send more info on the Windows install of Modula3 in a subsequent post= >. >> >>Regards, >>Randy Coleburn >> >>-----Original Message----- >>From: Mika Nystrom [mailto:mika at async.caltech.edu]=3D20 >>Sent: Saturday, February 26, 2011 12:55 PM >>To: Coleburn, Randy >>Cc: m3devel at elegosoft.com >>Subject: Re: [M3devel] results of threadtest program on Windows7=3D20 >> >>Hi Randy, >> >>Hm yes it looks like my Windows programming skills leave something >>to be desired. >> >>You can run the thread tester while skipping a test as follows >> >> threadtest -tests ALL,-fork >> >>(for instance) >> >>if you just run=3D20 >> >> threadtest -sadfassdaf >> >>it'll print the tests that are available. >> >>As it happens, I just had to upgrade my windows 2000 system to windows 7. >>Can you give me a very brief description of what you did to install Modula= >-=3D >>3 >>on this system? >> >> Mika >> >>"Coleburn, Randy" writes: >>>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>>Content-Type: text/plain; charset=3D3D"us-ascii" >>>Content-Transfer-Encoding: quoted-printable >>> >>>Mika: >>> >>>I've finally managed to get cm3 rebuilt on Windows 7 again. >>> >>>So, I ran your threadtest program. >>> >>>Here are the results. Note the "..." is where I cut out a bunch of the r= >e=3D >>p=3D3D >>>eating "ERROR FApply" messages. >>> >>>C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >>>Writing file...done >>>Creating read threads...done >>>Creating fork threads...done >>>Creating alloc threads...done >>>Creating lock threads...done >>>running...printing oldest/median age/newest >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>. >>>. >>>. >>>*** >>>*** runtime error: >>>*** An enumeration or subrange value was out of range. >>>*** file "..\src\Main.m3", line 340 >>>*** >>> >>>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The sy= >ste=3D >>m c=3D3D >>>annot find the file specified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>. >>>. >>>. >>>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The sy= >ste=3D >>m c=3D3D >>>annot find the file specified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>. >>>. >>>. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>Stack trace: >>> FP PC Procedure >>>--------- --------- ------------------------------- >>>0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >>>0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 >>> >>>Regards, >>>Randy Coleburn >>> >>>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>>Content-Type: text/html; charset=3D3D"us-ascii" >>>Content-Transfer-Encoding: quoted-printable >>> From rcolebur at SCIRES.COM Tue Feb 1 01:48:21 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Message-ID: Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. Do you have a plan to fix? Anything I can do to help? --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 01:53:21 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 00:53:21 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: References: Message-ID: Oops, I was just wondering to self if I had fixed that. I can get to it fairly soon. We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling convention (which imho is a good reason to make it a separate m3cg call...) (The input parameter is in eax, and the output is in esp.) (Tony, you really don't want a separate m3cg call?) - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. Do you have a plan to fix? Anything I can do to help? --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 01:55:07 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 00:55:07 +0000 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: , , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, , Message-ID: ps: I'm happy to report that my available time here has become significantly reduced. Partly with that in mind, but even w/o it, I am willing to go back to the old way. m3front is full of mystery and I'm not sure I'm keen on understanding it. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 15:26:20 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I think we need to have a way to get the scope of the containing procedure. TryStmt.Parse is not the right place. If I get some time soon I can take a look at this. On Jan 31, 2011, at 2:19 PM, Jay K wrote: Right..I also tried Scope.Insert, didn't seem to help. But I think I might be doing it too late. I'll try doing it in Parse instead of Check. Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat more closely resembles the old pattern: IF Foo() THEN RETURN ELSE TRY FINALLY in the old way would use the stack, so we could too. But the allocation is slower/larger now (alloca possibly aligning up more than needed). Anyway, this point isn't the problem. The problem is my inability to work with m3front. I'll look at it a bit more -- i.e. as I said, moving the code to Parse. The other improvement I'd like to make: currently, if I had it working, what I'm aiming for is: struct EF1{ ... jmpbuf* jb } ef1; jmpbuf* jb = 0; jb = jb ? jb : alloca(sizeof(*jb)); ef1.jb = jb; setjmp(ef1.jb); but it'd be better to have: struct EF1{ ... jmpbuf* jb } ef1; ef1.jb = 0; ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); setjmp(ef1.jb); ie. we don't need the extra pointer variable, the pointer in the record should suffice. [not sure where to put this in email] I tried figuring out how to determine the current vs. topmost-in-current-function scope. I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, so I kinda hoped it'd just use the correct scope. Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. Even if it doesn't get used, that might be ok, since there is tracking of use, and declaring it might not cause any waste if it isn't used. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 12:14:09 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? Hi Jay, I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Jan 30, 2011, at 5:45 AM, Jay K wrote: I'm trying to fix the inadvertant change I made where FOR i := 1 TO 10 DO TRY FINALLY END will call alloca 10 times. I'm not familiar with the frontend, and it is hard for me to understand. It seems like it is probably very well factored, written by someone who really knew what they were doing, but it is dense, and not verbosely commented. Something like this?? But this doesn't work. What am I missing? And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? ? 1.txt Index: src/misc/Marker.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v retrieving revision 1.1.1.1 diff -u -r1.1.1.1 Marker.i3 --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 @@ -64,9 +64,12 @@ PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); (* generate the call to acquire or release a mutex *) -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T); (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) PROCEDURE Reset (); +PROCEDURE NewJumpbuf (): Variable.T; + END Marker. Index: src/misc/Marker.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v retrieving revision 1.9 diff -u -r1.9 Marker.m3 --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 @@ -10,7 +10,7 @@ MODULE Marker; IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; TYPE Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, @@ -231,8 +231,10 @@ END; END CallFinallyHandler; -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T) = VAR new: BOOLEAN; + label := CG.Next_label (); BEGIN (* int setjmp(void* ); *) IF (setjmp = NIL) THEN @@ -262,13 +264,33 @@ Target.Word.size, Target.Word.align, Target.Word.cg_type, 0); END; - - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) + + (* IF jumpuf # NIL THEN + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); + * END + *) + + CG.Load_nil (); + Variable.Load (jumpbuf); + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); + CG.Start_call_direct (alloca, 0, Target.Address.cg_type); CG.Load_int (Target.Word.cg_type, Jumpbuf_size); CG.Pop_param (Target.Word.cg_type); CG.Call_direct (alloca, Target.Address.cg_type); + + (* FUTURE: We should actually have a Variable + * for the entire EF1, including initializing the jumpbuf + * in it. That would save us this extra load/store, + * and save a local variable. + *) + Variable.LoadLValue (jumpbuf); + CG.Swap (); + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); + Variable.Load (jumpbuf); + CG.Store_addr (frame, M3RT.EF1_jmpbuf); + CG.Set_label (label); (* setmp(frame.jmpbuf) *) CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); @@ -806,5 +828,15 @@ tos := 0; END Reset; +PROCEDURE NewJumpbuf (): Variable.T = + VAR jumpbuf: Variable.T; + BEGIN + jumpbuf := Variable.New (M3ID.NoID, TRUE); + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, + readonly := FALSE, open_array_ok := FALSE, + needs_init := TRUE); + RETURN jumpbuf; + END NewJumpbuf; + BEGIN END Marker. Index: src/stmts/TryFinStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v retrieving revision 1.6 diff -u -r1.6 TryFinStmt.m3 --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -10,6 +10,7 @@ IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; +IMPORT Variable; FROM Stmt IMPORT Outcome; TYPE @@ -20,6 +21,7 @@ viaProc : BOOLEAN; scope : Scope.T; handler : HandlerProc; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -63,6 +65,11 @@ RETURN p; END Parse; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + RETURN NOT (Target.Has_stack_walker OR p.viaProc); + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; BEGIN @@ -95,6 +102,9 @@ END; END; TryStmt.PopHandler (); + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE HandlerName (uid: INTEGER): TEXT = @@ -106,11 +116,16 @@ END HandlerName; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN - IF Target.Has_stack_walker THEN RETURN Compile1 (p); - ELSIF p.viaProc THEN RETURN Compile2 (p); - ELSE RETURN Compile3 (p); + IF Target.Has_stack_walker THEN result := Compile1 (p); + ELSIF p.viaProc THEN result := Compile2 (p); + ELSE usesSetjmp := TRUE; + result := Compile3 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -302,7 +317,9 @@ l := CG.Next_label (3); CG.Set_label (l, barrier := TRUE); Marker.PushFrame (frame, M3RT.HandlerClass.Finally); - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) Marker.PushFinally (l, l+1, frame); Index: src/stmts/TryStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v retrieving revision 1.3 diff -u -r1.3 TryStmt.m3 --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -22,6 +22,7 @@ hasElse : BOOLEAN; elseBody : Stmt.T; handled : ESet.T; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -153,6 +154,14 @@ p.handles := h3; END ReverseHandlers; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + IF (p.handles = NIL) AND (NOT p.hasElse) THEN + RETURN FALSE; + END; + RETURN NOT Target.Has_stack_walker; + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR h: Handler; handled: ESet.T; BEGIN @@ -182,6 +191,10 @@ WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; Stmt.TypeCheck (p.elseBody, cs); PopHandler (); + + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = @@ -245,14 +258,19 @@ END CheckHandler; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN IF (p.handles = NIL) AND (NOT p.hasElse) THEN - RETURN Stmt.Compile (p.body); - END; - IF Target.Has_stack_walker - THEN RETURN Compile1 (p); - ELSE RETURN Compile2 (p); + result := Stmt.Compile (p.body); + ELSIF Target.Has_stack_walker THEN + result := Compile1 (p); + ELSE + usesSetjmp := TRUE; + result := Compile2 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -423,7 +441,9 @@ END; (* capture the machine state *) - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) oc := Stmt.Compile (p.body); Thanks, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 04:39:36 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 31 Jan 2011 22:39:36 -0500 Subject: [M3devel] link failure on NT386 In-Reply-To: References: Message-ID: Why can't we just have a run-time hook that resolves differently on different targets? Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. On Jan 31, 2011, at 7:53 PM, Jay K wrote: > Oops, I was just wondering to self if I had fixed that. > I can get to it fairly soon. > We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling > convention (which imho is a good reason to > make it a separate m3cg call...) > (The input parameter is in eax, and the output is in esp.) > (Tony, you really don't want a separate m3cg call?) > > > - Jay > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 31 Jan 2011 19:48:21 -0500 > Subject: [M3devel] link failure on NT386 > > Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. > Do you have a plan to fix? > Anything I can do to help? > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 04:45:53 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Mon, 31 Jan 2011 22:45:53 -0500 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: , , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, , Message-ID: <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu> I'd like to do what you envisage. It is a significant improvement, and should not require too much effort: 1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3) 2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. On Jan 31, 2011, at 7:55 PM, Jay K wrote: > ps: I'm happy to report that my available time here > has become significantly reduced. Partly with that in mind, > but even w/o it, I am willing to go back to the old way. > m3front is full of mystery and I'm not sure I'm keen on understanding it. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 15:26:20 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I think we need to have a way to get the scope of the containing procedure. > TryStmt.Parse is not the right place. > If I get some time soon I can take a look at this. > > On Jan 31, 2011, at 2:19 PM, Jay K wrote: > > Right..I also tried Scope.Insert, didn't seem to help. > > But I think I might be doing it too late. I'll try doing it in Parse instead of Check. > Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. > I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat > more closely resembles the old pattern: > > IF Foo() THEN > RETURN > ELSE TRY FINALLY > > in the old way would use the stack, so we could too. > But the allocation is slower/larger now (alloca possibly aligning up more than needed). > Anyway, this point isn't the problem. The problem is my inability to work with m3front. > I'll look at it a bit more -- i.e. as I said, moving the code to Parse. > > The other improvement I'd like to make: > > currently, if I had it working, what I'm aiming for is: > struct EF1{ ... jmpbuf* jb } ef1; > jmpbuf* jb = 0; > > jb = jb ? jb : alloca(sizeof(*jb)); > ef1.jb = jb; > setjmp(ef1.jb); > > but it'd be better to have: > > struct EF1{ ... jmpbuf* jb } ef1; > ef1.jb = 0; > > ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); > setjmp(ef1.jb); > > > ie. we don't need the extra pointer variable, the pointer in the record should suffice. > > > [not sure where to put this in email] > I tried figuring out how to determine the current vs. topmost-in-current-function scope. > I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps > some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, > so I kinda hoped it'd just use the correct scope. > > > Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. > Even if it doesn't get used, that might be ok, since there is tracking of use, and > declaring it might not cause any waste if it isn't used. > > > Thanks, > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 12:14:09 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > Hi Jay, > > I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On Jan 30, 2011, at 5:45 AM, Jay K wrote: > > I'm trying to fix the inadvertant change I made where > FOR i := 1 TO 10 DO TRY FINALLY END > > will call alloca 10 times. > > I'm not familiar with the frontend, and it is hard for me to understand. > It seems like it is probably very well factored, written by someone who really knew what they were doing, > but it is dense, and not verbosely commented. > > Something like this?? But this doesn't work. What am I missing? > And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? > > > ? 1.txt > Index: src/misc/Marker.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v > retrieving revision 1.1.1.1 > diff -u -r1.1.1.1 Marker.i3 > --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 > +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 > @@ -64,9 +64,12 @@ > PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); > (* generate the call to acquire or release a mutex *) > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T); > (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) > > PROCEDURE Reset (); > > +PROCEDURE NewJumpbuf (): Variable.T; > + > END Marker. > Index: src/misc/Marker.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v > retrieving revision 1.9 > diff -u -r1.9 Marker.m3 > --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 > +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,7 +10,7 @@ > MODULE Marker; > > IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; > -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; > +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; > > TYPE > Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, > @@ -231,8 +231,10 @@ > END; > END CallFinallyHandler; > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T) = > VAR new: BOOLEAN; > + label := CG.Next_label (); > BEGIN > (* int setjmp(void* ); *) > IF (setjmp = NIL) THEN > @@ -262,13 +264,33 @@ > Target.Word.size, Target.Word.align, > Target.Word.cg_type, 0); > END; > - > - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) > + > + (* IF jumpuf # NIL THEN > + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); > + * END > + *) > + > + CG.Load_nil (); > + Variable.Load (jumpbuf); > + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); > + > CG.Start_call_direct (alloca, 0, Target.Address.cg_type); > CG.Load_int (Target.Word.cg_type, Jumpbuf_size); > CG.Pop_param (Target.Word.cg_type); > CG.Call_direct (alloca, Target.Address.cg_type); > + > + (* FUTURE: We should actually have a Variable > + * for the entire EF1, including initializing the jumpbuf > + * in it. That would save us this extra load/store, > + * and save a local variable. > + *) > + Variable.LoadLValue (jumpbuf); > + CG.Swap (); > + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); > + Variable.Load (jumpbuf); > + > CG.Store_addr (frame, M3RT.EF1_jmpbuf); > + CG.Set_label (label); > > (* setmp(frame.jmpbuf) *) > CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); > @@ -806,5 +828,15 @@ > tos := 0; > END Reset; > > +PROCEDURE NewJumpbuf (): Variable.T = > + VAR jumpbuf: Variable.T; > + BEGIN > + jumpbuf := Variable.New (M3ID.NoID, TRUE); > + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, > + readonly := FALSE, open_array_ok := FALSE, > + needs_init := TRUE); > + RETURN jumpbuf; > + END NewJumpbuf; > + > BEGIN > END Marker. > Index: src/stmts/TryFinStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v > retrieving revision 1.6 > diff -u -r1.6 TryFinStmt.m3 > --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 > +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,6 +10,7 @@ > > IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; > IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; > +IMPORT Variable; > FROM Stmt IMPORT Outcome; > > TYPE > @@ -20,6 +21,7 @@ > viaProc : BOOLEAN; > scope : Scope.T; > handler : HandlerProc; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -63,6 +65,11 @@ > RETURN p; > END Parse; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + RETURN NOT (Target.Has_stack_walker OR p.viaProc); > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; > BEGIN > @@ -95,6 +102,9 @@ > END; > END; > TryStmt.PopHandler (); > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE HandlerName (uid: INTEGER): TEXT = > @@ -106,11 +116,16 @@ > END HandlerName; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > - IF Target.Has_stack_walker THEN RETURN Compile1 (p); > - ELSIF p.viaProc THEN RETURN Compile2 (p); > - ELSE RETURN Compile3 (p); > + IF Target.Has_stack_walker THEN result := Compile1 (p); > + ELSIF p.viaProc THEN result := Compile2 (p); > + ELSE usesSetjmp := TRUE; > + result := Compile3 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -302,7 +317,9 @@ > l := CG.Next_label (3); > CG.Set_label (l, barrier := TRUE); > Marker.PushFrame (frame, M3RT.HandlerClass.Finally); > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > Marker.PushFinally (l, l+1, frame); > Index: src/stmts/TryStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v > retrieving revision 1.3 > diff -u -r1.3 TryStmt.m3 > --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 > +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -22,6 +22,7 @@ > hasElse : BOOLEAN; > elseBody : Stmt.T; > handled : ESet.T; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -153,6 +154,14 @@ > p.handles := h3; > END ReverseHandlers; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + IF (p.handles = NIL) AND (NOT p.hasElse) THEN > + RETURN FALSE; > + END; > + RETURN NOT Target.Has_stack_walker; > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR h: Handler; handled: ESet.T; > BEGIN > @@ -182,6 +191,10 @@ > WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; > Stmt.TypeCheck (p.elseBody, cs); > PopHandler (); > + > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = > @@ -245,14 +258,19 @@ > END CheckHandler; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > IF (p.handles = NIL) AND (NOT p.hasElse) THEN > - RETURN Stmt.Compile (p.body); > - END; > - IF Target.Has_stack_walker > - THEN RETURN Compile1 (p); > - ELSE RETURN Compile2 (p); > + result := Stmt.Compile (p.body); > + ELSIF Target.Has_stack_walker THEN > + result := Compile1 (p); > + ELSE > + usesSetjmp := TRUE; > + result := Compile2 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -423,7 +441,9 @@ > END; > > (* capture the machine state *) > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > oc := Stmt.Compile (p.body); > > Thanks, > - Jay > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 08:12:13 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 07:12:13 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: References: , , Message-ID: Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. Why are the atomics in m3cg? I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. alloca is simpler just by virtue of it only taking one parameter. Why aren't and/or/xor/add runtime hooks, or special functions? Or the set operations? I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. And that there may be a desire to in general avoid growing m3cg. esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. I'm also wishy washy. I can see it is easy to implement easily any way. I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. At some point, I realize, you can model just about everything as function calls, except, calling functions. :) I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. Since anything implemented in Modula-3 and exposed must be in some interface. However runtime hooks can be implemented in C as well. It depends somewhat if they are only called by generated code or also hand written Modula-3. You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, there is just relying on m3front and m3core agreeing. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:39:36 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] link failure on NT386 Why can't we just have a run-time hook that resolves differently on different targets?Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. On Jan 31, 2011, at 7:53 PM, Jay K wrote:Oops, I was just wondering to self if I had fixed that. I can get to it fairly soon. We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling convention (which imho is a good reason to make it a separate m3cg call...) (The input parameter is in eax, and the output is in esp.) (Tony, you really don't want a separate m3cg call?) - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier.Do you have a plan to fix?Anything I can do to help?--Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 08:16:40 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 07:16:40 +0000 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu> References: , , , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, , , , , , <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu> Message-ID: These look like good hints. I wouldn't know to make such slightly large changes, that the existing infrastructure doesn't suffice. I'll have a go at it..as my now-very-limited time permits. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:45:53 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I'd like to do what you envisage. It is a significant improvement, and should not require too much effort:1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3)2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. On Jan 31, 2011, at 7:55 PM, Jay K wrote:ps: I'm happy to report that my available time here has become significantly reduced. Partly with that in mind, but even w/o it, I am willing to go back to the old way. m3front is full of mystery and I'm not sure I'm keen on understanding it. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 15:26:20 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I think we need to have a way to get the scope of the containing procedure.TryStmt.Parse is not the right place.If I get some time soon I can take a look at this. On Jan 31, 2011, at 2:19 PM, Jay K wrote:Right..I also tried Scope.Insert, didn't seem to help. But I think I might be doing it too late. I'll try doing it in Parse instead of Check. Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat more closely resembles the old pattern: IF Foo() THEN RETURN ELSE TRY FINALLY in the old way would use the stack, so we could too. But the allocation is slower/larger now (alloca possibly aligning up more than needed). Anyway, this point isn't the problem. The problem is my inability to work with m3front. I'll look at it a bit more -- i.e. as I said, moving the code to Parse. The other improvement I'd like to make: currently, if I had it working, what I'm aiming for is: struct EF1{ ... jmpbuf* jb } ef1; jmpbuf* jb = 0; jb = jb ? jb : alloca(sizeof(*jb)); ef1.jb = jb; setjmp(ef1.jb); but it'd be better to have: struct EF1{ ... jmpbuf* jb } ef1; ef1.jb = 0; ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); setjmp(ef1.jb); ie. we don't need the extra pointer variable, the pointer in the record should suffice. [not sure where to put this in email] I tried figuring out how to determine the current vs. topmost-in-current-function scope. I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, so I kinda hoped it'd just use the correct scope. Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. Even if it doesn't get used, that might be ok, since there is tracking of use, and declaring it might not cause any waste if it isn't used. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 12:14:09 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? Hi Jay, I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. Antony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Jan 30, 2011, at 5:45 AM, Jay K wrote:I'm trying to fix the inadvertant change I made where FOR i := 1 TO 10 DO TRY FINALLY END will call alloca 10 times. I'm not familiar with the frontend, and it is hard for me to understand. It seems like it is probably very well factored, written by someone who really knew what they were doing, but it is dense, and not verbosely commented. Something like this?? But this doesn't work. What am I missing? And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? ? 1.txt Index: src/misc/Marker.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v retrieving revision 1.1.1.1 diff -u -r1.1.1.1 Marker.i3 --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 @@ -64,9 +64,12 @@ PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); (* generate the call to acquire or release a mutex *) -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T); (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) PROCEDURE Reset (); +PROCEDURE NewJumpbuf (): Variable.T; + END Marker. Index: src/misc/Marker.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v retrieving revision 1.9 diff -u -r1.9 Marker.m3 --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 @@ -10,7 +10,7 @@ MODULE Marker; IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; TYPE Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, @@ -231,8 +231,10 @@ END; END CallFinallyHandler; -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T) = VAR new: BOOLEAN; + label := CG.Next_label (); BEGIN (* int setjmp(void* ); *) IF (setjmp = NIL) THEN @@ -262,13 +264,33 @@ Target.Word.size, Target.Word.align, Target.Word.cg_type, 0); END; - - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) + + (* IF jumpuf # NIL THEN + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); + * END + *) + + CG.Load_nil (); + Variable.Load (jumpbuf); + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); + CG.Start_call_direct (alloca, 0, Target.Address.cg_type); CG.Load_int (Target.Word.cg_type, Jumpbuf_size); CG.Pop_param (Target.Word.cg_type); CG.Call_direct (alloca, Target.Address.cg_type); + + (* FUTURE: We should actually have a Variable + * for the entire EF1, including initializing the jumpbuf + * in it. That would save us this extra load/store, + * and save a local variable. + *) + Variable.LoadLValue (jumpbuf); + CG.Swap (); + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); + Variable.Load (jumpbuf); + CG.Store_addr (frame, M3RT.EF1_jmpbuf); + CG.Set_label (label); (* setmp(frame.jmpbuf) *) CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); @@ -806,5 +828,15 @@ tos := 0; END Reset; +PROCEDURE NewJumpbuf (): Variable.T = + VAR jumpbuf: Variable.T; + BEGIN + jumpbuf := Variable.New (M3ID.NoID, TRUE); + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, + readonly := FALSE, open_array_ok := FALSE, + needs_init := TRUE); + RETURN jumpbuf; + END NewJumpbuf; + BEGIN END Marker. Index: src/stmts/TryFinStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v retrieving revision 1.6 diff -u -r1.6 TryFinStmt.m3 --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -10,6 +10,7 @@ IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; +IMPORT Variable; FROM Stmt IMPORT Outcome; TYPE @@ -20,6 +21,7 @@ viaProc : BOOLEAN; scope : Scope.T; handler : HandlerProc; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -63,6 +65,11 @@ RETURN p; END Parse; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + RETURN NOT (Target.Has_stack_walker OR p.viaProc); + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; BEGIN @@ -95,6 +102,9 @@ END; END; TryStmt.PopHandler (); + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE HandlerName (uid: INTEGER): TEXT = @@ -106,11 +116,16 @@ END HandlerName; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN - IF Target.Has_stack_walker THEN RETURN Compile1 (p); - ELSIF p.viaProc THEN RETURN Compile2 (p); - ELSE RETURN Compile3 (p); + IF Target.Has_stack_walker THEN result := Compile1 (p); + ELSIF p.viaProc THEN result := Compile2 (p); + ELSE usesSetjmp := TRUE; + result := Compile3 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -302,7 +317,9 @@ l := CG.Next_label (3); CG.Set_label (l, barrier := TRUE); Marker.PushFrame (frame, M3RT.HandlerClass.Finally); - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) Marker.PushFinally (l, l+1, frame); Index: src/stmts/TryStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v retrieving revision 1.3 diff -u -r1.3 TryStmt.m3 --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -22,6 +22,7 @@ hasElse : BOOLEAN; elseBody : Stmt.T; handled : ESet.T; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -153,6 +154,14 @@ p.handles := h3; END ReverseHandlers; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + IF (p.handles = NIL) AND (NOT p.hasElse) THEN + RETURN FALSE; + END; + RETURN NOT Target.Has_stack_walker; + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR h: Handler; handled: ESet.T; BEGIN @@ -182,6 +191,10 @@ WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; Stmt.TypeCheck (p.elseBody, cs); PopHandler (); + + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = @@ -245,14 +258,19 @@ END CheckHandler; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN IF (p.handles = NIL) AND (NOT p.hasElse) THEN - RETURN Stmt.Compile (p.body); - END; - IF Target.Has_stack_walker - THEN RETURN Compile1 (p); - ELSE RETURN Compile2 (p); + result := Stmt.Compile (p.body); + ELSIF Target.Has_stack_walker THEN + result := Compile1 (p); + ELSE + usesSetjmp := TRUE; + result := Compile2 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -423,7 +441,9 @@ END; (* capture the machine state *) - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) oc := Stmt.Compile (p.body); Thanks, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 15:46:22 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 1 Feb 2011 09:46:22 -0500 Subject: [M3devel] link failure on NT386 In-Reply-To: References: Message-ID: <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu> The atomics are not necessarily functions. They approximate real instructions generated inline. Shall I just go ahead and add it? Sent from my iPad On 01/02/2011, at 2:12 AM, Jay K wrote: > Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? > In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. > Why are the atomics in m3cg? > I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. > alloca is simpler just by virtue of it only taking one parameter. > > Why aren't and/or/xor/add runtime hooks, or special functions? > Or the set operations? > > I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. > > And that there may be a desire to in general avoid growing m3cg. > esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. > > I'm also wishy washy. I can see it is easy to implement easily any way. > I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. > > > At some point, I realize, you can model just about everything as function calls, except, calling functions. :) > > I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. > Since anything implemented in Modula-3 and exposed must be in some interface. > However runtime hooks can be implemented in C as well. > It depends somewhat if they are only called by generated code or also hand written Modula-3. > You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, > there is just relying on m3front and m3core agreeing. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:39:36 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] link failure on NT386 > > Why can't we just have a run-time hook that resolves differently on different targets? > Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. > > On Jan 31, 2011, at 7:53 PM, Jay K wrote: > > Oops, I was just wondering to self if I had fixed that. > I can get to it fairly soon. > We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling > convention (which imho is a good reason to > make it a separate m3cg call...) > (The input parameter is in eax, and the output is in esp.) > (Tony, you really don't want a separate m3cg call?) > > > - Jay > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 31 Jan 2011 19:48:21 -0500 > Subject: [M3devel] link failure on NT386 > > Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. > Do you have a plan to fix? > Anything I can do to help? > --Randy Coleburn > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 15:48:02 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 1 Feb 2011 09:48:02 -0500 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu> <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu> Message-ID: Let me know if you need some help. Sent from my iPad On 01/02/2011, at 2:16 AM, Jay K wrote: > These look like good hints. I wouldn't know to make such slightly large changes, > that the existing infrastructure doesn't suffice. > I'll have a go at it..as my now-very-limited time permits. > > Thanks, > - Jay > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:45:53 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I'd like to do what you envisage. It is a significant improvement, and should not require too much effort: > 1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3) > 2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. > > On Jan 31, 2011, at 7:55 PM, Jay K wrote: > > ps: I'm happy to report that my available time here > has become significantly reduced. Partly with that in mind, > but even w/o it, I am willing to go back to the old way. > m3front is full of mystery and I'm not sure I'm keen on understanding it. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 15:26:20 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I think we need to have a way to get the scope of the containing procedure. > TryStmt.Parse is not the right place. > If I get some time soon I can take a look at this. > > On Jan 31, 2011, at 2:19 PM, Jay K wrote: > > Right..I also tried Scope.Insert, didn't seem to help. > > But I think I might be doing it too late. I'll try doing it in Parse instead of Check. > Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. > I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat > more closely resembles the old pattern: > > IF Foo() THEN > RETURN > ELSE TRY FINALLY > > in the old way would use the stack, so we could too. > But the allocation is slower/larger now (alloca possibly aligning up more than needed). > Anyway, this point isn't the problem. The problem is my inability to work with m3front. > I'll look at it a bit more -- i.e. as I said, moving the code to Parse. > > The other improvement I'd like to make: > > currently, if I had it working, what I'm aiming for is: > struct EF1{ ... jmpbuf* jb } ef1; > jmpbuf* jb = 0; > > jb = jb ? jb : alloca(sizeof(*jb)); > ef1.jb = jb; > setjmp(ef1.jb); > > but it'd be better to have: > > struct EF1{ ... jmpbuf* jb } ef1; > ef1.jb = 0; > > ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); > setjmp(ef1.jb); > > > ie. we don't need the extra pointer variable, the pointer in the record should suffice. > > > [not sure where to put this in email] > I tried figuring out how to determine the current vs. topmost-in-current-function scope. > I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps > some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, > so I kinda hoped it'd just use the correct scope. > > > Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. > Even if it doesn't get used, that might be ok, since there is tracking of use, and > declaring it might not cause any waste if it isn't used. > > > Thanks, > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 12:14:09 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > Hi Jay, > > I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On Jan 30, 2011, at 5:45 AM, Jay K wrote: > > I'm trying to fix the inadvertant change I made where > FOR i := 1 TO 10 DO TRY FINALLY END > > will call alloca 10 times. > > I'm not familiar with the frontend, and it is hard for me to understand. > It seems like it is probably very well factored, written by someone who really knew what they were doing, > but it is dense, and not verbosely commented. > > Something like this?? But this doesn't work. What am I missing? > And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? > > > ? 1.txt > Index: src/misc/Marker.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v > retrieving revision 1.1.1.1 > diff -u -r1.1.1.1 Marker.i3 > --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 > +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 > @@ -64,9 +64,12 @@ > PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); > (* generate the call to acquire or release a mutex *) > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T); > (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) > > PROCEDURE Reset (); > > +PROCEDURE NewJumpbuf (): Variable.T; > + > END Marker. > Index: src/misc/Marker.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v > retrieving revision 1.9 > diff -u -r1.9 Marker.m3 > --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 > +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,7 +10,7 @@ > MODULE Marker; > > IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; > -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; > +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; > > TYPE > Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, > @@ -231,8 +231,10 @@ > END; > END CallFinallyHandler; > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T) = > VAR new: BOOLEAN; > + label := CG.Next_label (); > BEGIN > (* int setjmp(void* ); *) > IF (setjmp = NIL) THEN > @@ -262,13 +264,33 @@ > Target.Word.size, Target.Word.align, > Target.Word.cg_type, 0); > END; > - > - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) > + > + (* IF jumpuf # NIL THEN > + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); > + * END > + *) > + > + CG.Load_nil (); > + Variable.Load (jumpbuf); > + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); > + > CG.Start_call_direct (alloca, 0, Target.Address.cg_type); > CG.Load_int (Target.Word.cg_type, Jumpbuf_size); > CG.Pop_param (Target.Word.cg_type); > CG.Call_direct (alloca, Target.Address.cg_type); > + > + (* FUTURE: We should actually have a Variable > + * for the entire EF1, including initializing the jumpbuf > + * in it. That would save us this extra load/store, > + * and save a local variable. > + *) > + Variable.LoadLValue (jumpbuf); > + CG.Swap (); > + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); > + Variable.Load (jumpbuf); > + > CG.Store_addr (frame, M3RT.EF1_jmpbuf); > + CG.Set_label (label); > > (* setmp(frame.jmpbuf) *) > CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); > @@ -806,5 +828,15 @@ > tos := 0; > END Reset; > > +PROCEDURE NewJumpbuf (): Variable.T = > + VAR jumpbuf: Variable.T; > + BEGIN > + jumpbuf := Variable.New (M3ID.NoID, TRUE); > + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, > + readonly := FALSE, open_array_ok := FALSE, > + needs_init := TRUE); > + RETURN jumpbuf; > + END NewJumpbuf; > + > BEGIN > END Marker. > Index: src/stmts/TryFinStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v > retrieving revision 1.6 > diff -u -r1.6 TryFinStmt.m3 > --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 > +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,6 +10,7 @@ > > IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; > IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; > +IMPORT Variable; > FROM Stmt IMPORT Outcome; > > TYPE > @@ -20,6 +21,7 @@ > viaProc : BOOLEAN; > scope : Scope.T; > handler : HandlerProc; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -63,6 +65,11 @@ > RETURN p; > END Parse; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + RETURN NOT (Target.Has_stack_walker OR p.viaProc); > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; > BEGIN > @@ -95,6 +102,9 @@ > END; > END; > TryStmt.PopHandler (); > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE HandlerName (uid: INTEGER): TEXT = > @@ -106,11 +116,16 @@ > END HandlerName; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > - IF Target.Has_stack_walker THEN RETURN Compile1 (p); > - ELSIF p.viaProc THEN RETURN Compile2 (p); > - ELSE RETURN Compile3 (p); > + IF Target.Has_stack_walker THEN result := Compile1 (p); > + ELSIF p.viaProc THEN result := Compile2 (p); > + ELSE usesSetjmp := TRUE; > + result := Compile3 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -302,7 +317,9 @@ > l := CG.Next_label (3); > CG.Set_label (l, barrier := TRUE); > Marker.PushFrame (frame, M3RT.HandlerClass.Finally); > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > Marker.PushFinally (l, l+1, frame); > Index: src/stmts/TryStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v > retrieving revision 1.3 > diff -u -r1.3 TryStmt.m3 > --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 > +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -22,6 +22,7 @@ > hasElse : BOOLEAN; > elseBody : Stmt.T; > handled : ESet.T; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -153,6 +154,14 @@ > p.handles := h3; > END ReverseHandlers; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + IF (p.handles = NIL) AND (NOT p.hasElse) THEN > + RETURN FALSE; > + END; > + RETURN NOT Target.Has_stack_walker; > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR h: Handler; handled: ESet.T; > BEGIN > @@ -182,6 +191,10 @@ > WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; > Stmt.TypeCheck (p.elseBody, cs); > PopHandler (); > + > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = > @@ -245,14 +258,19 @@ > END CheckHandler; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > IF (p.handles = NIL) AND (NOT p.hasElse) THEN > - RETURN Stmt.Compile (p.body); > - END; > - IF Target.Has_stack_walker > - THEN RETURN Compile1 (p); > - ELSE RETURN Compile2 (p); > + result := Stmt.Compile (p.body); > + ELSIF Target.Has_stack_walker THEN > + result := Compile1 (p); > + ELSE > + usesSetjmp := TRUE; > + result := Compile2 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -423,7 +441,9 @@ > END; > > (* capture the machine state *) > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > oc := Stmt.Compile (p.body); > > Thanks, > - Jay > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 20:20:50 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 19:20:50 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu> References: , <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu> Message-ID: Modeling something as a function in m3cg does not necessarily correlate to generating a function call. E.g. the existing m3_alloca. And vice versa. e.g some of the set operations. Almost anything can be modeled as a function call in m3cg, but not end up generating one. Many things can be not modeled as a function call, but end up generating one. gcc has a huge number of "builtin functions". So does Visual C++. In Visual C++, basically any "new instruction" (mmx, sse2/3/4, avx) ends up modeled as a function, but generates roughly one instruction per function calls. It's a slightly high level assembly -- i.e. you get to chose the important algorithmic instructions, the compiler does register allocation, stack/local allocation, etc. I can add it -- surely my diff from a few weeks ago wasn't terrible? I'm ok either way. I'm more curious to understand than to "do". I admit. - Jay CC: m3devel at elegosoft.com From: hosking at cs.purdue.edu Subject: Re: [M3devel] link failure on NT386 Date: Tue, 1 Feb 2011 09:46:22 -0500 To: jay.krell at cornell.edu The atomics are not necessarily functions. They approximate real instructions generated inline.Shall I just go ahead and add it? Sent from my iPad On 01/02/2011, at 2:12 AM, Jay K wrote: Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. Why are the atomics in m3cg? I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. alloca is simpler just by virtue of it only taking one parameter. Why aren't and/or/xor/add runtime hooks, or special functions? Or the set operations? I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. And that there may be a desire to in general avoid growing m3cg. esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. I'm also wishy washy. I can see it is easy to implement easily any way. I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. At some point, I realize, you can model just about everything as function calls, except, calling functions. :) I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. Since anything implemented in Modula-3 and exposed must be in some interface. However runtime hooks can be implemented in C as well. It depends somewhat if they are only called by generated code or also hand written Modula-3. You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, there is just relying on m3front and m3core agreeing. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:39:36 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] link failure on NT386 Why can't we just have a run-time hook that resolves differently on different targets?Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. On Jan 31, 2011, at 7:53 PM, Jay K wrote:Oops, I was just wondering to self if I had fixed that. I can get to it fairly soon. We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling convention (which imho is a good reason to make it a separate m3cg call...) (The input parameter is in eax, and the output is in esp.) (Tony, you really don't want a separate m3cg call?) - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier.Do you have a plan to fix?Anything I can do to help?--Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 1 21:01:44 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 1 Feb 2011 15:01:44 -0500 Subject: [M3devel] link failure on NT386 In-Reply-To: References: <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu> Message-ID: <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu> RTHooks makes most sense. Sent from my iPhone On Feb 1, 2011, at 2:20 PM, Jay K wrote: > Modeling something as a function in m3cg does not necessarily correlate to generating a function call. > E.g. the existing m3_alloca. > And vice versa. e.g some of the set operations. > Almost anything can be modeled as a function call in m3cg, but not end up generating one. > Many things can be not modeled as a function call, but end up generating one. > gcc has a huge number of "builtin functions". So does Visual C++. > In Visual C++, basically any "new instruction" (mmx, sse2/3/4, avx) ends up modeled > as a function, but generates roughly one instruction per function calls. > It's a slightly high level assembly -- i.e. you get to chose the important algorithmic instructions, > the compiler does register allocation, stack/local allocation, etc. > > I can add it -- surely my diff from a few weeks ago wasn't terrible? > > I'm ok either way. > I'm more curious to understand than to "do". I admit. > > - Jay > > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > Subject: Re: [M3devel] link failure on NT386 > Date: Tue, 1 Feb 2011 09:46:22 -0500 > To: jay.krell at cornell.edu > > The atomics are not necessarily functions. They approximate real instructions generated inline. > Shall I just go ahead and add it? > > Sent from my iPad > > On 01/02/2011, at 2:12 AM, Jay K wrote: > > Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? > In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. > Why are the atomics in m3cg? > I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. > alloca is simpler just by virtue of it only taking one parameter. > > Why aren't and/or/xor/add runtime hooks, or special functions? > Or the set operations? > > I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. > > And that there may be a desire to in general avoid growing m3cg. > esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. > > I'm also wishy washy. I can see it is easy to implement easily any way. > I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. > > > At some point, I realize, you can model just about everything as function calls, except, calling functions. :) > > I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. > Since anything implemented in Modula-3 and exposed must be in some interface. > However runtime hooks can be implemented in C as well. > It depends somewhat if they are only called by generated code or also hand written Modula-3. > You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, > there is just relying on m3front and m3core agreeing. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:39:36 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] link failure on NT386 > > Why can't we just have a run-time hook that resolves differently on different targets? > Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. > > On Jan 31, 2011, at 7:53 PM, Jay K wrote: > > Oops, I was just wondering to self if I had fixed that. > I can get to it fairly soon. > We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling > convention (which imho is a good reason to > make it a separate m3cg call...) > (The input parameter is in eax, and the output is in esp.) > (Tony, you really don't want a separate m3cg call?) > > > - Jay > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 31 Jan 2011 19:48:21 -0500 > Subject: [M3devel] link failure on NT386 > > Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. > Do you have a plan to fix? > Anything I can do to help? > --Randy Coleburn > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 1 22:12:05 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 1 Feb 2011 21:12:05 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu> References: , , , , <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu>, , <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu> Message-ID: Why? You do realize, though it doesn't really matter, that the function cannot be implemented in Modula-3, or C, and isn't implemented on most platforms. It is only implemented on NT, in assembly. Maybe RTHooks is the holder for most function calls from generated code? The interface between any library and m3front? No matter how and if they are implemented? (it could perhaps be implemented in C, void* m3_alloca(size_t n) { return alloca(n); } but I have a "problem" with that, not sure it is a real problem or not, that it strongly resembles returning the address of a local, and if tools had enough visibility into our code, they would flag it as a bug and/or misoptimize it.) As well, this still requires the backend to recognize and special-case the function. As well, it seems like atomics could fit the same model. Why are they in m3cg? I'm trying not to insist on anything in particular, but I'd like to understand why the different options. m3cg offers the most direct, easily implemented, easily optimized, special-case free approach, though it makes m3cg "wider" and m3cg is kind of scary and large. But making special-case functions is just a way to hide the interface widening..in a way that is easier to remove or omit per-target if applicable perhaps.. - Jay From: hosking at cs.purdue.edu Date: Tue, 1 Feb 2011 15:01:44 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] link failure on NT386 RTHooks makes most sense. Sent from my iPhone On Feb 1, 2011, at 2:20 PM, Jay K wrote: Modeling something as a function in m3cg does not necessarily correlate to generating a function call. E.g. the existing m3_alloca. And vice versa. e.g some of the set operations. Almost anything can be modeled as a function call in m3cg, but not end up generating one. Many things can be not modeled as a function call, but end up generating one. gcc has a huge number of "builtin functions". So does Visual C++. In Visual C++, basically any "new instruction" (mmx, sse2/3/4, avx) ends up modeled as a function, but generates roughly one instruction per function calls. It's a slightly high level assembly -- i.e. you get to chose the important algorithmic instructions, the compiler does register allocation, stack/local allocation, etc. I can add it -- surely my diff from a few weeks ago wasn't terrible? I'm ok either way. I'm more curious to understand than to "do". I admit. - Jay CC: m3devel at elegosoft.com From: hosking at cs.purdue.edu Subject: Re: [M3devel] link failure on NT386 Date: Tue, 1 Feb 2011 09:46:22 -0500 To: jay.krell at cornell.edu The atomics are not necessarily functions. They approximate real instructions generated inline. Shall I just go ahead and add it? Sent from my iPad On 01/02/2011, at 2:12 AM, Jay K wrote: Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. Why are the atomics in m3cg? I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. alloca is simpler just by virtue of it only taking one parameter. Why aren't and/or/xor/add runtime hooks, or special functions? Or the set operations? I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. And that there may be a desire to in general avoid growing m3cg. esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. I'm also wishy washy. I can see it is easy to implement easily any way. I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. At some point, I realize, you can model just about everything as function calls, except, calling functions. :) I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. Since anything implemented in Modula-3 and exposed must be in some interface. However runtime hooks can be implemented in C as well. It depends somewhat if they are only called by generated code or also hand written Modula-3. You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, there is just relying on m3front and m3core agreeing. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:39:36 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] link failure on NT386 Why can't we just have a run-time hook that resolves differently on different targets? Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. On Jan 31, 2011, at 7:53 PM, Jay K wrote: Oops, I was just wondering to self if I had fixed that. I can get to it fairly soon. We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling convention (which imho is a good reason to make it a separate m3cg call...) (The input parameter is in eax, and the output is in esp.) (Tony, you really don't want a separate m3cg call?) - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Mon, 31 Jan 2011 19:48:21 -0500 Subject: [M3devel] link failure on NT386 Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. Do you have a plan to fix? Anything I can do to help? --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Feb 2 00:57:37 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 1 Feb 2011 18:57:37 -0500 Subject: [M3devel] link failure on NT386 In-Reply-To: References: , , , , <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu>, , <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu> Message-ID: I still don't understand what you are proposing? M3CG.alloca? On Feb 1, 2011, at 4:12 PM, Jay K wrote: > Why? You do realize, though it doesn't really matter, that the function cannot be implemented in Modula-3, or C, and isn't implemented on most platforms. It is only implemented on NT, in assembly. > Maybe RTHooks is the holder for most function calls from generated code? > The interface between any library and m3front? > No matter how and if they are implemented? > > (it could perhaps be implemented in C, void* m3_alloca(size_t n) { return alloca(n); } but I have a "problem" with that, not sure it is a real problem or not, that it strongly resembles returning the address of a local, and if tools had enough visibility into our code, they would flag it as a bug and/or misoptimize it.) > > As well, this still requires the backend to recognize and special-case the function. > As well, it seems like atomics could fit the same model. Why are they in m3cg? > > I'm trying not to insist on anything in particular, but I'd like to understand why the different options. > m3cg offers the most direct, easily implemented, easily optimized, special-case free approach, though it makes m3cg "wider" and m3cg is kind of scary and large. But making special-case functions is just a way to hide the interface widening..in a way that is easier to remove or omit per-target if applicable perhaps.. > > > - Jay > > From: hosking at cs.purdue.edu > Date: Tue, 1 Feb 2011 15:01:44 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] link failure on NT386 > > RTHooks makes most sense. > > Sent from my iPhone > > On Feb 1, 2011, at 2:20 PM, Jay K wrote: > > Modeling something as a function in m3cg does not necessarily correlate to generating a function call. > E.g. the existing m3_alloca. > And vice versa. e.g some of the set operations. > Almost anything can be modeled as a function call in m3cg, but not end up generating one. > Many things can be not modeled as a function call, but end up generating one. > gcc has a huge number of "builtin functions". So does Visual C++. > In Visual C++, basically any "new instruction" (mmx, sse2/3/4, avx) ends up modeled > as a function, but generates roughly one instruction per function calls. > It's a slightly high level assembly -- i.e. you get to chose the important algorithmic instructions, > the compiler does register allocation, stack/local allocation, etc. > > I can add it -- surely my diff from a few weeks ago wasn't terrible? > > I'm ok either way. > I'm more curious to understand than to "do". I admit. > > - Jay > > CC: m3devel at elegosoft.com > From: hosking at cs.purdue.edu > Subject: Re: [M3devel] link failure on NT386 > Date: Tue, 1 Feb 2011 09:46:22 -0500 > To: jay.krell at cornell.edu > > The atomics are not necessarily functions. They approximate real instructions generated inline. > Shall I just go ahead and add it? > > Sent from my iPad > > On 01/02/2011, at 2:12 AM, Jay K wrote: > > Interesting. Me playing honestly dumb: how do you decide among runtime hooks vs. "special" functions recognized by the backend? > In either case, the best reason I think for an m3cg hook is that the calling convention is unusual. > Why are the atomics in m3cg? > I grant, a possible reason: in the atomics case, the transform from "regular function" to "special function" might be more significant. > alloca is simpler just by virtue of it only taking one parameter. > > Why aren't and/or/xor/add runtime hooks, or special functions? > Or the set operations? > > I'm very willing to accept the answer that I suspect is the truth: there isn't a cut and dry reason, and many approaches lead to almost identical results. > > And that there may be a desire to in general avoid growing m3cg. > esp. to speed up a case that is very slow for other reasons and that we'd like to speed up in a way that would remove the need for the m3cg hook. > > I'm also wishy washy. I can see it is easy to implement easily any way. > I'm just overly bothered when I see obvious easy to fix inefficiency, even it is just a micro-optimization. > > > At some point, I realize, you can model just about everything as function calls, except, calling functions. :) > > I think runtime hooks are somewhat preferred for stuff implemented in Modula-3. > Since anything implemented in Modula-3 and exposed must be in some interface. > However runtime hooks can be implemented in C as well. > It depends somewhat if they are only called by generated code or also hand written Modula-3. > You know, if it is just implemented in C, and only called from generated code, there isn't a strict interface/module system, > there is just relying on m3front and m3core agreeing. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:39:36 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] link failure on NT386 > > Why can't we just have a run-time hook that resolves differently on different targets? > Take a look at m3front/src/misc/Runtyme and m3core/src/runtime/RTHooks. > > On Jan 31, 2011, at 7:53 PM, Jay K wrote: > > Oops, I was just wondering to self if I had fixed that. > I can get to it fairly soon. > We "just" have to translate m3_alloca to _chkstk..and fiddle with the calling > convention (which imho is a good reason to > make it a separate m3cg call...) > (The input parameter is in eax, and the output is in esp.) > (Tony, you really don't want a separate m3cg call?) > > > - Jay > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 31 Jan 2011 19:48:21 -0500 > Subject: [M3devel] link failure on NT386 > > Jay: I note the HEAD branch is still broken for builds on NT386 due to the linker problem with unresolved symbol we talked about earlier. > Do you have a plan to fix? > Anything I can do to help? > --Randy Coleburn > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Feb 2 01:10:46 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 2 Feb 2011 00:10:46 +0000 Subject: [M3devel] link failure on NT386 In-Reply-To: References: , , , , , , , , <3C7FF40C-5A76-42F0-9E71-9087230DF4E8@cs.purdue.edu>, , , , <6959CC00-E300-4228-96F2-097CF4F90A86@cs.purdue.edu>, , Message-ID: I still don't understand what you are proposing? M3CG.alloca? Yes. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Thu Feb 3 03:02:27 2011 From: hendrik at topoi.pooq.com (Hendrik Boom) Date: Wed, 2 Feb 2011 21:02:27 -0500 Subject: [M3devel] About Juno-2 and automating debugging software In-Reply-To: <22454.41538.qm@web29715.mail.ird.yahoo.com> References: <459675.89026.qm@web29709.mail.ird.yahoo.com> <22454.41538.qm@web29715.mail.ird.yahoo.com> Message-ID: <20110203020227.GB1094@topoi.pooq.com> On Mon, Jan 31, 2011 at 04:20:58AM +0000, Daniel Alejandro Benavides D. wrote: > Hi all: > yes, I believe is doable especially if you think this issue comes again in another form, the AI guys thinking what went wrong, well nothing went wrong as I realize now, there is a lot of research certainly, but in the end is just the same thing like before, and you can see this like the reuse of concepts of prior knowledge as is mentioned between other by Dr Andrew Tanenbaum, it just happens when something like this: see, the GPGPU GPU stuff alike is a nother come of the concept of explicit parallelism and its implication in computing fields, surely more developed in terms of the hardware this was I believe, but at the end is just another way of calling Transputers in earlier times, is just like it, then it was not an abandoned thing like before, just a reuse of the concept, and in practical terms this what it is the Occam Occam2 and alike languages; there is one in particular of my interest here the one built for drawing using CSP of Hoare, the language > is called Armadillo see a slide show here, specially check for the Computer magazine backcover in p. 6 in: > https://lsl.unal.edu.co/fl18/conferencias/09.20_ArmadilloTortugav02.pdf The TUNES website is a good place to look for plans and musings in the direction of formally verified programming: http://tunes.org/ It's a project that aims to use formal verification for security, and has produces a lot of interesting theorizing about the issues, but not much practical code. At least, that was the case last time I looked at the web site seriously a few years ago. -- hendrik From dabenavidesd at yahoo.es Sun Feb 6 01:07:56 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 6 Feb 2011 00:07:56 +0000 (GMT) Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: <20110205221326.967B12474007@birch.elegosoft.com> Message-ID: <692616.32152.qm@web29703.mail.ird.yahoo.com> Hi all: it seems that by denying itself from being well behaved program is checking for the unmatched traces or good behaviour programs, sort of analyzing many times more (incorrect traces with correct one) than checking the traces are well behaved and just checking wheh its not, better to check when its not, and not asumming the traces are ill anyway, then most of the time is spend in good standing than in bad or ill behaved program nalysis. The question how do you get those traces in good state, for instance checking by itself several times you would gain a pattern is repeatable, opposite of and ill corrupt and unmatched execution traces. In that case it makes a trace pattern you are in good stands but of course if it doens't then you have sort of a problem there. The next step is where to get the break point before it goes into an unrepeatable trace, that is when you must have an static analyzer which run a simulation and verifies program is well stated at each assert of it. Then it seems like you were getting the opposite direction going to see where it is not working yet at each part it makes asserts it, sort of looking for not well behaved errors but rather irrepeatable ones, which I think is harder. A error one you get by denying it's safe guarded makes illegal steps in the program you state or forbidden ones, is an unsafe error, a safe error is if you get it to pass it but you can't get it works by its results (sort of visbile) so getting in the right results but not in its good states are plausible to find, but ill ones, sort of what you are looking for i.e. looking for forbidden states that are ill-behaved and get wrong results are almost the harder path to get it such a bug Just to let you know if you see this point. --- El s?b, 5/2/11, Mika Nystrom escribi?: > De: Mika Nystrom > Asunto: [M3commit] CVS Update: cm3 > Para: m3commit at elegosoft.com > Fecha: s?bado, 5 de febrero, 2011 18:13 > CVSROOT: /usr/cvs > Changes by: mika at birch. > 11/02/05 23:13:26 > > Modified files: > cm3/m3-libs/m3core/tests/thread/src/: > Main.m3 > > Log message: > removed a lot of unnecessary heap memory > allocation, which could interfere with the threading tests. > > Most output to Stdio.stdout, not sure if > this is right (compared with Stdio.stderr). > > From hendrik at topoi.pooq.com Sun Feb 6 01:49:35 2011 From: hendrik at topoi.pooq.com (Hendrik Boom) Date: Sat, 5 Feb 2011 19:49:35 -0500 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: <692616.32152.qm@web29703.mail.ird.yahoo.com> References: <20110205221326.967B12474007@birch.elegosoft.com> <692616.32152.qm@web29703.mail.ird.yahoo.com> Message-ID: <20110206004935.GA31175@topoi.pooq.com> On Sun, Feb 06, 2011 at 12:07:56AM +0000, Daniel Alejandro Benavides D. wrote: > Hi all: ... ... > Just to let you know if you see this point. It sounds as if you're saying it's possible to prove a program is faulty by running it, but it's not possible to say it's correct. And concurrency makes things nonrepeatable, especially when there's bugs. -- hendrik From dabenavidesd at yahoo.es Sun Feb 6 02:01:16 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 6 Feb 2011 01:01:16 +0000 (GMT) Subject: [M3devel] About Juno-2 and automating debugging software In-Reply-To: <20110203020227.GB1094@topoi.pooq.com> Message-ID: <749746.64343.qm@web29720.mail.ird.yahoo.com> Hi all: thanks for your asnwer I came to see this today: http://www.espresso-soft.org/main.php?button=supervisors expand the last (show/hide) abstract selector. I think definitively this guys looks better things than many of us !! Or not? In a sense is good I guess, but what if not? Just hope to see what it brings to it. :) --- El mi?, 2/2/11, Hendrik Boom escribi?: > De: Hendrik Boom > Asunto: Re: [M3devel] About Juno-2 and automating debugging software > Para: m3devel at elegosoft.com > Fecha: mi?rcoles, 2 de febrero, 2011 21:02 > On Mon, Jan 31, 2011 at 04:20:58AM > +0000, Daniel Alejandro Benavides D. wrote: > > Hi all: > > yes, I believe is doable especially if you think this > issue comes again in another form, the AI guys thinking what > went wrong, well nothing went wrong as I realize now, there > is a lot of research certainly, but in the end is just the > same thing like before, and you can see this like the reuse > of concepts of prior knowledge as is mentioned between other > by Dr Andrew Tanenbaum, it just happens when something like > this: see, the GPGPU GPU stuff alike is a nother come of the > concept of explicit parallelism and its implication in > computing fields, surely more developed in terms of the > hardware this was I believe, but at the end is just another > way of calling Transputers in earlier times, is just like > it, then it was not an abandoned thing like before, just a > reuse of the concept, and in practical terms this what it is > the Occam Occam2 and alike languages; there is one in > particular of my interest here the one built for drawing > using CSP of Hoare, the language > > is called Armadillo see a slide show here, > specially check for the Computer magazine backcover in p. 6 > in: > > https://lsl.unal.edu.co/fl18/conferencias/09.20_ArmadilloTortugav02.pdf > > The TUNES website is a good place to look for plans and > musings in the > direction of formally verified programming: > > http://tunes.org/ > > It's a project that aims to use formal verification for > security, and > has produces a lot of interesting theorizing about the > issues, but not > much practical code. At least, that was the case last > time I looked > at the web site seriously a few years ago. > > -- hendrik > From dabenavidesd at yahoo.es Sun Feb 6 02:14:01 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sun, 6 Feb 2011 01:14:01 +0000 (GMT) Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: <20110206004935.GA31175@topoi.pooq.com> Message-ID: <879742.59049.qm@web29710.mail.ird.yahoo.com> Hi all: yes in a sense, as we don't have the static checker or any static analyzer for its check yet, that is what it is for now, or Am I wrong? Surely, they showed that is possible to find; Damian Dominguez with Dr Nelson to see a thread deadlock in object oriented code but with a special tool very primitive, but in general we don't have that, I mean we just can get where a thread implementation makes this things illegal, not when the program is illegal by itself, that's the aim for me now. Better to think as you say even better. I don't dispute the last point though you can get a tool to analyze a sample of outputs and make a distribution of it thats what I think a professor told me a day when looking for one of threading without locking, just yielding its control, here surely the case is very different. Thanks in advance --- El s?b, 5/2/11, Hendrik Boom escribi?: > De: Hendrik Boom > Asunto: Re: [M3devel] [M3commit] CVS Update: cm3 > Para: m3devel at elegosoft.com > Fecha: s?bado, 5 de febrero, 2011 19:49 > On Sun, Feb 06, 2011 at 12:07:56AM > +0000, Daniel Alejandro Benavides D. wrote: > > Hi all: > ... > ... > > Just to let you know if you see this point. > > It sounds as if you're saying it's possible to prove a > program is faulty > by running it, but it's not possible to say it's correct. > > And concurrency makes things nonrepeatable, especially when > there's > bugs. > > -- hendrik > From jay.krell at cornell.edu Tue Feb 8 15:06:34 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 8 Feb 2011 14:06:34 +0000 Subject: [M3devel] assert in Modula-3 C code? Message-ID: It was recently suspect that asserts aren't enabled in the C code in m3core.h I don't think so. I think there might just be a missing check of a return value in m3core/thread/*.m3 However, if we want to dispell this fear now and forever, we could do e.g.: m3core.h: M3_DLL_LOCAL void __stdcall M3C__AssertFailed (const char* file, unsigned line, const char* expression); #define M3_ASSERT(expr) ((expr) ? ((void)0) ? M3C__AssertFailed ( \ __FILE__, __LINE__, #expression)) hand.c: M3_DLL_LOCAL void __stdcall M3C__AssertFailed (const char* file, unsigned line, const char* expression) { fprintf (stderr, "%s(%u): assertion failed: %s\n", file, line, expression); abort (); } This might slightly reduce quality on some platforms, e.g.: Assertion failed: (0), function main, file 1.c, line 2. Abort trap jbook2:libm3 jay$ cat 1.c #include int main() { assert(0); return 0; } that is -- see how it printed the file name? Anyway, we could get that back too, like #if __GNUC__ > whatever #define M3_FUNCTION something #elif _MSC_VER > something #define M3_FUNCTION __FUNCTION__ #else #define M3_FUNCTION 0 #endif ? But I think probably using standard assert is adequate or preferable. We could #undef NDEBUG in m3core.h, in case someone added to the C compiler on the config files. or #ifdef NDEBUG, #error. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Tue Feb 8 15:27:57 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Tue, 8 Feb 2011 09:27:57 -0500 Subject: [M3devel] assert in Modula-3 C code? In-Reply-To: References: Message-ID: I think I was mistaken. It does appear to be turned on. On Feb 8, 2011, at 9:06 AM, Jay K wrote: > It was recently suspect that asserts aren't enabled in the C code in m3core.h > I don't think so. I think there might just be a missing check of a return value in m3core/thread/*.m3 > > However, if we want to dispell this fear now and forever, we could do e.g.: > > m3core.h: > > M3_DLL_LOCAL > void > __stdcall > M3C__AssertFailed (const char* file, unsigned line, const char* expression); > > #define M3_ASSERT(expr) ((expr) ? ((void)0) ? M3C__AssertFailed ( \ > __FILE__, __LINE__, #expression)) > > > > hand.c: > > M3_DLL_LOCAL > void > __stdcall > M3C__AssertFailed (const char* file, unsigned line, const char* expression) > { > fprintf (stderr, "%s(%u): assertion failed: %s\n", file, line, expression); > abort (); > } > > This might slightly reduce quality on some platforms, e.g.: > > Assertion failed: (0), function main, file 1.c, line 2. > Abort trap > jbook2:libm3 jay$ cat 1.c > #include > int main() { assert(0); return 0; } > > > that is -- see how it printed the file name? Anyway, we could get that back too, like > #if __GNUC__ > whatever > #define M3_FUNCTION something > #elif _MSC_VER > something > #define M3_FUNCTION __FUNCTION__ > #else > #define M3_FUNCTION 0 > #endif > > > ? > > But I think probably using standard assert is adequate or preferable. > We could #undef NDEBUG in m3core.h, in case someone added to the C compiler on the config files. > or #ifdef NDEBUG, #error. > > - Jay > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 8 23:41:44 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 8 Feb 2011 22:41:44 +0000 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: ,,, , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, ,,, , , , , , , <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu>, Message-ID: "Are you sure?" I mean, are you sure the right thing wouldn't be to declare Variables in Try*.Parse or Try*.Check? Possibly they'll even be at a high enough scope as to be correct? ie: they don't really have to be at the function scope, as long as they are just outside the try and the except/finally...depending on how except/finally are generated..ok, this is the less important point. I can see putting them at function-scope being easy enough and more clearly correct. But I do still wonder about the approach using Variable.T and such, instead of something "custom" involving Cg.declare_local and separate hooks in ProcBody esp. for initialization. (Though I can see a "custom" approach having other benefits: e.g. doing a multiplication jmpbuf_size times number of trys and making one larger call to alloca. Furthermore, the approach that uses just one jmpbuf for the entire function, plus a local integer, would more reasonly need a special access to the function scope.) I have a bit more time to spend here now maybe..but I'm tempted to put in a temporary hack of using a 1K jmpbuf for all targets..at least that keeps the per-target code out... and an assert m3core that it is large enough (as was there already). We could also pick some liberal values per-processor, like 64 bytes for x86, 128 for amd64, 512 for sparc, 1K for powerpc/alpha/mips/other, along with the assertion. (powerpc does tend to need quite large jmpbuf, alpha/mips/other are rare and can suffer until a better solution is in place, 64 for x86 is still bloated but not so bad temporarily..)) Again, this is just a stopgap. The fact that very long running loops with a try allocate more and more stack is quite bad. I didn't realize it at the time, sorry. I also think the frontend should declare the EF1 type correctly and itself, so as to initialize the pointer in it, instead of having a separate pointer local variable that gets copied to it. - Jay From: jay.krell at cornell.edu To: hosking at cs.purdue.edu Date: Tue, 1 Feb 2011 07:16:40 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? These look like good hints. I wouldn't know to make such slightly large changes, that the existing infrastructure doesn't suffice. I'll have a go at it..as my now-very-limited time permits. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 22:45:53 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I'd like to do what you envisage. It is a significant improvement, and should not require too much effort: 1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3) 2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. On Jan 31, 2011, at 7:55 PM, Jay K wrote: ps: I'm happy to report that my available time here has become significantly reduced. Partly with that in mind, but even w/o it, I am willing to go back to the old way. m3front is full of mystery and I'm not sure I'm keen on understanding it. - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 15:26:20 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? I think we need to have a way to get the scope of the containing procedure. TryStmt.Parse is not the right place. If I get some time soon I can take a look at this. On Jan 31, 2011, at 2:19 PM, Jay K wrote: Right..I also tried Scope.Insert, didn't seem to help. But I think I might be doing it too late. I'll try doing it in Parse instead of Check. Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat more closely resembles the old pattern: IF Foo() THEN RETURN ELSE TRY FINALLY in the old way would use the stack, so we could too. But the allocation is slower/larger now (alloca possibly aligning up more than needed). Anyway, this point isn't the problem. The problem is my inability to work with m3front. I'll look at it a bit more -- i.e. as I said, moving the code to Parse. The other improvement I'd like to make: currently, if I had it working, what I'm aiming for is: struct EF1{ ... jmpbuf* jb } ef1; jmpbuf* jb = 0; jb = jb ? jb : alloca(sizeof(*jb)); ef1.jb = jb; setjmp(ef1.jb); but it'd be better to have: struct EF1{ ... jmpbuf* jb } ef1; ef1.jb = 0; ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); setjmp(ef1.jb); ie. we don't need the extra pointer variable, the pointer in the record should suffice. [not sure where to put this in email] I tried figuring out how to determine the current vs. topmost-in-current-function scope. I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, so I kinda hoped it'd just use the correct scope. Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. Even if it doesn't get used, that might be ok, since there is tracking of use, and declaring it might not cause any waste if it isn't used. Thanks, - Jay From: hosking at cs.purdue.edu Date: Mon, 31 Jan 2011 12:14:09 -0500 To: jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] fixing try/setjmp in a loop? Hi Jay, I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Jan 30, 2011, at 5:45 AM, Jay K wrote: I'm trying to fix the inadvertant change I made where FOR i := 1 TO 10 DO TRY FINALLY END will call alloca 10 times. I'm not familiar with the frontend, and it is hard for me to understand. It seems like it is probably very well factored, written by someone who really knew what they were doing, but it is dense, and not verbosely commented. Something like this?? But this doesn't work. What am I missing? And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? ? 1.txt Index: src/misc/Marker.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v retrieving revision 1.1.1.1 diff -u -r1.1.1.1 Marker.i3 --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 @@ -64,9 +64,12 @@ PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); (* generate the call to acquire or release a mutex *) -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T); (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) PROCEDURE Reset (); +PROCEDURE NewJumpbuf (): Variable.T; + END Marker. Index: src/misc/Marker.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v retrieving revision 1.9 diff -u -r1.9 Marker.m3 --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 @@ -10,7 +10,7 @@ MODULE Marker; IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; TYPE Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, @@ -231,8 +231,10 @@ END; END CallFinallyHandler; -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; + jumpbuf: Variable.T) = VAR new: BOOLEAN; + label := CG.Next_label (); BEGIN (* int setjmp(void* ); *) IF (setjmp = NIL) THEN @@ -262,13 +264,33 @@ Target.Word.size, Target.Word.align, Target.Word.cg_type, 0); END; - - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) + + (* IF jumpuf # NIL THEN + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); + * END + *) + + CG.Load_nil (); + Variable.Load (jumpbuf); + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); + CG.Start_call_direct (alloca, 0, Target.Address.cg_type); CG.Load_int (Target.Word.cg_type, Jumpbuf_size); CG.Pop_param (Target.Word.cg_type); CG.Call_direct (alloca, Target.Address.cg_type); + + (* FUTURE: We should actually have a Variable + * for the entire EF1, including initializing the jumpbuf + * in it. That would save us this extra load/store, + * and save a local variable. + *) + Variable.LoadLValue (jumpbuf); + CG.Swap (); + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); + Variable.Load (jumpbuf); + CG.Store_addr (frame, M3RT.EF1_jmpbuf); + CG.Set_label (label); (* setmp(frame.jmpbuf) *) CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); @@ -806,5 +828,15 @@ tos := 0; END Reset; +PROCEDURE NewJumpbuf (): Variable.T = + VAR jumpbuf: Variable.T; + BEGIN + jumpbuf := Variable.New (M3ID.NoID, TRUE); + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, + readonly := FALSE, open_array_ok := FALSE, + needs_init := TRUE); + RETURN jumpbuf; + END NewJumpbuf; + BEGIN END Marker. Index: src/stmts/TryFinStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v retrieving revision 1.6 diff -u -r1.6 TryFinStmt.m3 --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -10,6 +10,7 @@ IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; +IMPORT Variable; FROM Stmt IMPORT Outcome; TYPE @@ -20,6 +21,7 @@ viaProc : BOOLEAN; scope : Scope.T; handler : HandlerProc; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -63,6 +65,11 @@ RETURN p; END Parse; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + RETURN NOT (Target.Has_stack_walker OR p.viaProc); + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; BEGIN @@ -95,6 +102,9 @@ END; END; TryStmt.PopHandler (); + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE HandlerName (uid: INTEGER): TEXT = @@ -106,11 +116,16 @@ END HandlerName; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN - IF Target.Has_stack_walker THEN RETURN Compile1 (p); - ELSIF p.viaProc THEN RETURN Compile2 (p); - ELSE RETURN Compile3 (p); + IF Target.Has_stack_walker THEN result := Compile1 (p); + ELSIF p.viaProc THEN result := Compile2 (p); + ELSE usesSetjmp := TRUE; + result := Compile3 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -302,7 +317,9 @@ l := CG.Next_label (3); CG.Set_label (l, barrier := TRUE); Marker.PushFrame (frame, M3RT.HandlerClass.Finally); - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) Marker.PushFinally (l, l+1, frame); Index: src/stmts/TryStmt.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v retrieving revision 1.3 diff -u -r1.3 TryStmt.m3 --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 @@ -22,6 +22,7 @@ hasElse : BOOLEAN; elseBody : Stmt.T; handled : ESet.T; + jumpbuf : Variable.T := NIL; OVERRIDES check := Check; compile := Compile; @@ -153,6 +154,14 @@ p.handles := h3; END ReverseHandlers; +PROCEDURE UsesSetjmp (p: P): BOOLEAN = + BEGIN + IF (p.handles = NIL) AND (NOT p.hasElse) THEN + RETURN FALSE; + END; + RETURN NOT Target.Has_stack_walker; + END UsesSetjmp; + PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = VAR h: Handler; handled: ESet.T; BEGIN @@ -182,6 +191,10 @@ WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; Stmt.TypeCheck (p.elseBody, cs); PopHandler (); + + IF UsesSetjmp (p) THEN + p.jumpbuf := Marker.NewJumpbuf (); + END; END Check; PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = @@ -245,14 +258,19 @@ END CheckHandler; PROCEDURE Compile (p: P): Stmt.Outcomes = + VAR usesSetjmp := FALSE; + result: Stmt.Outcomes; BEGIN IF (p.handles = NIL) AND (NOT p.hasElse) THEN - RETURN Stmt.Compile (p.body); - END; - IF Target.Has_stack_walker - THEN RETURN Compile1 (p); - ELSE RETURN Compile2 (p); + result := Stmt.Compile (p.body); + ELSIF Target.Has_stack_walker THEN + result := Compile1 (p); + ELSE + usesSetjmp := TRUE; + result := Compile2 (p); END; + <* ASSERT usesSetjmp = UsesSetjmp (p) *> + RETURN result; END Compile; PROCEDURE Compile1 (p: P): Stmt.Outcomes = @@ -423,7 +441,9 @@ END; (* capture the machine state *) - Marker.CaptureState (frame, l+1); + <* ASSERT UsesSetjmp (p) *> + <* ASSERT p.jumpbuf # NIL *> + Marker.CaptureState (frame, l+1, p.jumpbuf); (* compile the body *) oc := Stmt.Compile (p.body); Thanks, - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Wed Feb 9 15:18:59 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Wed, 9 Feb 2011 09:18:59 -0500 Subject: [M3devel] fixing try/setjmp in a loop? In-Reply-To: References: , , , , <36BA1892-1828-43A7-B706-AE87B3D98EA1@cs.purdue.edu>, , , , , , , , , , <257A4227-D2C2-4ACD-B49C-6CE620E150CB@cs.purdue.edu>, Message-ID: <65642FC4-4F9C-4F8B-817E-EE32B93D4E11@cs.purdue.edu> That is overkill and not particularly modular. I'd prefer to keep the changes as localised as possible. On Feb 8, 2011, at 5:41 PM, Jay K wrote: > "Are you sure?" > I mean, are you sure the right thing wouldn't be to declare Variables in Try*.Parse or Try*.Check? > Possibly they'll even be at a high enough scope as to be correct? > > > ie: they don't really have to be at the function scope, as long as they are just outside the try and the except/finally...depending on how except/finally are generated..ok, this is the less important point. I can see putting them at function-scope being easy enough and more clearly correct. But I do still wonder about the approach using Variable.T and such, instead of something "custom" involving Cg.declare_local and separate hooks in ProcBody esp. for initialization. > > > (Though I can see a "custom" approach having other benefits: e.g. doing a multiplication jmpbuf_size times number of trys and making one larger call to alloca. Furthermore, the approach that uses just one jmpbuf for the entire function, plus a local integer, would more reasonly need a special access to the function scope.) > > > I have a bit more time to spend here now maybe..but I'm tempted to put in a temporary hack of using a 1K jmpbuf for all targets..at least that keeps the per-target code out... and an assert m3core that it is large enough (as was there already). > > > > We could also pick some liberal values per-processor, like 64 bytes for x86, 128 for amd64, 512 for sparc, 1K for powerpc/alpha/mips/other, along with the assertion. (powerpc does tend to need quite large jmpbuf, alpha/mips/other are rare and can suffer until a better solution is in place, 64 for x86 is still bloated but not so bad temporarily..)) Again, this is just a stopgap. The fact that very long running loops with a try allocate more and more stack is quite bad. I didn't realize it at the time, sorry. > > > I also think the frontend should declare the EF1 type correctly and itself, so as to initialize the pointer in it, instead of having a separate pointer local variable that gets copied to it. > > > - Jay > > > From: jay.krell at cornell.edu > To: hosking at cs.purdue.edu > Date: Tue, 1 Feb 2011 07:16:40 +0000 > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > These look like good hints. I wouldn't know to make such slightly large changes, > that the existing infrastructure doesn't suffice. > I'll have a go at it..as my now-very-limited time permits. > > Thanks, > - Jay > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 22:45:53 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I'd like to do what you envisage. It is a significant improvement, and should not require too much effort: > 1) Use Declare_local and track all marker declarations in Marker.m3 (similar to Tracer.m3) > 2) Have a callback from Procedure.GenBody (right after Scope.InitValues) to a new procedure Marker.EmitPending that emits the declarations and the initialisations of the variables to NIL. > > On Jan 31, 2011, at 7:55 PM, Jay K wrote: > > ps: I'm happy to report that my available time here > has become significantly reduced. Partly with that in mind, > but even w/o it, I am willing to go back to the old way. > m3front is full of mystery and I'm not sure I'm keen on understanding it. > > > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 15:26:20 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > I think we need to have a way to get the scope of the containing procedure. > TryStmt.Parse is not the right place. > If I get some time soon I can take a look at this. > > On Jan 31, 2011, at 2:19 PM, Jay K wrote: > > Right..I also tried Scope.Insert, didn't seem to help. > > But I think I might be doing it too late. I'll try doing it in Parse instead of Check. > Though I don't recall if Parse knows enough -- if it knows if CaptureState will be called. > I agree/disagree about alloca up front, with the obvious reasons -- it is smaller code to alloca up front, and it somewhat > more closely resembles the old pattern: > > IF Foo() THEN > RETURN > ELSE TRY FINALLY > > in the old way would use the stack, so we could too. > But the allocation is slower/larger now (alloca possibly aligning up more than needed). > Anyway, this point isn't the problem. The problem is my inability to work with m3front. > I'll look at it a bit more -- i.e. as I said, moving the code to Parse. > > The other improvement I'd like to make: > > currently, if I had it working, what I'm aiming for is: > struct EF1{ ... jmpbuf* jb } ef1; > jmpbuf* jb = 0; > > jb = jb ? jb : alloca(sizeof(*jb)); > ef1.jb = jb; > setjmp(ef1.jb); > > but it'd be better to have: > > struct EF1{ ... jmpbuf* jb } ef1; > ef1.jb = 0; > > ef1.jb = ef1.jb ? ef1.jb : alloca(sizeof(*jb)); > setjmp(ef1.jb); > > > ie. we don't need the extra pointer variable, the pointer in the record should suffice. > > > [not sure where to put this in email] > I tried figuring out how to determine the current vs. topmost-in-current-function scope. > I'm not sure if Scope.Top is about in a function or if it is module scope, or even perhaps > some outside-all-modules scope. But Scope.Insert doesn't let you specify a scope, > so I kinda hoped it'd just use the correct scope. > > > Anyway, I'll look again at introducing the Variable.New and Scope.Insert during Parse. > Even if it doesn't get used, that might be ok, since there is tracking of use, and > declaring it might not cause any waste if it isn't used. > > > Thanks, > - Jay > > > From: hosking at cs.purdue.edu > Date: Mon, 31 Jan 2011 12:14:09 -0500 > To: jay.krell at cornell.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] fixing try/setjmp in a loop? > > Hi Jay, > > I've not had a chance to digest this, but I do understand the problem. We need the Variable to be entered into the outermost scope of the procedure in which the TRY appears so that it will have initialization code (assigning NIL) generated for it on entry to the procedure. Then your usage of the variable should simply check for NIL and alloca if it is, as you already appear to do here. It would be possible to have the initialization code perform the alloca but then we would have an alloca for every TRY block in a procedure even if they never execute. Probably not the best plan. > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > On Jan 30, 2011, at 5:45 AM, Jay K wrote: > > I'm trying to fix the inadvertant change I made where > FOR i := 1 TO 10 DO TRY FINALLY END > > will call alloca 10 times. > > I'm not familiar with the frontend, and it is hard for me to understand. > It seems like it is probably very well factored, written by someone who really knew what they were doing, > but it is dense, and not verbosely commented. > > Something like this?? But this doesn't work. What am I missing? > And, we should go the next step and merge the jmpbuf pointer variable with the jmpbuf field in the EF1, right? > > > ? 1.txt > Index: src/misc/Marker.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.i3,v > retrieving revision 1.1.1.1 > diff -u -r1.1.1.1 Marker.i3 > --- src/misc/Marker.i3 14 Jan 2001 13:40:31 -0000 1.1.1.1 > +++ src/misc/Marker.i3 30 Jan 2011 10:41:18 -0000 > @@ -64,9 +64,12 @@ > PROCEDURE SetLock (acquire: BOOLEAN; var: CG.Var; offset: INTEGER); > (* generate the call to acquire or release a mutex *) > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label); > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T); > (* call 'setjmp' on 'frame's jmpbuf and branch to 'handler' on re-returns. *) > > PROCEDURE Reset (); > > +PROCEDURE NewJumpbuf (): Variable.T; > + > END Marker. > Index: src/misc/Marker.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/misc/Marker.m3,v > retrieving revision 1.9 > diff -u -r1.9 Marker.m3 > --- src/misc/Marker.m3 13 Jan 2011 14:58:28 -0000 1.9 > +++ src/misc/Marker.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,7 +10,7 @@ > MODULE Marker; > > IMPORT CG, Error, Type, Variable, ProcType, ESet, Expr, AssignStmt; > -IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host; > +IMPORT M3ID, M3RT, Target, Module, RunTyme, Procedure, Host, Addr; > > TYPE > Kind = { zFINALLY, zFINALLYPROC, zLOCK, zEXIT, zTRY, zTRYELSE, > @@ -231,8 +231,10 @@ > END; > END CallFinallyHandler; > > -PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label) = > +PROCEDURE CaptureState (frame: CG.Var; handler: CG.Label; > + jumpbuf: Variable.T) = > VAR new: BOOLEAN; > + label := CG.Next_label (); > BEGIN > (* int setjmp(void* ); *) > IF (setjmp = NIL) THEN > @@ -262,13 +264,33 @@ > Target.Word.size, Target.Word.align, > Target.Word.cg_type, 0); > END; > - > - (* frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); *) > + > + (* IF jumpuf # NIL THEN > + * frame.jmpbuf = alloca(Csetjmp__Jumpbuf_size); > + * END > + *) > + > + CG.Load_nil (); > + Variable.Load (jumpbuf); > + CG.If_compare (CG.Type.Addr, CG.Cmp.NE, label, CG.Maybe); > + > CG.Start_call_direct (alloca, 0, Target.Address.cg_type); > CG.Load_int (Target.Word.cg_type, Jumpbuf_size); > CG.Pop_param (Target.Word.cg_type); > CG.Call_direct (alloca, Target.Address.cg_type); > + > + (* FUTURE: We should actually have a Variable > + * for the entire EF1, including initializing the jumpbuf > + * in it. That would save us this extra load/store, > + * and save a local variable. > + *) > + Variable.LoadLValue (jumpbuf); > + CG.Swap (); > + CG.Store_indirect (CG.Type.Addr, 0, Target.Address.size); > + Variable.Load (jumpbuf); > + > CG.Store_addr (frame, M3RT.EF1_jmpbuf); > + CG.Set_label (label); > > (* setmp(frame.jmpbuf) *) > CG.Start_call_direct (setjmp, 0, Target.Integer.cg_type); > @@ -806,5 +828,15 @@ > tos := 0; > END Reset; > > +PROCEDURE NewJumpbuf (): Variable.T = > + VAR jumpbuf: Variable.T; > + BEGIN > + jumpbuf := Variable.New (M3ID.NoID, TRUE); > + Variable.BindType (jumpbuf, Addr.T, indirect := FALSE, > + readonly := FALSE, open_array_ok := FALSE, > + needs_init := TRUE); > + RETURN jumpbuf; > + END NewJumpbuf; > + > BEGIN > END Marker. > Index: src/stmts/TryFinStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryFinStmt.m3,v > retrieving revision 1.6 > diff -u -r1.6 TryFinStmt.m3 > --- src/stmts/TryFinStmt.m3 5 Jan 2011 14:34:54 -0000 1.6 > +++ src/stmts/TryFinStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -10,6 +10,7 @@ > > IMPORT M3ID, CG, Token, Scanner, Stmt, StmtRep, Marker, Target, Type, Addr; > IMPORT RunTyme, Procedure, ProcBody, M3RT, Scope, Fmt, Host, TryStmt, Module; > +IMPORT Variable; > FROM Stmt IMPORT Outcome; > > TYPE > @@ -20,6 +21,7 @@ > viaProc : BOOLEAN; > scope : Scope.T; > handler : HandlerProc; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -63,6 +65,11 @@ > RETURN p; > END Parse; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + RETURN NOT (Target.Has_stack_walker OR p.viaProc); > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR zz: Scope.T; oc: Stmt.Outcomes; name: INTEGER; > BEGIN > @@ -95,6 +102,9 @@ > END; > END; > TryStmt.PopHandler (); > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE HandlerName (uid: INTEGER): TEXT = > @@ -106,11 +116,16 @@ > END HandlerName; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > - IF Target.Has_stack_walker THEN RETURN Compile1 (p); > - ELSIF p.viaProc THEN RETURN Compile2 (p); > - ELSE RETURN Compile3 (p); > + IF Target.Has_stack_walker THEN result := Compile1 (p); > + ELSIF p.viaProc THEN result := Compile2 (p); > + ELSE usesSetjmp := TRUE; > + result := Compile3 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -302,7 +317,9 @@ > l := CG.Next_label (3); > CG.Set_label (l, barrier := TRUE); > Marker.PushFrame (frame, M3RT.HandlerClass.Finally); > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > Marker.PushFinally (l, l+1, frame); > Index: src/stmts/TryStmt.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3front/src/stmts/TryStmt.m3,v > retrieving revision 1.3 > diff -u -r1.3 TryStmt.m3 > --- src/stmts/TryStmt.m3 5 Jan 2011 14:34:54 -0000 1.3 > +++ src/stmts/TryStmt.m3 30 Jan 2011 10:41:18 -0000 > @@ -22,6 +22,7 @@ > hasElse : BOOLEAN; > elseBody : Stmt.T; > handled : ESet.T; > + jumpbuf : Variable.T := NIL; > OVERRIDES > check := Check; > compile := Compile; > @@ -153,6 +154,14 @@ > p.handles := h3; > END ReverseHandlers; > > +PROCEDURE UsesSetjmp (p: P): BOOLEAN = > + BEGIN > + IF (p.handles = NIL) AND (NOT p.hasElse) THEN > + RETURN FALSE; > + END; > + RETURN NOT Target.Has_stack_walker; > + END UsesSetjmp; > + > PROCEDURE Check (p: P; VAR cs: Stmt.CheckState) = > VAR h: Handler; handled: ESet.T; > BEGIN > @@ -182,6 +191,10 @@ > WHILE (h # NIL) DO CheckHandler (h, cs); h := h.next; END; > Stmt.TypeCheck (p.elseBody, cs); > PopHandler (); > + > + IF UsesSetjmp (p) THEN > + p.jumpbuf := Marker.NewJumpbuf (); > + END; > END Check; > > PROCEDURE CheckLabels (h: Handler; scope: Scope.T; VAR cs: Stmt.CheckState) = > @@ -245,14 +258,19 @@ > END CheckHandler; > > PROCEDURE Compile (p: P): Stmt.Outcomes = > + VAR usesSetjmp := FALSE; > + result: Stmt.Outcomes; > BEGIN > IF (p.handles = NIL) AND (NOT p.hasElse) THEN > - RETURN Stmt.Compile (p.body); > - END; > - IF Target.Has_stack_walker > - THEN RETURN Compile1 (p); > - ELSE RETURN Compile2 (p); > + result := Stmt.Compile (p.body); > + ELSIF Target.Has_stack_walker THEN > + result := Compile1 (p); > + ELSE > + usesSetjmp := TRUE; > + result := Compile2 (p); > END; > + <* ASSERT usesSetjmp = UsesSetjmp (p) *> > + RETURN result; > END Compile; > > PROCEDURE Compile1 (p: P): Stmt.Outcomes = > @@ -423,7 +441,9 @@ > END; > > (* capture the machine state *) > - Marker.CaptureState (frame, l+1); > + <* ASSERT UsesSetjmp (p) *> > + <* ASSERT p.jumpbuf # NIL *> > + Marker.CaptureState (frame, l+1, p.jumpbuf); > > (* compile the body *) > oc := Stmt.Compile (p.body); > > Thanks, > - Jay > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Wed Feb 9 23:00:57 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Wed, 09 Feb 2011 14:00:57 -0800 Subject: [M3devel] threading on Windows? Message-ID: <20110209220057.790671A2078@async.async.caltech.edu> Hi m3devel, I'm just curious if anyone out there who's running CM3 on Windows has had a chance to try my thread testing program yet. Current status on Unix (Linux, FreeBSD) appears to be that there are problems in pthreads (but I think only under heavy load) and that user threading works perfectly. So I now wonder how things are on the third threading platform (Windows). The thread testing program is at m3-libs/m3core/tests/thread . (Note however that there is a memory leak in TRY-EXCEPT in the current head so don't try to update everything to the CVS head. The previous release should be OK.) Also has anyone else noticed that debugging information has recently broken? m3gdb is very unhappy on all platforms for me... Mika From rcolebur at SCIRES.COM Wed Feb 9 23:25:20 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Wed, 9 Feb 2011 17:25:20 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <20110209220057.790671A2078@async.async.caltech.edu> References: <20110209220057.790671A2078@async.async.caltech.edu> Message-ID: Mika: Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. As soon as I can get the build problem resolved, I'll try out your test program. Regards, Randy Coleburn -----Original Message----- From: Mika Nystrom [mailto:mika at async.caltech.edu] Sent: Wednesday, February 09, 2011 5:01 PM To: m3devel at elegosoft.com Subject: [M3devel] threading on Windows? Hi m3devel, I'm just curious if anyone out there who's running CM3 on Windows has had a chance to try my thread testing program yet. Current status on Unix (Linux, FreeBSD) appears to be that there are problems in pthreads (but I think only under heavy load) and that user threading works perfectly. So I now wonder how things are on the third threading platform (Windows). The thread testing program is at m3-libs/m3core/tests/thread . (Note however that there is a memory leak in TRY-EXCEPT in the current head so don't try to update everything to the CVS head. The previous release should be OK.) Also has anyone else noticed that debugging information has recently broken? m3gdb is very unhappy on all platforms for me... Mika From jay.krell at cornell.edu Wed Feb 9 23:58:30 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 9 Feb 2011 22:58:30 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, Message-ID: Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Thu Feb 10 00:22:56 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, Message-ID: I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 10 03:01:22 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 10 Feb 2011 02:01:22 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , Message-ID: In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Thu Feb 10 04:28:45 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Wed, 9 Feb 2011 22:28:45 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , Message-ID: Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 10 05:40:03 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 10 Feb 2011 04:40:03 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , , Message-ID: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -0500 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Feb 10 06:48:20 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 10 Feb 2011 00:48:20 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , , Message-ID: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: > Of course not. > It is a made-up function that the frontend generates a call to. > That it never previously did. > Using the general function call interface. > You need to add a special case, in the general function call code, to do something different and specific > for this specific function. > > > - Jay > > From: rcolebur at SCIRES.COM > To: jay.krell at cornell.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 22:28:45 -0500 > Subject: Re: [M3devel] threading on Windows? > > Jay: > > I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. > > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 9:01 PM > To: Coleburn, Randy; m3devel > Subject: RE: [M3devel] threading on Windows? > > In m3back/src/M3x86.m3. > > - Jay > > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 18:22:56 -0500 > Subject: Re: [M3devel] threading on Windows? > > I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. > All I know is that everything was working fine until I checked out the HEAD repository. > When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? > Are we talking Modula-3 code, C code, Assembler, what? What package/module? > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 5:59 PM > To: Coleburn, Randy; Mika Nystrom; m3devel > Subject: RE: [M3devel] threading on Windows? > > Maybe someone else can do it? > The fix is: special case calls to the function "m3_alloca". > Change it to call "_chkstk" (or maybe "chkstk", whatever works). > The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. > To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. > > - Jay > > > From: rcolebur at SCIRES.COM > > To: mika at async.caltech.edu; m3devel at elegosoft.com > > Date: Wed, 9 Feb 2011 17:25:20 -0500 > > Subject: Re: [M3devel] threading on Windows? > > > > Mika: > > > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > > > As soon as I can get the build problem resolved, I'll try out your test program. > > > > Regards, > > Randy Coleburn > > > > -----Original Message----- > > From: Mika Nystrom [mailto:mika at async.caltech.edu] > > Sent: Wednesday, February 09, 2011 5:01 PM > > To: m3devel at elegosoft.com > > Subject: [M3devel] threading on Windows? > > > > Hi m3devel, > > > > I'm just curious if anyone out there who's running CM3 on Windows has had > > a chance to try my thread testing program yet. Current status on Unix > > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > > I think only under heavy load) and that user threading works perfectly. > > So I now wonder how things are on the third threading platform (Windows). > > The thread testing program is at m3-libs/m3core/tests/thread . > > > > (Note however that there is a memory leak in TRY-EXCEPT in the current > > head so don't try to update everything to the CVS head. The previous > > release should be OK.) > > > > Also has anyone else noticed that debugging information has recently > > broken? m3gdb is very unhappy on all platforms for me... > > > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 10 12:01:12 2011 From: jay.krell at cornell.edu (jay.krell at cornell.edu) Date: Thu, 10 Feb 2011 03:01:12 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu> Message-ID: <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: > What happened to fixing this with runtime hooks? > > > On Feb 9, 2011, at 11:40 PM, Jay K wrote: > >> Of course not. >> It is a made-up function that the frontend generates a call to. >> That it never previously did. >> Using the general function call interface. >> You need to add a special case, in the general function call code, to do something different and specific >> for this specific function. >> >> >> - Jay >> >> From: rcolebur at SCIRES.COM >> To: jay.krell at cornell.edu; m3devel at elegosoft.com >> Date: Wed, 9 Feb 2011 22:28:45 -050 >> Subject: Re: [M3devel] threading on Windows? >> >> Jay: >> >> I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. >> >> Regards, >> Randy >> >> From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K >> Sent: Wednesday, February 09, 2011 9:01 PM >> To: Coleburn, Randy; m3devel >> Subject: RE: [M3devel] threading on Windows? >> >> In m3back/src/M3x86.m3. >> >> - Jay >> >> >> From: rcolebur at SCIRES.COM >> To: m3devel at elegosoft.com >> Date: Wed, 9 Feb 2011 18:22:56 -0500 >> Subject: Re: [M3devel] threading on Windows? >> >> I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. >> All I know is that everything was working fine until I checked out the HEAD repository. >> When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? >> Are we talking Modula-3 code, C code, Assembler, what? What package/module? >> Regards, >> Randy >> >> From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K >> Sent: Wednesday, February 09, 2011 5:59 PM >> To: Coleburn, Randy; Mika Nystrom; m3devel >> Subject: RE: [M3devel] threading on Windows? >> >> Maybe someone else can do it? >> The fix is: special case calls to the function "m3_alloca". >> Change it to call "_chkstk" (or maybe "chkstk", whatever works). >> The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. >> To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. >> >> - Jay >> >> > From: rcolebur at SCIRES.COM >> > To: mika at async.caltech.edu; m3devel at elegosoft.com >> > Date: Wed, 9 Feb 2011 17:25:20 -0500 >> > Subject: Re: [M3devel] threading on Windows? >> > >> > Mika: >> > >> > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. >> > >> > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. >> > >> > As soon as I can get the build problem resolved, I'll try out your test program. >> > >> > Regards, >> > Randy Coleburn >> > >> > -----Original Message----- >> > From: Mika Nystrom [mailto:mika at async.caltech.edu] >> > Sent: Wednesday, February 09, 2011 5:01 PM >> > To: m3devel at elegosoft.com >> > Subject: [M3devel] threading on Windows? >> > >> > Hi m3devel, >> > >> > I'm just curious if anyone out there who's running CM3 on Windows has had >> > a chance to try my thread testing program yet. Current status on Unix >> > (Linux, FreeBSD) appears to be that there are problems in pthreads (but >> > I think only under heavy load) and that user threading works perfectly. >> > So I now wonder how things are on the third threading platform (Windows). >> > The thread testing program is at m3-libs/m3core/tests/thread . >> > >> > (Note however that there is a memory leak in TRY-EXCEPT in the current >> > head so don't try to update everything to the CVS head. The previous >> > release should be OK.) >> > >> > Also has anyone else noticed that debugging information has recently >> > broken? m3gdb is very unhappy on all platforms for me... >> > >> > Mika > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Feb 10 21:32:41 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 10 Feb 2011 15:32:41 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> Message-ID: <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: > 1. I never understood why this any better. > 2. It how things are currently. > 3. Why not m3cg.alloca? (but see #2) > > - Jay/iPad > > On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: > >> What happened to fixing this with runtime hooks? >> >> >> On Feb 9, 2011, at 11:40 PM, Jay K wrote: >> >>> Of course not. >>> It is a made-up function that the frontend generates a call to. >>> That it never previously did. >>> Using the general function call interface. >>> You need to add a special case, in the general function call code, to do something different and specific >>> for this specific function. >>> >>> >>> - Jay >>> >>> From: rcolebur at SCIRES.COM >>> To: jay.krell at cornell.edu; m3devel at elegosoft.com >>> Date: Wed, 9 Feb 2011 22:28:45 -050 >>> Subject: Re: [M3devel] threading on Windows? >>> >>> Jay: >>> >>> I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. >>> >>> Regards, >>> Randy >>> >>> From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K >>> Sent: Wednesday, February 09, 2011 9:01 PM >>> To: Coleburn, Randy; m3devel >>> Subject: RE: [M3devel] threading on Windows? >>> >>> In m3back/src/M3x86.m3. >>> >>> - Jay >>> >>> >>> From: rcolebur at SCIRES.COM >>> To: m3devel at elegosoft.com >>> Date: Wed, 9 Feb 2011 18:22:56 -0500 >>> Subject: Re: [M3devel] threading on Windows? >>> >>> I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. >>> All I know is that everything was working fine until I checked out the HEAD repository. >>> When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? >>> Are we talking Modula-3 code, C code, Assembler, what? What package/module? >>> Regards, >>> Randy >>> >>> From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K >>> Sent: Wednesday, February 09, 2011 5:59 PM >>> To: Coleburn, Randy; Mika Nystrom; m3devel >>> Subject: RE: [M3devel] threading on Windows? >>> >>> Maybe someone else can do it? >>> The fix is: special case calls to the function "m3_alloca". >>> Change it to call "_chkstk" (or maybe "chkstk", whatever works). >>> The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. >>> To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. >>> >>> - Jay >>> >>> > From: rcolebur at SCIRES.COM >>> > To: mika at async.caltech.edu; m3devel at elegosoft.com >>> > Date: Wed, 9 Feb 2011 17:25:20 -0500 >>> > Subject: Re: [M3devel] threading on Windows? >>> > >>> > Mika: >>> > >>> > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. >>> > >>> > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. >>> > >>> > As soon as I can get the build problem resolved, I'll try out your test program. >>> > >>> > Regards, >>> > Randy Coleburn >>> > >>> > -----Original Message----- >>> > From: Mika Nystrom [mailto:mika at async.caltech.edu] >>> > Sent: Wednesday, February 09, 2011 5:01 PM >>> > To: m3devel at elegosoft.com >>> > Subject: [M3devel] threading on Windows? >>> > >>> > Hi m3devel, >>> > >>> > I'm just curious if anyone out there who's running CM3 on Windows has had >>> > a chance to try my thread testing program yet. Current status on Unix >>> > (Linux, FreeBSD) appears to be that there are problems in pthreads (but >>> > I think only under heavy load) and that user threading works perfectly. >>> > So I now wonder how things are on the third threading platform (Windows). >>> > The thread testing program is at m3-libs/m3core/tests/thread . >>> > >>> > (Note however that there is a memory leak in TRY-EXCEPT in the current >>> > head so don't try to update everything to the CVS head. The previous >>> > release should be OK.) >>> > >>> > Also has anyone else noticed that debugging information has recently >>> > broken? m3gdb is very unhappy on all platforms for me... >>> > >>> > Mika >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Thu Feb 10 21:59:25 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Thu, 10 Feb 2011 15:59:25 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> Message-ID: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am "in the dark", both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I'm seeing several different symbols that don't resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking > wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Thu Feb 10 22:25:51 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Thu, 10 Feb 2011 16:25:51 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> Message-ID: <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu> Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: > Jay / Tony: > > I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. > > I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. > > I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. > > BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. > > Regards, > Randy > > From: Tony Hosking [mailto:hosking at cs.purdue.edu] > Sent: Thursday, February 10, 2011 3:33 PM > To: jay.krell at cornell.edu > Cc: Coleburn, Randy; m3devel > Subject: Re: [M3devel] threading on Windows? > > Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. > > On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: > > > 1. I never understood why this any better. > 2. It how things are currently. > 3. Why not m3cg.alloca? (but see #2) > > - Jay/iPad > > On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: > > What happened to fixing this with runtime hooks? > > > On Feb 9, 2011, at 11:40 PM, Jay K wrote: > > > Of course not. > It is a made-up function that the frontend generates a call to. > That it never previously did. > Using the general function call interface. > You need to add a special case, in the general function call code, to do something different and specific > for this specific function. > > > - Jay > > From: rcolebur at SCIRES.COM > To: jay.krell at cornell.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 22:28:45 -050 > Subject: Re: [M3devel] threading on Windows? > > Jay: > > I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. > > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 9:01 PM > To: Coleburn, Randy; m3devel > Subject: RE: [M3devel] threading on Windows? > > In m3back/src/M3x86.m3. > > - Jay > > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 18:22:56 -0500 > Subject: Re: [M3devel] threading on Windows? > > I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. > All I know is that everything was working fine until I checked out the HEAD repository. > When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? > Are we talking Modula-3 code, C code, Assembler, what? What package/module? > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 5:59 PM > To: Coleburn, Randy; Mika Nystrom; m3devel > Subject: RE: [M3devel] threading on Windows? > > Maybe someone else can do it? > The fix is: special case calls to the function "m3_alloca". > Change it to call "_chkstk" (or maybe "chkstk", whatever works). > The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. > To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. > > - Jay > > > From: rcolebur at SCIRES.COM > > To: mika at async.caltech.edu; m3devel at elegosoft.com > > Date: Wed, 9 Feb 2011 17:25:20 -0500 > > Subject: Re: [M3devel] threading on Windows? > > > > Mika: > > > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > > > As soon as I can get the build problem resolved, I'll try out your test program. > > > > Regards, > > Randy Coleburn > > > > -----Original Message----- > > From: Mika Nystrom [mailto:mika at async.caltech.edu] > > Sent: Wednesday, February 09, 2011 5:01 PM > > To: m3devel at elegosoft.com > > Subject: [M3devel] threading on Windows? > > > > Hi m3devel, > > > > I'm just curious if anyone out there who's running CM3 on Windows has had > > a chance to try my thread testing program yet. Current status on Unix > > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > > I think only under heavy load) and that user threading works perfectly. > > So I now wonder how things are on the third threading platform (Windows). > > The thread testing program is at m3-libs/m3core/tests/thread . > > > > (Note however that there is a memory leak in TRY-EXCEPT in the current > > head so don't try to update everything to the CVS head. The previous > > release should be OK.) > > > > Also has anyone else noticed that debugging information has recently > > broken? m3gdb is very unhappy on all platforms for me... > > > > Mika > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 10 23:11:55 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 10 Feb 2011 22:11:55 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> Message-ID: Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 11 00:13:57 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Thu, 10 Feb 2011 23:13:57 +0000 (GMT) Subject: [M3devel] threading on Windows? In-Reply-To: <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> Message-ID: <295326.94632.qm@web29715.mail.ird.yahoo.com> Hi all: by the way this must be a top one if you want cross platform compatibility rather than built-in platform specific fixes, why just we can make another thread runtime and put everything on it, e.g make this threading behavior consistent with the rest of pthread unsupported platforms (say DECthreads, or any other thread safe interface, was this ever supported i.e for SPIN trends threads or as now Pthreads are as thread safe?), at lower layer level, i.e like it was with the user threads, perhaps it is like the counter example you would need to test more deeply in the implementation not other ones' implementations, perhaps? Is there any test invalid currently, or any of it correct, if there isn't a "safe" pattern in the code why do you test unsafe threaded code (i.e all behave odly bad or all behave the same bad? This is to make another system thread safe implementation, like it was SPIN, sadly uniprocessor platform, this is better to test in an uniprocessor rather than multiprocessor indeed as its a simulation of the real code is running on it), still very impressive bench marks time in the user level or kernel level for DEC system SPIN , is like the same? for both user level ones and besides it behaves like it was a Mach 3.0 kernel user or a DECThreads client, as it had it underneath it layered, this proves a better runtime makes even better for correctness and efficiency, at least is what I see in the benchmarks, who knows with SMP systems, but just a point on it. Besides this SPIN trends threads were more abstract and more specialized e.g allow to select scheduling policy (currently at compile time in pthreads) and timing selected values. I wonder how anything of this would be manageable on a SMP machine, still worth the value to check as it is really faster than anything on the market for those days, see: http://qstream.org/~krasic/cs508-2006/summaries/paper17/SpinOS.ppt http://www.cs.cornell.edu/People/egs/papers/spin-strands.pdf Yet another hint for your thread testing, there is a static checker for threading, it used thread states in object oriented programs written in path expression constraints and proved to check threads synchronization consistency, one would be able to compile it to thread primitives to check for semantic analysis and optimization, just in case the references see below at the end of this paragraph, I must say this looks very interesting so your work would be to create annotations not trace them, or trace them if needed for create them, but still very promising if one is capable of creating the abstract annotations to some extent ahead of time or later if in execution time, as it was in some intelligent debuggers, at some extent should be the call graphs make the path expressions, perhaps abstract as through a Trestle Locking level abstraction and a thread safe condition? http://www.informatik.fh-luebeck.de/Seehusen/Publications/OS92/osna.ps http://130.203.133.150/viewdoc/versions;jsessionid=6B086982E4C53F40A01EDA48A003EE3A?doi=10.1.1.34.1092 (let it http://www.cs.rug.nl/~wim/pub/whh217.ps.gz ) and http://www.cs.rug.nl/~wim/pub/whh233.ps.gz and so http://books.google.com/books?id=Hb_pOpI8mJ8C&lpg=PP1&dq=Concurrent%20programming&pg=PA221#v=onepage&q&f=false http://www.cs.brown.edu/people/spr/fieldbook/fieldbook.html Thanks in advance --- El jue, 10/2/11, jay.krell at cornell.edu escribi?: De: jay.krell at cornell.edu Asunto: Re: [M3devel] threading on Windows? Para: "Tony Hosking" CC: "m3devel" , "Jay K" Fecha: jueves, 10 de febrero, 2011 06:01 1. I never understood why this any better. 2. It how things are currently.3. Why not m3cg.alloca? (but see #2) ?- Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. ? ? ?- Jay? ? From:?rcolebur at SCIRES.COM To:?jay.krell at cornell.edu;?m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay:?I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file.?Regards,Randy?From:?jayk123 at hotmail.com?[mailto:jayk123 at hotmail.com]?On Behalf Of?Jay K Sent:?Wednesday, February 09, 2011 9:01 PM To:?Coleburn, Randy; m3devel Subject:?RE: [M3devel] threading on Windows??In m3back/src/M3x86.m3. ? ?- Jay ?From:?rcolebur at SCIRES.COM To:?m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows?I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve.All I know is that everything was working fine until I checked out the HEAD repository.When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case??Are we talking Modula-3 code, C code, Assembler, what?? What package/module?Regards,Randy?From:?jayk123 at hotmail.com?[mailto:jayk123 at hotmail.com]?On Behalf Of?Jay K Sent:?Wednesday, February 09, 2011 5:59 PM To:?Coleburn, Randy; Mika Nystrom; m3devel Subject:?RE: [M3devel] threading on Windows??Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed?in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. ? ?- Jay ? > From:?rcolebur at SCIRES.COM > To:?mika at async.caltech.edu;?m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? >? > Mika: >? > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. >? > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. >? > As soon as I can get the build problem resolved, I'll try out your test program. >? > Regards, > Randy Coleburn >? > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu]? > Sent: Wednesday, February 09, 2011 5:01 PM > To:?m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? >? > Hi m3devel, >? > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . >? > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) >? > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... >? > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 11 00:40:48 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Thu, 10 Feb 2011 23:40:48 +0000 (GMT) Subject: [M3devel] threading on Windows? In-Reply-To: Message-ID: <161929.78691.qm@web29701.mail.ird.yahoo.com> Hi all: is useful to say here that the compiler linker approach at some extent was done in the same way as was in Modula-3, now Open Watcom, used same runtime structure and required as far as I know no better enlightment in the original versions of it, I read it some quite time ago in its sources, hard to tell you where it was, but if OpenWatcom sold it as a Windows solution in its runtime why not in the Modula-3 one, indeed how it would be to port its linker to work for Modula-3 and tune it for the? required? works of it. Is better to adapt to a better portable (Linux, etc) solution than? to adapt to a? non-portable still solution as I believe. Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] threading on Windows? Para: "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 17:11 Why is it better? ?Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) ?- Jay/iPad ? You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. ? ? Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. ? ? As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. ? ? Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. ? ? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Feb 11 01:36:12 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 11 Feb 2011 00:36:12 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <161929.78691.qm@web29701.mail.ird.yahoo.com> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> Message-ID: Daniel, you generally make very little sense to me. Please consider additional English training. "Modula-3" defines and implements its own threading library, it's own I/O library, it's own traced and untraced memory allocator (traced == garbage collected), it's own GUI library (Trestle). It is its own portabilty layer, akin to various other options, e.g. using pthreads and then implementing pthreads on Windows, using Tk or Qt, etc., and depending on them to be ported. etc. Yes, I am deliberately confusing "language" and "library". Which are people interested in? One, the other, or both? Now, granted, we are portable between essentially only two platforms Win32/NT and "Posix", where "Posix" is defined as including pthreads and X Windows, and even in a sense gcc. (Even if Modula-3 predates Posix, the dependencies are pretty light, such that they have mostly always been almost consistent across the board -- open/read/write/close, socket/accept/connect/recv/send/listen, a few variations for signals). (Personally I would like to change the backend to generate C...) One could "get out of this (library) business" and target another intermediate portability layer, i.e. write "bindings" for Tk, Qt, etc., and then use Modula-3 "just" as a language, and not an associated large set of libraries. In general, it is desired for portability layers to be thin in terms of performance and possibly state. Ideally, given the choice of writing to Win32 threads and getting the smallest fastest program that runs only on Wondows, or pthreads and getting the smallest fastest program that runs on Posix, nobody should shy away from using Modula-3 threads, since they run pretty much just as well as Win32 and pthreads. Pretty much. Actually condition variables can't be done super well on Win32 prior to Vista, but they are done in a pretty ok way now. Still, threads and mutexes at least are very thin layers. Implementing threading more completely yourself, on many systems, precludes you from actually running on more than one processor. Win7's "user mode scheduling" offers an option with more control and still the ability to run on multiple processors. But I suspect we aren't very interested, at least not currently. - Jay Date: Thu, 10 Feb 2011 23:40:48 +0000 From: dabenavidesd at yahoo.es To: hosking at cs.purdue.edu; jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Hi all: is useful to say here that the compiler linker approach at some extent was done in the same way as was in Modula-3, now Open Watcom, used same runtime structure and required as far as I know no better enlightment in the original versions of it, I read it some quite time ago in its sources, hard to tell you where it was, but if OpenWatcom sold it as a Windows solution in its runtime why not in the Modula-3 one, indeed how it would be to port its linker to work for Modula-3 and tune it for the required works of it. Is better to adapt to a better portable (Linux, etc) solution than to adapt to a non-portable still solution as I believe. Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] threading on Windows? Para: "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 17:11 Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 11 02:08:31 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Fri, 11 Feb 2011 01:08:31 +0000 (GMT) Subject: [M3devel] threading on Windows? In-Reply-To: Message-ID: <355223.44215.qm@web29702.mail.ird.yahoo.com> Hi all: thanks for your suggestion yet I learn also from you and others so I try to keep improving it, I may read some English grammar too, this is the best I can do for now, sorry for that and thanks for being open about it (other suggestions are most welcome). Please think about this, I believe this is transforming very quickly and we don't know pretty much yet where it would be in some not short time, see my point, 1.2 billions of smart phones are being ship, 3 million more than those of PCs, if you see this, this is a turn break point, you need to be able to see also this market becomes more important than some may think. Still about the PCs, the current trend is too much research being done at the GPGPU thing, with yet another revolution of this days in computation, this is my point precisely, will be able to tackle the opportunity of the change, or be overwhelmed by it, and just wait for another time. I just say in behalf of the things I have said that could be more common than we thought, yet there are good opportunities, who thought this was another revolution to come, I did not believe it but I'm more turned to think this is another "trend" in computing. And I think you would not believe but in a country like mine they thought to construct hardware assisted graphics chips, it didn't turn out to be real, but many things came after this, we have self configurable hardware, what else you one want? Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: RE: [M3devel] threading on Windows? Para: dabenavidesd at yahoo.es, "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 19:36 Daniel, you generally make very little sense to me. Please consider?additional?English training. ? ? "Modula-3" defines?and implements its?own threading library,?it's own I/O library, it's own traced and untraced memory allocator (traced == garbage collected), it's own GUI library (Trestle). ? It is its own portabilty layer, akin to various other options, e.g. using pthreads and?then implementing pthreads on Windows, using Tk or Qt, etc., and depending on them to be ported. etc. Yes, I am deliberately confusing "language" and "library". Which are people interested in? One, the other, or both? ? ? Now, granted, we are portable between essentially only two platforms Win32/NT and "Posix", where "Posix" is defined as including pthreads and X Windows, and even in a sense gcc. ? (Even if Modula-3 predates Posix, the dependencies are pretty light, such that they have mostly always been almost consistent across the board -- open/read/write/close, socket/accept/connect/recv/send/listen, a few variations for signals). ? (Personally I would like to change the backend to generate C...) ? ? One could "get out of this (library) business" and target another intermediate portability layer, i.e. write "bindings" for Tk, Qt, etc., and then use Modula-3 "just" as a language, and not an?associated large set of libraries. ? ? In general, it is desired for portability layers to be thin in terms of performance and possibly state. Ideally, given the choice of writing to Win32 threads and getting the smallest fastest program that runs only on Wondows, or pthreads and getting the smallest fastest program that runs on Posix, nobody should shy away from using Modula-3 threads, since they run pretty much just as well as Win32 and pthreads. Pretty much. Actually condition variables can't be done super well on Win32 prior to Vista, but they are done in a pretty ok way now. Still, threads and mutexes at least are very thin layers. ? ? Implementing threading more completely yourself, on many systems, precludes you from actually running on more than one processor. Win7's "user mode scheduling" offers an option with more control and still the ability to run on multiple processors. But I suspect we aren't very interested, at least not currently. ? ? ? ?- Jay ? Date: Thu, 10 Feb 2011 23:40:48 +0000 From: dabenavidesd at yahoo.es To: hosking at cs.purdue.edu; jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Hi all: is useful to say here that the compiler linker approach at some extent was done in the same way as was in Modula-3, now Open Watcom, used same runtime structure and required as far as I know no better enlightment in the original versions of it, I read it some quite time ago in its sources, hard to tell you where it was, but if OpenWatcom sold it as a Windows solution in its runtime why not in the Modula-3 one, indeed how it would be to port its linker to work for Modula-3 and tune it for the? required? works of it. Is better to adapt to a better portable (Linux, etc) solution than? to adapt to a? non-portable still solution as I believe. Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] threading on Windows? Para: "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 17:11 #yiv1956618167 .yiv1956618167ExternalClass #yiv1956618167ecxyiv295862095 .yiv1956618167ecxyiv295862095hmmessage P {padding:0px;} #yiv1956618167 .yiv1956618167ExternalClass #yiv1956618167ecxyiv295862095 .yiv1956618167ecxyiv295862095hmmessage {font-size:10pt;font-family:Tahoma;} Why is it better? ?Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) ?- Jay/iPad ? You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. ? ? Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. ? ? As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. ? ? Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. ? ? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 11 03:51:10 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Fri, 11 Feb 2011 02:51:10 +0000 (GMT) Subject: [M3devel] threading on Windows? In-Reply-To: Message-ID: <332041.74062.qm@web29718.mail.ird.yahoo.com> Hi all: Jay let me make a statement, perhaps if some can follow me to agree in this, a claim from a? Professor here, very pro FOSS anyway: "The cost of hardware are always descending and growing up in technology capabilities, this is not the case in software terms, there is little improvement in the techniques and yet they are more excessive expensive each time it has a new version". I would agree if one looks to some companies, like in M$ fields, this is the case certainly, as Win ships in every machine human has seen almost, ok but in other responsible houses this was not the case, Digital Equipment Corporation if I may say so, they had very good machines, Professor told you could specify very precisely what do you would want of it and they shipped and were very endurable. Of software though they didn't make a revolution they did the best achievements in the software I have seen so far, runtime capabilities of this compiler I have see so far anything like this and specially static analysis tools, very impressive, even some people would tell and have told certainly already and certainly they are greater than I am. Thanks in advance Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: RE: [M3devel] threading on Windows? Para: dabenavidesd at yahoo.es, "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 19:36 Daniel, you generally make very little sense to me. Please consider?additional?English training. ? ? "Modula-3" defines?and implements its?own threading library,?it's own I/O library, it's own traced and untraced memory allocator (traced == garbage collected), it's own GUI library (Trestle). ? It is its own portabilty layer, akin to various other options, e.g. using pthreads and?then implementing pthreads on Windows, using Tk or Qt, etc., and depending on them to be ported. etc. Yes, I am deliberately confusing "language" and "library". Which are people interested in? One, the other, or both? ? ? Now, granted, we are portable between essentially only two platforms Win32/NT and "Posix", where "Posix" is defined as including pthreads and X Windows, and even in a sense gcc. ? (Even if Modula-3 predates Posix, the dependencies are pretty light, such that they have mostly always been almost consistent across the board -- open/read/write/close, socket/accept/connect/recv/send/listen, a few variations for signals). ? (Personally I would like to change the backend to generate C...) ? ? One could "get out of this (library) business" and target another intermediate portability layer, i.e. write "bindings" for Tk, Qt, etc., and then use Modula-3 "just" as a language, and not an?associated large set of libraries. ? ? In general, it is desired for portability layers to be thin in terms of performance and possibly state. Ideally, given the choice of writing to Win32 threads and getting the smallest fastest program that runs only on Wondows, or pthreads and getting the smallest fastest program that runs on Posix, nobody should shy away from using Modula-3 threads, since they run pretty much just as well as Win32 and pthreads. Pretty much. Actually condition variables can't be done super well on Win32 prior to Vista, but they are done in a pretty ok way now. Still, threads and mutexes at least are very thin layers. ? ? Implementing threading more completely yourself, on many systems, precludes you from actually running on more than one processor. Win7's "user mode scheduling" offers an option with more control and still the ability to run on multiple processors. But I suspect we aren't very interested, at least not currently. ? ? ? ?- Jay ? Date: Thu, 10 Feb 2011 23:40:48 +0000 From: dabenavidesd at yahoo.es To: hosking at cs.purdue.edu; jay.krell at cornell.edu CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Hi all: is useful to say here that the compiler linker approach at some extent was done in the same way as was in Modula-3, now Open Watcom, used same runtime structure and required as far as I know no better enlightment in the original versions of it, I read it some quite time ago in its sources, hard to tell you where it was, but if OpenWatcom sold it as a Windows solution in its runtime why not in the Modula-3 one, indeed how it would be to port its linker to work for Modula-3 and tune it for the? required? works of it. Is better to adapt to a better portable (Linux, etc) solution than? to adapt to a? non-portable still solution as I believe. Thanks in advance --- El jue, 10/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] threading on Windows? Para: "Tony" CC: "m3devel" Fecha: jueves, 10 de febrero, 2011 17:11 #yiv863291047 .yiv863291047ExternalClass #yiv863291047ecxyiv295862095 .yiv863291047ecxyiv295862095hmmessage P {padding:0px;} #yiv863291047 .yiv863291047ExternalClass #yiv863291047ecxyiv295862095 .yiv863291047ecxyiv295862095hmmessage {font-size:10pt;font-family:Tahoma;} Why is it better? ?Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) ?- Jay/iPad ? You really think there's a way to do this w/o a special case in the compiler? I highly doubt that. Granted, I haven't looked at it and I will spend a few minutes doing so, but again, I highly doubt this can be anything but a very special case. Maybe runtime hooks allow for an enum comparison instead of a string comparison. m3cg.alloca allows the most direct route w/o discovering the special case in a general purpose path. ? ? Inlining...well, on NT, the stack pages must be touched in order. Code that does merely: void f() { char foo[4097]; } incurs a function calls in the prolog. The function called is _chkstk. It is a long standing bug in the NT/x86 backend that it doesn't do this. Given that jmpbuf is always way smaller than 4K, doing merely sub esp,Jmpbuf__size is probably reasonable. As well, oh...well, that's not m3cg.Alloca, that's reall m3cg.AllocaJmpbufSize...which is specific enough, that if the backend wants to know the target jmpbuf size, it can optimize slightly thereof. We could make it be m3cg.AllocaSmallValue(name: TEXT). I'm torn. Many many many almost identical options. ? ? As well, Target.i3 could contain a jumpbuf size, default it to 0. If it is zero, use the code I have now. If it isn't zero, use the old code. Something like that. Yes, I realize there are changes in the EH code possibly implied. ? ? Again, _chkstk has a highly custom calling convention, both its input parameter and output don't follow convention. RTHooks makes that not a special case? I'll look later.. ? ? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Fri Feb 11 22:26:52 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Fri, 11 Feb 2011 16:26:52 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu> Message-ID: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new "heap-based jmpbuf allocation" have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate "alloca" routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I "do" anything, I'll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the "heap based jumpbuf allocation scheme" was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am "in the dark", both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I'm seeing several different symbols that don't resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking > wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Feb 11 22:29:43 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 11 Feb 2011 16:29:43 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu> <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com> <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu> <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu> Message-ID: <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: > Tony: > > Thank you very much for this explanation. > > So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: > 1. Space leak problem. > 2. Binding to appropriate ?alloca? routine on Windows platform. > > Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. > > I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. > > Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. > > Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. > > Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. > > If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. > > Regards, > Randy Coleburn > > From: Tony Hosking [mailto:hosking at cs.purdue.edu] > Sent: Thursday, February 10, 2011 4:26 PM > To: Coleburn, Randy > Cc: m3devel > Subject: Re: [M3devel] threading on Windows? > > Hi Randy, > > A far as I know, the changes are as follows. > > TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. > > The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. > > To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. > > -- Tony > > Antony Hosking | Associate Professor | Computer Science | Purdue University > 305 N. University Street | West Lafayette | IN 47907 | USA > Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > > > On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: > > > Jay / Tony: > > I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. > > I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. > > I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. > > BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. > > Regards, > Randy > > From: Tony Hosking [mailto:hosking at cs.purdue.edu] > Sent: Thursday, February 10, 2011 3:33 PM > To: jay.krell at cornell.edu > Cc: Coleburn, Randy; m3devel > Subject: Re: [M3devel] threading on Windows? > > Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. > > On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: > > > > 1. I never understood why this any better. > 2. It how things are currently. > 3. Why not m3cg.alloca? (but see #2) > > - Jay/iPad > > On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: > > What happened to fixing this with runtime hooks? > > > On Feb 9, 2011, at 11:40 PM, Jay K wrote: > > > > Of course not. > It is a made-up function that the frontend generates a call to. > That it never previously did. > Using the general function call interface. > You need to add a special case, in the general function call code, to do something different and specific > for this specific function. > > > - Jay > > From: rcolebur at SCIRES.COM > To: jay.krell at cornell.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 22:28:45 -050 > Subject: Re: [M3devel] threading on Windows? > > Jay: > > I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. > > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 9:01 PM > To: Coleburn, Randy; m3devel > Subject: RE: [M3devel] threading on Windows? > > In m3back/src/M3x86.m3. > > - Jay > > > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 18:22:56 -0500 > Subject: Re: [M3devel] threading on Windows? > > I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. > All I know is that everything was working fine until I checked out the HEAD repository. > When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? > Are we talking Modula-3 code, C code, Assembler, what? What package/module? > Regards, > Randy > > From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K > Sent: Wednesday, February 09, 2011 5:59 PM > To: Coleburn, Randy; Mika Nystrom; m3devel > Subject: RE: [M3devel] threading on Windows? > > Maybe someone else can do it? > The fix is: special case calls to the function "m3_alloca". > Change it to call "_chkstk" (or maybe "chkstk", whatever works). > The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. > To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. > > - Jay > > > From: rcolebur at SCIRES.COM > > To: mika at async.caltech.edu; m3devel at elegosoft.com > > Date: Wed, 9 Feb 2011 17:25:20 -0500 > > Subject: Re: [M3devel] threading on Windows? > > > > Mika: > > > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > > > As soon as I can get the build problem resolved, I'll try out your test program. > > > > Regards, > > Randy Coleburn > > > > -----Original Message----- > > From: Mika Nystrom [mailto:mika at async.caltech.edu] > > Sent: Wednesday, February 09, 2011 5:01 PM > > To: m3devel at elegosoft.com > > Subject: [M3devel] threading on Windows? > > > > Hi m3devel, > > > > I'm just curious if anyone out there who's running CM3 on Windows has had > > a chance to try my thread testing program yet. Current status on Unix > > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > > I think only under heavy load) and that user threading works perfectly. > > So I now wonder how things are on the third threading platform (Windows). > > The thread testing program is at m3-libs/m3core/tests/thread . > > > > (Note however that there is a memory leak in TRY-EXCEPT in the current > > head so don't try to update everything to the CVS head. The previous > > release should be OK.) > > > > Also has anyone else noticed that debugging information has recently > > broken? m3gdb is very unhappy on all platforms for me... > > > > Mika > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Feb 11 22:51:37 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 11 Feb 2011 21:51:37 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> Message-ID: It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - Jay From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote:Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely:1. Space leak problem.2. Binding to appropriate ?alloca? routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards,Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- TonyAntony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. Regards,Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler.On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently.3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote:What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows?Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows?I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve.All I know is that everything was working fine until I checked out the HEAD repository.When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case??Are we talking Modula-3 code, C code, Assembler, what? What package/module?Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 00:09:17 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 11 Feb 2011 23:09:17 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> Message-ID: > me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows It is totally compatible. Someone just has to write a few lines of code to implement it. - Jay From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate ?alloca? routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Sat Feb 12 00:17:15 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Fri, 11 Feb 2011 18:17:15 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu> Message-ID: Jay: Ok, I don't mean to start a new issue or upset anyone. I see you checked in some new changes. I updated to latest HEAD and tried to build, but again I get unresolved symbols beginning with m3-libs\sysutils (see attached sysutils.lst) So, not sure what you mean by "it should be ok now". Are you saying you do, or do not, want me to work further on the solving the "problem," regardless of how we describe it ? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, February 11, 2011 4:52 PM To: Tony; Coleburn, Randy Cc: m3devel Subject: RE: [M3devel] threading on Windows? It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - Jay ________________________________ From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new "heap-based jmpbuf allocation" have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate "alloca" routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I "do" anything, I'll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the "heap based jumpbuf allocation scheme" was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am "in the dark", both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I'm seeing several different symbols that don't resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking > wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sysutils.lst Type: application/octet-stream Size: 4059 bytes Desc: sysutils.lst URL: From mika at async.caltech.edu Sat Feb 12 02:57:15 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 17:57:15 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> Message-ID: <20110212015715.A43471A2078@async.async.caltech.edu> Hello m3devel, I'm attempting to test the CVS head as of Jay's latest checkins with the thread tester. I'm afraid there seems to be something wrong. I get a seg fault when running a few threads that do the following: PROCEDURE TApply(cl : Closure) : REFANY = BEGIN Thread.Pause(InitPause); LOOP TRY WITH now = Time.Now() DO times1[cl.id]:= FLOOR(now); IF now < 0.0d0 THEN RAISE X END END EXCEPT X => <*ASSERT FALSE*> END END END TApply; The crash is as follows: (gdb) run -tests tryexcept Starting program: /home/mika/cm3-cvs/cm3/m3-libs/m3core/tests/thread/LINUXLIBC6/threadtest -tests tryexcept [Thread debugging using libthread_db enabled] Writing file...done Creating tryexcept threads...done running...printing oldest/median age/newest . Program received signal SIGSEGV, Segmentation fault. 0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 53 { (gdb) where #0 0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 #1 0x0808cb57 in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 #2 0x0804b875 in Main__TApply (cl=) at ../src/Main.m3:205 #3 0x08086c5e in ThreadPosix__RunThread () at ../src/thread/POSIX/ThreadPosix.m3:993 #4 0x400c343b in makecontext () from /lib/libc.so.6 #5 0x00000000 in ?? () (gdb) Note I am using user threads since pthreads are known to be buggy at the moment. I would like to find one configuration (any configuration!!) that lets me run a multithreaded program under AMD64_LINUX. I don't care what version of the compiler it is, whether the threads are user or pthreads, how many things I have to hack. Does anyone know what I need to do? Mika From mika at async.caltech.edu Sat Feb 12 03:08:36 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 18:08:36 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: <20110212015715.A43471A2078@async.async.caltech.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu> Message-ID: <20110212020836.97F7A1A207A@async.async.caltech.edu> I forgot to add that I was running that program on the platform LINUXLIBC6. On FreeBSD4 I get the following bizarre behavior (program hangs): (gdb) run -tests tryexcept Starting program: /big/home/mika/cm3-writable/cm3/m3-libs/m3core/tests/thread/FreeBSD4/threadtest -tests tryexcept Writing file...done Creating tryexcept threads...done running...printing oldest/median age/newest .^C Program received signal SIGINT, Interrupt. [Switching to Thread 1 (LWP 100315)] 0x681093f3 in pthread_testcancel () from /usr/lib/libpthread.so.1 (gdb) where #0 0x681093f3 in pthread_testcancel () from /usr/lib/libpthread.so.1 #1 0x680f917c in sigaction () from /usr/lib/libpthread.so.1 #2 0x680f9d03 in sigaction () from /usr/lib/libpthread.so.1 #3 0x680f850b in sigaction () from /usr/lib/libpthread.so.1 #4 0x680f828c in sigaction () from /usr/lib/libpthread.so.1 #5 0x6810192e in pthread_mutexattr_init () from /usr/lib/libpthread.so.1 #6 0x00000000 in ?? () (gdb) Even though I am using ThreadPosix, not ThreadPthread... Also on AMD64_LINUX, the following: (gdb) run -tests tryexcept Starting program: /home/mika/cm3/cm3/m3-libs/m3core/tests/thread/AMD64_LINUX/threadtest -tests tryexcept [Thread debugging using libthread_db enabled] Writing file...done Creating tryexcept threads...done running...printing oldest/median age/newest .[New Thread 0x2b7877844620 (LWP 24086)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x2b7877844620 (LWP 24086)] 0x00007fff44dff5b1 in ?? () (gdb) where #0 0x00007fff44dff5b1 in ?? () #1 0x00007fff44dff6a5 in gettimeofday () #2 0x00002b787589690a in gettimeofday () from /lib/libc.so.6 #3 0x000000000045148f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:58 #4 0x000000000045139e in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 #5 0x0000000000404da7 in Main__TApply (cl=Cannot access memory at address 0x2b7974014e68 ) at ../src/Main.m3:205 #6 0x000000000044b3fb in ThreadPosix__RunThread () at ../src/thread/POSIX/ThreadPosix.m3:993 #7 0x00002b787584b7b0 in ?? () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () (gdb) AMD64_FREEBSD seems to work, oddly enough, but slowly. I wonder if my compiler buidling has failed somehow (I am using the Python script following a recipe provided by Jay after modifying m3core/src/thread/m3makefile ) Mika Mika Nystrom writes: > >Hello m3devel, > >I'm attempting to test the CVS head as of Jay's latest checkins with >the thread tester. > >I'm afraid there seems to be something wrong. > >I get a seg fault when running a few threads that do the following: > >PROCEDURE TApply(cl : Closure) : REFANY = > BEGIN > Thread.Pause(InitPause); > LOOP > TRY > WITH now = Time.Now() DO > times1[cl.id]:= FLOOR(now); > IF now < 0.0d0 THEN RAISE X END > END > EXCEPT X => <*ASSERT FALSE*> > END > END > END TApply; > >The crash is as follows: > >(gdb) run -tests tryexcept >Starting program: /home/mika/cm3-cvs/cm3/m3-libs/m3core/tests/thread/LINUXLIBC >6/threadtest -tests tryexcept >[Thread debugging using libthread_db enabled] >Writing file...done >Creating tryexcept threads...done >running...printing oldest/median age/newest >. >Program received signal SIGSEGV, Segmentation fault. >0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 >53 { >(gdb) where >#0 0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 >#1 0x0808cb57 in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 >#2 0x0804b875 in Main__TApply (cl=) > at ../src/Main.m3:205 >#3 0x08086c5e in ThreadPosix__RunThread () > at ../src/thread/POSIX/ThreadPosix.m3:993 >#4 0x400c343b in makecontext () from /lib/libc.so.6 >#5 0x00000000 in ?? () >(gdb) > >Note I am using user threads since pthreads are known to be buggy at the >moment. > >I would like to find one configuration (any configuration!!) that lets >me run a multithreaded program under AMD64_LINUX. I don't care what >version of the compiler it is, whether the threads are user or pthreads, >how many things I have to hack. Does anyone know what I need to do? > > Mika From mika at async.caltech.edu Sat Feb 12 03:50:13 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 18:50:13 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: <20110212020836.97F7A1A207A@async.async.caltech.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu> <20110212020836.97F7A1A207A@async.async.caltech.edu> Message-ID: <20110212025013.916EA1A2078@async.async.caltech.edu> Another odd datapoint is that even rather aggressive testing seems not to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user threads, of course. So status right now of user threads seems to be: FreeBSD4 : strange failure in libpthread as reported by gdb LINUXLIBC6 : seg fault AMD64_LINUX : seg fault AMD64_FREEBSD : OK I386_DARWIN : OK I am never, ever 100% certain of my bootstrapping procedure so it's possible I've done something wrong... however I'm using upgrade.py and when I look for Thread\*.*o files I only see the ThreadPosix file and not ThreadPThread. I'll try bootstrapping again :-) Mika Mika Nystrom writes: > >I forgot to add that I was running that program on the platform LINUXLIBC6. > >On FreeBSD4 I get the following bizarre behavior (program hangs): > >(gdb) run -tests tryexcept >Starting program: /big/home/mika/cm3-writable/cm3/m3-libs/m3core/tests/thread/ >FreeBSD4/threadtest -tests tryexcept >Writing file...done >Creating tryexcept threads...done >running...printing oldest/median age/newest >.^C >Program received signal SIGINT, Interrupt. >[Switching to Thread 1 (LWP 100315)] >0x681093f3 in pthread_testcancel () from /usr/lib/libpthread.so.1 > >(gdb) where >#0 0x681093f3 in pthread_testcancel () from /usr/lib/libpthread.so.1 >#1 0x680f917c in sigaction () from /usr/lib/libpthread.so.1 >#2 0x680f9d03 in sigaction () from /usr/lib/libpthread.so.1 >#3 0x680f850b in sigaction () from /usr/lib/libpthread.so.1 >#4 0x680f828c in sigaction () from /usr/lib/libpthread.so.1 >#5 0x6810192e in pthread_mutexattr_init () from /usr/lib/libpthread.so.1 >#6 0x00000000 in ?? () >(gdb) > >Even though I am using ThreadPosix, not ThreadPthread... > >Also on AMD64_LINUX, the following: > >(gdb) run -tests tryexcept >Starting program: /home/mika/cm3/cm3/m3-libs/m3core/tests/thread/AMD64_LINUX/t >hreadtest -tests tryexcept >[Thread debugging using libthread_db enabled] >Writing file...done >Creating tryexcept threads...done >running...printing oldest/median age/newest >.[New Thread 0x2b7877844620 (LWP 24086)] > >Program received signal SIGSEGV, Segmentation fault. >[Switching to Thread 0x2b7877844620 (LWP 24086)] >0x00007fff44dff5b1 in ?? () >(gdb) where >#0 0x00007fff44dff5b1 in ?? () >#1 0x00007fff44dff6a5 in gettimeofday () >#2 0x00002b787589690a in gettimeofday () from /lib/libc.so.6 >#3 0x000000000045148f in TimePosix__Now () > at ../src/time/POSIX/TimePosixC.c:58 >#4 0x000000000045139e in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 >#5 0x0000000000404da7 in Main__TApply (cl=Cannot access memory at address 0x2 >b7974014e68 >) at ../src/Main.m3:205 >#6 0x000000000044b3fb in ThreadPosix__RunThread () > at ../src/thread/POSIX/ThreadPosix.m3:993 >#7 0x00002b787584b7b0 in ?? () from /lib/libc.so.6 >#8 0x0000000000000000 in ?? () >(gdb) > >AMD64_FREEBSD seems to work, oddly enough, but slowly. > >I wonder if my compiler buidling has failed somehow (I am using the >Python script following a recipe provided by Jay after modifying >m3core/src/thread/m3makefile ) > > Mika > > >Mika Nystrom writes: >> >>Hello m3devel, >> >>I'm attempting to test the CVS head as of Jay's latest checkins with >>the thread tester. >> >>I'm afraid there seems to be something wrong. >> >>I get a seg fault when running a few threads that do the following: >> >>PROCEDURE TApply(cl : Closure) : REFANY = >> BEGIN >> Thread.Pause(InitPause); >> LOOP >> TRY >> WITH now = Time.Now() DO >> times1[cl.id]:= FLOOR(now); >> IF now < 0.0d0 THEN RAISE X END >> END >> EXCEPT X => <*ASSERT FALSE*> >> END >> END >> END TApply; >> >>The crash is as follows: >> >>(gdb) run -tests tryexcept >>Starting program: /home/mika/cm3-cvs/cm3/m3-libs/m3core/tests/thread/LINUXLIB >C >>6/threadtest -tests tryexcept >>[Thread debugging using libthread_db enabled] >>Writing file...done >>Creating tryexcept threads...done >>running...printing oldest/median age/newest >>. >>Program received signal SIGSEGV, Segmentation fault. >>0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 >>53 { >>(gdb) where >>#0 0x0808cc0f in TimePosix__Now () at ../src/time/POSIX/TimePosixC.c:53 >>#1 0x0808cb57 in Time__Now () at ../src/time/POSIX/TimePosix.m3:14 >>#2 0x0804b875 in Main__TApply (cl=) >> at ../src/Main.m3:205 >>#3 0x08086c5e in ThreadPosix__RunThread () >> at ../src/thread/POSIX/ThreadPosix.m3:993 >>#4 0x400c343b in makecontext () from /lib/libc.so.6 >>#5 0x00000000 in ?? () >>(gdb) >> >>Note I am using user threads since pthreads are known to be buggy at the >>moment. >> >>I would like to find one configuration (any configuration!!) that lets >>me run a multithreaded program under AMD64_LINUX. I don't care what >>version of the compiler it is, whether the threads are user or pthreads, >>how many things I have to hack. Does anyone know what I need to do? >> >> Mika From mika at async.caltech.edu Sat Feb 12 04:05:25 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 19:05:25 -0800 Subject: [M3devel] pthread_atfork Message-ID: <20110212030525.822411A2078@async.async.caltech.edu> Tony, What's RTProcessC.c doing? I note the following cryptic comment: /* NOTE: Even userthreads now depends * on availability of pthreads. * This can be fixed if need be. */ now further down we can read: #if defined(_WIN32) \ || defined(__vms) \ || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) return 0; #else while (1) { int i = pthread_atfork(prepare, parent, child); if (i != EAGAIN) return i; sleep(0); } #endif so on FreeBSD 5 (what I'm running on my "FreeBSD4" system), RTProcess.RegisterForkHandlers does nothing? Hmmm..... Mika From mika at async.caltech.edu Sat Feb 12 07:48:12 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 11 Feb 2011 22:48:12 -0800 Subject: [M3devel] threading on Windows? Message-ID: <20110212064812.286E61A2078@async.async.caltech.edu> Well, with the following changes: re-bootstrapping on the two Linux systems and removal of the -pthread flag on FreeBSD4, all is well. User threading works on all the architectures listed below, at least as far as the thread testing program can test it. (It seems that because of the way pthread_atfork is coded, pthread is not needed at all on FreeBSD4.) Mika Mika Nystrom writes: >Another odd datapoint is that even rather aggressive testing seems not >to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user >threads, of course. > >So status right now of user threads seems to be: > >FreeBSD4 : strange failure in libpthread as reported by gdb >LINUXLIBC6 : seg fault >AMD64_LINUX : seg fault >AMD64_FREEBSD : OK >I386_DARWIN : OK From jay.krell at cornell.edu Sat Feb 12 11:24:01 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 10:24:01 +0000 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212030525.822411A2078@async.async.caltech.edu> References: <20110212030525.822411A2078@async.async.caltech.edu> Message-ID: I wrote that. As I recall, FreeBSD < 6 doesn't have pthread_atfork. I think that was based on reading the man pages. So, yes. What would you suggest? All this atfork stuff came about as part of getting cvsupd to work. With pthreads/kernel threads. It historically worked, with Modula-3 user threads. Modula-3 user threads have this funny property that all threads survive work. With pthreads/kernel threads, fork gives you a new process with just one thread. (Solaris has fork1 and forkall, and fork maybe used to be forkall, but now it is fork1). cvsupd is one of those slightly unusual programs that does "fork and then do more work" as supposed to the more typical "fork and exec". Which is to say..I don't know. Maybe, don't expect pthreads to work on FreeBSD < 6? At least not in a process that ever calls fork? Or does "fork and then more work"? In reality, i think a lot of libraries have trouble with "fork and then do more work", but I don't know. I recall vague notices in the Apple/Darwin manpages not to assume this works -- the obvious implication that there is plenty of use of pthreads e.g. mutexes, but relatively little use of pthread_atfork. You could perhaps replace all uses of fork with a function that called the fork handlers, then fork, then the other handlers. That is probably what I meant in the comment about this being easy to fix. You might though then run into the problem that callers of fork can't do much before exec...oh, that's vfork I bet actually. - Jay > To: hosking at cs.purdue.edu > Date: Fri, 11 Feb 2011 19:05:25 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com; jay.krell at cornell.edu > Subject: [M3devel] pthread_atfork > > Tony, > > What's RTProcessC.c doing? > > I note the following cryptic comment: > > /* NOTE: Even userthreads now depends > * on availability of pthreads. > * This can be fixed if need be. > */ > > now further down we can read: > > #if defined(_WIN32) \ > || defined(__vms) \ > || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > return 0; > #else > while (1) > { > int i = pthread_atfork(prepare, parent, child); > if (i != EAGAIN) > return i; > sleep(0); > } > #endif > > so on FreeBSD 5 (what I'm running on my "FreeBSD4" system), > RTProcess.RegisterForkHandlers does nothing? > > Hmmm..... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 11:39:06 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 10:39:06 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , , , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, , <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , , , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , , , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu>, , Message-ID: TextUtils.mo : error LNK2001: unresolved external symbol __alloca TextUtils.mo : error LNK2001: unresolved external symbol _Csetjmp__Jumpbuf_size sysutils.dll : fatal error LNK1120: 2 unresolved externals Most likely you didn't build it correctly. Go back to a working cm3/m3core/libm3 and run upgrade.py. There should be no longer any references to alloca of Csetjmp__Jumpbuf_size. Really, Csetjmp__Jumpbuf_size should never have been a problem. That likely also indicates some sort of incorrect bootstrap. > Are you saying you do, or do not, want me to work further on the solving the ?problem,? regardless of how we describe it ? I still want alloca to work, either via a function call or a specific m3cg method. (I still don't see that RTHooks.Alloca makes better sense; I still don't see a way to avoid hacking the compiler, given the custom calling convention.) It will be difficult to test the fix until/unless m3front is changed back. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Fri, 11 Feb 2011 18:17:15 -0500 Subject: Re: [M3devel] threading on Windows? Jay: Ok, I don?t mean to start a new issue or upset anyone. I see you checked in some new changes. I updated to latest HEAD and tried to build, but again I get unresolved symbols beginning with m3-libs\sysutils (see attached sysutils.lst) So, not sure what you mean by ?it should be ok now?. Are you saying you do, or do not, want me to work further on the solving the ?problem,? regardless of how we describe it ? Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, February 11, 2011 4:52 PM To: Tony; Coleburn, Randy Cc: m3devel Subject: RE: [M3devel] threading on Windows? It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - JayFrom: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely:1. Space leak problem.2. Binding to appropriate ?alloca? routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards,Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- TonyAntony Hosking | Associate Professor | Computer Science | Purdue University305 N. University Street | West Lafayette | IN 47907 | USAOffice +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. Regards,Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler.On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently.3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote:What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows?Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows?I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve.All I know is that everything was working fine until I checked out the HEAD repository.When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case??Are we talking Modula-3 code, C code, Assembler, what? What package/module?Regards,Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Sat Feb 12 13:02:11 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Sat, 12 Feb 2011 13:02:11 +0100 Subject: [M3devel] pthread_atfork In-Reply-To: References: <20110212030525.822411A2078@async.async.caltech.edu> Message-ID: <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com> I think it would be acceptable to say that pthread threading is broken on FreeBSD6 (as it is not officially supported any more). All the stuff below should be documented in the 'known problems' section. Especially the different behaviours of fork for user and pthreads. This should also be mentioned in the M3 interfaces (source code)! Also that to use use threads with FreeBSD requires to remove the -pthread flag, as Mika noted. BTW, do we have a chapter explaining what to do to use user threads instead of the default pthreads? Olaf PS: Hint: this would be a great task for anybody lurking on the lists and trying to get involved :-) Quoting Jay K : > > I wrote that. > > As I recall, FreeBSD < 6 doesn't have pthread_atfork. > > I think that was based on reading the man pages. > > So, yes. What would you suggest? > > > > > > All this atfork stuff came about as part of getting cvsupd to work. > > With pthreads/kernel threads. It historically worked, with > Modula-3 user threads. > > Modula-3 user threads have this funny property that all threads > survive work. > > With pthreads/kernel threads, fork gives you a new process with > just one thread. > > (Solaris has fork1 and forkall, and fork maybe used to be > forkall, but now it is fork1). > > > > > > cvsupd is one of those slightly unusual programs that does "fork and > then do more work" > > as supposed to the more typical "fork and exec". > > > > Which is to say..I don't know. Maybe, don't expect pthreads to work > on FreeBSD < 6? > > At least not in a process that ever calls fork? Or does "fork and > then more work"? > > > > > > In reality, i think a lot of libraries have trouble with "fork and > then do more work", but > > I don't know. I recall vague notices in the Apple/Darwin manpages > not to assume > > this works -- the obvious implication that there is plenty of use of > pthreads e.g. mutexes, > > but relatively little use of pthread_atfork. > > > > > > You could perhaps replace all uses of fork with a function that > called the fork handlers, > > then fork, then the other handlers. That is probably what I meant in > the comment > > about this being easy to fix. > > You might though then run into the problem that callers of fork can't do much > > before exec...oh, that's vfork I bet actually. > > > > > > - Jay > > > > > > >> To: hosking at cs.purdue.edu >> Date: Fri, 11 Feb 2011 19:05:25 -0800 >> From: mika at async.caltech.edu >> CC: m3devel at elegosoft.com; jay.krell at cornell.edu >> Subject: [M3devel] pthread_atfork >> >> Tony, >> >> What's RTProcessC.c doing? >> >> I note the following cryptic comment: >> >> /* NOTE: Even userthreads now depends >> * on availability of pthreads. >> * This can be fixed if need be. >> */ >> >> now further down we can read: >> >> #if defined(_WIN32) \ >> || defined(__vms) \ >> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) >> return 0; >> #else >> while (1) >> { >> int i = pthread_atfork(prepare, parent, child); >> if (i != EAGAIN) >> return i; >> sleep(0); >> } >> #endif >> >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system), >> RTProcess.RegisterForkHandlers does nothing? >> >> Hmmm..... >> >> Mika > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Sat Feb 12 14:38:33 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 13:38:33 +0000 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com> References: <20110212030525.822411A2078@async.async.caltech.edu>, , <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com> Message-ID: > Especially the different behaviours of fork for user and pthreads The behavior has been unified. User threads no longer survive fork. Except perhaps on FreeBSD 5? (Off by one somewhere in the discussion!) It looks like it is ok on FreeBSD 6. I think we could fix it on FreeBSD 5. At least if we changed all of our fork() to be RTProcess.Fork() and called the handlers in there, at least on FreeBSD 5. I believe the difference is only noticable if you fork-and-do-work. Not if you fork-and-exec. I am very tempted to fix this, so that nobody would be discussing where it did or didn't work. (Though, fork() would still vary in behavior, but we'd never use it directly..) Threads not surviving fork is new with using pthreads. Threads never survived fork on Windows. For user threads, the change was a bit late, didn't occur until we worked on getting cvsupd to work with pthreads. - Jay > Date: Sat, 12 Feb 2011 13:02:11 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: Re: [M3devel] pthread_atfork > > I think it would be acceptable to say that pthread threading is > broken on FreeBSD6 (as it is not officially supported any more). > All the stuff below should be documented in the 'known problems' > section. Especially the different behaviours of fork for > user and pthreads. This should also be mentioned in the M3 > interfaces (source code)! > > Also that to use use threads with FreeBSD requires to remove the > -pthread flag, as Mika noted. > > BTW, do we have a chapter explaining what to do to use user threads > instead of the default pthreads? > > Olaf > > PS: Hint: this would be a great task for anybody lurking on the lists > and trying to get involved :-) > > Quoting Jay K : > > > > > I wrote that. > > > > As I recall, FreeBSD < 6 doesn't have pthread_atfork. > > > > I think that was based on reading the man pages. > > > > So, yes. What would you suggest? > > > > > > > > > > > > All this atfork stuff came about as part of getting cvsupd to work. > > > > With pthreads/kernel threads. It historically worked, with > > Modula-3 user threads. > > > > Modula-3 user threads have this funny property that all threads > > survive work. > > > > With pthreads/kernel threads, fork gives you a new process with > > just one thread. > > > > (Solaris has fork1 and forkall, and fork maybe used to be > > forkall, but now it is fork1). > > > > > > > > > > > > cvsupd is one of those slightly unusual programs that does "fork and > > then do more work" > > > > as supposed to the more typical "fork and exec". > > > > > > > > Which is to say..I don't know. Maybe, don't expect pthreads to work > > on FreeBSD < 6? > > > > At least not in a process that ever calls fork? Or does "fork and > > then more work"? > > > > > > > > > > > > In reality, i think a lot of libraries have trouble with "fork and > > then do more work", but > > > > I don't know. I recall vague notices in the Apple/Darwin manpages > > not to assume > > > > this works -- the obvious implication that there is plenty of use of > > pthreads e.g. mutexes, > > > > but relatively little use of pthread_atfork. > > > > > > > > > > > > You could perhaps replace all uses of fork with a function that > > called the fork handlers, > > > > then fork, then the other handlers. That is probably what I meant in > > the comment > > > > about this being easy to fix. > > > > You might though then run into the problem that callers of fork can't do much > > > > before exec...oh, that's vfork I bet actually. > > > > > > > > > > > > - Jay > > > > > > > > > > > > > >> To: hosking at cs.purdue.edu > >> Date: Fri, 11 Feb 2011 19:05:25 -0800 > >> From: mika at async.caltech.edu > >> CC: m3devel at elegosoft.com; jay.krell at cornell.edu > >> Subject: [M3devel] pthread_atfork > >> > >> Tony, > >> > >> What's RTProcessC.c doing? > >> > >> I note the following cryptic comment: > >> > >> /* NOTE: Even userthreads now depends > >> * on availability of pthreads. > >> * This can be fixed if need be. > >> */ > >> > >> now further down we can read: > >> > >> #if defined(_WIN32) \ > >> || defined(__vms) \ > >> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > >> return 0; > >> #else > >> while (1) > >> { > >> int i = pthread_atfork(prepare, parent, child); > >> if (i != EAGAIN) > >> return i; > >> sleep(0); > >> } > >> #endif > >> > >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system), > >> RTProcess.RegisterForkHandlers does nothing? > >> > >> Hmmm..... > >> > >> Mika > > > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 15:09:59 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 14:09:59 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <20110212064812.286E61A2078@async.async.caltech.edu> References: <20110212064812.286E61A2078@async.async.caltech.edu> Message-ID: > >LINUXLIBC6 : seg fault I've seen this now. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb3200b90 (LWP 18154)] 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) at ../src/runtime/common/RTCollector.m3:409 409 IF hdr.typecode = RT0.TextLitTypecode THEN RETURN END; 1: x/i $pc 0xb6dee58f : mov (%eax),%eax (gdb) bt 9 #0 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) at ../src/runtime/common/RTCollector.m3:409 #1 0xb6deaa46 in RTHeapMap__Walk (x=0xb6a20014, pc=0xb7738648, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:202 #2 0xb6dea341 in RTHeapMap__DoWalkRef (t=0xb7738694, a=0xb6a20014, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:62 #3 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739c54, a=0xb6a2000c, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 #4 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739e68, a=0xb6a2000c, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 #5 0xb6dea2bf in RTHeapMap__WalkRef (h=0xb6a20008, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:47 #6 0xb6df0636 in RTCollector__CleanBetween (h=0xb6a20008, he=0xb6a30000, clean=0 '\0') at ../src/runtime/common/RTCollector.m3:1091 #7 0xb6df0473 in RTCollector__CleanPage (page=0xb6a20000) at ../src/runtime/common/RTCollector.m3:1064 #8 0xb6defc96 in RTCollector__CollectSomeInStateZero () at ../src/runtime/common/RTCollector.m3:885 (More stack frames follow...) (gdb) p $eax $2 = 2097148 (gdb) p/x $eax $3 = 0x1ffffc (gdb) x/x $eax+4 0x200000: Cannot access memory at address 0x200000 (gdb) x/x $eax 0x1ffffc: Cannot access memory at address 0x1ffffc Maybe I should try with a correctly sized jmpbuf to assure it isn't stack overflow? But notice that the next page doesn't work either. Actually I tried adding a lot to $eax and still not accessible. Maybe maybe maybe maybe I'll dig into this. But I'm very busy lately and maybe indefinitely into future.. Thanks, - Jay > To: jay.krell at cornell.edu; hosking at cs.purdue.edu > Date: Fri, 11 Feb 2011 22:48:12 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] threading on Windows? > > Well, with the following changes: re-bootstrapping on the two Linux > systems and removal of the -pthread flag on FreeBSD4, all is well. > User threading works on all the architectures listed below, at least > as far as the thread testing program can test it. (It seems that because > of the way pthread_atfork is coded, pthread is not needed at all on > FreeBSD4.) > > Mika > > Mika Nystrom writes: > >Another odd datapoint is that even rather aggressive testing seems not > >to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user > >threads, of course. > > > >So status right now of user threads seems to be: > > > >FreeBSD4 : strange failure in libpthread as reported by gdb > >LINUXLIBC6 : seg fault > >AMD64_LINUX : seg fault > >AMD64_FREEBSD : OK > >I386_DARWIN : OK -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 15:10:13 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 14:10:13 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110212064812.286E61A2078@async.async.caltech.edu>, Message-ID: (ps: using kernel/pthreads) From: jay.krell at cornell.edu To: mika at async.caltech.edu; hosking at cs.purdue.edu CC: m3devel at elegosoft.com Subject: RE: [M3devel] threading on Windows? Date: Sat, 12 Feb 2011 14:09:59 +0000 > >LINUXLIBC6 : seg fault I've seen this now. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb3200b90 (LWP 18154)] 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) at ../src/runtime/common/RTCollector.m3:409 409 IF hdr.typecode = RT0.TextLitTypecode THEN RETURN END; 1: x/i $pc 0xb6dee58f : mov (%eax),%eax (gdb) bt 9 #0 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) at ../src/runtime/common/RTCollector.m3:409 #1 0xb6deaa46 in RTHeapMap__Walk (x=0xb6a20014, pc=0xb7738648, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:202 #2 0xb6dea341 in RTHeapMap__DoWalkRef (t=0xb7738694, a=0xb6a20014, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:62 #3 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739c54, a=0xb6a2000c, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 #4 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739e68, a=0xb6a2000c, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 #5 0xb6dea2bf in RTHeapMap__WalkRef (h=0xb6a20008, v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:47 #6 0xb6df0636 in RTCollector__CleanBetween (h=0xb6a20008, he=0xb6a30000, clean=0 '\0') at ../src/runtime/common/RTCollector.m3:1091 #7 0xb6df0473 in RTCollector__CleanPage (page=0xb6a20000) at ../src/runtime/common/RTCollector.m3:1064 #8 0xb6defc96 in RTCollector__CollectSomeInStateZero () at ../src/runtime/common/RTCollector.m3:885 (More stack frames follow...) (gdb) p $eax $2 = 2097148 (gdb) p/x $eax $3 = 0x1ffffc (gdb) x/x $eax+4 0x200000: Cannot access memory at address 0x200000 (gdb) x/x $eax 0x1ffffc: Cannot access memory at address 0x1ffffc Maybe I should try with a correctly sized jmpbuf to assure it isn't stack overflow? But notice that the next page doesn't work either. Actually I tried adding a lot to $eax and still not accessible. Maybe maybe maybe maybe I'll dig into this. But I'm very busy lately and maybe indefinitely into future.. Thanks, - Jay > To: jay.krell at cornell.edu; hosking at cs.purdue.edu > Date: Fri, 11 Feb 2011 22:48:12 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] threading on Windows? > > Well, with the following changes: re-bootstrapping on the two Linux > systems and removal of the -pthread flag on FreeBSD4, all is well. > User threading works on all the architectures listed below, at least > as far as the thread testing program can test it. (It seems that because > of the way pthread_atfork is coded, pthread is not needed at all on > FreeBSD4.) > > Mika > > Mika Nystrom writes: > >Another odd datapoint is that even rather aggressive testing seems not > >to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user > >threads, of course. > > > >So status right now of user threads seems to be: > > > >FreeBSD4 : strange failure in libpthread as reported by gdb > >LINUXLIBC6 : seg fault > >AMD64_LINUX : seg fault > >AMD64_FREEBSD : OK > >I386_DARWIN : OK -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 12 17:59:30 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 08:59:30 -0800 Subject: [M3devel] pthread_atfork In-Reply-To: References: <20110212030525.822411A2078@async.async.caltech.edu> Message-ID: <20110212165930.DC32C1A2078@async.async.caltech.edu> Hi Jay, I think "fork and do work" qualifies as a bug in the application, if the application wants to be portable. Why can't it fork and exec instead? If it wants to do more work it should just fork and exec another copy of itself. fork() is also not part of Modula-3... Process.Create() is, however. Don't tell me you broke that as part of "fixing" things for cvsupd... There is still one aspect of your comment I don't understand. What does the call to pthread_atfork actually accomplish when running with user threads? You say below that "don't expect pthreads to work on FreeBSD < 6 [using the fork-and-do-work pattern]". Fair enough. But why would user threads work? We're skipping pthread_atfork for them too. Do we not need pthread_atfork with user threads? What you're saying suggests that FreeBSD >= 6 and all other platforms need pthread_atfork with user threads. Is that really what you mean? At the moment I'm just concerned with getting user threads to work as I know there are bugs with pthreads. I also think we should not waste time and effort making the "fork and do work" pattern work unless it's really, really necessary (which I really don't see why it would be). Fix the application instead of introducing bugs in the libraries and language implementation! I've been trying for a long, long time to switch from PM3 to CM3 (almost ten years, I think). These bugs from "enhancements" have stopped me every time. Maybe things are finally working well enough... (some comments inline in your quoted text below) Mika Jay K writes: >--_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >I wrote that. > >As I recall=2C FreeBSD < 6 doesn't have pthread_atfork. > >I think that was based on reading the man pages. > >So=2C yes. What would you suggest? > > > > > >All this atfork stuff came about as part of getting cvsupd to work. > > With pthreads/kernel threads. It historically worked=2C with Modula-3 use= >r threads. > > Modula-3 user threads have this funny property that all threads survive w= >ork. you mean survive fork don't you? > > With pthreads/kernel threads=2C fork gives you a new process with just on= >e thread. > > (Solaris has fork1 and forkall=2C and fork maybe used to be forkall=2C b= >ut now it is fork1). > > > > > >cvsupd is one of those slightly unusual programs that does "fork and then d= >o more work" > >as supposed to the more typical "fork and exec". > > > >Which is to say..I don't know. Maybe=2C don't expect pthreads to work on Fr= >eeBSD < 6? > >At least not in a process that ever calls fork? Or does "fork and then more= > work"? > > > > > >In reality=2C i think a lot of libraries have trouble with "fork and then d= >o more work"=2C but > >I don't know. I recall vague notices in the Apple/Darwin manpages not to as= >sume > >this works -- the obvious implication that there is plenty of use of pthrea= >ds e.g. mutexes=2C > >but relatively little use of pthread_atfork. > > > > > >You could perhaps replace all uses of fork with a function that called the = >fork handlers=2C > >then fork=2C then the other handlers. That is probably what I meant in the = >comment > >about this being easy to fix. > >You might though then run into the problem that callers of fork can't do mu= >ch > >before exec...oh=2C that's vfork I bet actually. I don't understand this paragraph either :( > > > > > > - Jay > > > > > > >> To: hosking at cs.purdue.edu >> Date: Fri=2C 11 Feb 2011 19:05:25 -0800 >> From: mika at async.caltech.edu >> CC: m3devel at elegosoft.com=3B jay.krell at cornell.edu >> Subject: [M3devel] pthread_atfork >>=20 >> Tony=2C >>=20 >> What's RTProcessC.c doing? >>=20 >> I note the following cryptic comment: >>=20 >> /* NOTE: Even userthreads now depends >> * on availability of pthreads. >> * This can be fixed if need be. >> */ >>=20 >> now further down we can read: >>=20 >> #if defined(_WIN32) \ >> || defined(__vms) \ >> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) >> return 0=3B >> #else >> while (1) >> { >> int i =3D pthread_atfork(prepare=2C parent=2C child)=3B >> if (i !=3D EAGAIN) >> return i=3B >> sleep(0)=3B >> } >> #endif >>=20 >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=2C >> RTProcess.RegisterForkHandlers does nothing? >>=20 >> Hmmm..... >>=20 >> Mika > = > >--_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >I wrote that.
>As I recall=2C FreeBSD <=3B 6 doesn't have pthread_atfork.
>I think that was based on reading the man pages.
>So=2C yes. What would you suggest?
>
>
>All this atfork stuff came about as part of getting cvsupd to work.
> =3B With pthreads/kernel threads. It historically worked=2C with Modul= >a-3 user threads.
> =3B Modula-3 user threads have this funny property that all threads su= >rvive work.
> =3B With pthreads/kernel threads=2C fork gives you a new process with = >just one thread.
> =3B =3B (Solaris has fork1 and forkall=2C and fork maybe used to b= >e forkall=2C but now it is fork1).
>
>
>cvsupd is one of those slightly unusual programs that does "fork and then d= >o more work"
>as supposed to the more typical "fork and exec".
>
>Which is to say..I don't know. Maybe=2C don't expect pthreads to work on Fr= >eeBSD <=3B 6?
>At least not in a process that ever calls fork? Or does "fork and then more= > work"?
>
>
>In reality=2C i think a lot of libraries have trouble with "fork and then d= >o more work"=2C but
>I don't know. I recall vague notices in the Apple/Darwin manpages not to as= >sume
>this works -- the obvious implication that there is plenty of use of pthrea= >ds e.g. mutexes=2C
>but relatively little use of pthread_atfork.
>
>
>You could perhaps replace all uses of fork with a function that called the = >fork handlers=2C
>then fork=2C then the other handlers. That is probably what I meant in the = >comment
>about this being easy to fix.
>You might though then run into the problem that callers of fork can't do mu= >ch
>before exec...oh=2C that's vfork I bet actually.
>
>
> =3B- Jay






>=3B To: hosking at cs.purdue.edur>>=3B Date: Fri=2C 11 Feb 2011 19:05:25 -0800
>=3B From: mika at async= >.caltech.edu
>=3B CC: m3devel at elegosoft.com=3B jay.krell at cornell.edur>>=3B Subject: [M3devel] pthread_atfork
>=3B
>=3B Tony=2C
= >>=3B
>=3B What's RTProcessC.c doing?
>=3B
>=3B I note th= >e following cryptic comment:
>=3B
>=3B /* NOTE: Even userthreads= > now depends
>=3B * on availability of pthreads.
>=3B * This ca= >n be fixed if need be.
>=3B */
>=3B
>=3B now further down = >we can read:
>=3B
>=3B #if defined(_WIN32) \
>=3B |= >| defined(__vms) \
>=3B || (defined(__FreeBSD__) &=3B&= >=3B (__FreeBSD__ <=3B 6))
>=3B return 0=3B
>=3B #else
&g= >t=3B while (1)
>=3B {
>=3B int i =3D pthread_atfork= >(prepare=2C parent=2C child)=3B
>=3B if (i !=3D EAGAIN)
>= >=3B return i=3B
>=3B sleep(0)=3B
>=3B }
>= >=3B #endif
>=3B
>=3B so on FreeBSD 5 (what I'm running on my "Fr= >eeBSD4" system)=2C
>=3B RTProcess.RegisterForkHandlers does nothing?r>>=3B
>=3B Hmmm.....
>=3B
>=3B Mika
> = > >= > >--_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_-- From wagner at elegosoft.com Sat Feb 12 20:55:27 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Sat, 12 Feb 2011 20:55:27 +0100 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212165930.DC32C1A2078@async.async.caltech.edu> References: <20110212030525.822411A2078@async.async.caltech.edu> <20110212165930.DC32C1A2078@async.async.caltech.edu> Message-ID: <20110212205527.e6xetojo0ssgocg4@mail.elegosoft.com> Well, I think actually the fork-and-do-work pattern has been the default for a long time with the M3 user threads implementation. It wasn't specified though, and I think no application except CVSup did use it. Jay is not responsible for any breakage here, he just tried to fix CVSup and establish a consistent behaviour after our change to system pthreads on most platforms. Olaf Quoting Mika Nystrom : > Hi Jay, > > I think "fork and do work" qualifies as a bug in the application, if > the application wants to be portable. Why can't it fork and exec instead? > If it wants to do more work it should just fork and exec another copy > of itself. > > fork() is also not part of Modula-3... Process.Create() is, however. > Don't tell me you broke that as part of "fixing" things for cvsupd... > > There is still one aspect of your comment I don't understand. What does > the call to pthread_atfork actually accomplish when running with user > threads? You say below that "don't expect pthreads to work on FreeBSD < > 6 [using the fork-and-do-work pattern]". Fair enough. But why would > user threads work? We're skipping pthread_atfork for them too. > > Do we not need pthread_atfork with user threads? What you're saying > suggests that FreeBSD >= 6 and all other platforms need pthread_atfork > with user threads. Is that really what you mean? > > At the moment I'm just concerned with getting user threads to work as > I know there are bugs with pthreads. > > I also think we should not waste time and effort making the "fork and do > work" pattern work unless it's really, really necessary (which I really > don't see why it would be). Fix the application instead of introducing > bugs in the libraries and language implementation! > > I've been trying for a long, long time to switch from PM3 to CM3 (almost > ten years, I think). These bugs from "enhancements" have stopped me > every time. Maybe things are finally working well enough... > > (some comments inline in your quoted text below) > > Mika > > > Jay K writes: >> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >> Content-Type: text/plain; charset="iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> >> I wrote that. >> >> As I recall=2C FreeBSD < 6 doesn't have pthread_atfork. >> >> I think that was based on reading the man pages. >> >> So=2C yes. What would you suggest? >> >> >> >> >> >> All this atfork stuff came about as part of getting cvsupd to work. >> >> With pthreads/kernel threads. It historically worked=2C with Modula-3 use= >> r threads. >> >> Modula-3 user threads have this funny property that all threads survive w= >> ork. > > you mean survive fork don't you? > >> >> With pthreads/kernel threads=2C fork gives you a new process with just on= >> e thread. >> >> (Solaris has fork1 and forkall=2C and fork maybe used to be forkall=2C b= >> ut now it is fork1). >> >> >> >> >> >> cvsupd is one of those slightly unusual programs that does "fork and then d= >> o more work" >> >> as supposed to the more typical "fork and exec". >> >> >> >> Which is to say..I don't know. Maybe=2C don't expect pthreads to work on Fr= >> eeBSD < 6? >> >> At least not in a process that ever calls fork? Or does "fork and then more= >> work"? >> >> >> >> >> >> In reality=2C i think a lot of libraries have trouble with "fork and then d= >> o more work"=2C but >> >> I don't know. I recall vague notices in the Apple/Darwin manpages not to as= >> sume >> >> this works -- the obvious implication that there is plenty of use of pthrea= >> ds e.g. mutexes=2C >> >> but relatively little use of pthread_atfork. >> >> >> >> >> >> You could perhaps replace all uses of fork with a function that called the = >> fork handlers=2C >> >> then fork=2C then the other handlers. That is probably what I meant in the = >> comment >> >> about this being easy to fix. >> >> You might though then run into the problem that callers of fork can't do mu= >> ch >> >> before exec...oh=2C that's vfork I bet actually. > > I don't understand this paragraph either :( > >> >> >> >> >> >> - Jay >> >> >> >> >> >> >>> To: hosking at cs.purdue.edu >>> Date: Fri=2C 11 Feb 2011 19:05:25 -0800 >>> From: mika at async.caltech.edu >>> CC: m3devel at elegosoft.com=3B jay.krell at cornell.edu >>> Subject: [M3devel] pthread_atfork >>> =20 >>> Tony=2C >>> =20 >>> What's RTProcessC.c doing? >>> =20 >>> I note the following cryptic comment: >>> =20 >>> /* NOTE: Even userthreads now depends >>> * on availability of pthreads. >>> * This can be fixed if need be. >>> */ >>> =20 >>> now further down we can read: >>> =20 >>> #if defined(_WIN32) \ >>> || defined(__vms) \ >>> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) >>> return 0=3B >>> #else >>> while (1) >>> { >>> int i =3D pthread_atfork(prepare=2C parent=2C child)=3B >>> if (i !=3D EAGAIN) >>> return i=3B >>> sleep(0)=3B >>> } >>> #endif >>> =20 >>> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=2C >>> RTProcess.RegisterForkHandlers does nothing? >>> =20 >>> Hmmm..... >>> =20 >>> Mika >> = >> >> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >> Content-Type: text/html; charset="iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> >> >> >> >> >> I wrote that.
>> As I recall=2C FreeBSD <=3B 6 doesn't have pthread_atfork.
>> I think that was based on reading the man pages.
>> So=2C yes. What would you suggest?
>>
>>
>> All this atfork stuff came about as part of getting cvsupd to work.
>>  =3B With pthreads/kernel threads. It historically worked=2C with Modul= >> a-3 user threads.
>>  =3B Modula-3 user threads have this funny property that all threads su= >> rvive work.
>>  =3B With pthreads/kernel threads=2C fork gives you a new process with = >> just one thread.
>>  =3B =3B (Solaris has fork1 and forkall=2C and fork maybe used to b= >> e forkall=2C but now it is fork1).
>>
>>
>> cvsupd is one of those slightly unusual programs that does "fork and then d= >> o more work"
>> as supposed to the more typical "fork and exec".
>>
>> Which is to say..I don't know. Maybe=2C don't expect pthreads to work on Fr= >> eeBSD <=3B 6?
>> At least not in a process that ever calls fork? Or does "fork and then more= >> work"?
>>
>>
>> In reality=2C i think a lot of libraries have trouble with "fork and then d= >> o more work"=2C but
>> I don't know. I recall vague notices in the Apple/Darwin manpages not to as= >> sume
>> this works -- the obvious implication that there is plenty of use of pthrea= >> ds e.g. mutexes=2C
>> but relatively little use of pthread_atfork.
>>
>>
>> You could perhaps replace all uses of fork with a function that called the = >> fork handlers=2C
>> then fork=2C then the other handlers. That is probably what I meant in the = >> comment
>> about this being easy to fix.
>> You might though then run into the problem that callers of fork can't do mu= >> ch
>> before exec...oh=2C that's vfork I bet actually.
>>
>>
>>  =3B- Jay






>=3B To: hosking at cs.purdue.edu> r>>=3B Date: Fri=2C 11 Feb 2011 19:05:25 -0800
>=3B From: mika at async= >> .caltech.edu
>=3B CC: m3devel at elegosoft.com=3B jay.krell at cornell.edu> r>>=3B Subject: [M3devel] pthread_atfork
>=3B
>=3B Tony=2C
= >> >=3B
>=3B What's RTProcessC.c doing?
>=3B
>=3B I note th= >> e following cryptic comment:
>=3B
>=3B /* NOTE: Even userthreads= >> now depends
>=3B * on availability of pthreads.
>=3B * This ca= >> n be fixed if need be.
>=3B */
>=3B
>=3B now further down = >> we can read:
>=3B
>=3B #if defined(_WIN32) \
>=3B |= >> | defined(__vms) \
>=3B || (defined(__FreeBSD__) &=3B&= >> =3B (__FreeBSD__ <=3B 6))
>=3B return 0=3B
>=3B #else
&g= >> t=3B while (1)
>=3B {
>=3B int i =3D pthread_atfork= >> (prepare=2C parent=2C child)=3B
>=3B if (i !=3D EAGAIN)
>= >> =3B return i=3B
>=3B sleep(0)=3B
>=3B }
>= >> =3B #endif
>=3B
>=3B so on FreeBSD 5 (what I'm running on my "Fr= >> eeBSD4" system)=2C
>=3B RTProcess.RegisterForkHandlers does nothing?> r>>=3B
>=3B Hmmm.....
>=3B
>=3B Mika
>> = >> >> = >> >> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_-- > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Sat Feb 12 21:25:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 12:25:50 -0800 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212205527.e6xetojo0ssgocg4@mail.elegosoft.com> References: <20110212030525.822411A2078@async.async.caltech.edu> <20110212165930.DC32C1A2078@async.async.caltech.edu> <20110212205527.e6xetojo0ssgocg4@mail.elegosoft.com> Message-ID: <20110212202550.542D41A2078@async.async.caltech.edu> How do you do "fork-and-do-work" from Modula-3? All my programs that create new Unix processes do it with Process.Create, which as far as I know does a fork-and-exec... I'm not sure what if anything is broken here, either. But then again I didn't fully understand Jay's last mail on the subject. Mika Olaf Wagner writes: >Well, I think actually the fork-and-do-work pattern has been >the default for a long time with the M3 user threads implementation. >It wasn't specified though, and I think no application except >CVSup did use it. > >Jay is not responsible for any breakage here, he just tried to fix >CVSup and establish a consistent behaviour after our change to >system pthreads on most platforms. > >Olaf > >Quoting Mika Nystrom : > >> Hi Jay, >> >> I think "fork and do work" qualifies as a bug in the application, if >> the application wants to be portable. Why can't it fork and exec instead? >> If it wants to do more work it should just fork and exec another copy >> of itself. >> >> fork() is also not part of Modula-3... Process.Create() is, however. >> Don't tell me you broke that as part of "fixing" things for cvsupd... >> >> There is still one aspect of your comment I don't understand. What does >> the call to pthread_atfork actually accomplish when running with user >> threads? You say below that "don't expect pthreads to work on FreeBSD < >> 6 [using the fork-and-do-work pattern]". Fair enough. But why would >> user threads work? We're skipping pthread_atfork for them too. >> >> Do we not need pthread_atfork with user threads? What you're saying >> suggests that FreeBSD >=3D 6 and all other platforms need pthread_atfork >> with user threads. Is that really what you mean? >> >> At the moment I'm just concerned with getting user threads to work as >> I know there are bugs with pthreads. >> >> I also think we should not waste time and effort making the "fork and do >> work" pattern work unless it's really, really necessary (which I really >> don't see why it would be). Fix the application instead of introducing >> bugs in the libraries and language implementation! >> >> I've been trying for a long, long time to switch from PM3 to CM3 (almost >> ten years, I think). These bugs from "enhancements" have stopped me >> every time. Maybe things are finally working well enough... >> >> (some comments inline in your quoted text below) >> >> Mika >> >> >> Jay K writes: >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >>> Content-Type: text/plain; charset=3D"iso-8859-1" >>> Content-Transfer-Encoding: quoted-printable >>> >>> >>> I wrote that. >>> >>> As I recall=3D2C FreeBSD < 6 doesn't have pthread_atfork. >>> >>> I think that was based on reading the man pages. >>> >>> So=3D2C yes. What would you suggest? >>> >>> >>> >>> >>> >>> All this atfork stuff came about as part of getting cvsupd to work. >>> >>> With pthreads/kernel threads. It historically worked=3D2C with Modula-3 = >use=3D >>> r threads. >>> >>> Modula-3 user threads have this funny property that all threads survive = >w=3D >>> ork. >> >> you mean survive fork don't you? >> >>> >>> With pthreads/kernel threads=3D2C fork gives you a new process with just= > on=3D >>> e thread. >>> >>> (Solaris has fork1 and forkall=3D2C and fork maybe used to be forkall= >=3D2C b=3D >>> ut now it is fork1). >>> >>> >>> >>> >>> >>> cvsupd is one of those slightly unusual programs that does "fork and then= > d=3D >>> o more work" >>> >>> as supposed to the more typical "fork and exec". >>> >>> >>> >>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o= >n Fr=3D >>> eeBSD < 6? >>> >>> At least not in a process that ever calls fork? Or does "fork and then mo= >re=3D >>> work"? >>> >>> >>> >>> >>> >>> In reality=3D2C i think a lot of libraries have trouble with "fork and th= >en d=3D >>> o more work"=3D2C but >>> >>> I don't know. I recall vague notices in the Apple/Darwin manpages not to = >as=3D >>> sume >>> >>> this works -- the obvious implication that there is plenty of use of pthr= >ea=3D >>> ds e.g. mutexes=3D2C >>> >>> but relatively little use of pthread_atfork. >>> >>> >>> >>> >>> >>> You could perhaps replace all uses of fork with a function that called th= >e =3D >>> fork handlers=3D2C >>> >>> then fork=3D2C then the other handlers. That is probably what I meant in = >the =3D >>> comment >>> >>> about this being easy to fix. >>> >>> You might though then run into the problem that callers of fork can't do = >mu=3D >>> ch >>> >>> before exec...oh=3D2C that's vfork I bet actually. >> >> I don't understand this paragraph either :( >> >>> >>> >>> >>> >>> >>> - Jay >>> >>> >>> >>> >>> >>> >>>> To: hosking at cs.purdue.edu >>>> Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800 >>>> From: mika at async.caltech.edu >>>> CC: m3devel at elegosoft.com=3D3B jay.krell at cornell.edu >>>> Subject: [M3devel] pthread_atfork >>>> =3D20 >>>> Tony=3D2C >>>> =3D20 >>>> What's RTProcessC.c doing? >>>> =3D20 >>>> I note the following cryptic comment: >>>> =3D20 >>>> /* NOTE: Even userthreads now depends >>>> * on availability of pthreads. >>>> * This can be fixed if need be. >>>> */ >>>> =3D20 >>>> now further down we can read: >>>> =3D20 >>>> #if defined(_WIN32) \ >>>> || defined(__vms) \ >>>> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) >>>> return 0=3D3B >>>> #else >>>> while (1) >>>> { >>>> int i =3D3D pthread_atfork(prepare=3D2C parent=3D2C child)=3D3B >>>> if (i !=3D3D EAGAIN) >>>> return i=3D3B >>>> sleep(0)=3D3B >>>> } >>>> #endif >>>> =3D20 >>>> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=3D2C >>>> RTProcess.RegisterForkHandlers does nothing? >>>> =3D20 >>>> Hmmm..... >>>> =3D20 >>>> Mika >>> =09=09 =09 =09=09 =3D >>> >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ >>> Content-Type: text/html; charset=3D"iso-8859-1" >>> Content-Transfer-Encoding: quoted-printable >>> >>> >>> >>> >>> >>> >>> I wrote that.
>>> As I recall=3D2C FreeBSD <=3D3B 6 doesn't have pthread_atfork.
>>> I think that was based on reading the man pages.
>>> So=3D2C yes. What would you suggest?
>>>
>>>
>>> All this atfork stuff came about as part of getting cvsupd to work.
>>>  =3D3B With pthreads/kernel threads. It historically worked=3D2C with= > Modul=3D >>> a-3 user threads.
>>>  =3D3B Modula-3 user threads have this funny property that all thread= >s su=3D >>> rvive work.
>>>  =3D3B With pthreads/kernel threads=3D2C fork gives you a new process= > with =3D >>> just one thread.
>>>  =3D3B =3D3B (Solaris has fork1 and forkall=3D2C and fork maybe u= >sed to b=3D >>> e forkall=3D2C but now it is fork1).
>>>
>>>
>>> cvsupd is one of those slightly unusual programs that does "fork and then= > d=3D >>> o more work"
>>> as supposed to the more typical "fork and exec".
>>>
>>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o= >n Fr=3D >>> eeBSD <=3D3B 6?
>>> At least not in a process that ever calls fork? Or does "fork and then mo= >re=3D >>> work"?
>>>
>>>
>>> In reality=3D2C i think a lot of libraries have trouble with "fork and th= >en d=3D >>> o more work"=3D2C but
>>> I don't know. I recall vague notices in the Apple/Darwin manpages not to = >as=3D >>> sume
>>> this works -- the obvious implication that there is plenty of use of pthr= >ea=3D >>> ds e.g. mutexes=3D2C
>>> but relatively little use of pthread_atfork.
>>>
>>>
>>> You could perhaps replace all uses of fork with a function that called th= >e =3D >>> fork handlers=3D2C
>>> then fork=3D2C then the other handlers. That is probably what I meant in = >the =3D >>> comment
>>> about this being easy to fix.
>>> You might though then run into the problem that callers of fork can't do = >mu=3D >>> ch
>>> before exec...oh=3D2C that's vfork I bet actually.
>>>
>>>
>>>  =3D3B- Jay






>=3D3B To: hosking at cs.purdue= >.edu>> r>>=3D3B Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800
>=3D3B From: mi= >ka at async=3D >>> .caltech.edu
>=3D3B CC: m3devel at elegosoft.com=3D3B jay.krell at cornell= >.edu>> r>>=3D3B Subject: [M3devel] pthread_atfork
>=3D3B
>=3D3B Ton= >y=3D2C
=3D >>> >=3D3B
>=3D3B What's RTProcessC.c doing?
>=3D3B
>=3D3B= > I note th=3D >>> e following cryptic comment:
>=3D3B
>=3D3B /* NOTE: Even usert= >hreads=3D >>> now depends
>=3D3B * on availability of pthreads.
>=3D3B * Th= >is ca=3D >>> n be fixed if need be.
>=3D3B */
>=3D3B
>=3D3B now furth= >er down =3D >>> we can read:
>=3D3B
>=3D3B #if defined(_WIN32) \
>=3D3B = > |=3D >>> | defined(__vms) \
>=3D3B || (defined(__FreeBSD__) &=3D3B= >&=3D >>> =3D3B (__FreeBSD__ <=3D3B 6))
>=3D3B return 0=3D3B
>=3D3B= > #else
&g=3D >>> t=3D3B while (1)
>=3D3B {
>=3D3B int i =3D3D pthr= >ead_atfork=3D >>> (prepare=3D2C parent=3D2C child)=3D3B
>=3D3B if (i !=3D3D EAGA= >IN)
>=3D >>> =3D3B return i=3D3B
>=3D3B sleep(0)=3D3B
>=3D3B = > }
>=3D >>> =3D3B #endif
>=3D3B
>=3D3B so on FreeBSD 5 (what I'm running o= >n my "Fr=3D >>> eeBSD4" system)=3D2C
>=3D3B RTProcess.RegisterForkHandlers does noth= >ing?>> r>>=3D3B
>=3D3B Hmmm.....
>=3D3B
>=3D3B Mika
>>> =09 =09 =09=09 =3D >>> >>> =3D >>> >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_-- >> > > > >--=20 >Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa= >ny >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz= >: Berlin >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Sat Feb 12 21:43:02 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 12:43:02 -0800 Subject: [M3devel] Performance issues with Process.Wait under userthreads Message-ID: <20110212204302.19C781A2078@async.async.caltech.edu> Hi again m3devel (especially Tony), I have finally taken a bite of a problem that has been annoying me for a very, very long time. Under user threads, the code for Process.Wait is as follows (see ThreadPosix.m3): PROCEDURE WaitProcess (pid: int; VAR status: int): int = (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) CONST Delay = 0.1D0; BEGIN LOOP WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO IF r # 0 THEN RETURN r END; END; Pause(Delay); END; END WaitProcess; It inserts a 0.1 second delay after each failed waitpid. This is extremely annoying for programs that start a long sequence of child processes and wait for them in sequence. Namely, the compiler itself. As a result the cm3 compiler (and PM3's m3build) are normally very very slow when using user threads. For about the last ten years, I've had a hacked up m3build (for my PM3 installation) that skips the Pause and busy-waits instead. Note there is another problem here. Since the Modula-3 runtime ignores SIGCHLD, no zombie processes are created since the Unix system automatically reaps the child processes. I can see this would be a problem since PIDs are eventually reused and ... couldn't Uexec.waitpid wind up referring to the wrong process?? I will further note that the comment in Process.i3 reads as follows: PROCEDURE Wait(p: T): ExitCode; Wait until the process with handle p terminates, then free the operating system resources associated with the process and return an exit code indicating the reason for its termination. It is a checked runtime error to call Wait twice on the same process handle. I am going to take this as fair warning that Process.Create *may* use resources that are not going to be released until Process.Wait has been called. I have modified (in my local copy of CM3) the system as follows. I have come up with a semi-general mechanism for immediately unblocking a thread on the receipt of a unix signal. 1. The system relies on changing ThreadPosix.XPause such that if a signal is allowed to wake up a threads, that fact is recorded in a new field in the thread's descriptor record (of type ThreadPosix.T). 2. On receipt of a waited-for unix signal, a mask is set and control is passed to the thread scheduler which maintains the non-zero mask for exactly one iteration through the thread ring. 3. If a thread is paused and waiting for EITHER a signal or some time, the thread is released for running and the thread's waiting state is cleared. The changes are more or less as follows: 1. I have added a new field of type "int" to ThreadPosix.T: (* if state = pausing, the time at which we can restart *) waitingForTime: Time.T; + (* if state = pausing, the signal that truncates the pause *) + waitingForSig: int := -1; + (* true if we are waiting during an AlertWait or AlertJoin or AlertPause *) alertable: BOOLEAN := FALSE; 2. Modifications to pause: + PROCEDURE SigPause(n: LONGREAL; sig: int)= + <*FATAL Alerted*> + VAR until := n + Time.Now (); + BEGIN + XPause(until, FALSE, sig); + END SigPause; + PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= VAR until := n + Time.Now (); BEGIN XPause(until, TRUE); END AlertPause; ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) ! RAISES {Alerted} = BEGIN INC (inCritical); self.waitingForTime := until; self.alertable := alertable; + IF sig # -1 THEN + self.waitingForSig := sig + END; ICannotRun (State.pausing); DEC (inCritical); InternalYield (); 3. The received-signals mask: ! CONST MaxSigs = 64; ! TYPE Sig = [ 0..MaxSigs-1 ]; ! ! (* in order to listen to other signals, they have to be enabled in ! allow_sigvtalrm as well *) ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); ! ! gotSigs := SET OF Sig { }; ! ValueOfSIGCHLD() is a C function used to get the value of the SIGCHLD constant without guessing at it (in ThreadPosixC.c). 4. changes to the signal handler: ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); ! ! INC(inCritical); ! (* mark signal as being delivered *) ! IF sig >= 0 AND sig < MaxSigs THEN ! gotSigs := gotSigs + SET OF Sig { sig } ! END; ! DEC(inCritical); ! ! IF inCritical = 0 AND heapState.inCritical = 0 THEN ! InternalYield () ! END; END switch_thread; Note that I don't know if INC/DEC(inCritical) does exactly the right thing here. 5. changes to the scheduler: a. thread wakeup IF t.alertable AND t.alertPending THEN CanRun (t); EXIT; + + ELSIF t.waitingForSig IN gotSigs THEN + t.waitingForSig := -1; + CanRun(t); + EXIT; ELSIF t.waitingForTime <= now THEN CanRun (t); EXIT; ! b. clearing the mask END; END; + gotSigs := SET OF Sig {}; + IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN IF perfOn THEN PerfRunning (t.id); END; (* At least one thread wants to run; transfer to it *) 6. changes to WaitProcess (Process.Wait): PROCEDURE WaitProcess (pid: int; VAR status: int): int = (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) ! CONST Delay = 10.0D0; BEGIN LOOP WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO IF r # 0 THEN RETURN r END; END; ! SigPause(Delay,SIGCHLD); END; END WaitProcess; 7. install signal handler even if program is single-threaded: BEGIN + (* we need to call set up the signal handler for other reasons than + just thread switching now *) + setup_sigvtalrm (switch_thread); END ThreadPosix. 8. modify signal handler in ThreadPosixC.c to catch SIGCHLD: sigemptyset(&ThreadSwitchSignal); sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); + sigaddset(&ThreadSwitchSignal, SIGCHLD); act.sa_handler = handler; act.sa_flags = SA_RESTART; sigemptyset(&(act.sa_mask)); if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); + if (sigaction (SIGCHLD, &act, NULL)) abort(); I'll send the complete diff in a separate message for those who want to study it more closely. I propose the above changes for inclusion in the current CM3 repository. Mika From mika at async.caltech.edu Sat Feb 12 21:44:06 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 12:44:06 -0800 Subject: [M3devel] SIGCHLD diff Message-ID: <20110212204406.88A661A2078@async.async.caltech.edu> Index: ThreadPosix.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v retrieving revision 1.16 diff -c -r1.16 ThreadPosix.i3 *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 *************** *** 39,42 **** --- 39,45 ---- (*---------------------------------------------------------------------------*) + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> + PROCEDURE ValueOfSIGCHLD(): int; + END ThreadPosix. Index: ThreadPosix.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v retrieving revision 1.73 diff -c -r1.73 ThreadPosix.m3 *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 *************** *** 97,102 **** --- 97,105 ---- (* if state = pausing, the time at which we can restart *) waitingForTime: Time.T; + (* if state = pausing, the signal that truncates the pause *) + waitingForSig: int := -1; + (* true if we are waiting during an AlertWait or AlertJoin or AlertPause *) alertable: BOOLEAN := FALSE; *************** *** 147,152 **** --- 150,160 ---- defaultStackSize := 3000; + (* note that even though the "heavy machinery" is only used for + multipleThreads, we still need to set up the signal handler so + that we can catch signals from other sources than thread switching. + e.g., we use SIGCHLD to speed up Process.Wait *) + VAR stats: RECORD n_forks := 0; *************** *** 511,527 **** XPause(until, FALSE); END Pause; PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= VAR until := n + Time.Now (); BEGIN XPause(until, TRUE); END AlertPause; ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE) RAISES {Alerted} = BEGIN INC (inCritical); self.waitingForTime := until; self.alertable := alertable; ICannotRun (State.pausing); DEC (inCritical); InternalYield (); --- 519,546 ---- XPause(until, FALSE); END Pause; + PROCEDURE SigPause(n: LONGREAL; sig: int)= + <*FATAL Alerted*> + VAR until := n + Time.Now (); + BEGIN + XPause(until, FALSE, sig); + END SigPause; + PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= VAR until := n + Time.Now (); BEGIN XPause(until, TRUE); END AlertPause; ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) ! RAISES {Alerted} = BEGIN INC (inCritical); self.waitingForTime := until; self.alertable := alertable; + IF sig # -1 THEN + self.waitingForSig := sig + END; ICannotRun (State.pausing); DEC (inCritical); InternalYield (); *************** *** 703,712 **** END; END StartSwitching; ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); ! IF inCritical = 0 AND heapState.inCritical = 0 THEN InternalYield () END; END switch_thread; (*------------------------------------------------------------- scheduler ---*) --- 722,750 ---- END; END StartSwitching; ! CONST MaxSigs = 64; ! TYPE Sig = [ 0..MaxSigs-1 ]; ! ! (* in order to listen to other signals, they have to be enabled in ! allow_sigvtalrm as well *) ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); ! ! gotSigs := SET OF Sig { }; ! ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); ! ! INC(inCritical); ! (* mark signal as being delivered *) ! IF sig >= 0 AND sig < MaxSigs THEN ! gotSigs := gotSigs + SET OF Sig { sig } ! END; ! DEC(inCritical); ! ! IF inCritical = 0 AND heapState.inCritical = 0 THEN ! InternalYield () ! END; END switch_thread; (*------------------------------------------------------------- scheduler ---*) *************** *** 782,792 **** IF t.alertable AND t.alertPending THEN CanRun (t); EXIT; ELSIF t.waitingForTime <= now THEN CanRun (t); EXIT; ! ELSIF NOT somePausing THEN earliest := t.waitingForTime; somePausing := TRUE; --- 820,835 ---- IF t.alertable AND t.alertPending THEN CanRun (t); EXIT; + + ELSIF t.waitingForSig IN gotSigs THEN + t.waitingForSig := -1; + CanRun(t); + EXIT; ELSIF t.waitingForTime <= now THEN CanRun (t); EXIT; ! ELSIF NOT somePausing THEN earliest := t.waitingForTime; somePausing := TRUE; *************** *** 886,891 **** --- 929,936 ---- END; END; + gotSigs := SET OF Sig {}; + IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN IF perfOn THEN PerfRunning (t.id); END; (* At least one thread wants to run; transfer to it *) *************** *** 948,960 **** PROCEDURE WaitProcess (pid: int; VAR status: int): int = (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) ! CONST Delay = 0.1D0; BEGIN LOOP WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO IF r # 0 THEN RETURN r END; END; ! Pause(Delay); END; END WaitProcess; --- 993,1005 ---- PROCEDURE WaitProcess (pid: int; VAR status: int): int = (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) ! CONST Delay = 10.0D0; BEGIN LOOP WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO IF r # 0 THEN RETURN r END; END; ! SigPause(Delay,SIGCHLD); END; END WaitProcess; *************** *** 1361,1364 **** --- 1406,1412 ---- VAR debug := RTParams.IsPresent ("debugthreads"); BEGIN + (* we need to call set up the signal handler for other reasons than + just thread switching now *) + setup_sigvtalrm (switch_thread); END ThreadPosix. Index: ThreadPosixC.c =================================================================== RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v retrieving revision 1.59 diff -c -r1.59 ThreadPosixC.c *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 *************** *** 78,88 **** --- 78,97 ---- sigemptyset(&ThreadSwitchSignal); sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); + sigaddset(&ThreadSwitchSignal, SIGCHLD); act.sa_handler = handler; act.sa_flags = SA_RESTART; sigemptyset(&(act.sa_mask)); if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); + if (sigaction (SIGCHLD, &act, NULL)) abort(); + } + + int + __cdecl + ThreadPosix__value_of_SIGCHLD(void) + { + return SIGCHLD; } void From dabenavidesd at yahoo.es Sat Feb 12 22:08:13 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Sat, 12 Feb 2011 21:08:13 +0000 (GMT) Subject: [M3devel] pthread_atfork In-Reply-To: <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com> Message-ID: <651549.53394.qm@web29714.mail.ird.yahoo.com> Hi all: by the way you told, do user space interfaces need to be linked differently from user space threads library than the one in kernel threads, so the behaviour can be selected better at runtime if more than one available processor (i.e a virtualized one, perhaps two if there are virtualized GPU, GPGPU, etc) is available on demand (in example the collector has its own user space thread allocated control and one in kernel space that user can control from the Runtime), could it work in such a way to allow a Thread to live in spaces that are accesible through like an NetObj.T, thus allowing it to communciate them through such an abstraction? Perhaps this could allow to run the kernel threads with Modula-3 governance and if so put them to work or switch to kernel them to accelerate its process according to its demands and such debug them in a special API like the one in the SPIN OS, fairly more comprenhensive thread implementation of Mach, Cthreads (aka C Pthreads calls) and own strends. Also scheduling could be handled to manage both user/kernel level synchronization, the threads and also allow to change from having a specialized hardware-assisted virtualized processor to a non hardware-assissted virtualized processor (like an NetObj.T again), which is like the same situation for Modula-3 with just one processor so in case it needs, it does so. In any case thread and object migration are safely defined over Obliq. Granted, this is same situation like was before the enter of virtualization hardware but again the whole idea of hardware-assisted virtualization was not new, even there is an article in which a PDP-10 is claimed has requirements to make it virtual in hardware (and I believe in software too). So SPIN could be a platform, like for hardware assissted virtualization and/or operating-system level virtualization, if so, we would be able to trace system calls for a given OS client running on it, rather than barely, so it makes the implementation with hardware-assisted source-level debugger ready, which is related to where programs are segfaulting now (no system-level breakpoints, that is, no hardware assisted source-level debugger for virtualization user, different from Modula-3 situation in current user-level space threading or SPIN gdb DEC TTD Topaz Tele Debugger at source-level service and not to forget the ldb cross-platform source-level remote debugger) in pthread current implementations, likewise the whole idea of customizable scheduler and kernel allocator weren't and aren't new nor bad at all, neither for us nor were for SPINers; remember the CSA Occam2 TRAC sourced OS SuperSetPlus machine which had from 16 up to 256 individual processors independently executable as according to their product info add it could be connected over LAN up to 16 Mac, PC, and Sun workstations, pretty much like a compilation and execution engine in one, pretty and good mussle to have to see running. They offered besides Occam2, C, C++, Modula-2, Pascal and FORTRAN, prolog and Ada with source level debuggers and Operating Systems as their Software products too. I hope this would be an scenario in some years to come, at least for any of us, if solet's see where and what this takes to/from us. The main question will be whether this guys will allow enogh detail to drive their GPGPU or CPU, I hope this history doesn't repeat like it is and wasn't in the past of machine OSes sources, i.e braking traditional development etc. I'm sorry for the long of the message. Thanks in advance --- El s?b, 12/2/11, Olaf Wagner escribi?: > De: Olaf Wagner > Asunto: Re: [M3devel] pthread_atfork > Para: m3devel at elegosoft.com > Fecha: s?bado, 12 de febrero, 2011 07:02 > I think it would be acceptable to say > that pthread threading is > broken on FreeBSD6 (as it is not officially supported any > more). > All the stuff below should be documented in the 'known > problems' > section. Especially the different behaviours of fork for > user and pthreads. This should also be mentioned in the M3 > interfaces (source code)! > > Also that to use use threads with FreeBSD requires to > remove the > -pthread flag, as Mika noted. > > BTW, do we have a chapter explaining what to do to use user > threads > instead of the default pthreads? > > Olaf > > PS: Hint: this would be a great task for anybody lurking on > the lists > and trying to get involved :-) > > Quoting Jay K : > > > > > I wrote that. > > > > As I recall, FreeBSD < 6 doesn't have > pthread_atfork. > > > > I think that was based on reading the man pages. > > > > So, yes. What would you suggest? > > > > > > > > > > > > All this atfork stuff came about as part of getting > cvsupd to work. > > > > With pthreads/kernel threads. It > historically worked, with > > Modula-3 user threads. > > > > Modula-3 user threads have this funny > property that all threads > > survive work. > > > > With pthreads/kernel threads, fork > gives you a new process with > > just one thread. > > > > (Solaris has fork1 and forkall, and fork > maybe used to be > > forkall, but now it is fork1). > > > > > > > > > > > > cvsupd is one of those slightly unusual programs that > does "fork and > > then do more work" > > > > as supposed to the more typical "fork and exec". > > > > > > > > Which is to say..I don't know. Maybe, don't expect > pthreads to work > > on FreeBSD < 6? > > > > At least not in a process that ever calls fork? Or > does "fork and > > then more work"? > > > > > > > > > > > > In reality, i think a lot of libraries have trouble > with "fork and > > then do more work", but > > > > I don't know. I recall vague notices in the > Apple/Darwin manpages > > not to assume > > > > this works -- the obvious implication that there is > plenty of use of > > pthreads e.g. mutexes, > > > > but relatively little use of pthread_atfork. > > > > > > > > > > > > You could perhaps replace all uses of fork with a > function that > > called the fork handlers, > > > > then fork, then the other handlers. That is probably > what I meant in > > the comment > > > > about this being easy to fix. > > > > You might though then run into the problem that > callers of fork can't do much > > > > before exec...oh, that's vfork I bet actually. > > > > > > > > > > > > - Jay > > > > > > > > > > > > > >> To: hosking at cs.purdue.edu > >> Date: Fri, 11 Feb 2011 19:05:25 -0800 > >> From: mika at async.caltech.edu > >> CC: m3devel at elegosoft.com; > jay.krell at cornell.edu > >> Subject: [M3devel] pthread_atfork > >> > >> Tony, > >> > >> What's RTProcessC.c doing? > >> > >> I note the following cryptic comment: > >> > >> /* NOTE: Even userthreads now depends > >> * on availability of pthreads. > >> * This can be fixed if need be. > >> */ > >> > >> now further down we can read: > >> > >> #if defined(_WIN32) \ > >> || > defined(__vms) \ > >> || > (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > >> return 0; > >> #else > >> while (1) > >> { > >> int i = > pthread_atfork(prepare, parent, child); > >> if (i != EAGAIN) > >> return i; > >> sleep(0); > >> } > >> #endif > >> > >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" > system), > >> RTProcess.RegisterForkHandlers does nothing? > >> > >> Hmmm..... > >> > >> Mika > > > > > > -- > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 > 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | > Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | > USt-IdNr: DE163214194 > > From jay.krell at cornell.edu Sat Feb 12 22:45:18 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 21:45:18 +0000 Subject: [M3devel] pthread_atfork In-Reply-To: <651549.53394.qm@web29714.mail.ird.yahoo.com> References: <20110212130211.0h7j5eev44gs8w00@mail.elegosoft.com>, <651549.53394.qm@web29714.mail.ird.yahoo.com> Message-ID: > so the behaviour can be selected better at runtime if more than one available processor I would like user vs. kernel threads to be selectable at link and/or run-time, but Tony has nixed the small changes required to make it so. It has also been suggested you could copy around different m3core.so files to switch, but that isn't the case either. Thread.T has a different typeid in the two systems. Anyway, the better behavior is almost always kernel threads. You should really just generally stick with them. You do forever ongoing debate on these matters, and the "pendulum" has seemingly swung many times, so I can't really say everyone agrees in kernel threads always. Windows 7 adds "user mode scheduling", which is still kernel threads -- there is still a one-to-one mapping between some usermode state and kernelmode state, but the scheduling decisions are made in usermode. I believe pthreads on OpenVMS use the "m:n" approach. But Linux, Solaris by default, Darwin, I suspect but am not sure FreeBSD and NetBSD, Win32 all use the simple "1:1" model. Where every thread is a kernel thread. All scheduling is done by the kernel. Win32 also has "fibers" but these are fairly...surprisingly difficult to use, but in a way that Tony et. al. wouldn't be surprised by. When you use fibers, you have to write your own scheduler, and your own locking primitives. Normal Win32 functions like GetCurrentThreadId(), EnterCriticalSection() become sort of useless/broken in the face of fibers. Basically the same as what it takes to implement Modula-3 user threads. Though you can run on multiple processors I believe fairly easily. Hm. The "m:n" model is actually not difficult to implement on Win32 and w/ pthreads. But still, probably not a good idea. - Jay > Date: Sat, 12 Feb 2011 21:08:13 +0000 > From: dabenavidesd at yahoo.es > To: m3devel at elegosoft.com; wagner at elegosoft.com > Subject: Re: [M3devel] pthread_atfork > > Hi all: > by the way you told, do user space interfaces need to be linked differently from user space threads library than the one in kernel threads, so the behaviour can be selected better at runtime if more than one available processor (i.e a virtualized one, perhaps two if there are virtualized GPU, GPGPU, etc) is available on demand (in example the collector has its own user space thread allocated control and one in kernel space that user can control from the Runtime), could it work in such a way to allow a Thread to live in spaces that are accesible through like an NetObj.T, thus allowing it to communciate them through such an abstraction? > Perhaps this could allow to run the kernel threads with Modula-3 governance and if so put them to work or switch to kernel them to accelerate its process according to its demands and such debug them in a special API like the one in the SPIN OS, fairly more comprenhensive thread implementation of Mach, Cthreads (aka C Pthreads calls) and own strends. Also scheduling could be handled to manage both user/kernel level synchronization, the threads and also allow to change from having a specialized hardware-assisted virtualized processor to a non hardware-assissted virtualized processor (like an NetObj.T again), which is like the same situation for Modula-3 with just one processor so in case it needs, it does so. In any case thread and object migration are safely defined over Obliq. > Granted, this is same situation like was before the enter of virtualization hardware but again the whole idea of hardware-assisted virtualization was not new, even there is an article in which a PDP-10 is claimed has requirements to make it virtual in hardware (and I believe in software too). So SPIN could be a platform, like for hardware assissted virtualization and/or operating-system level virtualization, if so, we would be able to trace system calls for a given OS client running on it, rather than barely, so it makes the implementation with hardware-assisted source-level debugger ready, which is related to where programs are segfaulting now (no system-level breakpoints, that is, no hardware assisted source-level debugger for virtualization user, different from Modula-3 situation in current user-level space threading or SPIN gdb DEC TTD Topaz Tele Debugger at source-level service and not to forget the ldb cross-platform source-level remote debugger) > in pthread current implementations, likewise the whole idea of customizable scheduler and kernel allocator weren't and aren't new nor bad at all, neither for us nor were for SPINers; remember the CSA Occam2 TRAC sourced OS SuperSetPlus machine which had from 16 up to 256 individual processors independently executable as according to their product info add it could be connected over LAN up to 16 Mac, PC, and Sun workstations, pretty much like a compilation and execution engine in one, pretty and good mussle to have to see running. They offered besides Occam2, C, C++, Modula-2, Pascal and FORTRAN, prolog and Ada with source level debuggers and Operating Systems as their Software products too. I hope this would be an scenario in some years to come, at least for any of us, if solet's see where and what this takes to/from us. > The main question will be whether this guys will allow enogh detail to drive their GPGPU or CPU, I hope this history doesn't repeat like it is and wasn't in the past of machine OSes sources, i.e braking traditional development etc. > I'm sorry for the long of the message. > Thanks in advance > > --- El s?b, 12/2/11, Olaf Wagner escribi?: > > > De: Olaf Wagner > > Asunto: Re: [M3devel] pthread_atfork > > Para: m3devel at elegosoft.com > > Fecha: s?bado, 12 de febrero, 2011 07:02 > > I think it would be acceptable to say > > that pthread threading is > > broken on FreeBSD6 (as it is not officially supported any > > more). > > All the stuff below should be documented in the 'known > > problems' > > section. Especially the different behaviours of fork for > > user and pthreads. This should also be mentioned in the M3 > > interfaces (source code)! > > > > Also that to use use threads with FreeBSD requires to > > remove the > > -pthread flag, as Mika noted. > > > > BTW, do we have a chapter explaining what to do to use user > > threads > > instead of the default pthreads? > > > > Olaf > > > > PS: Hint: this would be a great task for anybody lurking on > > the lists > > and trying to get involved :-) > > > > Quoting Jay K : > > > > > > > > I wrote that. > > > > > > As I recall, FreeBSD < 6 doesn't have > > pthread_atfork. > > > > > > I think that was based on reading the man pages. > > > > > > So, yes. What would you suggest? > > > > > > > > > > > > > > > > > > All this atfork stuff came about as part of getting > > cvsupd to work. > > > > > > With pthreads/kernel threads. It > > historically worked, with > > > Modula-3 user threads. > > > > > > Modula-3 user threads have this funny > > property that all threads > > > survive work. > > > > > > With pthreads/kernel threads, fork > > gives you a new process with > > > just one thread. > > > > > > (Solaris has fork1 and forkall, and fork > > maybe used to be > > > forkall, but now it is fork1). > > > > > > > > > > > > > > > > > > cvsupd is one of those slightly unusual programs that > > does "fork and > > > then do more work" > > > > > > as supposed to the more typical "fork and exec". > > > > > > > > > > > > Which is to say..I don't know. Maybe, don't expect > > pthreads to work > > > on FreeBSD < 6? > > > > > > At least not in a process that ever calls fork? Or > > does "fork and > > > then more work"? > > > > > > > > > > > > > > > > > > In reality, i think a lot of libraries have trouble > > with "fork and > > > then do more work", but > > > > > > I don't know. I recall vague notices in the > > Apple/Darwin manpages > > > not to assume > > > > > > this works -- the obvious implication that there is > > plenty of use of > > > pthreads e.g. mutexes, > > > > > > but relatively little use of pthread_atfork. > > > > > > > > > > > > > > > > > > You could perhaps replace all uses of fork with a > > function that > > > called the fork handlers, > > > > > > then fork, then the other handlers. That is probably > > what I meant in > > > the comment > > > > > > about this being easy to fix. > > > > > > You might though then run into the problem that > > callers of fork can't do much > > > > > > before exec...oh, that's vfork I bet actually. > > > > > > > > > > > > > > > > > > - Jay > > > > > > > > > > > > > > > > > > > > >> To: hosking at cs.purdue.edu > > >> Date: Fri, 11 Feb 2011 19:05:25 -0800 > > >> From: mika at async.caltech.edu > > >> CC: m3devel at elegosoft.com; > > jay.krell at cornell.edu > > >> Subject: [M3devel] pthread_atfork > > >> > > >> Tony, > > >> > > >> What's RTProcessC.c doing? > > >> > > >> I note the following cryptic comment: > > >> > > >> /* NOTE: Even userthreads now depends > > >> * on availability of pthreads. > > >> * This can be fixed if need be. > > >> */ > > >> > > >> now further down we can read: > > >> > > >> #if defined(_WIN32) \ > > >> || > > defined(__vms) \ > > >> || > > (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > > >> return 0; > > >> #else > > >> while (1) > > >> { > > >> int i = > > pthread_atfork(prepare, parent, child); > > >> if (i != EAGAIN) > > >> return i; > > >> sleep(0); > > >> } > > >> #endif > > >> > > >> so on FreeBSD 5 (what I'm running on my "FreeBSD4" > > system), > > >> RTProcess.RegisterForkHandlers does nothing? > > >> > > >> Hmmm..... > > >> > > >> Mika > > > > > > > > > > > -- > > Olaf Wagner -- elego Software Solutions GmbH > > > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > > phone: +49 30 23 45 86 96 mobile: +49 177 2345 > > 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | > > Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > > Handelregister: Amtsgericht Charlottenburg HRB 77719 | > > USt-IdNr: DE163214194 > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 22:50:28 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 21:50:28 +0000 Subject: [M3devel] Performance issues with Process.Wait under userthreads In-Reply-To: <20110212204302.19C781A2078@async.async.caltech.edu> References: <20110212204302.19C781A2078@async.async.caltech.edu> Message-ID: I don't believe there is a good solution here, other than using pthreads. Is there? Maybe tune the wait down smaller? I understand it stinks either way. I don't understand the SIGCHLD stuff, at a quick glance. Nor your changes. However you might be the only user of user threads. Is there any downside to your change? (Other than more code, ok.) Any loss of performance in any situation? Any loss of portability? - Jay > To: m3devel at elegosoft.com > Date: Sat, 12 Feb 2011 12:43:02 -0800 > From: mika at async.caltech.edu > Subject: [M3devel] Performance issues with Process.Wait under userthreads > > Hi again m3devel (especially Tony), > > I have finally taken a bite of a problem that has been annoying me > for a very, very long time. > > Under user threads, the code for Process.Wait is as follows (see > ThreadPosix.m3): > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > CONST Delay = 0.1D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > Pause(Delay); > END; > END WaitProcess; > > It inserts a 0.1 second delay after each failed waitpid. This is extremely > annoying for programs that start a long sequence of child processes and > wait for them in sequence. Namely, the compiler itself. As a result > the cm3 compiler (and PM3's m3build) are normally very very slow when > using user threads. For about the last ten years, I've had a hacked > up m3build (for my PM3 installation) that skips the Pause and busy-waits > instead. > > Note there is another problem here. Since the Modula-3 runtime ignores > SIGCHLD, no zombie processes are created since the Unix system automatically > reaps the child processes. I can see this would be a problem since PIDs > are eventually reused and ... couldn't Uexec.waitpid wind up referring to > the wrong process?? > > I will further note that the comment in Process.i3 reads as follows: > > PROCEDURE Wait(p: T): ExitCode; > > Wait until the process with handle p terminates, then free the operating system resources associated with the process and return an exit code indicating the reason for its termination. It is a checked runtime error to call Wait twice on the same process handle. > > I am going to take this as fair warning that Process.Create *may* use > resources that are not going to be released until Process.Wait has been > called. > > I have modified (in my local copy of CM3) the system as follows. > I have come up with a semi-general mechanism for immediately unblocking > a thread on the receipt of a unix signal. > > 1. The system relies on changing > ThreadPosix.XPause such that if a signal is allowed to wake up a threads, > that fact is recorded in a new field in the thread's descriptor > record (of type ThreadPosix.T). > > 2. On receipt of a waited-for unix signal, a mask is set and control > is passed to the thread scheduler which maintains the non-zero mask for > exactly one iteration through the thread ring. > > 3. If a thread is paused and waiting for EITHER a signal or some time, > the thread is released for running and the thread's waiting state is > cleared. > > The changes are more or less as follows: > > 1. I have added a new field of type "int" to ThreadPosix.T: > > (* if state = pausing, the time at which we can restart *) > waitingForTime: Time.T; > > + (* if state = pausing, the signal that truncates the pause *) > + waitingForSig: int := -1; > + > (* true if we are waiting during an AlertWait or AlertJoin > or AlertPause *) > alertable: BOOLEAN := FALSE; > > > 2. Modifications to pause: > > + PROCEDURE SigPause(n: LONGREAL; sig: int)= > + <*FATAL Alerted*> > + VAR until := n + Time.Now (); > + BEGIN > + XPause(until, FALSE, sig); > + END SigPause; > + > PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= > VAR until := n + Time.Now (); > BEGIN > XPause(until, TRUE); > END AlertPause; > > ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) > ! RAISES {Alerted} = > BEGIN > INC (inCritical); > self.waitingForTime := until; > self.alertable := alertable; > + IF sig # -1 THEN > + self.waitingForSig := sig > + END; > ICannotRun (State.pausing); > DEC (inCritical); > InternalYield (); > > 3. The received-signals mask: > > ! CONST MaxSigs = 64; > ! TYPE Sig = [ 0..MaxSigs-1 ]; > ! > ! (* in order to listen to other signals, they have to be enabled in > ! allow_sigvtalrm as well *) > ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); > ! > ! gotSigs := SET OF Sig { }; > ! > > ValueOfSIGCHLD() is a C function used to get the value of the SIGCHLD > constant without guessing at it (in ThreadPosixC.c). > > 4. changes to the signal handler: > > ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); > ! > ! INC(inCritical); > ! (* mark signal as being delivered *) > ! IF sig >= 0 AND sig < MaxSigs THEN > ! gotSigs := gotSigs + SET OF Sig { sig } > ! END; > ! DEC(inCritical); > ! > ! IF inCritical = 0 AND heapState.inCritical = 0 THEN > ! InternalYield () > ! END; > END switch_thread; > > Note that I don't know if INC/DEC(inCritical) does exactly the right > thing here. > > 5. changes to the scheduler: > > a. thread wakeup > IF t.alertable AND t.alertPending THEN > CanRun (t); > EXIT; > + > + ELSIF t.waitingForSig IN gotSigs THEN > + t.waitingForSig := -1; > + CanRun(t); > + EXIT; > > ELSIF t.waitingForTime <= now THEN > CanRun (t); > EXIT; > ! > > b. clearing the mask > > END; > END; > > + gotSigs := SET OF Sig {}; > + > IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN > IF perfOn THEN PerfRunning (t.id); END; > (* At least one thread wants to run; transfer to it *) > > 6. changes to WaitProcess (Process.Wait): > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > ! CONST Delay = 10.0D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > ! SigPause(Delay,SIGCHLD); > END; > END WaitProcess; > > 7. install signal handler even if program is single-threaded: > > BEGIN > + (* we need to call set up the signal handler for other reasons than > + just thread switching now *) > + setup_sigvtalrm (switch_thread); > END ThreadPosix. > > 8. modify signal handler in ThreadPosixC.c to catch SIGCHLD: > > sigemptyset(&ThreadSwitchSignal); > sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); > + sigaddset(&ThreadSwitchSignal, SIGCHLD); > > act.sa_handler = handler; > act.sa_flags = SA_RESTART; > sigemptyset(&(act.sa_mask)); > if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); > + if (sigaction (SIGCHLD, &act, NULL)) abort(); > > I'll send the complete diff in a separate message for those who want > to study it more closely. > > I propose the above changes for inclusion in the current CM3 repository. > > Mika > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 12 23:08:42 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 22:08:42 +0000 Subject: [M3devel] pthread_atfork In-Reply-To: <20110212202550.542D41A2078@async.async.caltech.edu> References: <20110212030525.822411A2078@async.async.caltech.edu>, , <20110212165930.DC32C1A2078@async.async.caltech.edu>, <20110212205527.e6xetojo0ssgocg4@mail.elegosoft.com>, <20110212202550.542D41A2078@async.async.caltech.edu> Message-ID: "fork() and do more work" is portable, to Posix, which is many systems. It is easily done in Modula-3 by calling fork(). Given a choice between fixing fork() and do more work, vs. changing cvsupd to not depend on it, and given what I know about fixing fork() and do more work, I chose fork() and do more work. Or just abandon cvsupd. cvsupd is far from the only user of this pattern, in the wider non-Modula-3 world. > But why would user threads work? We're skipping pthread_atfork for them too. It is quite possible that fork and do more work + user threads + FreeBSD < 6, doesn't work. It is quite possible that it works intermittently. Let me try to explain what pthread_atfork is for. Though the opengroup spec does discuss it in detail and is clear (Bing for "opengroup pthread_atfork"; the first hit is it). In a multithreaded system, arbitrary mutexes are held at arbitrary times. Aribrary libraries spawn arbitrary worker threads, and use mutexes therein. As well, such programs might fork and do more work. There is not necessarily any central arbitrator among worker threads and the thread that calls fork(). That is, when you fork, you might be holding multiple mutexes. There isn't a way for the forking thread to request all other threads to enter some quiescent state in which no mutexes are held. If you fork and do more work, the new process has only one thread, the thread that called fork(). But any mutex ownership at the time of fork() is inherited. As a result, attemps to use mutexes in the new child process will have a high propensity to hang/deadlock. Therefore...go back..and do introduce an arbitrator. The implementation of fork() is therefore integrated with the implementation of pthreads. pthread_atfork establishes a set of callbacks. 3 sets actually. fork() calls them at the appropriate times -- before fork in the parent, and after fork in both the parent and the child. What the callbacks are meant to do is acquire all locks in the parent before the fork, and release them all in the parent and child after the work. And possibly reinitialize globals in the child after the fork. In the Modula-3 user thread system, threads exist by virtue of having memory allocated for their stack, small memory allocated for their processor context/registers/etc., and by being on a global linked list. Scheduling is triggered by an occasional timer. Historically, upon fork(), all that memory and the global linked list were unchanged. Therefore the child process would continue to have all the same threads as the parent. But compare this to pthreads. Solaris has an option to be either way, but in everything else I've seen -- OpenBSD, FreeBSD, NetBSD, Darwin, Linux -- and I think Cygwin, HP-UX, Irix, AIX, and probably the Posix spec -- fork() is specified as creating a child process with only one thread. Solaris has fork1() and forkall() and the behavior of plain fork() has varied from release to release. So, again, upon fork(), historical Modula-3 user threads retain all threads. If a thread held a mutex at fork() time, that's ok, it will continue running in the child and eventually release the mutex. No hang. If fork() is followed shortly by exec, well, then indeed, even with user threads, alll threads are gone, globals get reinitialized (to zero), there is a new thread. I guess you just have to be a bit careful what you do between fork() and exec(). Notice that things like Process.Create() suspend scheduling and probably after that avoid allocating any memory and possibly avoid touching garbage collected memory and probably avoid taking any locks. That is, upon suspending scheduling, they don't depend on any mutex to need to be released by any other thread. The various threads/mutexes can be in an arbitrary state. In the child they will go away shortly (upon exec). In the parent, scheduling will shortly resume. Does some of this make sense? > It is quite possible that fork and do more work + user threads + FreeBSD < 6, doesn't work. Historically it did work -- child process retained all the threads. These days I expect..I expect you still get that same behavior. We could fix it, I believe, by never calling fork() directly. Have a wrapper Process.Fork(). On most systems, it would just call fork. But on OpenVMS and FreeBSD < 6...there is also the Register function. On most systems, it'd just call pthread_atfork. On OpenVMS and FreeBSD < 6, it should maintain the lists itself. And then Process.Fork on those systems, make the call outs. Indeed, the OpenVMS/FreeBSD<6 implementation could be the one and only implementation. If fork() was disallowed and Process.Fork() the only allowed replacement. Or, it could be tested that way, and then actually commited the other way. I very well might do this. - Jay > To: wagner at elegosoft.com > Date: Sat, 12 Feb 2011 12:25:50 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] pthread_atfork > > How do you do "fork-and-do-work" from Modula-3? > > All my programs that create new Unix processes do it with Process.Create, > which as far as I know does a fork-and-exec... > > I'm not sure what if anything is broken here, either. But then again I > didn't fully understand Jay's last mail on the subject. > > Mika > > Olaf Wagner writes: > >Well, I think actually the fork-and-do-work pattern has been > >the default for a long time with the M3 user threads implementation. > >It wasn't specified though, and I think no application except > >CVSup did use it. > > > >Jay is not responsible for any breakage here, he just tried to fix > >CVSup and establish a consistent behaviour after our change to > >system pthreads on most platforms. > > > >Olaf > > > >Quoting Mika Nystrom : > > > >> Hi Jay, > >> > >> I think "fork and do work" qualifies as a bug in the application, if > >> the application wants to be portable. Why can't it fork and exec instead? > >> If it wants to do more work it should just fork and exec another copy > >> of itself. > >> > >> fork() is also not part of Modula-3... Process.Create() is, however. > >> Don't tell me you broke that as part of "fixing" things for cvsupd... > >> > >> There is still one aspect of your comment I don't understand. What does > >> the call to pthread_atfork actually accomplish when running with user > >> threads? You say below that "don't expect pthreads to work on FreeBSD < > >> 6 [using the fork-and-do-work pattern]". Fair enough. But why would > >> user threads work? We're skipping pthread_atfork for them too. > >> > >> Do we not need pthread_atfork with user threads? What you're saying > >> suggests that FreeBSD >=3D 6 and all other platforms need pthread_atfork > >> with user threads. Is that really what you mean? > >> > >> At the moment I'm just concerned with getting user threads to work as > >> I know there are bugs with pthreads. > >> > >> I also think we should not waste time and effort making the "fork and do > >> work" pattern work unless it's really, really necessary (which I really > >> don't see why it would be). Fix the application instead of introducing > >> bugs in the libraries and language implementation! > >> > >> I've been trying for a long, long time to switch from PM3 to CM3 (almost > >> ten years, I think). These bugs from "enhancements" have stopped me > >> every time. Maybe things are finally working well enough... > >> > >> (some comments inline in your quoted text below) > >> > >> Mika > >> > >> > >> Jay K writes: > >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ > >>> Content-Type: text/plain; charset=3D"iso-8859-1" > >>> Content-Transfer-Encoding: quoted-printable > >>> > >>> > >>> I wrote that. > >>> > >>> As I recall=3D2C FreeBSD < 6 doesn't have pthread_atfork. > >>> > >>> I think that was based on reading the man pages. > >>> > >>> So=3D2C yes. What would you suggest? > >>> > >>> > >>> > >>> > >>> > >>> All this atfork stuff came about as part of getting cvsupd to work. > >>> > >>> With pthreads/kernel threads. It historically worked=3D2C with Modula-3 = > >use=3D > >>> r threads. > >>> > >>> Modula-3 user threads have this funny property that all threads survive = > >w=3D > >>> ork. > >> > >> you mean survive fork don't you? > >> > >>> > >>> With pthreads/kernel threads=3D2C fork gives you a new process with just= > > on=3D > >>> e thread. > >>> > >>> (Solaris has fork1 and forkall=3D2C and fork maybe used to be forkall= > >=3D2C b=3D > >>> ut now it is fork1). > >>> > >>> > >>> > >>> > >>> > >>> cvsupd is one of those slightly unusual programs that does "fork and then= > > d=3D > >>> o more work" > >>> > >>> as supposed to the more typical "fork and exec". > >>> > >>> > >>> > >>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o= > >n Fr=3D > >>> eeBSD < 6? > >>> > >>> At least not in a process that ever calls fork? Or does "fork and then mo= > >re=3D > >>> work"? > >>> > >>> > >>> > >>> > >>> > >>> In reality=3D2C i think a lot of libraries have trouble with "fork and th= > >en d=3D > >>> o more work"=3D2C but > >>> > >>> I don't know. I recall vague notices in the Apple/Darwin manpages not to = > >as=3D > >>> sume > >>> > >>> this works -- the obvious implication that there is plenty of use of pthr= > >ea=3D > >>> ds e.g. mutexes=3D2C > >>> > >>> but relatively little use of pthread_atfork. > >>> > >>> > >>> > >>> > >>> > >>> You could perhaps replace all uses of fork with a function that called th= > >e =3D > >>> fork handlers=3D2C > >>> > >>> then fork=3D2C then the other handlers. That is probably what I meant in = > >the =3D > >>> comment > >>> > >>> about this being easy to fix. > >>> > >>> You might though then run into the problem that callers of fork can't do = > >mu=3D > >>> ch > >>> > >>> before exec...oh=3D2C that's vfork I bet actually. > >> > >> I don't understand this paragraph either :( > >> > >>> > >>> > >>> > >>> > >>> > >>> - Jay > >>> > >>> > >>> > >>> > >>> > >>> > >>>> To: hosking at cs.purdue.edu > >>>> Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800 > >>>> From: mika at async.caltech.edu > >>>> CC: m3devel at elegosoft.com=3D3B jay.krell at cornell.edu > >>>> Subject: [M3devel] pthread_atfork > >>>> =3D20 > >>>> Tony=3D2C > >>>> =3D20 > >>>> What's RTProcessC.c doing? > >>>> =3D20 > >>>> I note the following cryptic comment: > >>>> =3D20 > >>>> /* NOTE: Even userthreads now depends > >>>> * on availability of pthreads. > >>>> * This can be fixed if need be. > >>>> */ > >>>> =3D20 > >>>> now further down we can read: > >>>> =3D20 > >>>> #if defined(_WIN32) \ > >>>> || defined(__vms) \ > >>>> || (defined(__FreeBSD__) && (__FreeBSD__ < 6)) > >>>> return 0=3D3B > >>>> #else > >>>> while (1) > >>>> { > >>>> int i =3D3D pthread_atfork(prepare=3D2C parent=3D2C child)=3D3B > >>>> if (i !=3D3D EAGAIN) > >>>> return i=3D3B > >>>> sleep(0)=3D3B > >>>> } > >>>> #endif > >>>> =3D20 > >>>> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=3D2C > >>>> RTProcess.RegisterForkHandlers does nothing? > >>>> =3D20 > >>>> Hmmm..... > >>>> =3D20 > >>>> Mika > >>> =09=09 =09 =09=09 =3D > >>> > >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_ > >>> Content-Type: text/html; charset=3D"iso-8859-1" > >>> Content-Transfer-Encoding: quoted-printable > >>> > >>> > >>> > >>> > >>> > >>> > >>> I wrote that.
> >>> As I recall=3D2C FreeBSD <=3D3B 6 doesn't have pthread_atfork.
> >>> I think that was based on reading the man pages.
> >>> So=3D2C yes. What would you suggest?
> >>>
> >>>
> >>> All this atfork stuff came about as part of getting cvsupd to work.
> >>>  =3D3B With pthreads/kernel threads. It historically worked=3D2C with= > > Modul=3D > >>> a-3 user threads.
> >>>  =3D3B Modula-3 user threads have this funny property that all thread= > >s su=3D > >>> rvive work.
> >>>  =3D3B With pthreads/kernel threads=3D2C fork gives you a new process= > > with =3D > >>> just one thread.
> >>>  =3D3B =3D3B (Solaris has fork1 and forkall=3D2C and fork maybe u= > >sed to b=3D > >>> e forkall=3D2C but now it is fork1).
> >>>
> >>>
> >>> cvsupd is one of those slightly unusual programs that does "fork and then= > > d=3D > >>> o more work"
> >>> as supposed to the more typical "fork and exec".
> >>>
> >>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o= > >n Fr=3D > >>> eeBSD <=3D3B 6?
> >>> At least not in a process that ever calls fork? Or does "fork and then mo= > >re=3D > >>> work"?
> >>>
> >>>
> >>> In reality=3D2C i think a lot of libraries have trouble with "fork and th= > >en d=3D > >>> o more work"=3D2C but
> >>> I don't know. I recall vague notices in the Apple/Darwin manpages not to = > >as=3D > >>> sume
> >>> this works -- the obvious implication that there is plenty of use of pthr= > >ea=3D > >>> ds e.g. mutexes=3D2C
> >>> but relatively little use of pthread_atfork.
> >>>
> >>>
> >>> You could perhaps replace all uses of fork with a function that called th= > >e =3D > >>> fork handlers=3D2C
> >>> then fork=3D2C then the other handlers. That is probably what I meant in = > >the =3D > >>> comment
> >>> about this being easy to fix.
> >>> You might though then run into the problem that callers of fork can't do = > >mu=3D > >>> ch
> >>> before exec...oh=3D2C that's vfork I bet actually.
> >>>
> >>>
> >>>  =3D3B- Jay






>=3D3B To: hosking at cs.purdue= > >.edu >>> r>>=3D3B Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800
>=3D3B From: mi= > >ka at async=3D > >>> .caltech.edu
>=3D3B CC: m3devel at elegosoft.com=3D3B jay.krell at cornell= > >.edu >>> r>>=3D3B Subject: [M3devel] pthread_atfork
>=3D3B
>=3D3B Ton= > >y=3D2C
=3D > >>> >=3D3B
>=3D3B What's RTProcessC.c doing?
>=3D3B
>=3D3B= > > I note th=3D > >>> e following cryptic comment:
>=3D3B
>=3D3B /* NOTE: Even usert= > >hreads=3D > >>> now depends
>=3D3B * on availability of pthreads.
>=3D3B * Th= > >is ca=3D > >>> n be fixed if need be.
>=3D3B */
>=3D3B
>=3D3B now furth= > >er down =3D > >>> we can read:
>=3D3B
>=3D3B #if defined(_WIN32) \
>=3D3B = > > |=3D > >>> | defined(__vms) \
>=3D3B || (defined(__FreeBSD__) &=3D3B= > >&=3D > >>> =3D3B (__FreeBSD__ <=3D3B 6))
>=3D3B return 0=3D3B
>=3D3B= > > #else
&g=3D > >>> t=3D3B while (1)
>=3D3B {
>=3D3B int i =3D3D pthr= > >ead_atfork=3D > >>> (prepare=3D2C parent=3D2C child)=3D3B
>=3D3B if (i !=3D3D EAGA= > >IN)
>=3D > >>> =3D3B return i=3D3B
>=3D3B sleep(0)=3D3B
>=3D3B = > > }
>=3D > >>> =3D3B #endif
>=3D3B
>=3D3B so on FreeBSD 5 (what I'm running o= > >n my "Fr=3D > >>> eeBSD4" system)=3D2C
>=3D3B RTProcess.RegisterForkHandlers does noth= > >ing? >>> r>>=3D3B
>=3D3B Hmmm.....
>=3D3B
>=3D3B Mika
> >>> =09 =09 =09=09 =3D > >>> > >>> =3D > >>> > >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_-- > >> > > > > > > > >--=20 > >Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa= > >ny > >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz= > >: Berlin > >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sun Feb 13 00:27:43 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sat, 12 Feb 2011 18:27:43 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <20110212015715.A43471A2078@async.async.caltech.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu> Message-ID: <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> Mika, I have no idea what the issue is here. But I do note that AMD64_LINUX with pthreads used to run just fine. I ran a bunch or stress tests when first ported to that target and all seemed good. What is preventing you using pthreads on this target? -- Tony On Feb 11, 2011, at 8:57 PM, Mika Nystrom wrote: > I would like to find one configuration (any configuration!!) that lets > me run a multithreaded program under AMD64_LINUX. I don't care what > version of the compiler it is, whether the threads are user or pthreads, > how many things I have to hack. Does anyone know what I need to do? From jay.krell at cornell.edu Sun Feb 13 00:37:12 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 12 Feb 2011 23:37:12 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu>, <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> Message-ID: I assume he means "the thread tester should work"? But agreed, the vast vast vast vast majority of things are ok and have been ok for the vast vast vast majority of Modula-3's existance. Even in recent years. pthreads and user threads. Infinite loops with TRY in January 2011 would use infinite stack. Understood. Finite loops would merely use more than they should and reclaim upon function return. cvsupd didn't work for a while with pthreads. Because of its use of "fork and do more work" and its assumption that all threads survive fork. Relatively few programs do that and they were all broken by pthreads. The new thread tester probably needs more attention. - Jay > Subject: Re: [M3devel] threading on Windows? > From: hosking at cs.purdue.edu > Date: Sat, 12 Feb 2011 18:27:43 -0500 > CC: jay.krell at cornell.edu; m3devel at elegosoft.com > To: mika at async.caltech.edu > > Mika, > > I have no idea what the issue is here. But I do note that AMD64_LINUX with pthreads used to run just fine. I ran a bunch or stress tests when first ported to that target and all seemed good. What is preventing you using pthreads on this target? > > -- Tony > > On Feb 11, 2011, at 8:57 PM, Mika Nystrom wrote: > > > I would like to find one configuration (any configuration!!) that lets > > me run a multithreaded program under AMD64_LINUX. I don't care what > > version of the compiler it is, whether the threads are user or pthreads, > > how many things I have to hack. Does anyone know what I need to do? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 13 00:39:15 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 15:39:15 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu> <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> Message-ID: <20110212233915.A8EEE1A2078@async.async.caltech.edu> Tony, I have yet to find a single system on which my thread testing program actually runs properly under pthreads. I have however managed to get it to work everywhere with user threading. My last email to you on the issue said the following: read: segfault after a long time fork: OK alloc: deadlock creat: appears OK for a long time then hangs hard (can't get into debugger) lock: OK I think Jay has now fixed the "read" problem, which had nothing to do with threading but was a TRY-EXCEPT issue. The fact that the runtime deadlocks when NEWing objects I think is the most serious problem. Could you try it, please? If you run "threadtest -n 10 -tests all" you should see something like this (the following is AMD64_FREEBSD with user threading---the numbers denote how much threads are lagging, in seconds). The program is utterly standard Modula-3. /big/home/mika/cm3-writable/cm3/m3-libs/m3core/tests/thread/AMD64_FREEBSD/threadtest -n 10 -tests all Writing file...done Creating read threads...done Creating nxread threads...done Creating tryexcept threads...done Creating fork threads...done Creating alloc threads...done Creating creat threads...done Creating lock threads...done running...printing oldest/median age/newest ..........laziest thread is 56/13/0 (tests: read 1/1/0 nxread 3/2/2 tryexcept 12/8/3 fork 13/13/13 alloc 17/15/13 creat 38/17/17 lock 56/18/17) ..........laziest thread is 57/13/0 (tests: read 2/1/0 nxread 3/2/2 tryexcept 12/8/3 fork 13/13/13 alloc 17/15/13 creat 36/17/17 lock 57/36/17) ..........laziest thread is 40/13/0 (tests: read 1/1/0 nxread 3/2/1 tryexcept 12/8/3 fork 13/13/13 alloc 17/15/13 creat 37/17/17 lock 40/37/17) ..........laziest thread is 34/14/0 (tests: read 1/0/0 nxread 3/2/1 tryexcept 13/9/3 fork 14/14/14 alloc 17/15/14 creat 31/17/17 lock 34/31/17) ..........laziest thread is 56/13/0 (tests: read 2/1/0 nxread 3/2/2 tryexcept 12/8/3 fork 13/13/13 alloc 16/15/13 creat 37/16/16 lock 56/18/16) ..........laziest thread is 37/14/0 (tests: read 1/0/0 nxread 3/2/1 tryexcept 13/8/3 fork 14/14/14 alloc 18/15/14 creat 37/19/19 lock 37/19/19) ..........laziest thread is 34/15/0 (tests: read 2/1/0 nxread 3/3/2 tryexcept 14/9/3 fork 15/15/15 alloc 16/16/15 creat 34/17/17 lock 34/17/17) ..........laziest thread is 34/14/0 (tests: read 1/1/0 nxread 3/2/1 tryexcept 13/8/3 fork 14/14/14 alloc 16/15/14 creat 34/16/16 lock 34/16/16) ..........laziest thread is 35/14/0 (tests: read 2/1/0 nxread 3/2/2 tryexcept 13/9/3 fork 14/14/14 alloc 17/16/14 creat 35/17/17 lock 35/17/17) ..........laziest thread is 37/14/0 (tests: read 2/1/0 nxread 3/2/2 tryexcept 13/8/3 fork 14/14/14 alloc 16/15/14 creat 37/17/17 lock 37/17/17) All tests complete. Congratulations. Mika Tony Hosking writes: >Mika, > >I have no idea what the issue is here. But I do note that AMD64_LINUX = >with pthreads used to run just fine. I ran a bunch or stress tests when = >first ported to that target and all seemed good. What is preventing you = >using pthreads on this target? > >-- Tony > >On Feb 11, 2011, at 8:57 PM, Mika Nystrom wrote: > >> I would like to find one configuration (any configuration!!) that lets >> me run a multithreaded program under AMD64_LINUX. I don't care what >> version of the compiler it is, whether the threads are user or = >pthreads, >> how many things I have to hack. Does anyone know what I need to do? From mika at async.caltech.edu Sun Feb 13 01:33:57 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 16:33:57 -0800 Subject: [M3devel] threading on Windows? In-Reply-To: References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu>, <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> Message-ID: <20110213003358.00D2A1A2078@async.async.caltech.edu> Jay K writes: >--_879a340b-b2bb-415e-936c-be8d5cdd855c_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >I assume he means "the thread tester should work"? >But agreed=2C the vast vast vast vast majority of things are ok and have be= >en ok for the vast vast vast majority of Modula-3's existance. > Even in recent years. pthreads and user threads. > Infinite loops with TRY in January 2011 would use infinite stack. Underst= >ood. > Finite loops would merely use more than they should and reclaim upon func= >tion return. >cvsupd didn't work for a while with pthreads. Because of its use of "fork a= >nd do more work" and its assumption >that all threads survive fork. Relatively few programs do that and they wer= >e all broken by pthreads. >=20 >The new thread tester probably needs more attention. >=20 > - Jay My thread tester wasn't developed "ex nihilo". It's based on analyzing the failure modes of a larger application that I am prevented from distributing the source code for, both for reasons of confidentiality and practicality. I have purposely written the tester to be "obviously correct" and I would like nothing more than for someone to show me where I have made a mistake and that pthreads are working as they should. For example, here is the "alloc" thread: PROCEDURE AApply(cl : Closure) : REFANY = BEGIN Thread.Pause(InitPause); LOOP VAR arr := NEW(REF ARRAY OF INTEGER, 1025); BEGIN FOR i := FIRST(arr^)+1 TO LAST(arr^) DO arr[i] := arr[i] - arr[i-1] END; times1[cl.id] := FLOOR(Time.Now()) END END END AApply; Is there anything here that is wrong or bad? When I tried pthreads just now on AMD64_LINUX, running two instances of AApply at the same time caused the runtime to deadlock. [BTW, I have never seen a problem with user threads except for building issues and the TRY-EXCEPT thing you just fixed.] Unfortunately, since just about every Modula-3 program allocates memory, my analysis is that because it appears you can deadlock the runtime by allocating memory, nothing is *really* OK with pthreads. Please tell me I'm wrong! Mika P.S. backtraces of deadlock: Thread 3 (Thread 0x40862950 (LWP 19974)): #0 0x00002b9d49f730e1 in nanosleep () from /lib/libpthread.so.0 #1 0x000000000045806d in ThreadPThread__Nanosleep (req=Cannot access memory at address 0x140861b08 ) at ../src/thread/PTHREAD/ThreadPThreadC.c:500 #2 0x0000000000454294 in ThreadPThread__CommonSleep () at ../src/thread/PTHREAD/ThreadPThread.m3:551 #3 0x0000000000456438 in ThreadPThread__StopWorld () at ../src/thread/PTHREAD/ThreadPThread.m3:1086 #4 0x0000000000455787 in RTThread__SuspendOthers () at ../src/thread/PTHREAD/ThreadPThread.m3:812 #5 0x000000000043970c in RTCollector__CollectSomeInStateZero () at ../src/runtime/common/RTCollector.m3:746 #6 0x00000000004396b8 in RTCollector__CollectSome () at ../src/runtime/common/RTCollector.m3:720 #7 0x00000000004390b9 in RTHeapRep__CollectEnough () at ../src/runtime/common/RTCollector.m3:654 #8 0x0000000000436111 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=Cannot access memory at address 0x140861dc8 ) at ../src/runtime/common/RTAllocator.m3:363 #9 0x000000000043596f in RTAllocator__GetOpenArray (M3_Eic7CK_def=Cannot access memory at address 0x140861eb8 ) at ../src/runtime/common/RTAllocator.m3:292 #10 0x0000000000434ab1 in RTHooks__AllocateOpenArray (M3_AJWxb1_defn=Cannot access memory at address 0x140861f38 ) at ../src/runtime/common/RTAllocator.m3:141 #11 0x0000000000407030 in Main__AApply (M3_AP7a1g_cl=Cannot access memory at address 0x140861f88 ) at ../src/Main.m3:234 #12 0x0000000000453b3f in ThreadPThread__RunThread (M3_DMxDjQ_me=Cannot access memory at address 0x140862028 ) at ../src/thread/PTHREAD/ThreadPThread.m3:475 #13 0x00000000004537d3 in ThreadPThread__ThreadBase (M3_AJWxb1_param=Cannot access memory at address 0x1408620d8 ) at ../src/thread/PTHREAD/ThreadPThread.m3:447 #14 0x00002b9d49f6bfc7 in start_thread () from /lib/libpthread.so.0 #15 0x00002b9d4b3bd64d in clone () from /lib/libc.so.6 #16 0x0000000000000000 in ?? () Thread 2 (Thread 0x40661950 (LWP 19973)): #0 0x00002b9d49f72384 in __lll_lock_wait () from /lib/libpthread.so.0 #1 0x00002b9d49f6dbf0 in _L_lock_102 () from /lib/libpthread.so.0 #2 0x00002b9d49f6d4fe in pthread_mutex_lock () from /lib/libpthread.so.0 #3 0x000000000045815e in ThreadPThread__pthread_mutex_lock (m=Cannot access memory at address 0x140660d68 ) at ../src/thread/PTHREAD/ThreadPThreadC.c:557 #4 0x00000000004575af in RTOS__LockHeap () at ../src/thread/PTHREAD/ThreadPThread.m3:1434 #5 0x0000000000436107 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=Cannot access memory at address 0x140660dc8 ) at ../src/runtime/common/RTAllocator.m3:361 #6 0x000000000043596f in RTAllocator__GetOpenArray (M3_Eic7CK_def=Cannot access memory at address 0x140660eb8 ) at ../src/runtime/common/RTAllocator.m3:292 #7 0x0000000000434ab1 in RTHooks__AllocateOpenArray (M3_AJWxb1_defn=Cannot access memory at address 0x140660f38 ) at ../src/runtime/common/RTAllocator.m3:141 #8 0x0000000000407030 in Main__AApply (M3_AP7a1g_cl=Cannot access memory at address 0x140660f88 ) at ../src/Main.m3:234 #9 0x0000000000453b3f in ThreadPThread__RunThread (M3_DMxDjQ_me=Cannot access memory at address 0x140661028 ) at ../src/thread/PTHREAD/ThreadPThread.m3:475 #10 0x00000000004537d3 in ThreadPThread__ThreadBase (M3_AJWxb1_param=Cannot access memory at address 0x1406610d8 ) at ../src/thread/PTHREAD/ThreadPThread.m3:447 #11 0x00002b9d49f6bfc7 in start_thread () from /lib/libpthread.so.0 #12 0x00002b9d4b3bd64d in clone () from /lib/libc.so.6 #13 0x0000000000000000 in ?? () Thread 1 (Thread 0x2b9d4d32a1a0 (LWP 19965)): #0 0x00002b9d4b3202a2 in sigsuspend () from /lib/libc.so.6 #1 0x0000000000457ba9 in ThreadPThread__sigsuspend () at ../src/thread/PTHREAD/ThreadPThreadC.c:141 #2 0x0000000000456a0d in ThreadPThread__SignalHandler (M3_DLS2Hj_sig=Cannot access memory at address 0x80001dde057c ) at ../src/thread/PTHREAD/ThreadPThread.m3:1205 #3 #4 0x00002b9d49f6ffad in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #5 0x00000000004580b3 in ThreadPThread__pthread_cond_timedwait (cond=Cannot access memory at address 0x80001dde0a88 ) at ../src/thread/PTHREAD/ThreadPThreadC.c:515 #6 0x00000000004544cf in ThreadPThread__XPause (M3_DMxDjQ_self=Cannot access memory at address 0x80001dde0ab8 ) at ../src/thread/PTHREAD/ThreadPThread.m3:580 #7 0x00000000004545b2 in Thread__Pause (M3_CtKayy_n=Cannot access memory at address 0x80001dde0b08 ) at ../src/thread/PTHREAD/ThreadPThread.m3:595 #8 0x00000000004097ee in Main_M3 (M3_AcxOUs_mode=Cannot access memory at address 0x80001dde0bc8 ) at ../src/Main.m3:449 #9 0x0000000000445799 in RTLinker__RunMainBody (M3_DjPxE3_m=Cannot access memory at address 0x80001dde0f58 ) at ../src/runtime/common/RTLinker.m3:406 #10 0x00000000004448f2 in RTLinker__AddUnitI (M3_DjPxE3_m=Cannot access memory at address 0x80001dde0ff8 ) at ../src/runtime/common/RTLinker.m3:113 #11 0x0000000000444990 in RTLinker__AddUnit (M3_DjPxE5_b=Cannot access memory at address 0x80001dde1018 ) at ../src/runtime/common/RTLinker.m3:122 #12 0x000000000040652c in main (argc=Cannot access memory at address 0x80001dde1068 ) at _m3main.mc:4 From mika at async.caltech.edu Sun Feb 13 07:29:55 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 22:29:55 -0800 Subject: [M3devel] Performance issues with Process.Wait under userthreads In-Reply-To: References: <20110212204302.19C781A2078@async.async.caltech.edu> Message-ID: <20110213062955.4F84B1A2078@async.async.caltech.edu> Jay K writes: >--_5ec25af1-29a7-4f8a-b7ad-e61068941159_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >I don't believe there is a good solution here=2C other than using pthreads. >Is there? Maybe tune the wait down smaller? >I understand it stinks either way. Well as I keep saying, I'm looking for a reliable runtime. I really need this---and soon. I can't keep futzing around with PM3 forever. I'd hope pthreads is it but for now the only thing that passes my tests is user threading. We (group at Caltech) determined many years ago that tuning the wait does nothing, unless you set it to zero (busy waiting). Possibly this has to do with the time quantum of the user threads scheduler, I don't know. >=20 >=20 >I don't understand the SIGCHLD stuff=2C at a quick glance. The idea is this. We have SIGVTALRM coming in already to do threadswitching. So I am enabling SIGCHLD and then taking SIGCHLD the way SIGVTALRM is taken, that is, in the scheduler. The other change is that a paused thread can wait for a condition such as "either after X delay OR after Y signal" and then we allow the paused thread to become runnable on the Y signal rather than after the X delay. >Nor your changes. >However you might be the only user of user threads. >Is there any downside to your change? > (Other than more code=2C ok.) >Any loss of performance in any situation? >Any loss of portability? I am not sure. Is SIGCHLD completely standard? I am worried about one thing. The thread switcher turns signals back on as its first action. If a large number of children exit at the same time, I fear that we get a very deeply nested signal stack, which doesn't sound good. Perhaps signals should not be turned back on immediately. Or SIGCHLD should be set to SIG_IGN and then re-enabled? (Simulating old-fashioned "unreliable" SysV signals!) I'm not sure. In any case I was just trying to parallelize the compiler and ran into some very odd things related to having lots of child processes running around... The SIGCHLD change speeds up the CM3 compiler by about a factor of 3 when using user threads. Mika >=20 >=20 > - Jay >=20 >> To: m3devel at elegosoft.com >> Date: Sat=2C 12 Feb 2011 12:43:02 -0800 >> From: mika at async.caltech.edu >> Subject: [M3devel] Performance issues with Process.Wait under userthreads >>=20 >> Hi again m3devel (especially Tony)=2C=20 >>=20 >> I have finally taken a bite of a problem that has been annoying me >> for a very=2C very long time. >>=20 >> Under user threads=2C the code for Process.Wait is as follows (see=20 >> ThreadPosix.m3): >>=20 >> PROCEDURE WaitProcess (pid: int=3B VAR status: int): int =3D >> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >> CONST Delay =3D 0.1D0=3B >> BEGIN >> LOOP >> WITH r =3D Uexec.waitpid(pid=2C ADR(status)=2C Uexec.WNOHANG) DO >> IF r # 0 THEN RETURN r END=3B >> END=3B >> Pause(Delay)=3B >> END=3B >> END WaitProcess=3B >>=20 >> It inserts a 0.1 second delay after each failed waitpid. This is extremel= >y >> annoying for programs that start a long sequence of child processes and >> wait for them in sequence. Namely=2C the compiler itself. As a result >> the cm3 compiler (and PM3's m3build) are normally very very slow when >> using user threads. For about the last ten years=2C I've had a hacked >> up m3build (for my PM3 installation) that skips the Pause and busy-waits >> instead.=20 >>=20 >> Note there is another problem here. Since the Modula-3 runtime ignores >> SIGCHLD=2C no zombie processes are created since the Unix system automati= >cally >> reaps the child processes. I can see this would be a problem since PIDs >> are eventually reused and ... couldn't Uexec.waitpid wind up referring to >> the wrong process?? >>=20 >> I will further note that the comment in Process.i3 reads as follows: >>=20 >> PROCEDURE Wait(p: T): ExitCode=3B >>=20 >> Wait until the process with handle p terminates=2C then free the operatin= >g system resources associated with the process and return an exit code indi= >cating the reason for its termination. It is a checked runtime error to cal= >l Wait twice on the same process handle.=20 >>=20 >> I am going to take this as fair warning that Process.Create *may* use >> resources that are not going to be released until Process.Wait has been >> called. >>=20 >> I have modified (in my local copy of CM3) the system as follows. >> I have come up with a semi-general mechanism for immediately unblocking >> a thread on the receipt of a unix signal.=20 >>=20 >> 1. The system relies on changing >> ThreadPosix.XPause such that if a signal is allowed to wake up a threads= >=2C >> that fact is recorded in a new field in the thread's descriptor >> record (of type ThreadPosix.T).=20 >>=20 >> 2. On receipt of a waited-for unix signal=2C a mask is set and control >> is passed to the thread scheduler which maintains the non-zero mask for >> exactly one iteration through the thread ring. >>=20 >> 3. If a thread is paused and waiting for EITHER a signal or some time=2C >> the thread is released for running and the thread's waiting state is=20 >> cleared. >>=20 >> The changes are more or less as follows: >>=20 >> 1. I have added a new field of type "int" to ThreadPosix.T: >>=20 >> (* if state =3D pausing=2C the time at which we can restart *) >> waitingForTime: Time.T=3B >>=20 >> + (* if state =3D pausing=2C the signal that truncates the pause *) >> + waitingForSig: int :=3D -1=3B >> + >> (* true if we are waiting during an AlertWait or AlertJoin >> or AlertPause *) >> alertable: BOOLEAN :=3D FALSE=3B >>=20 >>=20 >> 2. Modifications to pause: >>=20 >> + PROCEDURE SigPause(n: LONGREAL=3B sig: int)=3D >> + <*FATAL Alerted*> >> + VAR until :=3D n + Time.Now ()=3B >> + BEGIN >> + XPause(until=2C FALSE=2C sig)=3B >> + END SigPause=3B >> + >> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >> VAR until :=3D n + Time.Now ()=3B >> BEGIN >> XPause(until=2C TRUE)=3B >> END AlertPause=3B >>=20 >> ! PROCEDURE XPause (READONLY until: Time.T=3B alertable :=3D FALSE=3B sig= >:int :=3D -1) >> ! RAISES {Alerted} =3D >> BEGIN >> INC (inCritical)=3B >> self.waitingForTime :=3D until=3B >> self.alertable :=3D alertable=3B >> + IF sig # -1 THEN >> + self.waitingForSig :=3D sig >> + END=3B >> ICannotRun (State.pausing)=3B >> DEC (inCritical)=3B >> InternalYield ()=3B >>=20 >> 3. The received-signals mask: >>=20 >> ! CONST MaxSigs =3D 64=3B >> ! TYPE Sig =3D [ 0..MaxSigs-1 ]=3B >> ! >> ! (* in order to listen to other signals=2C they have to be enabled in >> ! allow_sigvtalrm as well *) >> ! VAR (*CONST*) SIGCHLD :=3D ValueOfSIGCHLD()=3B >> ! >> ! gotSigs :=3D SET OF Sig { }=3B >> ! >>=20 >> ValueOfSIGCHLD() is a C function used to get the value of the SIGCHLD >> constant without guessing at it (in ThreadPosixC.c). >>=20 >> 4. changes to the signal handler: >>=20 >> ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >> BEGIN >> allow_sigvtalrm ()=3B >> ! >> ! INC(inCritical)=3B >> ! (* mark signal as being delivered *) >> ! IF sig >=3D 0 AND sig < MaxSigs THEN >> ! gotSigs :=3D gotSigs + SET OF Sig { sig } >> ! END=3B >> ! DEC(inCritical)=3B >> ! >> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >> ! InternalYield () >> ! END=3B >> END switch_thread=3B >>=20 >> Note that I don't know if INC/DEC(inCritical) does exactly the right >> thing here. >>=20 >> 5. changes to the scheduler: >>=20 >> a. thread wakeup >> IF t.alertable AND t.alertPending THEN >> CanRun (t)=3B >> EXIT=3B >> + >> + ELSIF t.waitingForSig IN gotSigs THEN >> + t.waitingForSig :=3D -1=3B >> + CanRun(t)=3B >> + EXIT=3B >>=20 >> ELSIF t.waitingForTime <=3D now THEN >> CanRun (t)=3B >> EXIT=3B >> ! >>=20 >> b. clearing the mask >>=20 >> END=3B >> END=3B >>=20 >> + gotSigs :=3D SET OF Sig {}=3B >> + >> IF t.state =3D State.alive AND (scanned OR NOT someBlocking) THEN >> IF perfOn THEN PerfRunning (t.id)=3B END=3B >> (* At least one thread wants to run=3B transfer to it *) >>=20 >> 6. changes to WaitProcess (Process.Wait): >>=20 >> PROCEDURE WaitProcess (pid: int=3B VAR status: int): int =3D >> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >> ! CONST Delay =3D 10.0D0=3B >> BEGIN >> LOOP >> WITH r =3D Uexec.waitpid(pid=2C ADR(status)=2C Uexec.WNOHANG) DO >> IF r # 0 THEN RETURN r END=3B >> END=3B >> ! SigPause(Delay=2CSIGCHLD)=3B >> END=3B >> END WaitProcess=3B >>=20 >> 7. install signal handler even if program is single-threaded: >>=20 >> BEGIN >> + (* we need to call set up the signal handler for other reasons than >> + just thread switching now *) >> + setup_sigvtalrm (switch_thread)=3B >> END ThreadPosix. >>=20 >> 8. modify signal handler in ThreadPosixC.c to catch SIGCHLD: >>=20 >> sigemptyset(&ThreadSwitchSignal)=3B >> sigaddset(&ThreadSwitchSignal=2C SIG_TIMESLICE)=3B >> + sigaddset(&ThreadSwitchSignal=2C SIGCHLD)=3B >>=20 >> act.sa_handler =3D handler=3B >> act.sa_flags =3D SA_RESTART=3B >> sigemptyset(&(act.sa_mask))=3B >> if (sigaction (SIG_TIMESLICE=2C &act=2C NULL)) abort()=3B >> + if (sigaction (SIGCHLD=2C &act=2C NULL)) abort()=3B >>=20 >> I'll send the complete diff in a separate message for those who want >> to study it more closely. >>=20 >> I propose the above changes for inclusion in the current CM3 repository. >>=20 >> Mika >>=20 > = > >--_5ec25af1-29a7-4f8a-b7ad-e61068941159_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >I don't believe there is a good solution here=2C other than using pthreads.= >
>Is there? Maybe tune the wait down smaller?
>I understand it stinks either way.
> =3B
> =3B
>I don't =3Bunderstand the SIGCHLD stuff=2C at a quick glance.
>Nor your changes.
>However you might be the only user of user threads.
>Is there any downside to your change?
> =3B(Other than more code=2C ok.)
>Any loss of performance in any situation?
>Any loss of portability?
> =3B
> =3B
> =3B- Jay
 =3B
>>=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 12 Feb 2011 12:43:0= >2 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > Performance issues with Process.Wait under userthreads
>=3B
>= >=3B Hi again m3devel (especially Tony)=2C
>=3B
>=3B I have fina= >lly taken a bite of a problem that has been annoying me
>=3B for a ver= >y=2C very long time.
>=3B
>=3B Under user threads=2C the code fo= >r Process.Wait is as follows (see
>=3B ThreadPosix.m3):
>=3B >>=3B PROCEDURE WaitProcess (pid: int=3B VAR status: int): int =3D
>= >=3B (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *)
>=3B C= >ONST Delay =3D 0.1D0=3B
>=3B BEGIN
>=3B LOOP
>=3B WITH r =3D= > Uexec.waitpid(pid=2C ADR(status)=2C Uexec.WNOHANG) DO
>=3B IF r # 0 T= >HEN RETURN r END=3B
>=3B END=3B
>=3B Pause(Delay)=3B
>=3B EN= >D=3B
>=3B END WaitProcess=3B
>=3B
>=3B It inserts a 0.1 sec= >ond delay after each failed waitpid. This is extremely
>=3B annoying f= >or programs that start a long sequence of child processes and
>=3B wai= >t for them in sequence. Namely=2C the compiler itself. As a result
>= >=3B the cm3 compiler (and PM3's m3build) are normally very very slow whenR>>=3B using user threads. For about the last ten years=2C I've had a hac= >ked
>=3B up m3build (for my PM3 installation) that skips the Pause and= > busy-waits
>=3B instead.
>=3B
>=3B Note there is another = >problem here. Since the Modula-3 runtime ignores
>=3B SIGCHLD=2C no zo= >mbie processes are created since the Unix system automatically
>=3B re= >aps the child processes. I can see this would be a problem since PIDs
&g= >t=3B are eventually reused and ... couldn't Uexec.waitpid wind up referring= > to
>=3B the wrong process??
>=3B
>=3B I will further note = >that the comment in Process.i3 reads as follows:
>=3B
>=3B PROCE= >DURE Wait(p: T): ExitCode=3B
>=3B
>=3B Wait until the process wi= >th handle p terminates=2C then free the operating system resources associat= >ed with the process and return an exit code indicating the reason for its t= >ermination. It is a checked runtime error to call Wait twice on the same pr= >ocess handle.
>=3B
>=3B I am going to take this as fair warning= > that Process.Create *may* use
>=3B resources that are not going to be= > released until Process.Wait has been
>=3B called.
>=3B
>= >=3B I have modified (in my local copy of CM3) the system as follows.
>= >=3B I have come up with a semi-general mechanism for immediately unblocking= >
>=3B a thread on the receipt of a unix signal.
>=3B
>=3B = >1. The system relies on changing
>=3B ThreadPosix.XPause such that if = >a signal is allowed to wake up a threads=2C
>=3B that fact is recorded= > in a new field in the thread's descriptor
>=3B record (of type Thread= >Posix.T).
>=3B
>=3B 2. On receipt of a waited-for unix signal= >=2C a mask is set and control
>=3B is passed to the thread scheduler w= >hich maintains the non-zero mask for
>=3B exactly one iteration throug= >h the thread ring.
>=3B
>=3B 3. If a thread is paused and waitin= >g for EITHER a signal or some time=2C
>=3B the thread is released for = >running and the thread's waiting state is
>=3B cleared.
>=3B >>=3B The changes are more or less as follows:
>=3B
>=3B 1. I = >have added a new field of type "int" to ThreadPosix.T:
>=3B
>=3B= > (* if state =3D pausing=2C the time at which we can restart *)
>=3B w= >aitingForTime: Time.T=3B
>=3B
>=3B + (* if state =3D pausing=2C = >the signal that truncates the pause *)
>=3B + waitingForSig: int :=3D = >-1=3B
>=3B +
>=3B (* true if we are waiting during an AlertWait o= >r AlertJoin
>=3B or AlertPause *)
>=3B alertable: BOOLEAN :=3D FA= >LSE=3B
>=3B
>=3B
>=3B 2. Modifications to pause:
>=3B= >
>=3B + PROCEDURE SigPause(n: LONGREAL=3B sig: int)=3D
>=3B + &l= >t=3B*FATAL Alerted*>=3B
>=3B + VAR until :=3D n + Time.Now ()=3B
= >>=3B + BEGIN
>=3B + XPause(until=2C FALSE=2C sig)=3B
>=3B + END= > SigPause=3B
>=3B +
>=3B PROCEDURE AlertPause(n: LONGREAL) RAISES= > {Alerted}=3D
>=3B VAR until :=3D n + Time.Now ()=3B
>=3B BEGINR>>=3B XPause(until=2C TRUE)=3B
>=3B END AlertPause=3B
>=3B >>=3B ! PROCEDURE XPause (READONLY until: Time.T=3B alertable :=3D FALSE= >=3B sig:int :=3D -1)
>=3B ! RAISES {Alerted} =3D
>=3B BEGIN
&g= >t=3B INC (inCritical)=3B
>=3B self.waitingForTime :=3D until=3B
>= >=3B self.alertable :=3D alertable=3B
>=3B + IF sig # -1 THEN
>=3B= > + self.waitingForSig :=3D sig
>=3B + END=3B
>=3B ICannotRun (Sta= >te.pausing)=3B
>=3B DEC (inCritical)=3B
>=3B InternalYield ()=3B<= >BR>>=3B
>=3B 3. The received-signals mask:
>=3B
>=3B ! C= >ONST MaxSigs =3D 64=3B
>=3B ! TYPE Sig =3D [ 0..MaxSigs-1 ]=3B
>= >=3B !
>=3B ! (* in order to listen to other signals=2C they have to be= > enabled in
>=3B ! allow_sigvtalrm as well *)
>=3B ! VAR (*CONST*= >) SIGCHLD :=3D ValueOfSIGCHLD()=3B
>=3B !
>=3B ! gotSigs :=3D SET= > OF Sig { }=3B
>=3B !
>=3B
>=3B ValueOfSIGCHLD() is a C fun= >ction used to get the value of the SIGCHLD
>=3B constant without guess= >ing at it (in ThreadPosixC.c).
>=3B
>=3B 4. changes to the signa= >l handler:
>=3B
>=3B ! PROCEDURE switch_thread (sig: int) RAISES= > {Alerted} =3D
>=3B BEGIN
>=3B allow_sigvtalrm ()=3B
>=3B !<= >BR>>=3B ! INC(inCritical)=3B
>=3B ! (* mark signal as being delivere= >d *)
>=3B ! IF sig >=3B=3D 0 AND sig <=3B MaxSigs THEN
>=3B != > gotSigs :=3D gotSigs + SET OF Sig { sig }
>=3B ! END=3B
>=3B ! D= >EC(inCritical)=3B
>=3B !
>=3B ! IF inCritical =3D 0 AND heapState= >.inCritical =3D 0 THEN
>=3B ! InternalYield ()
>=3B ! END=3B
&= >gt=3B END switch_thread=3B
>=3B
>=3B Note that I don't know if I= >NC/DEC(inCritical) does exactly the right
>=3B thing here.
>=3B <= >BR>>=3B 5. changes to the scheduler:
>=3B
>=3B a. thread wakeu= >p
>=3B IF t.alertable AND t.alertPending THEN
>=3B CanRun (t)=3B<= >BR>>=3B EXIT=3B
>=3B +
>=3B + ELSIF t.waitingForSig IN gotSigs = >THEN
>=3B + t.waitingForSig :=3D -1=3B
>=3B + CanRun(t)=3B
>= >=3B + EXIT=3B
>=3B
>=3B ELSIF t.waitingForTime <=3B=3D now THE= >N
>=3B CanRun (t)=3B
>=3B EXIT=3B
>=3B !
>=3B
>= >=3B b. clearing the mask
>=3B
>=3B END=3B
>=3B END=3B
&g= >t=3B
>=3B + gotSigs :=3D SET OF Sig {}=3B
>=3B +
>=3B IF t.= >state =3D State.alive AND (scanned OR NOT someBlocking) THEN
>=3B IF p= >erfOn THEN PerfRunning (t.id)=3B END=3B
>=3B (* At least one thread wa= >nts to run=3B transfer to it *)
>=3B
>=3B 6. changes to WaitProc= >ess (Process.Wait):
>=3B
>=3B PROCEDURE WaitProcess (pid: int=3B= > VAR status: int): int =3D
>=3B (* ThreadPThread.m3 and ThreadPosix.m3= > are very similar. *)
>=3B ! CONST Delay =3D 10.0D0=3B
>=3B BEGIN= >
>=3B LOOP
>=3B WITH r =3D Uexec.waitpid(pid=2C ADR(status)=2C Ue= >xec.WNOHANG) DO
>=3B IF r # 0 THEN RETURN r END=3B
>=3B END=3B>>=3B ! SigPause(Delay=2CSIGCHLD)=3B
>=3B END=3B
>=3B END WaitP= >rocess=3B
>=3B
>=3B 7. install signal handler even if program is= > single-threaded:
>=3B
>=3B BEGIN
>=3B + (* we need to call= > set up the signal handler for other reasons than
>=3B + just thread s= >witching now *)
>=3B + setup_sigvtalrm (switch_thread)=3B
>=3B EN= >D ThreadPosix.
>=3B
>=3B 8. modify signal handler in ThreadPosix= >C.c to catch SIGCHLD:
>=3B
>=3B sigemptyset(&=3BThreadSwitchS= >ignal)=3B
>=3B sigaddset(&=3BThreadSwitchSignal=2C SIG_TIMESLICE)= >=3B
>=3B + sigaddset(&=3BThreadSwitchSignal=2C SIGCHLD)=3B
>= >=3B
>=3B act.sa_handler =3D handler=3B
>=3B act.sa_flags =3D SA_= >RESTART=3B
>=3B sigemptyset(&=3B(act.sa_mask))=3B
>=3B if (sig= >action (SIG_TIMESLICE=2C &=3Bact=2C NULL)) abort()=3B
>=3B + if (si= >gaction (SIGCHLD=2C &=3Bact=2C NULL)) abort()=3B
>=3B
>=3B I'= >ll send the complete diff in a separate message for those who want
>= >=3B to study it more closely.
>=3B
>=3B I propose the above chan= >ges for inclusion in the current CM3 repository.
>=3B
>=3B Mika<= >BR>>=3B
>= > >--_5ec25af1-29a7-4f8a-b7ad-e61068941159_-- From mika at async.caltech.edu Sun Feb 13 08:04:14 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 12 Feb 2011 23:04:14 -0800 Subject: [M3devel] More on threading Message-ID: <20110213070414.DD0341A2078@async.async.caltech.edu> Hi again m3devel, Well I thought I would relate another story having to do with threading and CM3. It's half-baked at the moment, but it only reflects a few hours' work on my part, sorry... I'm about to give up now because I don't know if the problem is in my code, or in threading, or what... maybe it is the atfork problem Jay talked about? Programmers spend a lot of time waiting for their compilers to run, so it would be nice if the compilation process were faster. After perusing the source code for the CM3 I realized that there is no feedback at all from the back-end and assembler steps to the rest of the compiler, and I believe that with the GCC back-end the vast majority of the runtime of the compiler is spent in these stages. So.... how hard could this be to parallelize? I decided to try making the two stages (cm3cg1 and as) into "promises" rather than running them on the spot. The QMachine adds a promise to call the back-end to a RefSeq.T called "promises", to which the Builder can also add promises. Then, after making all the promises, collect them and run them ("force them" a true Schemer would say) in parallel. Of course something goes wrong somewhere. With 10 threads I am able to run 31 seconds of CPU time in 19 seconds of wallclock time (using an average 160% of CPU) but then something goes wrong and my process is stuck in state "uxmxn". This is using user threads on AMD64_FREEBSD on a machine with four CPUs. Mika P.S. diff attached, it's a bit ugly, but maybe it works on some other system.... the code works just fine if you run one thread at a time. I.e., if you change IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END; to IF threads.size() > 0 THEN EVAL Thread.Join(threads.remlo()) END; Index: cm3/src/Builder.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/cm3/src/Builder.m3,v retrieving revision 1.36 diff -c -r1.36 Builder.m3 *** cm3/src/Builder.m3 24 Aug 2010 05:24:24 -0000 1.36 --- cm3/src/Builder.m3 13 Feb 2011 06:47:40 -0000 *************** *** 15,20 **** --- 15,22 ---- IMPORT QIdent; FROM Target IMPORT M3BackendMode_t, BackendAssembly, BackendModeStrings; FROM M3Path IMPORT OSKind, OSKindStrings; + IMPORT Pathname; + IMPORT RefSeq; TYPE UK = M3Unit.Kind; *************** *** 131,136 **** --- 133,139 ---- link_coverage : TEXT; (* coverage library *) m3_front_flags: Arg.List; (* configuration options for the front *) m3_options : Arg.List; (* misc. user options for the frontend *) + delayBackend := FALSE; END; TYPE *************** *** 932,937 **** --- 935,956 ---- (*------------------------------------------------------------ compilation --*) + TYPE MarkerPromise = QMachine.Promise BRANDED OBJECT OVERRIDES fulfil := FulfilNothing END; + + PROCEDURE FulfilNothing(<*UNUSED*>mp : MarkerPromise) = BEGIN END FulfilNothing; + + TYPE SeqClosure = Thread.Closure OBJECT seq : RefSeq.T; OVERRIDES apply := SeqApply END; + + PROCEDURE SeqApply(cl : SeqClosure) : REFANY = + BEGIN + FOR i := 0 TO cl.seq.size()-1 DO + WITH p = NARROW(cl.seq.get(i),QMachine.Promise) DO + p.fulfil() + END + END; + RETURN NIL + END SeqApply; + PROCEDURE CompileEverything (s: State; schedule: SourceList) = VAR u: M3Unit.T; BEGIN *************** *** 941,948 **** (* compile all the sources using the initial schedule *) FOR i := 0 TO LAST (schedule^) DO ! CompileOne (s, schedule[i]); END; FlushPending (s); (* recompile any interfaces where we goofed on the exports *) --- 960,1000 ---- (* compile all the sources using the initial schedule *) FOR i := 0 TO LAST (schedule^) DO ! s.delayBackend := TRUE; ! TRY ! CompileOne (s, schedule[i]); ! FINALLY ! s.delayBackend := FALSE; ! END; ! ! s.machine.promises.addhi(NEW(MarkerPromise)); ! ! END; ! ! VAR ! curSeq := NEW(RefSeq.T).init(); ! threads := NEW(RefSeq.T).init(); ! BEGIN ! FOR i := 0 TO s.machine.promises.size()-1 DO ! WITH p = s.machine.promises.get(i) DO ! curSeq.addhi(p); ! IF i = s.machine.promises.size()-1 OR ISTYPE(p,MarkerPromise) THEN ! WITH cl = NEW(SeqClosure, seq := curSeq) DO ! threads.addhi (Thread.Fork(cl)); ! ! IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END; ! ! curSeq := NEW(RefSeq.T).init() ! END ! END ! END ! END; ! WHILE threads.size() > 0 DO EVAL Thread.Join(threads.remlo()) END; END; + + EVAL s.machine.promises.init(); + + FlushPending (s); (* recompile any interfaces where we goofed on the exports *) *************** *** 1151,1156 **** --- 1203,1227 ---- END; END CompileM3; + TYPE + NotePromise = QMachine.Promise OBJECT + nam : Pathname.T; + OVERRIDES + fulfil := FulfilNP; + END; + + RemovePromise = QMachine.Promise OBJECT + nam : Pathname.T; + OVERRIDES + fulfil := FulfilRP; + END; + + PROCEDURE FulfilNP(np : NotePromise) = + BEGIN Utils.NoteTempFile(np.nam) END FulfilNP; + + PROCEDURE FulfilRP(rp : RemovePromise) = + BEGIN Utils.Remove(rp.nam) END FulfilRP; + PROCEDURE PushOneM3 (s: State; u: M3Unit.T): BOOLEAN = VAR tmpC, tmpS: TEXT; *************** *** 1191,1208 **** | 3 => (* -bootstrap, +m3back, +asm *) tmpC := TempCName (u); tmpS := TempSName (u); IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; IF RunM3 (s, u, tmpC) THEN ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) ! AND RunAsm (s, tmpS, u.object) THEN END; need_merge := TRUE; END; IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; | 6, (* +bootstrap, +m3back, -asm *) 7 => (* +bootstrap, +m3back, +asm *) --- 1262,1318 ---- | 3 => (* -bootstrap, +m3back, +asm *) + IF s.delayBackend THEN tmpC := TempCName (u); tmpS := TempSName (u); + (* IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; + *) + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(NotePromise, nam := tmpC)) + END; + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(NotePromise, nam := tmpS)) + END; + IF RunM3 (s, u, tmpC) THEN ! s.machine.record(TRUE); ! TRY ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) ! AND RunAsm (s, tmpS, u.object) THEN ! END; ! FINALLY ! s.machine.record(FALSE) END; + need_merge := TRUE; END; + (* IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; + *) + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpC)) + END; + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpS)) + END; + + ELSE + tmpC := TempCName (u); + tmpS := TempSName (u); + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; + IF RunM3 (s, u, tmpC) THEN + IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) + AND RunAsm (s, tmpS, u.object) THEN + END; + need_merge := TRUE; + END; + IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; + IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; + END | 6, (* +bootstrap, +m3back, -asm *) 7 => (* +bootstrap, +m3back, +asm *) Index: cm3/src/m3makefile =================================================================== RCS file: /usr/cvs/cm3/m3-sys/cm3/src/m3makefile,v retrieving revision 1.24 diff -c -r1.24 m3makefile *** cm3/src/m3makefile 8 Dec 2010 07:30:57 -0000 1.24 --- cm3/src/m3makefile 13 Feb 2011 06:47:42 -0000 *************** *** 29,34 **** --- 29,35 ---- end module ("Arg") module ("Builder") module ("Dirs") module ("M3Build") module ("M3Loc") Index: m3quake/src/QMachine.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.i3,v retrieving revision 1.6 diff -c -r1.6 QMachine.i3 *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -0000 1.6 --- m3quake/src/QMachine.i3 13 Feb 2011 06:47:43 -0000 *************** *** 8,13 **** --- 8,14 ---- IMPORT Thread, Wr, QValue, QCode; FROM Quake IMPORT Machine, Error, ID, IDMap; + IMPORT RefSeq; REVEAL T <: T_; *************** *** 15,20 **** --- 16,22 ---- T = Machine; T_ = OBJECT map: IDMap := NIL; (* READONLY *) + promises : RefSeq.T; METHODS init (map: IDMap): T; evaluate (s: QCode.Stream) RAISES {Error, Thread.Alerted}; *************** *** 37,42 **** --- 39,46 ---- set_wr (wr: Wr.T); exec_echo (b: BOOLEAN): BOOLEAN; trace (b: BOOLEAN); + + record(on : BOOLEAN); (* instead of performing certain acts, promise *) END; PROCEDURE PushBool (t: T; b: BOOLEAN); *************** *** 51,54 **** --- 55,63 ---- PROCEDURE GetEnv (default, v0, v1, v2, v3, v4: TEXT := NIL): TEXT; + TYPE Promise = OBJECT METHODS fulfil() RAISES { Error } END; + END QMachine. + + + Index: m3quake/src/QMachine.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3,v retrieving revision 1.35 diff -c -r1.35 QMachine.m3 *** m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35 --- m3quake/src/QMachine.m3 13 Feb 2011 06:47:44 -0000 *************** *** 16,21 **** --- 16,22 ---- IMPORT TextUtils, FSUtils, System, DirStack; (* sysutils *) IMPORT Compiler; IMPORT M3Path; + IMPORT RefSeq; CONST OnUnix = (Compiler.ThisOS = Compiler.OS.POSIX); *************** *** 44,49 **** --- 45,52 ---- shell : TEXT := NIL; sh_option : TEXT := NIL; tmp_dir : TEXT := NIL; + + doRecord := FALSE; OVERRIDES init := Init; evaluate := Evaluate; *************** *** 66,73 **** --- 69,81 ---- set_wr := SetWr; exec_echo := ExecEcho; trace := Trace; + + record := Record; END; + PROCEDURE Record(t : T; on : BOOLEAN) = + BEGIN t.doRecord := on END Record; + TYPE Registers = RECORD cp : QCode.Stream := NIL; (* code pointer *) *************** *** 139,144 **** --- 147,154 ---- t.globals := NEW (IntRefTbl.Default).init (); t.default_wr := Stdio.stdout; + t.promises := NEW(RefSeq.T).init(); + InitOSEnv (t); InitBuiltins (t); *************** *** 1555,1564 **** END; ELSE FlushIO (); ! Process.GetStandardFileHandles (stdin, stdout, stderr); ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), ! stdin := stdin, stdout := stdout, ! stderr := stderr); END; EXCEPT | Thread.Alerted => --- 1565,1594 ---- END; ELSE FlushIO (); ! IF t.doRecord THEN ! handle := NIL; ! WITH a = NEW(REF ARRAY OF TEXT, n_shell_args) DO ! a^ := SUBARRAY(args,0,n_shell_args); ! VAR wrx : Wr.T; BEGIN ! IF echo OR t.do_echo THEN ! wrx := wr ! ELSE ! wrx := NIL ! END; ! t.promises.addhi(NEW(ExecPromise, ! cmd := t.shell, ! wr := wrx, ! args := a, ! t := t, ! ignore_errors := ignore_errors)) ! END ! END ! ELSE ! Process.GetStandardFileHandles (stdin, stdout, stderr); ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), ! stdin := stdin, stdout := stdout, ! stderr := stderr); ! END; END; EXCEPT | Thread.Alerted => *************** *** 1573,1579 **** END; (* wait for everything to shutdown... *) ! exit_code := Process.Wait (handle); END; IF onlyTry THEN --- 1603,1613 ---- END; (* wait for everything to shutdown... *) ! IF handle = NIL THEN ! exit_code := 0 ! ELSE ! exit_code := Process.Wait (handle); ! END; (* else we're only promising *) END; IF onlyTry THEN *************** *** 1589,1594 **** --- 1623,1664 ---- END ExecCommand; + TYPE + ExecPromise = Promise OBJECT + cmd : TEXT; + args : REF ARRAY OF TEXT; + t : T; + wr : Wr.T; + ignore_errors : BOOLEAN; + OVERRIDES + fulfil := FulfilExecPromise; + END; + + PROCEDURE FulfilExecPromise(ep : ExecPromise) RAISES { Error } = + VAR + stdin, stdout, stderr: File.T; + BEGIN + Process.GetStandardFileHandles (stdin, stdout, stderr); + TRY + IF ep.wr # NIL THEN + Wr.PutText (ep.wr, ep.args[1]); + Wr.PutText (ep.wr, Wr.EOL); + FlushIO (); + END; + WITH handle = Process.Create (ep.cmd, ep.args^, + stdin := stdin, stdout := stdout, + stderr := stderr), + exit_code = Process.Wait(handle) DO + IF exit_code # 0 AND NOT ep.ignore_errors THEN + Err (ep.t, Fmt.F("exit %s: %s", Fmt.Int(exit_code), ep.cmd)) + END + END + EXCEPT + OSError.E (ec) => + Err (ep.t, Fmt.F ("exec failed%s *** %s", OSErr (ec), ep.cmd)); + END + END FulfilExecPromise; + PROCEDURE KillProcess (handle: Process.T) = BEGIN IF (handle # NIL) THEN From jay.krell at cornell.edu Sun Feb 13 08:41:23 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 13 Feb 2011 07:41:23 +0000 Subject: [M3devel] More on threading In-Reply-To: <20110213070414.DD0341A2078@async.async.caltech.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> Message-ID: Yes, gcc is slow and using as is wasteful. Yes this should all be multi-threaded. However it seems to me that m3front has many globals and therefore really making things multi-threaded isn't going to be easy. I think you are mostly wasting your time on user threads though. Have you tried the NT/x86 system? It is very fast. In the gcc system, each .mc file is run through m3cgc1 to produce an .ms file, which is run through as to produce an object file .mo. In the NT/x86 system, cm3 outputs .obj files directly. No .mc files. No m3cgc1 or as. Granted, I'd just as soon output .c files, and hope for a C compiler that doesn't run a separate assembler. My timing might loosen up a bit and I hope to spend some on the thread stress tester using pthreads. Next weekend. Am I right though that Darwin works fine? Solaris? FreeBSD? NetBSD? OpenBSD? Linux? I know, not Linux. You generally don't need any of these systems. All of the x86/AMD64 systems can run in a virtual machine. Solaris/sparc we have good access. - Jay > To: m3devel at elegosoft.com > Date: Sat, 12 Feb 2011 23:04:14 -0800 > From: mika at async.caltech.edu > CC: jay.krell at cornell.edu > Subject: [M3devel] More on threading > > > Hi again m3devel, > > Well I thought I would relate another story having to do with threading > and CM3. It's half-baked at the moment, but it only reflects a few hours' > work on my part, sorry... I'm about to give up now because I don't know > if the problem is in my code, or in threading, or what... maybe it is > the atfork problem Jay talked about? > > Programmers spend a lot of time waiting for their compilers to run, so > it would be nice if the compilation process were faster. After perusing > the source code for the CM3 I realized that there is no feedback at all > from the back-end and assembler steps to the rest of the compiler, and > I believe that with the GCC back-end the vast majority of the runtime > of the compiler is spent in these stages. > > So.... how hard could this be to parallelize? I decided to try making > the two stages (cm3cg1 and as) into "promises" rather than running them > on the spot. The QMachine adds a promise to call the back-end to a > RefSeq.T called "promises", to which the Builder can also add promises. > Then, after making all the promises, collect them and run them ("force > them" a true Schemer would say) in parallel. > > Of course something goes wrong somewhere. With 10 threads I am able to > run 31 seconds of CPU time in 19 seconds of wallclock time (using an > average 160% of CPU) but then something goes wrong and my process > is stuck in state "uxmxn". This is using user threads on AMD64_FREEBSD > on a machine with four CPUs. > > Mika > > P.S. diff attached, it's a bit ugly, but maybe it works on some other > system.... the code works just fine if you run one thread at a time. > I.e., if you change > > IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END; > > to > > IF threads.size() > 0 THEN EVAL Thread.Join(threads.remlo()) END; > > > > Index: cm3/src/Builder.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/cm3/src/Builder.m3,v > retrieving revision 1.36 > diff -c -r1.36 Builder.m3 > *** cm3/src/Builder.m3 24 Aug 2010 05:24:24 -0000 1.36 > --- cm3/src/Builder.m3 13 Feb 2011 06:47:40 -0000 > *************** > *** 15,20 **** > --- 15,22 ---- > IMPORT QIdent; > FROM Target IMPORT M3BackendMode_t, BackendAssembly, BackendModeStrings; > FROM M3Path IMPORT OSKind, OSKindStrings; > + IMPORT Pathname; > + IMPORT RefSeq; > > TYPE > UK = M3Unit.Kind; > *************** > *** 131,136 **** > --- 133,139 ---- > link_coverage : TEXT; (* coverage library *) > m3_front_flags: Arg.List; (* configuration options for the front *) > m3_options : Arg.List; (* misc. user options for the frontend *) > + delayBackend := FALSE; > END; > > TYPE > *************** > *** 932,937 **** > --- 935,956 ---- > > (*------------------------------------------------------------ compilation --*) > > + TYPE MarkerPromise = QMachine.Promise BRANDED OBJECT OVERRIDES fulfil := FulfilNothing END; > + > + PROCEDURE FulfilNothing(<*UNUSED*>mp : MarkerPromise) = BEGIN END FulfilNothing; > + > + TYPE SeqClosure = Thread.Closure OBJECT seq : RefSeq.T; OVERRIDES apply := SeqApply END; > + > + PROCEDURE SeqApply(cl : SeqClosure) : REFANY = > + BEGIN > + FOR i := 0 TO cl.seq.size()-1 DO > + WITH p = NARROW(cl.seq.get(i),QMachine.Promise) DO > + p.fulfil() > + END > + END; > + RETURN NIL > + END SeqApply; > + > PROCEDURE CompileEverything (s: State; schedule: SourceList) = > VAR u: M3Unit.T; > BEGIN > *************** > *** 941,948 **** > > (* compile all the sources using the initial schedule *) > FOR i := 0 TO LAST (schedule^) DO > ! CompileOne (s, schedule[i]); > END; > FlushPending (s); > > (* recompile any interfaces where we goofed on the exports *) > --- 960,1000 ---- > > (* compile all the sources using the initial schedule *) > FOR i := 0 TO LAST (schedule^) DO > ! s.delayBackend := TRUE; > ! TRY > ! CompileOne (s, schedule[i]); > ! FINALLY > ! s.delayBackend := FALSE; > ! END; > ! > ! s.machine.promises.addhi(NEW(MarkerPromise)); > ! > ! END; > ! > ! VAR > ! curSeq := NEW(RefSeq.T).init(); > ! threads := NEW(RefSeq.T).init(); > ! BEGIN > ! FOR i := 0 TO s.machine.promises.size()-1 DO > ! WITH p = s.machine.promises.get(i) DO > ! curSeq.addhi(p); > ! IF i = s.machine.promises.size()-1 OR ISTYPE(p,MarkerPromise) THEN > ! WITH cl = NEW(SeqClosure, seq := curSeq) DO > ! threads.addhi (Thread.Fork(cl)); > ! > ! IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END; > ! > ! curSeq := NEW(RefSeq.T).init() > ! END > ! END > ! END > ! END; > ! WHILE threads.size() > 0 DO EVAL Thread.Join(threads.remlo()) END; > END; > + > + EVAL s.machine.promises.init(); > + > + > FlushPending (s); > > (* recompile any interfaces where we goofed on the exports *) > *************** > *** 1151,1156 **** > --- 1203,1227 ---- > END; > END CompileM3; > > + TYPE > + NotePromise = QMachine.Promise OBJECT > + nam : Pathname.T; > + OVERRIDES > + fulfil := FulfilNP; > + END; > + > + RemovePromise = QMachine.Promise OBJECT > + nam : Pathname.T; > + OVERRIDES > + fulfil := FulfilRP; > + END; > + > + PROCEDURE FulfilNP(np : NotePromise) = > + BEGIN Utils.NoteTempFile(np.nam) END FulfilNP; > + > + PROCEDURE FulfilRP(rp : RemovePromise) = > + BEGIN Utils.Remove(rp.nam) END FulfilRP; > + > PROCEDURE PushOneM3 (s: State; u: M3Unit.T): BOOLEAN = > VAR > tmpC, tmpS: TEXT; > *************** > *** 1191,1208 **** > > > | 3 => (* -bootstrap, +m3back, +asm *) > tmpC := TempCName (u); > tmpS := TempSName (u); > IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; > IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; > IF RunM3 (s, u, tmpC) THEN > ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) > ! AND RunAsm (s, tmpS, u.object) THEN > END; > need_merge := TRUE; > END; > IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; > IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; > > | 6, (* +bootstrap, +m3back, -asm *) > 7 => (* +bootstrap, +m3back, +asm *) > --- 1262,1318 ---- > > > | 3 => (* -bootstrap, +m3back, +asm *) > + IF s.delayBackend THEN > tmpC := TempCName (u); > tmpS := TempSName (u); > + (* > IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; > IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; > + *) > + IF (NOT s.keep_files) THEN > + s.machine.promises.addhi(NEW(NotePromise, nam := tmpC)) > + END; > + IF (NOT s.keep_files) THEN > + s.machine.promises.addhi(NEW(NotePromise, nam := tmpS)) > + END; > + > IF RunM3 (s, u, tmpC) THEN > ! s.machine.record(TRUE); > ! TRY > ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) > ! AND RunAsm (s, tmpS, u.object) THEN > ! END; > ! FINALLY > ! s.machine.record(FALSE) > END; > + > need_merge := TRUE; > END; > + (* > IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; > IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; > + *) > + IF (NOT s.keep_files) THEN > + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpC)) > + END; > + IF (NOT s.keep_files) THEN > + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpS)) > + END; > + > + ELSE > + tmpC := TempCName (u); > + tmpS := TempSName (u); > + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; > + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; > + IF RunM3 (s, u, tmpC) THEN > + IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) > + AND RunAsm (s, tmpS, u.object) THEN > + END; > + need_merge := TRUE; > + END; > + IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; > + IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; > + END > > | 6, (* +bootstrap, +m3back, -asm *) > 7 => (* +bootstrap, +m3back, +asm *) > Index: cm3/src/m3makefile > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/cm3/src/m3makefile,v > retrieving revision 1.24 > diff -c -r1.24 m3makefile > *** cm3/src/m3makefile 8 Dec 2010 07:30:57 -0000 1.24 > --- cm3/src/m3makefile 13 Feb 2011 06:47:42 -0000 > *************** > *** 29,34 **** > --- 29,35 ---- > end > module ("Arg") > module ("Builder") > module ("Dirs") > module ("M3Build") > module ("M3Loc") > Index: m3quake/src/QMachine.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.i3,v > retrieving revision 1.6 > diff -c -r1.6 QMachine.i3 > *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -0000 1.6 > --- m3quake/src/QMachine.i3 13 Feb 2011 06:47:43 -0000 > *************** > *** 8,13 **** > --- 8,14 ---- > > IMPORT Thread, Wr, QValue, QCode; > FROM Quake IMPORT Machine, Error, ID, IDMap; > + IMPORT RefSeq; > > REVEAL > T <: T_; > *************** > *** 15,20 **** > --- 16,22 ---- > T = Machine; > T_ = OBJECT > map: IDMap := NIL; (* READONLY *) > + promises : RefSeq.T; > METHODS > init (map: IDMap): T; > evaluate (s: QCode.Stream) RAISES {Error, Thread.Alerted}; > *************** > *** 37,42 **** > --- 39,46 ---- > set_wr (wr: Wr.T); > exec_echo (b: BOOLEAN): BOOLEAN; > trace (b: BOOLEAN); > + > + record(on : BOOLEAN); (* instead of performing certain acts, promise *) > END; > > PROCEDURE PushBool (t: T; b: BOOLEAN); > *************** > *** 51,54 **** > --- 55,63 ---- > > PROCEDURE GetEnv (default, v0, v1, v2, v3, v4: TEXT := NIL): TEXT; > > + TYPE Promise = OBJECT METHODS fulfil() RAISES { Error } END; > + > END QMachine. > + > + > + > Index: m3quake/src/QMachine.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3,v > retrieving revision 1.35 > diff -c -r1.35 QMachine.m3 > *** m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35 > --- m3quake/src/QMachine.m3 13 Feb 2011 06:47:44 -0000 > *************** > *** 16,21 **** > --- 16,22 ---- > IMPORT TextUtils, FSUtils, System, DirStack; (* sysutils *) > IMPORT Compiler; > IMPORT M3Path; > + IMPORT RefSeq; > > CONST > OnUnix = (Compiler.ThisOS = Compiler.OS.POSIX); > *************** > *** 44,49 **** > --- 45,52 ---- > shell : TEXT := NIL; > sh_option : TEXT := NIL; > tmp_dir : TEXT := NIL; > + > + doRecord := FALSE; > OVERRIDES > init := Init; > evaluate := Evaluate; > *************** > *** 66,73 **** > --- 69,81 ---- > set_wr := SetWr; > exec_echo := ExecEcho; > trace := Trace; > + > + record := Record; > END; > > + PROCEDURE Record(t : T; on : BOOLEAN) = > + BEGIN t.doRecord := on END Record; > + > TYPE > Registers = RECORD > cp : QCode.Stream := NIL; (* code pointer *) > *************** > *** 139,144 **** > --- 147,154 ---- > t.globals := NEW (IntRefTbl.Default).init (); > t.default_wr := Stdio.stdout; > > + t.promises := NEW(RefSeq.T).init(); > + > InitOSEnv (t); > InitBuiltins (t); > > *************** > *** 1555,1564 **** > END; > ELSE > FlushIO (); > ! Process.GetStandardFileHandles (stdin, stdout, stderr); > ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), > ! stdin := stdin, stdout := stdout, > ! stderr := stderr); > END; > EXCEPT > | Thread.Alerted => > --- 1565,1594 ---- > END; > ELSE > FlushIO (); > ! IF t.doRecord THEN > ! handle := NIL; > ! WITH a = NEW(REF ARRAY OF TEXT, n_shell_args) DO > ! a^ := SUBARRAY(args,0,n_shell_args); > ! VAR wrx : Wr.T; BEGIN > ! IF echo OR t.do_echo THEN > ! wrx := wr > ! ELSE > ! wrx := NIL > ! END; > ! t.promises.addhi(NEW(ExecPromise, > ! cmd := t.shell, > ! wr := wrx, > ! args := a, > ! t := t, > ! ignore_errors := ignore_errors)) > ! END > ! END > ! ELSE > ! Process.GetStandardFileHandles (stdin, stdout, stderr); > ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), > ! stdin := stdin, stdout := stdout, > ! stderr := stderr); > ! END; > END; > EXCEPT > | Thread.Alerted => > *************** > *** 1573,1579 **** > END; > > (* wait for everything to shutdown... *) > ! exit_code := Process.Wait (handle); > END; > > IF onlyTry THEN > --- 1603,1613 ---- > END; > > (* wait for everything to shutdown... *) > ! IF handle = NIL THEN > ! exit_code := 0 > ! ELSE > ! exit_code := Process.Wait (handle); > ! END; (* else we're only promising *) > END; > > IF onlyTry THEN > *************** > *** 1589,1594 **** > --- 1623,1664 ---- > > END ExecCommand; > > + TYPE > + ExecPromise = Promise OBJECT > + cmd : TEXT; > + args : REF ARRAY OF TEXT; > + t : T; > + wr : Wr.T; > + ignore_errors : BOOLEAN; > + OVERRIDES > + fulfil := FulfilExecPromise; > + END; > + > + PROCEDURE FulfilExecPromise(ep : ExecPromise) RAISES { Error } = > + VAR > + stdin, stdout, stderr: File.T; > + BEGIN > + Process.GetStandardFileHandles (stdin, stdout, stderr); > + TRY > + IF ep.wr # NIL THEN > + Wr.PutText (ep.wr, ep.args[1]); > + Wr.PutText (ep.wr, Wr.EOL); > + FlushIO (); > + END; > + WITH handle = Process.Create (ep.cmd, ep.args^, > + stdin := stdin, stdout := stdout, > + stderr := stderr), > + exit_code = Process.Wait(handle) DO > + IF exit_code # 0 AND NOT ep.ignore_errors THEN > + Err (ep.t, Fmt.F("exit %s: %s", Fmt.Int(exit_code), ep.cmd)) > + END > + END > + EXCEPT > + OSError.E (ec) => > + Err (ep.t, Fmt.F ("exec failed%s *** %s", OSErr (ec), ep.cmd)); > + END > + END FulfilExecPromise; > + > PROCEDURE KillProcess (handle: Process.T) = > BEGIN > IF (handle # NIL) THEN > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 13 19:17:17 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 10:17:17 -0800 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu> Message-ID: <20110213181717.4F0E91A2078@async.async.caltech.edu> Jay K writes: >--_b09106d1-2a84-4ffe-876c-0b2e4cdfd0b0_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Yes=2C gcc is slow and using as is wasteful. >Yes this should all be multi-threaded. >However it seems to me that m3front has many globals and therefore >really making things multi-threaded isn't going to be easy. OK but as long as the back-end is taking up most of the cycles the property that the output of the front-end is used only by cmcg1 (m3back/m3cgc/whatever you call it) and the assembler means that what I did works for a pretty big speedup. In fact you can see that it works just fine (from the point of view of generating the correct output) since you can re-order running the backend with respect to all the steps done by the front-end. What you lose is (perhaps) some error messages, and of course, in general, any implicit dependencies between the error messages. I don't understand why the program gets stuck in uxmxn though. Maybe a problem with, sigh, user threads... >I think you are mostly wasting your time on user threads though. I hope so... I too would really like to use pthreads. > > >Have you tried the NT/x86 system? >It is very fast. > > >In the gcc system=2C each .mc file is run through m3cgc1 to produce an .ms = >file=2C >which is run through as to produce an object file .mo. > > >In the NT/x86 system=2C cm3 outputs .obj files directly. >No .mc files. No m3cgc1 or as. >Granted=2C I'd just as soon output .c files=2C and hope for a C compiler th= >at doesn't run a separate assembler. The NT compiler sounds faster but of course it's easier to parallelize the unix one using the method I outlined :-) Well unless you're using pthreads of course, then you can just fork a new thread for the integrated NT backend as long as it doesn't share too many variables with the front-end. But no I don't use Windows. I was asking on here how to get started with Modula-3 on Windows once but my machine was/is a Windows 2000 system and apparently CM3 no longer works on win2k..? I used to use PM3/Klagenfurt a lot on Windows (Cygwin) but thankfully my clients on that project let me switch the system to Unix so I don't have to touch that anymore. > > >My timing might loosen up a bit and I hope to spend some on the thread stre= >ss tester using pthreads. Next weekend. >Am I right though that Darwin works fine? >Solaris? FreeBSD? NetBSD? OpenBSD? Linux? I know=2C not Linux. >You generally don't need any of these systems. That would be great. I don't think pthreads works properly anywhere at the moment. My guess is that it never has, if you stress it hard enough. I first ran into problems on Darwin and then switched to AMD64 simply because I have better access to faster machines with AMD64. At first there were some obvious problems that Tony fixed quickly but then I kept running into odd behavior with my application and I added various other tests until I could reproduce everything I'd seen go wrong with my application, and I think at that point I was seeing problems on every platform (except user threads). The uxmxn problem I see with user threads on FreeBSD suggests that perhaps I need to add some more tests, however. Mika >All of the x86/AMD64 systems can run in a virtual machine. >Solaris/sparc we have good access. > > > - Jay > > >> To: m3devel at elegosoft.com >> Date: Sat=2C 12 Feb 2011 23:04:14 -0800 >> From: mika at async.caltech.edu >> CC: jay.krell at cornell.edu >> Subject: [M3devel] More on threading >>=20 >>=20 >> Hi again m3devel=2C >>=20 >> Well I thought I would relate another story having to do with threading >> and CM3. It's half-baked at the moment=2C but it only reflects a few hou= >rs' >> work on my part=2C sorry... I'm about to give up now because I don't know >> if the problem is in my code=2C or in threading=2C or what... maybe it is >> the atfork problem Jay talked about? >>=20 >> Programmers spend a lot of time waiting for their compilers to run=2C so >> it would be nice if the compilation process were faster. After perusing >> the source code for the CM3 I realized that there is no feedback at all >> from the back-end and assembler steps to the rest of the compiler=2C and >> I believe that with the GCC back-end the vast majority of the runtime >> of the compiler is spent in these stages. >>=20 >> So.... how hard could this be to parallelize? I decided to try making >> the two stages (cm3cg1 and as) into "promises" rather than running them >> on the spot. The QMachine adds a promise to call the back-end to a >> RefSeq.T called "promises"=2C to which the Builder can also add promises. >> Then=2C after making all the promises=2C collect them and run them ("forc= >e >> them" a true Schemer would say) in parallel. >>=20 >> Of course something goes wrong somewhere. With 10 threads I am able to >> run 31 seconds of CPU time in 19 seconds of wallclock time (using an >> average 160% of CPU) but then something goes wrong and my process >> is stuck in state "uxmxn". This is using user threads on AMD64_FREEBSD >> on a machine with four CPUs. >>=20 >> Mika >>=20 >> P.S. diff attached=2C it's a bit ugly=2C but maybe it works on some other >> system.... the code works just fine if you run one thread at a time. >> I.e.=2C if you change >>=20 >> IF threads.size() > 10 THEN EVAL Thread.Join(threads.remlo()) END=3B >>=20 >> to >>=20 >> IF threads.size() > 0 THEN EVAL Thread.Join(threads.remlo()) END=3B >>=20 >>=20 >>=20 >> Index: cm3/src/Builder.m3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: /usr/cvs/cm3/m3-sys/cm3/src/Builder.m3=2Cv >> retrieving revision 1.36 >> diff -c -r1.36 Builder.m3 >> *** cm3/src/Builder.m3 24 Aug 2010 05:24:24 -0000 1.36 >> --- cm3/src/Builder.m3 13 Feb 2011 06:47:40 -0000 >> *************** >> *** 15=2C20 **** >> --- 15=2C22 ---- >> IMPORT QIdent=3B >> FROM Target IMPORT M3BackendMode_t=2C BackendAssembly=2C BackendModeStr= >ings=3B >> FROM M3Path IMPORT OSKind=2C OSKindStrings=3B >> + IMPORT Pathname=3B >> + IMPORT RefSeq=3B >> =20 >> TYPE >> UK =3D M3Unit.Kind=3B >> *************** >> *** 131=2C136 **** >> --- 133=2C139 ---- >> link_coverage : TEXT=3B (* coverage library *) >> m3_front_flags: Arg.List=3B (* configuration options for = >the front *) >> m3_options : Arg.List=3B (* misc. user options for the= > frontend *) >> + delayBackend :=3D FALSE=3B >> END=3B >> =20 >> TYPE >> *************** >> *** 932=2C937 **** >> --- 935=2C956 ---- >> =20 >> (*------------------------------------------------------------ compilat= >ion --*) >> =20 >> + TYPE MarkerPromise =3D QMachine.Promise BRANDED OBJECT OVERRIDES fulfil= > :=3D FulfilNothing END=3B >> +=20 >> + PROCEDURE FulfilNothing(<*UNUSED*>mp : MarkerPromise) =3D BEGIN END Ful= >filNothing=3B >> +=20 >> + TYPE SeqClosure =3D Thread.Closure OBJECT seq : RefSeq.T=3B OVERRIDES a= >pply :=3D SeqApply END=3B >> +=20 >> + PROCEDURE SeqApply(cl : SeqClosure) : REFANY =3D >> + BEGIN >> + FOR i :=3D 0 TO cl.seq.size()-1 DO >> + WITH p =3D NARROW(cl.seq.get(i)=2CQMachine.Promise) DO >> + p.fulfil() >> + END >> + END=3B >> + RETURN NIL >> + END SeqApply=3B >> +=20 >> PROCEDURE CompileEverything (s: State=3B schedule: SourceList) =3D >> VAR u: M3Unit.T=3B >> BEGIN >> *************** >> *** 941=2C948 **** >> =20 >> (* compile all the sources using the initial schedule *) >> FOR i :=3D 0 TO LAST (schedule^) DO >> ! CompileOne (s=2C schedule[i])=3B >> END=3B >> FlushPending (s)=3B >> =20 >> (* recompile any interfaces where we goofed on the exports *) >> --- 960=2C1000 ---- >> =20 >> (* compile all the sources using the initial schedule *) >> FOR i :=3D 0 TO LAST (schedule^) DO >> ! s.delayBackend :=3D TRUE=3B >> ! TRY >> ! CompileOne (s=2C schedule[i])=3B >> ! FINALLY >> ! s.delayBackend :=3D FALSE=3B >> ! END=3B >> !=20 >> ! s.machine.promises.addhi(NEW(MarkerPromise))=3B >> !=20 >> ! END=3B >> !=20 >> ! VAR >> ! curSeq :=3D NEW(RefSeq.T).init()=3B >> ! threads :=3D NEW(RefSeq.T).init()=3B >> ! BEGIN >> ! FOR i :=3D 0 TO s.machine.promises.size()-1 DO >> ! WITH p =3D s.machine.promises.get(i) DO >> ! curSeq.addhi(p)=3B >> ! IF i =3D s.machine.promises.size()-1 OR ISTYPE(p=2CMarkerProm= >ise) THEN >> ! WITH cl =3D NEW(SeqClosure=2C seq :=3D curSeq) DO >> ! threads.addhi (Thread.Fork(cl))=3B >> !=20 >> ! IF threads.size() > 10 THEN EVAL Thread.Join(threads.reml= >o()) END=3B >> ! =20 >> ! curSeq :=3D NEW(RefSeq.T).init() >> ! END >> ! END >> ! END >> ! END=3B >> ! WHILE threads.size() > 0 DO EVAL Thread.Join(threads.remlo()) END= >=3B >> END=3B >> + =20 >> + EVAL s.machine.promises.init()=3B >> +=20 >> +=20 >> FlushPending (s)=3B >> =20 >> (* recompile any interfaces where we goofed on the exports *) >> *************** >> *** 1151=2C1156 **** >> --- 1203=2C1227 ---- >> END=3B >> END CompileM3=3B >> =20 >> + TYPE >> + NotePromise =3D QMachine.Promise OBJECT >> + nam : Pathname.T=3B >> + OVERRIDES >> + fulfil :=3D FulfilNP=3B >> + END=3B >> +=20 >> + RemovePromise =3D QMachine.Promise OBJECT >> + nam : Pathname.T=3B >> + OVERRIDES >> + fulfil :=3D FulfilRP=3B >> + END=3B >> +=20 >> + PROCEDURE FulfilNP(np : NotePromise) =3D=20 >> + BEGIN Utils.NoteTempFile(np.nam) END FulfilNP=3B >> +=20 >> + PROCEDURE FulfilRP(rp : RemovePromise) =3D=20 >> + BEGIN Utils.Remove(rp.nam) END FulfilRP=3B >> +=20 >> PROCEDURE PushOneM3 (s: State=3B u: M3Unit.T): BOOLEAN =3D >> VAR >> tmpC=2C tmpS: TEXT=3B >> *************** >> *** 1191=2C1208 **** >> =20 >> =20 >> | 3 =3D> (* -bootstrap=2C +m3back=2C +asm *) >> tmpC :=3D TempCName (u)=3B >> tmpS :=3D TempSName (u)=3B >> IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3B >> IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END=3B >> IF RunM3 (s=2C u=2C tmpC) THEN >> ! IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.optimize) >> ! AND RunAsm (s=2C tmpS=2C u.object) THEN >> END=3B >> need_merge :=3D TRUE=3B >> END=3B >> IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END=3B >> IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END=3B >> =20 >> | 6=2C (* +bootstrap=2C +m3back=2C -asm *) >> 7 =3D> (* +bootstrap=2C +m3back=2C +asm *) >> --- 1262=2C1318 ---- >> =20 >> =20 >> | 3 =3D> (* -bootstrap=2C +m3back=2C +asm *) >> + IF s.delayBackend THEN >> tmpC :=3D TempCName (u)=3B >> tmpS :=3D TempSName (u)=3B >> + (* >> IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3B >> IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END=3B >> + *) >> + IF (NOT s.keep_files) THEN=20 >> + s.machine.promises.addhi(NEW(NotePromise=2C nam :=3D tmpC))=20 >> + END=3B >> + IF (NOT s.keep_files) THEN=20 >> + s.machine.promises.addhi(NEW(NotePromise=2C nam :=3D tmpS))=20 >> + END=3B >> +=20 >> IF RunM3 (s=2C u=2C tmpC) THEN >> ! s.machine.record(TRUE)=3B >> ! TRY >> ! IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.optimize) >> ! AND RunAsm (s=2C tmpS=2C u.object) THEN >> ! END=3B >> ! FINALLY >> ! s.machine.record(FALSE) >> END=3B >> +=20 >> need_merge :=3D TRUE=3B >> END=3B >> + (* >> IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END=3B >> IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END=3B >> + *) >> + IF (NOT s.keep_files) THEN=20 >> + s.machine.promises.addhi(NEW(RemovePromise=2C nam :=3D tmpC))= >=20 >> + END=3B >> + IF (NOT s.keep_files) THEN=20 >> + s.machine.promises.addhi(NEW(RemovePromise=2C nam :=3D tmpS))= >=20 >> + END=3B >> +=20 >> + ELSE >> + tmpC :=3D TempCName (u)=3B >> + tmpS :=3D TempSName (u)=3B >> + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3B >> + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END=3B >> + IF RunM3 (s=2C u=2C tmpC) THEN >> + IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.optimize) >> + AND RunAsm (s=2C tmpS=2C u.object) THEN >> + END=3B >> + need_merge :=3D TRUE=3B >> + END=3B >> + IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END=3B >> + IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END=3B >> + END >> =20 >> | 6=2C (* +bootstrap=2C +m3back=2C -asm *) >> 7 =3D> (* +bootstrap=2C +m3back=2C +asm *) >> Index: cm3/src/m3makefile >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: /usr/cvs/cm3/m3-sys/cm3/src/m3makefile=2Cv >> retrieving revision 1.24 >> diff -c -r1.24 m3makefile >> *** cm3/src/m3makefile 8 Dec 2010 07:30:57 -0000 1.24 >> --- cm3/src/m3makefile 13 Feb 2011 06:47:42 -0000 >> *************** >> *** 29=2C34 **** >> --- 29=2C35 ---- >> end >> module ("Arg") >> module ("Builder") >> module ("Dirs") >> module ("M3Build") >> module ("M3Loc") >> Index: m3quake/src/QMachine.i3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.i3=2Cv >> retrieving revision 1.6 >> diff -c -r1.6 QMachine.i3 >> *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -0000 1.6 >> --- m3quake/src/QMachine.i3 13 Feb 2011 06:47:43 -0000 >> *************** >> *** 8=2C13 **** >> --- 8=2C14 ---- >> =20 >> IMPORT Thread=2C Wr=2C QValue=2C QCode=3B >> FROM Quake IMPORT Machine=2C Error=2C ID=2C IDMap=3B >> + IMPORT RefSeq=3B >> =20 >> REVEAL >> T <: T_=3B >> *************** >> *** 15=2C20 **** >> --- 16=2C22 ---- >> T =3D Machine=3B >> T_ =3D OBJECT >> map: IDMap :=3D NIL=3B (* READONLY *) >> + promises : RefSeq.T=3B >> METHODS >> init (map: IDMap): T=3B >> evaluate (s: QCode.Stream) RAISES {Error=2C Th= >read.Alerted}=3B >> *************** >> *** 37=2C42 **** >> --- 39=2C46 ---- >> set_wr (wr: Wr.T)=3B >> exec_echo (b: BOOLEAN): BOOLEAN=3B >> trace (b: BOOLEAN)=3B >> + =20 >> + record(on : BOOLEAN)=3B (* instead of performing certain acts= >=2C promise *) >> END=3B >> =20 >> PROCEDURE PushBool (t: T=3B b: BOOLEAN)=3B >> *************** >> *** 51=2C54 **** >> --- 55=2C63 ---- >> =20 >> PROCEDURE GetEnv (default=2C v0=2C v1=2C v2=2C v3=2C v4: TEXT :=3D NIL)= >: TEXT=3B >> =20 >> + TYPE Promise =3D OBJECT METHODS fulfil() RAISES { Error } END=3B >> +=20 >> END QMachine. >> +=20 >> +=20 >> +=20 >> Index: m3quake/src/QMachine.m3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3=2Cv >> retrieving revision 1.35 >> diff -c -r1.35 QMachine.m3 >> *** m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35 >> --- m3quake/src/QMachine.m3 13 Feb 2011 06:47:44 -0000 >> *************** >> *** 16=2C21 **** >> --- 16=2C22 ---- >> IMPORT TextUtils=2C FSUtils=2C System=2C DirStack=3B (* sysutils *) >> IMPORT Compiler=3B >> IMPORT M3Path=3B >> + IMPORT RefSeq=3B >> =20 >> CONST >> OnUnix =3D (Compiler.ThisOS =3D Compiler.OS.POSIX)=3B >> *************** >> *** 44=2C49 **** >> --- 45=2C52 ---- >> shell : TEXT :=3D NIL=3B >> sh_option : TEXT :=3D NIL=3B >> tmp_dir : TEXT :=3D NIL=3B >> +=20 >> + doRecord :=3D FALSE=3B >> OVERRIDES >> init :=3D Init=3B >> evaluate :=3D Evaluate=3B >> *************** >> *** 66=2C73 **** >> --- 69=2C81 ---- >> set_wr :=3D SetWr=3B >> exec_echo :=3D ExecEcho=3B >> trace :=3D Trace=3B >> +=20 >> + record :=3D Record=3B >> END=3B >> =20 >> + PROCEDURE Record(t : T=3B on : BOOLEAN) =3D=20 >> + BEGIN t.doRecord :=3D on END Record=3B >> +=20 >> TYPE >> Registers =3D RECORD >> cp : QCode.Stream :=3D NIL=3B (* code pointer *) >> *************** >> *** 139=2C144 **** >> --- 147=2C154 ---- >> t.globals :=3D NEW (IntRefTbl.Default).init ()=3B >> t.default_wr :=3D Stdio.stdout=3B >> =20 >> + t.promises :=3D NEW(RefSeq.T).init()=3B >> +=20 >> InitOSEnv (t)=3B >> InitBuiltins (t)=3B >> =20 >> *************** >> *** 1555=2C1564 **** >> END=3B >> ELSE >> FlushIO ()=3B >> ! Process.GetStandardFileHandles (stdin=2C stdout=2C stderr)=3B >> ! handle :=3D Process.Create (t.shell=2C SUBARRAY (args=2C 0=2C= > n_shell_args)=2C >> ! stdin :=3D stdin=2C stdout :=3D std= >out=2C >> ! stderr :=3D stderr)=3B >> END=3B >> EXCEPT >> | Thread.Alerted =3D> >> --- 1565=2C1594 ---- >> END=3B >> ELSE >> FlushIO ()=3B >> ! IF t.doRecord THEN >> ! handle :=3D NIL=3B >> ! WITH a =3D NEW(REF ARRAY OF TEXT=2C n_shell_args) DO >> ! a^ :=3D SUBARRAY(args=2C0=2Cn_shell_args)=3B >> ! VAR wrx : Wr.T=3B BEGIN >> ! IF echo OR t.do_echo THEN >> ! wrx :=3D wr >> ! ELSE >> ! wrx :=3D NIL >> ! END=3B >> ! t.promises.addhi(NEW(ExecPromise=2C >> ! cmd :=3D t.shell=2C >> ! wr :=3D wrx=2C >> ! args :=3D a=2C >> ! t :=3D t=2C >> ! ignore_errors :=3D ignore_errors)) >> ! END >> ! END >> ! ELSE >> ! Process.GetStandardFileHandles (stdin=2C stdout=2C stderr)= >=3B >> ! handle :=3D Process.Create (t.shell=2C SUBARRAY (args=2C 0= >=2C n_shell_args)=2C >> ! stdin :=3D stdin=2C stdout :=3D s= >tdout=2C >> ! stderr :=3D stderr)=3B >> ! END=3B >> END=3B >> EXCEPT >> | Thread.Alerted =3D> >> *************** >> *** 1573=2C1579 **** >> END=3B >> =20 >> (* wait for everything to shutdown... *) >> ! exit_code :=3D Process.Wait (handle)=3B >> END=3B >> =20 >> IF onlyTry THEN >> --- 1603=2C1613 ---- >> END=3B >> =20 >> (* wait for everything to shutdown... *) >> ! IF handle =3D NIL THEN >> ! exit_code :=3D 0=20 >> ! ELSE >> ! exit_code :=3D Process.Wait (handle)=3B >> ! END=3B (* else we're only promising *) >> END=3B >> =20 >> IF onlyTry THEN >> *************** >> *** 1589=2C1594 **** >> --- 1623=2C1664 ---- >> =20 >> END ExecCommand=3B >> =20 >> + TYPE=20 >> + ExecPromise =3D Promise OBJECT >> + cmd : TEXT=3B >> + args : REF ARRAY OF TEXT=3B >> + t : T=3B >> + wr : Wr.T=3B >> + ignore_errors : BOOLEAN=3B >> + OVERRIDES >> + fulfil :=3D FulfilExecPromise=3B >> + END=3B >> +=20 >> + PROCEDURE FulfilExecPromise(ep : ExecPromise) RAISES { Error } =3D=20 >> + VAR >> + stdin=2C stdout=2C stderr: File.T=3B >> + BEGIN >> + Process.GetStandardFileHandles (stdin=2C stdout=2C stderr)=3B >> + TRY >> + IF ep.wr # NIL THEN >> + Wr.PutText (ep.wr=2C ep.args[1])=3B >> + Wr.PutText (ep.wr=2C Wr.EOL)=3B >> + FlushIO ()=3B >> + END=3B >> + WITH handle =3D Process.Create (ep.cmd=2C ep.args^=2C >> + stdin :=3D stdin=2C stdout :=3D std= >out=2C >> + stderr :=3D stderr)=2C >> + exit_code =3D Process.Wait(handle) DO >> + IF exit_code # 0 AND NOT ep.ignore_errors THEN >> + Err (ep.t=2C Fmt.F("exit %s: %s"=2C Fmt.Int(exit_code)=2C ep.= >cmd)) >> + END >> + END >> + EXCEPT >> + OSError.E (ec) =3D> >> + Err (ep.t=2C Fmt.F ("exec failed%s *** %s"=2C OSErr (ec)=2C e= >p.cmd))=3B >> + END >> + END FulfilExecPromise=3B >> +=20 >> PROCEDURE KillProcess (handle: Process.T) =3D >> BEGIN >> IF (handle # NIL) THEN >>=20 > = > >--_b09106d1-2a84-4ffe-876c-0b2e4cdfd0b0_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >Yes=2C gcc is slow and using as is wasteful.
Yes this should all be mult= >i-threaded.
However it seems to me that m3front has many globals and the= >refore
really making things multi-threaded isn't going to be easy.
I = >think you are mostly wasting your time on user threads though.


H= >ave you tried the NT/x86 system?
It is very fast.


In the gcc = >system=2C each .mc file is run through m3cgc1 to produce an .ms file=2C
= >which is run through as to produce an object file .mo.


In the NT= >/x86 system=2C cm3 outputs .obj files directly.
No .mc files. No m3cgc1 = >or as.
Granted=2C I'd just as soon output .c files=2C and hope for a C c= >ompiler that doesn't run a separate assembler.


My timing might l= >oosen up a bit and I hope to spend some on the thread stress tester using p= >threads. Next weekend.
Am I right though that Darwin works fine?
Sola= >ris? FreeBSD? NetBSD? OpenBSD? Linux? I know=2C not Linux.
You generally= > don't need any of these systems.
All of the x86/AMD64 systems can run i= >n a virtual machine.
Solaris/sparc we have good access.


 = >=3B- Jay


>=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C= > 12 Feb 2011 23:04:14 -0800
>=3B From: mika at async.caltech.edu
>= >=3B CC: jay.krell at cornell.edu
>=3B Subject: [M3devel] More on threadin= >g
>=3B
>=3B
>=3B Hi again m3devel=2C
>=3B
>=3B = >Well I thought I would relate another story having to do with threading
= >>=3B and CM3. It's half-baked at the moment=2C but it only reflects a fe= >w hours'
>=3B work on my part=2C sorry... I'm about to give up now bec= >ause I don't know
>=3B if the problem is in my code=2C or in threading= >=2C or what... maybe it is
>=3B the atfork problem Jay talked about?r>>=3B
>=3B Programmers spend a lot of time waiting for their compi= >lers to run=2C so
>=3B it would be nice if the compilation process wer= >e faster. After perusing
>=3B the source code for the CM3 I realized = >that there is no feedback at all
>=3B from the back-end and assembler = >steps to the rest of the compiler=2C and
>=3B I believe that with the = >GCC back-end the vast majority of the runtime
>=3B of the compiler is = >spent in these stages.
>=3B
>=3B So.... how hard could this be t= >o parallelize? I decided to try making
>=3B the two stages (cm3cg1 an= >d as) into "promises" rather than running them
>=3B on the spot. The = >QMachine adds a promise to call the back-end to a
>=3B RefSeq.T called= > "promises"=2C to which the Builder can also add promises.
>=3B Then= >=2C after making all the promises=2C collect them and run them ("force
&= >gt=3B them" a true Schemer would say) in parallel.
>=3B
>=3B Of = >course something goes wrong somewhere. With 10 threads I am able to
>= >=3B run 31 seconds of CPU time in 19 seconds of wallclock time (using an>>=3B average 160% of CPU) but then something goes wrong and my processr>>=3B is stuck in state "uxmxn". This is using user threads on AMD64_FR= >EEBSD
>=3B on a machine with four CPUs.
>=3B
>=3B Mika= >
>=3B
>=3B P.S. diff attached=2C it's a bit ugly=2C but maybe it= > works on some other
>=3B system.... the code works just fine if you = >run one thread at a time.
>=3B I.e.=2C if you change
>=3B
>= >=3B IF threads.size() >=3B 10 THEN EVAL Thread.Join(threads.remlo())= > END=3B
>=3B
>=3B to
>=3B
>=3B IF threads.size()= > >=3B 0 THEN EVAL Thread.Join(threads.remlo()) END=3B
>=3B
>= >=3B
>=3B
>=3B Index: cm3/src/Builder.m3
>=3B =3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=3B RCS file: /usr/cvs/cm3/m3= >-sys/cm3/src/Builder.m3=2Cv
>=3B retrieving revision 1.36
>=3B di= >ff -c -r1.36 Builder.m3
>=3B *** cm3/src/Builder.m3 24 Aug 2010 05: >24:= >24 -0000 1.36
>=3B --- cm3/src/Builder.m3 13 Feb 2011 06:47:40 -0 >000r>>=3B ***************
>=3B *** 15=2C20 ****
>=3B --- 15=2C22 -= >---
>=3B IMPORT QIdent=3B
>=3B FROM Target IMPORT M3BackendMo= >de_t=2C BackendAssembly=2C BackendModeStrings=3B
>=3B FROM M3Path IM= >PORT OSKind=2C OSKindStrings=3B
>=3B + IMPORT Pathname=3B
>=3B + = >IMPORT RefSeq=3B
>=3B
>=3B TYPE
>=3B UK =3D M3Unit.= >Kind=3B
>=3B ***************
>=3B *** 131=2C136 ****
>=3B --= >- 133=2C139 ----
>=3B link_coverage : TEXT=3B (* c= >overage library *)
>=3B m3_front_flags: Arg.List=3B (*= > configuration options for the front *)
>=3B m3_options : Arg= >.List=3B (* misc. user options for the frontend *)
>=3B + = > delayBackend :=3D FALSE=3B
>=3B END=3B
>=3B
>=3B = >TYPE
>=3B ***************
>=3B *** 932=2C937 ****
>=3B --- 9= >35=2C956 ----
>=3B
>=3B (*----------------------------------= >-------------------------- compilation --*)
>=3B
>=3B + TYPE M= >arkerPromise =3D QMachine.Promise BRANDED OBJECT OVERRIDES fulfil :=3D Fulf= >ilNothing END=3B
>=3B +
>=3B + PROCEDURE FulfilNothing(<=3B*UN= >USED*>=3Bmp : MarkerPromise) =3D BEGIN END FulfilNothing=3B
>=3B + <= >br>>=3B + TYPE SeqClosure =3D Thread.Closure OBJECT seq : RefSeq.T=3B OVE= >RRIDES apply :=3D SeqApply END=3B
>=3B +
>=3B + PROCEDURE SeqApp= >ly(cl : SeqClosure) : REFANY =3D
>=3B + BEGIN
>=3B + FOR i = >:=3D 0 TO cl.seq.size()-1 DO
>=3B + WITH p =3D NARROW(cl.seq.get= >(i)=2CQMachine.Promise) DO
>=3B + p.fulfil()
>=3B + = > END
>=3B + END=3B
>=3B + RETURN NIL
>=3B + END Se= >qApply=3B
>=3B +
>=3B PROCEDURE CompileEverything (s: State=3B= > schedule: SourceList) =3D
>=3B VAR u: M3Unit.T=3B
>=3B = >BEGIN
>=3B ***************
>=3B *** 941=2C948 ****
>=3B r>>=3B (* compile all the sources using the initial schedule *)
= >>=3B FOR i :=3D 0 TO LAST (schedule^) DO
>=3B ! CompileO= >ne (s=2C schedule[i])=3B
>=3B END=3B
>=3B FlushPendin= >g (s)=3B
>=3B
>=3B (* recompile any interfaces where we = >goofed on the exports *)
>=3B --- 960=2C1000 ----
>=3B
>= >=3B (* compile all the sources using the initial schedule *)
>= >=3B FOR i :=3D 0 TO LAST (schedule^) DO
>=3B ! s.delayBack= >end :=3D TRUE=3B
>=3B ! TRY
>=3B ! CompileOne (s=2C= > schedule[i])=3B
>=3B ! FINALLY
>=3B ! s.delayBacke= >nd :=3D FALSE=3B
>=3B ! END=3B
>=3B !
>=3B ! s.= >machine.promises.addhi(NEW(MarkerPromise))=3B
>=3B !
>=3B ! = >END=3B
>=3B !
>=3B ! VAR
>=3B ! curSeq :=3D NEW(= >RefSeq.T).init()=3B
>=3B ! threads :=3D NEW(RefSeq.T).init()=3B<= >br>>=3B ! BEGIN
>=3B ! FOR i :=3D 0 TO s.machine.promises.= >size()-1 DO
>=3B ! WITH p =3D s.machine.promises.get(i) DO
= >>=3B ! curSeq.addhi(p)=3B
>=3B ! IF i =3D s.mach= >ine.promises.size()-1 OR ISTYPE(p=2CMarkerPromise) THEN
>=3B ! = > WITH cl =3D NEW(SeqClosure=2C seq :=3D curSeq) DO
>=3B ! = > threads.addhi (Thread.Fork(cl))=3B
>=3B !
>=3B ! = > IF threads.size() >=3B 10 THEN EVAL Thread.Join(threads.remlo()) END= >=3B
>=3B !
>=3B ! curSeq :=3D NEW(Re= >fSeq.T).init()
>=3B ! END
>=3B ! END
>= >=3B ! END
>=3B ! END=3B
>=3B ! WHILE threads.= >size() >=3B 0 DO EVAL Thread.Join(threads.remlo()) END=3B
>=3B = > END=3B
>=3B +
>=3B + EVAL s.machine.promises.in= >it()=3B
>=3B +
>=3B +
>=3B FlushPending (s)=3B
&g= >t=3B
>=3B (* recompile any interfaces where we goofed on the = >exports *)
>=3B ***************
>=3B *** 1151=2C1156 ****
>= >=3B --- 1203=2C1227 ----
>=3B END=3B
>=3B END CompileM3= >=3B
>=3B
>=3B + TYPE
>=3B + NotePromise =3D QMachine.Pr= >omise OBJECT
>=3B + nam : Pathname.T=3B
>=3B + OVERRIDES>>=3B + fulfil :=3D FulfilNP=3B
>=3B + END=3B
>=3B +
= >>=3B + RemovePromise =3D QMachine.Promise OBJECT
>=3B + nam : = >Pathname.T=3B
>=3B + OVERRIDES
>=3B + fulfil :=3D FulfilRP= >=3B
>=3B + END=3B
>=3B +
>=3B + PROCEDURE FulfilNP(np : N= >otePromise) =3D
>=3B + BEGIN Utils.NoteTempFile(np.nam) END FulfilN= >P=3B
>=3B +
>=3B + PROCEDURE FulfilRP(rp : RemovePromise) =3D r>>=3B + BEGIN Utils.Remove(rp.nam) END FulfilRP=3B
>=3B +
>= >=3B PROCEDURE PushOneM3 (s: State=3B u: M3Unit.T): BOOLEAN =3D
>=3B= > VAR
>=3B tmpC=2C tmpS: TEXT=3B
>=3B ***************>>=3B *** 1191=2C1208 ****
>=3B
>=3B
>=3B | 3 = >=3D>=3B (* -bootstrap=2C +m3back=2C +asm *)
>=3B tmpC := >=3D TempCName (u)=3B
>=3B tmpS :=3D TempSName (u)=3B
>= >=3B IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3Br>>=3B IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END= >=3B
>=3B IF RunM3 (s=2C u=2C tmpC) THEN
>=3B ! = > IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.optimize)
>=3B ! = > AND RunAsm (s=2C tmpS=2C u.object) THEN
>=3B END= >=3B
>=3B need_merge :=3D TRUE=3B
>=3B END= >=3B
>=3B IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END= >=3B
>=3B IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END= >=3B
>=3B
>=3B | 6=2C (* +bootstrap=2C +m3back=2C -asm= > *)
>=3B 7 =3D>=3B (* +bootstrap=2C +m3back=2C +asm *)
&= >gt=3B --- 1262=2C1318 ----
>=3B
>=3B
>=3B | 3 = >=3D>=3B (* -bootstrap=2C +m3back=2C +asm *)
>=3B + IF s.delay= >Backend THEN
>=3B tmpC :=3D TempCName (u)=3B
>=3B = > tmpS :=3D TempSName (u)=3B
>=3B + (*
>=3B IF (NOT = >s.keep_files) THEN Utils.NoteTempFile (tmpC) END=3B
>=3B IF = >(NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END=3B
>=3B + *)
= >>=3B + IF (NOT s.keep_files) THEN
>=3B + s.machin= >e.promises.addhi(NEW(NotePromise=2C nam :=3D tmpC))
>=3B + EN= >D=3B
>=3B + IF (NOT s.keep_files) THEN
>=3B + = >s.machine.promises.addhi(NEW(NotePromise=2C nam :=3D tmpS))
>=3B + = > END=3B
>=3B +
>=3B IF RunM3 (s=2C u=2C tmpC) THE= >N
>=3B ! s.machine.record(TRUE)=3B
>=3B ! TRY= >
>=3B ! IF RunM3Back (s=2C tmpC=2C tmpS=2C u.debug=2C u.o= >ptimize)
>=3B ! AND RunAsm (s=2C tmpS=2C u.object) THEN>>=3B ! END=3B
>=3B ! FINALLY
>=3B ! = > s.machine.record(FALSE)
>=3B END=3B
>=3B + r>>=3B need_merge :=3D TRUE=3B
>=3B END=3B
= >>=3B + (*
>=3B IF (NOT s.keep_files) THEN Utils.Remove (tm= >pC) END=3B
>=3B IF (NOT s.keep_files) THEN Utils.Remove (tmp= >S) END=3B
>=3B + *)
>=3B + IF (NOT s.keep_files) THEN >>=3B + s.machine.promises.addhi(NEW(RemovePromise=2C nam :=3D = >tmpC))
>=3B + END=3B
>=3B + IF (NOT s.keep_files= >) THEN
>=3B + s.machine.promises.addhi(NEW(RemovePromise=2C= > nam :=3D tmpS))
>=3B + END=3B
>=3B +
>=3B + = > ELSE
>=3B + tmpC :=3D TempCName (u)=3B
>=3B + = > tmpS :=3D TempSName (u)=3B
>=3B + IF (NOT s.keep_files) TH= >EN Utils.NoteTempFile (tmpC) END=3B
>=3B + IF (NOT s.keep_fi= >les) THEN Utils.NoteTempFile (tmpS) END=3B
>=3B + IF RunM3 (= >s=2C u=2C tmpC) THEN
>=3B + IF RunM3Back (s=2C tmpC=2C tm= >pS=2C u.debug=2C u.optimize)
>=3B + AND RunAsm (s=2C tmpS= >=2C u.object) THEN
>=3B + END=3B
>=3B + n= >eed_merge :=3D TRUE=3B
>=3B + END=3B
>=3B + I= >F (NOT s.keep_files) THEN Utils.Remove (tmpC) END=3B
>=3B + = >IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END=3B
>=3B + END= >
>=3B
>=3B | 6=2C (* +bootstrap=2C +m3back=2C -asm *)= >
>=3B 7 =3D>=3B (* +bootstrap=2C +m3back=2C +asm *)
>= >=3B Index: cm3/src/m3makefile
>=3B =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D
>=3B RCS file: /usr/cvs/cm3/m3-sys/cm3/src/m3makefi= >le=2Cv
>=3B retrieving revision 1.24
>=3B diff -c -r1.24 m3makefi= >le
>=3B *** cm3/src/m3makefile 8 Dec 2010 07:30:57 -0000 1.24
>>= >=3B --- cm3/src/m3makefile 13 Feb 2011 06:47:42 -0000
>=3B ********** >*= >****
>=3B *** 29=2C34 ****
>=3B --- 29=2C35 ----
>=3B end<= >br>>=3B module ("Arg")
>=3B module ("Builder")
>=3B modul= >e ("Dirs")
>=3B module ("M3Build")
>=3B module ("M3Loc")
&= >gt=3B Index: m3quake/src/QMachine.i3
>=3B =3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D
>=3B RCS file: /usr/cvs/cm3/m3-sys/m3quake/sr= >c/QMachine.i3=2Cv
>=3B retrieving revision 1.6
>=3B diff -c -r1.6= > QMachine.i3
>=3B *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -00 >0= >0 1.6
>=3B --- m3quake/src/QMachine.i3 13 Feb 2011 06:47:43 -0 >000
&= >gt=3B ***************
>=3B *** 8=2C13 ****
>=3B --- 8=2C14 ----r>>=3B
>=3B IMPORT Thread=2C Wr=2C QValue=2C QCode=3B
>=3B= > FROM Quake IMPORT Machine=2C Error=2C ID=2C IDMap=3B
>=3B + IMPORT = >RefSeq=3B
>=3B
>=3B REVEAL
>=3B T <=3B: T_=3B
= >>=3B ***************
>=3B *** 15=2C20 ****
>=3B --- 16=2C22 ---= >-
>=3B T =3D Machine=3B
>=3B T_ =3D OBJECT
>=3B = > map: IDMap :=3D NIL=3B (* READONLY *)
>=3B + promises : RefSeq.T= >=3B
>=3B METHODS
>=3B init (map: IDMap): T=3B
&= >gt=3B evaluate (s: QCode.Stream) RAISES {Error= >=2C Thread.Alerted}=3B
>=3B ***************
>=3B *** 37=2C42 ****= >
>=3B --- 39=2C46 ----
>=3B set_wr (wr: Wr.T)=3B
>= >=3B exec_echo (b: BOOLEAN): BOOLEAN=3B
>=3B trace (b: = >BOOLEAN)=3B
>=3B +
>=3B + record(on : BOOLEAN)=3B (*= > instead of performing certain acts=2C promise *)
>=3B END=3B
&= >gt=3B
>=3B PROCEDURE PushBool (t: T=3B b: BOOLEAN)=3B
>=3B = >***************
>=3B *** 51=2C54 ****
>=3B --- 55=2C63 ----
&g= >t=3B
>=3B PROCEDURE GetEnv (default=2C v0=2C v1=2C v2=2C v3=2C v4= >: TEXT :=3D NIL): TEXT=3B
>=3B
>=3B + TYPE Promise =3D OBJECT = >METHODS fulfil() RAISES { Error } END=3B
>=3B +
>=3B END QMac= >hine.
>=3B +
>=3B +
>=3B +
>=3B Index: m3quake/src/Q= >Machine.m3
>=3B =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D>>=3B RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3=2Cv
>=3B= > retrieving revision 1.35
>=3B diff -c -r1.35 QMachine.m3
>=3B **= >* m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35
>=3B >--- m3qu= >ake/src/QMachine.m3 13 Feb 2011 06:47:44 -0000
>=3B ***************
= >>>=3B *** 16=2C21 ****
>=3B --- 16=2C22 ----
>=3B IMPORT Text= >Utils=2C FSUtils=2C System=2C DirStack=3B (* sysutils *)
>=3B IMPORT= > Compiler=3B
>=3B IMPORT M3Path=3B
>=3B + IMPORT RefSeq=3B
&= >gt=3B
>=3B CONST
>=3B OnUnix =3D (Compiler.ThisOS =3D Co= >mpiler.OS.POSIX)=3B
>=3B ***************
>=3B *** 44=2C49 ****>>=3B --- 45=2C52 ----
>=3B shell : TEXT :=3D NIL= >=3B
>=3B sh_option : TEXT :=3D NIL=3B
>=3B tm= >p_dir : TEXT :=3D NIL=3B
>=3B +
>=3B + doRecord := >=3D FALSE=3B
>=3B OVERRIDES
>=3B init :=3D Init=3B= >
>=3B evaluate :=3D Evaluate=3B
>=3B ***************
&g= >t=3B *** 66=2C73 ****
>=3B --- 69=2C81 ----
>=3B set_wr = >:=3D SetWr=3B
>=3B exec_echo :=3D ExecEcho=3B
>=3B tr= >ace :=3D Trace=3B
>=3B +
>=3B + record :=3D Record=3B>>=3B END=3B
>=3B
>=3B + PROCEDURE Record(t : T=3B on : = >BOOLEAN) =3D
>=3B + BEGIN t.doRecord :=3D on END Record=3B
>= >=3B +
>=3B TYPE
>=3B Registers =3D RECORD
>=3B = >cp : QCode.Stream :=3D NIL=3B (* code pointer *)
>=3B **************= >*
>=3B *** 139=2C144 ****
>=3B --- 147=2C154 ----
>=3B = > t.globals :=3D NEW (IntRefTbl.Default).init ()=3B
>=3B t.def= >ault_wr :=3D Stdio.stdout=3B
>=3B
>=3B + t.promises :=3D N= >EW(RefSeq.T).init()=3B
>=3B +
>=3B InitOSEnv (t)=3B
>= >=3B InitBuiltins (t)=3B
>=3B
>=3B ***************
>= >=3B *** 1555=2C1564 ****
>=3B END=3B
>=3B E= >LSE
>=3B FlushIO ()=3B
>=3B ! Process.GetSt= >andardFileHandles (stdin=2C stdout=2C stderr)=3B
>=3B ! hand= >le :=3D Process.Create (t.shell=2C SUBARRAY (args=2C 0=2C n_shell_args)=2C<= >br>>=3B ! stdin :=3D stdin=2C stdout = >:=3D stdout=2C
>=3B ! stderr :=3D = >stderr)=3B
>=3B END=3B
>=3B EXCEPT
>=3B = > | Thread.Alerted =3D>=3B
>=3B --- 1565=2C1594 ----
>=3B = > END=3B
>=3B ELSE
>=3B FlushIO ()= >=3B
>=3B ! IF t.doRecord THEN
>=3B ! handle= > :=3D NIL=3B
>=3B ! WITH a =3D NEW(REF ARRAY OF TEXT=2C n_= >shell_args) DO
>=3B ! a^ :=3D SUBARRAY(args=2C0=2Cn_shel= >l_args)=3B
>=3B ! VAR wrx : Wr.T=3B BEGIN
>=3B ! = > IF echo OR t.do_echo THEN
>=3B ! wrx := >=3D wr
>=3B ! ELSE
>=3B ! wrx := >=3D NIL
>=3B ! END=3B
>=3B ! t.pr= >omises.addhi(NEW(ExecPromise=2C
>=3B ! = > cmd :=3D t.shell=2C
>=3B ! = >wr :=3D wrx=2C
>=3B ! args :=3D a= >=2C
>=3B ! t :=3D t=2C
>=3B = >! ignore_errors :=3D ignore_errors))r>>=3B ! END
>=3B ! END
>=3B ! = > ELSE
>=3B ! Process.GetStandardFileHandles (stdin=2C s= >tdout=2C stderr)=3B
>=3B ! handle :=3D Process.Create (t.s= >hell=2C SUBARRAY (args=2C 0=2C n_shell_args)=2C
>=3B ! = > stdin :=3D stdin=2C stdout :=3D stdout=2C
>=3B = >! stderr :=3D stderr)=3B
>=3B ! = > END=3B
>=3B END=3B
>=3B EXCEPT
>= >=3B | Thread.Alerted =3D>=3B
>=3B ***************
>=3B = >*** 1573=2C1579 ****
>=3B END=3B
>=3B
>=3B = > (* wait for everything to shutdown... *)
>=3B ! exit_code :=3D= > Process.Wait (handle)=3B
>=3B END=3B
>=3B
>=3B = > IF onlyTry THEN
>=3B --- 1603=2C1613 ----
>=3B END=3B<= >br>>=3B
>=3B (* wait for everything to shutdown... *)
= >>=3B ! IF handle =3D NIL THEN
>=3B ! exit_code :=3D 0 = >
>=3B ! ELSE
>=3B ! exit_code :=3D Process.Wait (ha= >ndle)=3B
>=3B ! END=3B (* else we're only promising *)
>=3B= > END=3B
>=3B
>=3B IF onlyTry THEN
>=3B ******= >*********
>=3B *** 1589=2C1594 ****
>=3B --- 1623=2C1664 ----
= >>=3B
>=3B END ExecCommand=3B
>=3B
>=3B + TYPE r>>=3B + ExecPromise =3D Promise OBJECT
>=3B + cmd : TEXT=3Br>>=3B + args : REF ARRAY OF TEXT=3B
>=3B + t : T=3B
>= >=3B + wr : Wr.T=3B
>=3B + ignore_errors : BOOLEAN=3B
>=3B= > + OVERRIDES
>=3B + fulfil :=3D FulfilExecPromise=3B
>=3B += > END=3B
>=3B +
>=3B + PROCEDURE FulfilExecPromise(ep : ExecPro= >mise) RAISES { Error } =3D
>=3B + VAR
>=3B + stdin=2C stdo= >ut=2C stderr: File.T=3B
>=3B + BEGIN
>=3B + Process.GetStan= >dardFileHandles (stdin=2C stdout=2C stderr)=3B
>=3B + TRY
>= >=3B + IF ep.wr # NIL THEN
>=3B + Wr.PutText (ep.wr=2C ep= >.args[1])=3B
>=3B + Wr.PutText (ep.wr=2C Wr.EOL)=3B
>=3B = >+ FlushIO ()=3B
>=3B + END=3B
>=3B + WITH han= >dle =3D Process.Create (ep.cmd=2C ep.args^=2C
>=3B + = > stdin :=3D stdin=2C stdout :=3D stdout=2C
>=3B + = > stderr :=3D stderr)=2C
>=3B + = > exit_code =3D Process.Wait(handle) DO
>=3B + IF exit_code= > # 0 AND NOT ep.ignore_errors THEN
>=3B + Err (ep.t=2C Fmt.F= >("exit %s: %s"=2C Fmt.Int(exit_code)=2C ep.cmd))
>=3B + END>>=3B + END
>=3B + EXCEPT
>=3B + OSError.E (ec)= > =3D>=3B
>=3B + Err (ep.t=2C Fmt.F ("exec failed%s *** %s"= >=2C OSErr (ec)=2C ep.cmd))=3B
>=3B + END
>=3B + END FulfilE= >xecPromise=3B
>=3B +
>=3B PROCEDURE KillProcess (handle: Proce= >ss.T) =3D
>=3B BEGIN
>=3B IF (handle # NIL) THEN
>= >=3B
>= > >--_b09106d1-2a84-4ffe-876c-0b2e4cdfd0b0_-- From hosking at cs.purdue.edu Sun Feb 13 21:18:03 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:18:03 -0500 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110212204406.88A661A2078@async.async.caltech.edu> References: <20110212204406.88A661A2078@async.async.caltech.edu> Message-ID: Hi Mika, Why was this change needed (I haven't looked closely)? I don't think we diverged much from previous implementations of user threads, which presumably used to work for you. On Feb 12, 2011, at 3:44 PM, Mika Nystrom wrote: > > Index: ThreadPosix.i3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v > retrieving revision 1.16 > diff -c -r1.16 ThreadPosix.i3 > *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 > --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 > *************** > *** 39,42 **** > --- 39,45 ---- > > (*---------------------------------------------------------------------------*) > > + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> > + PROCEDURE ValueOfSIGCHLD(): int; > + > END ThreadPosix. > Index: ThreadPosix.m3 > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v > retrieving revision 1.73 > diff -c -r1.73 ThreadPosix.m3 > *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 > --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 > *************** > *** 97,102 **** > --- 97,105 ---- > (* if state = pausing, the time at which we can restart *) > waitingForTime: Time.T; > > + (* if state = pausing, the signal that truncates the pause *) > + waitingForSig: int := -1; > + > (* true if we are waiting during an AlertWait or AlertJoin > or AlertPause *) > alertable: BOOLEAN := FALSE; > *************** > *** 147,152 **** > --- 150,160 ---- > > defaultStackSize := 3000; > > + (* note that even though the "heavy machinery" is only used for > + multipleThreads, we still need to set up the signal handler so > + that we can catch signals from other sources than thread switching. > + e.g., we use SIGCHLD to speed up Process.Wait *) > + > VAR > stats: RECORD > n_forks := 0; > *************** > *** 511,527 **** > XPause(until, FALSE); > END Pause; > > PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= > VAR until := n + Time.Now (); > BEGIN > XPause(until, TRUE); > END AlertPause; > > ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE) RAISES {Alerted} = > BEGIN > INC (inCritical); > self.waitingForTime := until; > self.alertable := alertable; > ICannotRun (State.pausing); > DEC (inCritical); > InternalYield (); > --- 519,546 ---- > XPause(until, FALSE); > END Pause; > > + PROCEDURE SigPause(n: LONGREAL; sig: int)= > + <*FATAL Alerted*> > + VAR until := n + Time.Now (); > + BEGIN > + XPause(until, FALSE, sig); > + END SigPause; > + > PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= > VAR until := n + Time.Now (); > BEGIN > XPause(until, TRUE); > END AlertPause; > > ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) > ! RAISES {Alerted} = > BEGIN > INC (inCritical); > self.waitingForTime := until; > self.alertable := alertable; > + IF sig # -1 THEN > + self.waitingForSig := sig > + END; > ICannotRun (State.pausing); > DEC (inCritical); > InternalYield (); > *************** > *** 703,712 **** > END; > END StartSwitching; > > ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); > ! IF inCritical = 0 AND heapState.inCritical = 0 THEN InternalYield () END; > END switch_thread; > > (*------------------------------------------------------------- scheduler ---*) > --- 722,750 ---- > END; > END StartSwitching; > > ! CONST MaxSigs = 64; > ! TYPE Sig = [ 0..MaxSigs-1 ]; > ! > ! (* in order to listen to other signals, they have to be enabled in > ! allow_sigvtalrm as well *) > ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); > ! > ! gotSigs := SET OF Sig { }; > ! > ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); > ! > ! INC(inCritical); > ! (* mark signal as being delivered *) > ! IF sig >= 0 AND sig < MaxSigs THEN > ! gotSigs := gotSigs + SET OF Sig { sig } > ! END; > ! DEC(inCritical); > ! > ! IF inCritical = 0 AND heapState.inCritical = 0 THEN > ! InternalYield () > ! END; > END switch_thread; > > (*------------------------------------------------------------- scheduler ---*) > *************** > *** 782,792 **** > IF t.alertable AND t.alertPending THEN > CanRun (t); > EXIT; > > ELSIF t.waitingForTime <= now THEN > CanRun (t); > EXIT; > ! > ELSIF NOT somePausing THEN > earliest := t.waitingForTime; > somePausing := TRUE; > --- 820,835 ---- > IF t.alertable AND t.alertPending THEN > CanRun (t); > EXIT; > + > + ELSIF t.waitingForSig IN gotSigs THEN > + t.waitingForSig := -1; > + CanRun(t); > + EXIT; > > ELSIF t.waitingForTime <= now THEN > CanRun (t); > EXIT; > ! > ELSIF NOT somePausing THEN > earliest := t.waitingForTime; > somePausing := TRUE; > *************** > *** 886,891 **** > --- 929,936 ---- > END; > END; > > + gotSigs := SET OF Sig {}; > + > IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN > IF perfOn THEN PerfRunning (t.id); END; > (* At least one thread wants to run; transfer to it *) > *************** > *** 948,960 **** > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > ! CONST Delay = 0.1D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > ! Pause(Delay); > END; > END WaitProcess; > > --- 993,1005 ---- > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > ! CONST Delay = 10.0D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > ! SigPause(Delay,SIGCHLD); > END; > END WaitProcess; > > *************** > *** 1361,1364 **** > --- 1406,1412 ---- > VAR debug := RTParams.IsPresent ("debugthreads"); > > BEGIN > + (* we need to call set up the signal handler for other reasons than > + just thread switching now *) > + setup_sigvtalrm (switch_thread); > END ThreadPosix. > Index: ThreadPosixC.c > =================================================================== > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v > retrieving revision 1.59 > diff -c -r1.59 ThreadPosixC.c > *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 > --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 > *************** > *** 78,88 **** > --- 78,97 ---- > > sigemptyset(&ThreadSwitchSignal); > sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); > + sigaddset(&ThreadSwitchSignal, SIGCHLD); > > act.sa_handler = handler; > act.sa_flags = SA_RESTART; > sigemptyset(&(act.sa_mask)); > if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); > + if (sigaction (SIGCHLD, &act, NULL)) abort(); > + } > + > + int > + __cdecl > + ThreadPosix__value_of_SIGCHLD(void) > + { > + return SIGCHLD; > } > > void From hosking at cs.purdue.edu Sun Feb 13 21:22:13 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:22:13 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110212064812.286E61A2078@async.async.caltech.edu> Message-ID: <35125A4E-2BB1-48B9-87EC-58CFCFA5D5AF@cs.purdue.edu> Looks like stack overflow to me. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 12, 2011, at 9:09 AM, Jay K wrote: > > >LINUXLIBC6 : seg fault > > I've seen this now. > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0xb3200b90 (LWP 18154)] > 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) > at ../src/runtime/common/RTCollector.m3:409 > 409 IF hdr.typecode = RT0.TextLitTypecode THEN RETURN END; > 1: x/i $pc > 0xb6dee58f : mov (%eax),%eax > (gdb) bt 9 > #0 0xb6dee58f in RTCollector__Move (self=0xb6a9000c, cp=0xb6a20014) > at ../src/runtime/common/RTCollector.m3:409 > #1 0xb6deaa46 in RTHeapMap__Walk (x=0xb6a20014, pc=0xb7738648, v=0xb6a9000c) > at ../src/runtime/common/RTHeapMap.m3:202 > #2 0xb6dea341 in RTHeapMap__DoWalkRef (t=0xb7738694, a=0xb6a20014, > v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:62 > #3 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739c54, a=0xb6a2000c, > v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 > #4 0xb6dea318 in RTHeapMap__DoWalkRef (t=0xb7739e68, a=0xb6a2000c, > v=0xb6a9000c) at ../src/runtime/common/RTHeapMap.m3:57 > #5 0xb6dea2bf in RTHeapMap__WalkRef (h=0xb6a20008, v=0xb6a9000c) > at ../src/runtime/common/RTHeapMap.m3:47 > #6 0xb6df0636 in RTCollector__CleanBetween (h=0xb6a20008, he=0xb6a30000, > clean=0 '\0') at ../src/runtime/common/RTCollector.m3:1091 > #7 0xb6df0473 in RTCollector__CleanPage (page=0xb6a20000) > at ../src/runtime/common/RTCollector.m3:1064 > #8 0xb6defc96 in RTCollector__CollectSomeInStateZero () > at ../src/runtime/common/RTCollector.m3:885 > (More stack frames follow...) > (gdb) p $eax > $2 = 2097148 > (gdb) p/x $eax > $3 = 0x1ffffc > (gdb) x/x $eax+4 > 0x200000: Cannot access memory at address 0x200000 > (gdb) x/x $eax > 0x1ffffc: Cannot access memory at address 0x1ffffc > > > Maybe I should try with a correctly sized jmpbuf to assure it isn't stack overflow? > But notice that the next page doesn't work either. Actually I tried adding > a lot to $eax and still not accessible. > > > Maybe maybe maybe maybe I'll dig into this. But I'm very busy lately and maybe indefinitely into future.. > > > Thanks, > - Jay > > > > To: jay.krell at cornell.edu; hosking at cs.purdue.edu > > Date: Fri, 11 Feb 2011 22:48:12 -0800 > > From: mika at async.caltech.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] threading on Windows? > > > > Well, with the following changes: re-bootstrapping on the two Linux > > systems and removal of the -pthread flag on FreeBSD4, all is well. > > User threading works on all the architectures listed below, at least > > as far as the thread testing program can test it. (It seems that because > > of the way pthread_atfork is coded, pthread is not needed at all on > > FreeBSD4.) > > > > Mika > > > > Mika Nystrom writes: > > >Another odd datapoint is that even rather aggressive testing seems not > > >to have any trouble on I386_DARWIN (same as AMD64_FREEBSD). With user > > >threads, of course. > > > > > >So status right now of user threads seems to be: > > > > > >FreeBSD4 : strange failure in libpthread as reported by gdb > > >LINUXLIBC6 : seg fault > > >AMD64_LINUX : seg fault > > >AMD64_FREEBSD : OK > > >I386_DARWIN : OK -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Sun Feb 13 21:32:05 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:32:05 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: <20110213003358.00D2A1A2078@async.async.caltech.edu> References: , <161929.78691.qm@web29701.mail.ird.yahoo.com> <20110212015715.A43471A2078@async.async.caltech.edu>, <8479B3DD-5BB3-4738-9F41-D2D88ED306CA@cs.purdue.edu> <20110213003358.00D2A1A2078@async.async.caltech.edu> Message-ID: I am confused. This backtrace looks like it is from the release, which we know has problems. On Feb 12, 2011, at 7:33 PM, Mika Nystrom wrote: > P.S. backtraces of deadlock: > > Thread 3 (Thread 0x40862950 (LWP 19974)): > #0 0x00002b9d49f730e1 in nanosleep () from /lib/libpthread.so.0 > #1 0x000000000045806d in ThreadPThread__Nanosleep (req=Cannot access memory at address 0x140861b08 > ) at ../src/thread/PTHREAD/ThreadPThreadC.c:500 > #2 0x0000000000454294 in ThreadPThread__CommonSleep () at ../src/thread/PTHREAD/ThreadPThread.m3:551 > #3 0x0000000000456438 in ThreadPThread__StopWorld () at ../src/thread/PTHREAD/ThreadPThread.m3:1086 > #4 0x0000000000455787 in RTThread__SuspendOthers () at ../src/thread/PTHREAD/ThreadPThread.m3:812 > #5 0x000000000043970c in RTCollector__CollectSomeInStateZero () at ../src/runtime/common/RTCollector.m3:746 > #6 0x00000000004396b8 in RTCollector__CollectSome () at ../src/runtime/common/RTCollector.m3:720 > #7 0x00000000004390b9 in RTHeapRep__CollectEnough () at ../src/runtime/common/RTCollector.m3:654 > #8 0x0000000000436111 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=Cannot access memory at address 0x140861dc8 > ) at ../src/runtime/common/RTAllocator.m3:363 > #9 0x000000000043596f in RTAllocator__GetOpenArray (M3_Eic7CK_def=Cannot access memory at address 0x140861eb8 > ) at ../src/runtime/common/RTAllocator.m3:292 > #10 0x0000000000434ab1 in RTHooks__AllocateOpenArray (M3_AJWxb1_defn=Cannot access memory at address 0x140861f38 > ) at ../src/runtime/common/RTAllocator.m3:141 > #11 0x0000000000407030 in Main__AApply (M3_AP7a1g_cl=Cannot access memory at address 0x140861f88 > ) at ../src/Main.m3:234 > #12 0x0000000000453b3f in ThreadPThread__RunThread (M3_DMxDjQ_me=Cannot access memory at address 0x140862028 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:475 > #13 0x00000000004537d3 in ThreadPThread__ThreadBase (M3_AJWxb1_param=Cannot access memory at address 0x1408620d8 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:447 > #14 0x00002b9d49f6bfc7 in start_thread () from /lib/libpthread.so.0 > #15 0x00002b9d4b3bd64d in clone () from /lib/libc.so.6 > #16 0x0000000000000000 in ?? () > > Thread 2 (Thread 0x40661950 (LWP 19973)): > #0 0x00002b9d49f72384 in __lll_lock_wait () from /lib/libpthread.so.0 > #1 0x00002b9d49f6dbf0 in _L_lock_102 () from /lib/libpthread.so.0 > #2 0x00002b9d49f6d4fe in pthread_mutex_lock () from /lib/libpthread.so.0 > #3 0x000000000045815e in ThreadPThread__pthread_mutex_lock (m=Cannot access memory at address 0x140660d68 > ) at ../src/thread/PTHREAD/ThreadPThreadC.c:557 > #4 0x00000000004575af in RTOS__LockHeap () at ../src/thread/PTHREAD/ThreadPThread.m3:1434 > #5 0x0000000000436107 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=Cannot access memory at address 0x140660dc8 > ) at ../src/runtime/common/RTAllocator.m3:361 > #6 0x000000000043596f in RTAllocator__GetOpenArray (M3_Eic7CK_def=Cannot access memory at address 0x140660eb8 > ) at ../src/runtime/common/RTAllocator.m3:292 > #7 0x0000000000434ab1 in RTHooks__AllocateOpenArray (M3_AJWxb1_defn=Cannot access memory at address 0x140660f38 > ) at ../src/runtime/common/RTAllocator.m3:141 > #8 0x0000000000407030 in Main__AApply (M3_AP7a1g_cl=Cannot access memory at address 0x140660f88 > ) at ../src/Main.m3:234 > #9 0x0000000000453b3f in ThreadPThread__RunThread (M3_DMxDjQ_me=Cannot access memory at address 0x140661028 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:475 > #10 0x00000000004537d3 in ThreadPThread__ThreadBase (M3_AJWxb1_param=Cannot access memory at address 0x1406610d8 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:447 > #11 0x00002b9d49f6bfc7 in start_thread () from /lib/libpthread.so.0 > #12 0x00002b9d4b3bd64d in clone () from /lib/libc.so.6 > #13 0x0000000000000000 in ?? () > > Thread 1 (Thread 0x2b9d4d32a1a0 (LWP 19965)): > #0 0x00002b9d4b3202a2 in sigsuspend () from /lib/libc.so.6 > #1 0x0000000000457ba9 in ThreadPThread__sigsuspend () at ../src/thread/PTHREAD/ThreadPThreadC.c:141 > #2 0x0000000000456a0d in ThreadPThread__SignalHandler (M3_DLS2Hj_sig=Cannot access memory at address 0x80001dde057c > ) at ../src/thread/PTHREAD/ThreadPThread.m3:1205 > #3 > #4 0x00002b9d49f6ffad in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 > #5 0x00000000004580b3 in ThreadPThread__pthread_cond_timedwait (cond=Cannot access memory at address 0x80001dde0a88 > ) at ../src/thread/PTHREAD/ThreadPThreadC.c:515 > #6 0x00000000004544cf in ThreadPThread__XPause (M3_DMxDjQ_self=Cannot access memory at address 0x80001dde0ab8 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:580 > #7 0x00000000004545b2 in Thread__Pause (M3_CtKayy_n=Cannot access memory at address 0x80001dde0b08 > ) at ../src/thread/PTHREAD/ThreadPThread.m3:595 > #8 0x00000000004097ee in Main_M3 (M3_AcxOUs_mode=Cannot access memory at address 0x80001dde0bc8 > ) at ../src/Main.m3:449 > #9 0x0000000000445799 in RTLinker__RunMainBody (M3_DjPxE3_m=Cannot access memory at address 0x80001dde0f58 > ) at ../src/runtime/common/RTLinker.m3:406 > #10 0x00000000004448f2 in RTLinker__AddUnitI (M3_DjPxE3_m=Cannot access memory at address 0x80001dde0ff8 > ) at ../src/runtime/common/RTLinker.m3:113 > #11 0x0000000000444990 in RTLinker__AddUnit (M3_DjPxE5_b=Cannot access memory at address 0x80001dde1018 > ) at ../src/runtime/common/RTLinker.m3:122 > #12 0x000000000040652c in main (argc=Cannot access memory at address 0x80001dde1068 > ) at _m3main.mc:4 From mika at async.caltech.edu Sun Feb 13 21:37:26 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 12:37:26 -0800 Subject: [M3devel] SIGCHLD diff In-Reply-To: References: <20110212204406.88A661A2078@async.async.caltech.edu> Message-ID: <20110213203726.82ADB1A2078@async.async.caltech.edu> Tony, The old user threads has an old problem with Process.Wait. It has a Thread.Pause(0.1d0) in it. This causes the compiler to run three times slower than it should. The m3build I've been using for years has that Thread.Pause removed (so it uses 100% CPU instead, busy-waiting...) But yeah user threads works the same as it used to (except for now calling pthread_atfork, which I'll return to in another email...) Mika Tony Hosking writes: >Hi Mika, Why was this change needed (I haven't looked closely)? I don't = >think we diverged much from previous implementations of user threads, = >which presumably used to work for you. > >On Feb 12, 2011, at 3:44 PM, Mika Nystrom wrote: > >>=20 >> Index: ThreadPosix.i3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: = >/usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v >> retrieving revision 1.16 >> diff -c -r1.16 ThreadPosix.i3 >> *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 >> --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 >> *************** >> *** 39,42 **** >> --- 39,45 ---- >>=20 >> = >(*------------------------------------------------------------------------= >---*) >>=20 >> + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> >> + PROCEDURE ValueOfSIGCHLD(): int; >> +=20 >> END ThreadPosix. >> Index: ThreadPosix.m3 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: = >/usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v >> retrieving revision 1.73 >> diff -c -r1.73 ThreadPosix.m3 >> *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 >> --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 >> *************** >> *** 97,102 **** >> --- 97,105 ---- >> (* if state =3D pausing, the time at which we can restart *) >> waitingForTime: Time.T; >>=20 >> + (* if state =3D pausing, the signal that truncates the pause *) >> + waitingForSig: int :=3D -1; >> +=20 >> (* true if we are waiting during an AlertWait or AlertJoin=20 >> or AlertPause *) >> alertable: BOOLEAN :=3D FALSE; >> *************** >> *** 147,152 **** >> --- 150,160 ---- >>=20 >> defaultStackSize :=3D 3000; >>=20 >> + (* note that even though the "heavy machinery" is only used for=20 >> + multipleThreads, we still need to set up the signal handler so >> + that we can catch signals from other sources than thread = >switching. >> + e.g., we use SIGCHLD to speed up Process.Wait *) >> +=20 >> VAR >> stats: RECORD >> n_forks :=3D 0; >> *************** >> *** 511,527 **** >> XPause(until, FALSE); >> END Pause; >>=20 >> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >> VAR until :=3D n + Time.Now (); >> BEGIN >> XPause(until, TRUE); >> END AlertPause; >>=20 >> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D FALSE) = >RAISES {Alerted} =3D >> BEGIN >> INC (inCritical); >> self.waitingForTime :=3D until; >> self.alertable :=3D alertable; >> ICannotRun (State.pausing); >> DEC (inCritical); >> InternalYield (); >> --- 519,546 ---- >> XPause(until, FALSE); >> END Pause; >>=20 >> + PROCEDURE SigPause(n: LONGREAL; sig: int)=3D >> + <*FATAL Alerted*> >> + VAR until :=3D n + Time.Now (); >> + BEGIN >> + XPause(until, FALSE, sig); >> + END SigPause; >> +=20 >> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >> VAR until :=3D n + Time.Now (); >> BEGIN >> XPause(until, TRUE); >> END AlertPause; >>=20 >> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D FALSE; = >sig:int :=3D -1)=20 >> ! RAISES {Alerted} =3D >> BEGIN >> INC (inCritical); >> self.waitingForTime :=3D until; >> self.alertable :=3D alertable; >> + IF sig # -1 THEN >> + self.waitingForSig :=3D sig >> + END; >> ICannotRun (State.pausing); >> DEC (inCritical); >> InternalYield (); >> *************** >> *** 703,712 **** >> END; >> END StartSwitching; >>=20 >> ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} =3D >> BEGIN >> allow_sigvtalrm (); >> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN = >InternalYield () END; >> END switch_thread; >>=20 >> (*------------------------------------------------------------- = >scheduler ---*) >> --- 722,750 ---- >> END; >> END StartSwitching; >>=20 >> ! CONST MaxSigs =3D 64; >> ! TYPE Sig =3D [ 0..MaxSigs-1 ]; >> !=20 >> ! (* in order to listen to other signals, they have to be enabled in >> ! allow_sigvtalrm as well *) >> ! VAR (*CONST*) SIGCHLD :=3D ValueOfSIGCHLD(); >> ! =20 >> ! gotSigs :=3D SET OF Sig { }; >> !=20 >> ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >> BEGIN >> allow_sigvtalrm (); >> !=20 >> ! INC(inCritical); >> ! (* mark signal as being delivered *) >> ! IF sig >=3D 0 AND sig < MaxSigs THEN >> ! gotSigs :=3D gotSigs + SET OF Sig { sig } >> ! END; >> ! DEC(inCritical); >> !=20 >> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN=20 >> ! InternalYield ()=20 >> ! END; >> END switch_thread; >>=20 >> (*------------------------------------------------------------- = >scheduler ---*) >> *************** >> *** 782,792 **** >> IF t.alertable AND t.alertPending THEN >> CanRun (t); >> EXIT; >>=20 >> ELSIF t.waitingForTime <=3D now THEN >> CanRun (t); >> EXIT; >> ! =20 >> ELSIF NOT somePausing THEN >> earliest :=3D t.waitingForTime; >> somePausing :=3D TRUE; >> --- 820,835 ---- >> IF t.alertable AND t.alertPending THEN >> CanRun (t); >> EXIT; >> + =20 >> + ELSIF t.waitingForSig IN gotSigs THEN >> + t.waitingForSig :=3D -1; >> + CanRun(t); >> + EXIT; >>=20 >> ELSIF t.waitingForTime <=3D now THEN >> CanRun (t); >> EXIT; >> !=20 >> ELSIF NOT somePausing THEN >> earliest :=3D t.waitingForTime; >> somePausing :=3D TRUE; >> *************** >> *** 886,891 **** >> --- 929,936 ---- >> END; >> END; >>=20 >> + gotSigs :=3D SET OF Sig {}; >> +=20 >> IF t.state =3D State.alive AND (scanned OR NOT someBlocking) THEN >> IF perfOn THEN PerfRunning (t.id); END; >> (* At least one thread wants to run; transfer to it *) >> *************** >> *** 948,960 **** >>=20 >> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D >> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >> ! CONST Delay =3D 0.1D0; >> BEGIN >> LOOP >> WITH r =3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >> IF r # 0 THEN RETURN r END; >> END; >> ! Pause(Delay); >> END; >> END WaitProcess; >>=20 >> --- 993,1005 ---- >>=20 >> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D >> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >> ! CONST Delay =3D 10.0D0; >> BEGIN >> LOOP >> WITH r =3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >> IF r # 0 THEN RETURN r END; >> END; >> ! SigPause(Delay,SIGCHLD); >> END; >> END WaitProcess; >>=20 >> *************** >> *** 1361,1364 **** >> --- 1406,1412 ---- >> VAR debug :=3D RTParams.IsPresent ("debugthreads"); >>=20 >> BEGIN >> + (* we need to call set up the signal handler for other reasons = >than >> + just thread switching now *) >> + setup_sigvtalrm (switch_thread); >> END ThreadPosix. >> Index: ThreadPosixC.c >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> RCS file: = >/usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v >> retrieving revision 1.59 >> diff -c -r1.59 ThreadPosixC.c >> *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 >> --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 >> *************** >> *** 78,88 **** >> --- 78,97 ---- >>=20 >> sigemptyset(&ThreadSwitchSignal); >> sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); >> + sigaddset(&ThreadSwitchSignal, SIGCHLD); >>=20 >> act.sa_handler =3D handler; >> act.sa_flags =3D SA_RESTART; >> sigemptyset(&(act.sa_mask)); >> if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); >> + if (sigaction (SIGCHLD, &act, NULL)) abort(); >> + } >> +=20 >> + int >> + __cdecl >> + ThreadPosix__value_of_SIGCHLD(void) >> + { >> + return SIGCHLD; >> } >>=20 >> void From hosking at cs.purdue.edu Sun Feb 13 21:37:32 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:37:32 -0500 Subject: [M3devel] Performance issues with Process.Wait under userthreads In-Reply-To: <20110212204302.19C781A2078@async.async.caltech.edu> References: <20110212204302.19C781A2078@async.async.caltech.edu> Message-ID: <80DB3538-EAFF-4C03-A117-75EE49BB5D2C@cs.purdue.edu> This all seems pretty reasonable. On Feb 12, 2011, at 3:43 PM, Mika Nystrom wrote: > Hi again m3devel (especially Tony), > > I have finally taken a bite of a problem that has been annoying me > for a very, very long time. > > Under user threads, the code for Process.Wait is as follows (see > ThreadPosix.m3): > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > CONST Delay = 0.1D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > Pause(Delay); > END; > END WaitProcess; > > It inserts a 0.1 second delay after each failed waitpid. This is extremely > annoying for programs that start a long sequence of child processes and > wait for them in sequence. Namely, the compiler itself. As a result > the cm3 compiler (and PM3's m3build) are normally very very slow when > using user threads. For about the last ten years, I've had a hacked > up m3build (for my PM3 installation) that skips the Pause and busy-waits > instead. > > Note there is another problem here. Since the Modula-3 runtime ignores > SIGCHLD, no zombie processes are created since the Unix system automatically > reaps the child processes. I can see this would be a problem since PIDs > are eventually reused and ... couldn't Uexec.waitpid wind up referring to > the wrong process?? > > I will further note that the comment in Process.i3 reads as follows: > > PROCEDURE Wait(p: T): ExitCode; > > Wait until the process with handle p terminates, then free the operating system resources associated with the process and return an exit code indicating the reason for its termination. It is a checked runtime error to call Wait twice on the same process handle. > > I am going to take this as fair warning that Process.Create *may* use > resources that are not going to be released until Process.Wait has been > called. > > I have modified (in my local copy of CM3) the system as follows. > I have come up with a semi-general mechanism for immediately unblocking > a thread on the receipt of a unix signal. > > 1. The system relies on changing > ThreadPosix.XPause such that if a signal is allowed to wake up a threads, > that fact is recorded in a new field in the thread's descriptor > record (of type ThreadPosix.T). > > 2. On receipt of a waited-for unix signal, a mask is set and control > is passed to the thread scheduler which maintains the non-zero mask for > exactly one iteration through the thread ring. > > 3. If a thread is paused and waiting for EITHER a signal or some time, > the thread is released for running and the thread's waiting state is > cleared. > > The changes are more or less as follows: > > 1. I have added a new field of type "int" to ThreadPosix.T: > > (* if state = pausing, the time at which we can restart *) > waitingForTime: Time.T; > > + (* if state = pausing, the signal that truncates the pause *) > + waitingForSig: int := -1; > + > (* true if we are waiting during an AlertWait or AlertJoin > or AlertPause *) > alertable: BOOLEAN := FALSE; > > > 2. Modifications to pause: > > + PROCEDURE SigPause(n: LONGREAL; sig: int)= > + <*FATAL Alerted*> > + VAR until := n + Time.Now (); > + BEGIN > + XPause(until, FALSE, sig); > + END SigPause; > + > PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}= > VAR until := n + Time.Now (); > BEGIN > XPause(until, TRUE); > END AlertPause; > > ! PROCEDURE XPause (READONLY until: Time.T; alertable := FALSE; sig:int := -1) > ! RAISES {Alerted} = > BEGIN > INC (inCritical); > self.waitingForTime := until; > self.alertable := alertable; > + IF sig # -1 THEN > + self.waitingForSig := sig > + END; > ICannotRun (State.pausing); > DEC (inCritical); > InternalYield (); > > 3. The received-signals mask: > > ! CONST MaxSigs = 64; > ! TYPE Sig = [ 0..MaxSigs-1 ]; > ! > ! (* in order to listen to other signals, they have to be enabled in > ! allow_sigvtalrm as well *) > ! VAR (*CONST*) SIGCHLD := ValueOfSIGCHLD(); > ! > ! gotSigs := SET OF Sig { }; > ! > > ValueOfSIGCHLD() is a C function used to get the value of the SIGCHLD > constant without guessing at it (in ThreadPosixC.c). > > 4. changes to the signal handler: > > ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); > ! > ! INC(inCritical); > ! (* mark signal as being delivered *) > ! IF sig >= 0 AND sig < MaxSigs THEN > ! gotSigs := gotSigs + SET OF Sig { sig } > ! END; > ! DEC(inCritical); > ! > ! IF inCritical = 0 AND heapState.inCritical = 0 THEN > ! InternalYield () > ! END; > END switch_thread; > > Note that I don't know if INC/DEC(inCritical) does exactly the right > thing here. > > 5. changes to the scheduler: > > a. thread wakeup > IF t.alertable AND t.alertPending THEN > CanRun (t); > EXIT; > + > + ELSIF t.waitingForSig IN gotSigs THEN > + t.waitingForSig := -1; > + CanRun(t); > + EXIT; > > ELSIF t.waitingForTime <= now THEN > CanRun (t); > EXIT; > ! > > b. clearing the mask > > END; > END; > > + gotSigs := SET OF Sig {}; > + > IF t.state = State.alive AND (scanned OR NOT someBlocking) THEN > IF perfOn THEN PerfRunning (t.id); END; > (* At least one thread wants to run; transfer to it *) > > 6. changes to WaitProcess (Process.Wait): > > PROCEDURE WaitProcess (pid: int; VAR status: int): int = > (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) > ! CONST Delay = 10.0D0; > BEGIN > LOOP > WITH r = Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO > IF r # 0 THEN RETURN r END; > END; > ! SigPause(Delay,SIGCHLD); > END; > END WaitProcess; > > 7. install signal handler even if program is single-threaded: > > BEGIN > + (* we need to call set up the signal handler for other reasons than > + just thread switching now *) > + setup_sigvtalrm (switch_thread); > END ThreadPosix. > > 8. modify signal handler in ThreadPosixC.c to catch SIGCHLD: > > sigemptyset(&ThreadSwitchSignal); > sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); > + sigaddset(&ThreadSwitchSignal, SIGCHLD); > > act.sa_handler = handler; > act.sa_flags = SA_RESTART; > sigemptyset(&(act.sa_mask)); > if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); > + if (sigaction (SIGCHLD, &act, NULL)) abort(); > > I'll send the complete diff in a separate message for those who want > to study it more closely. > > I propose the above changes for inclusion in the current CM3 repository. > > Mika From hosking at cs.purdue.edu Sun Feb 13 21:44:11 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:44:11 -0500 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110213203726.82ADB1A2078@async.async.caltech.edu> References: <20110212204406.88A661A2078@async.async.caltech.edu> <20110213203726.82ADB1A2078@async.async.caltech.edu> Message-ID: Sorry, I missed you e-mail describing this change. Looks good. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: > Tony, > > The old user threads has an old problem with Process.Wait. It has a > Thread.Pause(0.1d0) in it. This causes the compiler to run three times > slower than it should. The m3build I've been using for years has > that Thread.Pause removed (so it uses 100% CPU instead, busy-waiting...) > > But yeah user threads works the same as it used to (except for > now calling pthread_atfork, which I'll return to in another email...) > > Mika > > Tony Hosking writes: >> Hi Mika, Why was this change needed (I haven't looked closely)? I don't = >> think we diverged much from previous implementations of user threads, = >> which presumably used to work for you. >> >> On Feb 12, 2011, at 3:44 PM, Mika Nystrom wrote: >> >>> =20 >>> Index: ThreadPosix.i3 >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> RCS file: = >> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v >>> retrieving revision 1.16 >>> diff -c -r1.16 ThreadPosix.i3 >>> *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 >>> --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 >>> *************** >>> *** 39,42 **** >>> --- 39,45 ---- >>> =20 >>> = >> (*------------------------------------------------------------------------= >> ---*) >>> =20 >>> + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> >>> + PROCEDURE ValueOfSIGCHLD(): int; >>> +=20 >>> END ThreadPosix. >>> Index: ThreadPosix.m3 >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> RCS file: = >> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v >>> retrieving revision 1.73 >>> diff -c -r1.73 ThreadPosix.m3 >>> *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 >>> --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 >>> *************** >>> *** 97,102 **** >>> --- 97,105 ---- >>> (* if state =3D pausing, the time at which we can restart *) >>> waitingForTime: Time.T; >>> =20 >>> + (* if state =3D pausing, the signal that truncates the pause *) >>> + waitingForSig: int :=3D -1; >>> +=20 >>> (* true if we are waiting during an AlertWait or AlertJoin=20 >>> or AlertPause *) >>> alertable: BOOLEAN :=3D FALSE; >>> *************** >>> *** 147,152 **** >>> --- 150,160 ---- >>> =20 >>> defaultStackSize :=3D 3000; >>> =20 >>> + (* note that even though the "heavy machinery" is only used for=20 >>> + multipleThreads, we still need to set up the signal handler so >>> + that we can catch signals from other sources than thread = >> switching. >>> + e.g., we use SIGCHLD to speed up Process.Wait *) >>> +=20 >>> VAR >>> stats: RECORD >>> n_forks :=3D 0; >>> *************** >>> *** 511,527 **** >>> XPause(until, FALSE); >>> END Pause; >>> =20 >>> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >>> VAR until :=3D n + Time.Now (); >>> BEGIN >>> XPause(until, TRUE); >>> END AlertPause; >>> =20 >>> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D FALSE) = >> RAISES {Alerted} =3D >>> BEGIN >>> INC (inCritical); >>> self.waitingForTime :=3D until; >>> self.alertable :=3D alertable; >>> ICannotRun (State.pausing); >>> DEC (inCritical); >>> InternalYield (); >>> --- 519,546 ---- >>> XPause(until, FALSE); >>> END Pause; >>> =20 >>> + PROCEDURE SigPause(n: LONGREAL; sig: int)=3D >>> + <*FATAL Alerted*> >>> + VAR until :=3D n + Time.Now (); >>> + BEGIN >>> + XPause(until, FALSE, sig); >>> + END SigPause; >>> +=20 >>> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D >>> VAR until :=3D n + Time.Now (); >>> BEGIN >>> XPause(until, TRUE); >>> END AlertPause; >>> =20 >>> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D FALSE; = >> sig:int :=3D -1)=20 >>> ! RAISES {Alerted} =3D >>> BEGIN >>> INC (inCritical); >>> self.waitingForTime :=3D until; >>> self.alertable :=3D alertable; >>> + IF sig # -1 THEN >>> + self.waitingForSig :=3D sig >>> + END; >>> ICannotRun (State.pausing); >>> DEC (inCritical); >>> InternalYield (); >>> *************** >>> *** 703,712 **** >>> END; >>> END StartSwitching; >>> =20 >>> ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} =3D >>> BEGIN >>> allow_sigvtalrm (); >>> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN = >> InternalYield () END; >>> END switch_thread; >>> =20 >>> (*------------------------------------------------------------- = >> scheduler ---*) >>> --- 722,750 ---- >>> END; >>> END StartSwitching; >>> =20 >>> ! CONST MaxSigs =3D 64; >>> ! TYPE Sig =3D [ 0..MaxSigs-1 ]; >>> !=20 >>> ! (* in order to listen to other signals, they have to be enabled in >>> ! allow_sigvtalrm as well *) >>> ! VAR (*CONST*) SIGCHLD :=3D ValueOfSIGCHLD(); >>> ! =20 >>> ! gotSigs :=3D SET OF Sig { }; >>> !=20 >>> ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>> BEGIN >>> allow_sigvtalrm (); >>> !=20 >>> ! INC(inCritical); >>> ! (* mark signal as being delivered *) >>> ! IF sig >=3D 0 AND sig < MaxSigs THEN >>> ! gotSigs :=3D gotSigs + SET OF Sig { sig } >>> ! END; >>> ! DEC(inCritical); >>> !=20 >>> ! IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN=20 >>> ! InternalYield ()=20 >>> ! END; >>> END switch_thread; >>> =20 >>> (*------------------------------------------------------------- = >> scheduler ---*) >>> *************** >>> *** 782,792 **** >>> IF t.alertable AND t.alertPending THEN >>> CanRun (t); >>> EXIT; >>> =20 >>> ELSIF t.waitingForTime <=3D now THEN >>> CanRun (t); >>> EXIT; >>> ! =20 >>> ELSIF NOT somePausing THEN >>> earliest :=3D t.waitingForTime; >>> somePausing :=3D TRUE; >>> --- 820,835 ---- >>> IF t.alertable AND t.alertPending THEN >>> CanRun (t); >>> EXIT; >>> + =20 >>> + ELSIF t.waitingForSig IN gotSigs THEN >>> + t.waitingForSig :=3D -1; >>> + CanRun(t); >>> + EXIT; >>> =20 >>> ELSIF t.waitingForTime <=3D now THEN >>> CanRun (t); >>> EXIT; >>> !=20 >>> ELSIF NOT somePausing THEN >>> earliest :=3D t.waitingForTime; >>> somePausing :=3D TRUE; >>> *************** >>> *** 886,891 **** >>> --- 929,936 ---- >>> END; >>> END; >>> =20 >>> + gotSigs :=3D SET OF Sig {}; >>> +=20 >>> IF t.state =3D State.alive AND (scanned OR NOT someBlocking) THEN >>> IF perfOn THEN PerfRunning (t.id); END; >>> (* At least one thread wants to run; transfer to it *) >>> *************** >>> *** 948,960 **** >>> =20 >>> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D >>> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >>> ! CONST Delay =3D 0.1D0; >>> BEGIN >>> LOOP >>> WITH r =3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >>> IF r # 0 THEN RETURN r END; >>> END; >>> ! Pause(Delay); >>> END; >>> END WaitProcess; >>> =20 >>> --- 993,1005 ---- >>> =20 >>> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D >>> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >>> ! CONST Delay =3D 10.0D0; >>> BEGIN >>> LOOP >>> WITH r =3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >>> IF r # 0 THEN RETURN r END; >>> END; >>> ! SigPause(Delay,SIGCHLD); >>> END; >>> END WaitProcess; >>> =20 >>> *************** >>> *** 1361,1364 **** >>> --- 1406,1412 ---- >>> VAR debug :=3D RTParams.IsPresent ("debugthreads"); >>> =20 >>> BEGIN >>> + (* we need to call set up the signal handler for other reasons = >> than >>> + just thread switching now *) >>> + setup_sigvtalrm (switch_thread); >>> END ThreadPosix. >>> Index: ThreadPosixC.c >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> RCS file: = >> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v >>> retrieving revision 1.59 >>> diff -c -r1.59 ThreadPosixC.c >>> *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 >>> --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 >>> *************** >>> *** 78,88 **** >>> --- 78,97 ---- >>> =20 >>> sigemptyset(&ThreadSwitchSignal); >>> sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); >>> + sigaddset(&ThreadSwitchSignal, SIGCHLD); >>> =20 >>> act.sa_handler =3D handler; >>> act.sa_flags =3D SA_RESTART; >>> sigemptyset(&(act.sa_mask)); >>> if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); >>> + if (sigaction (SIGCHLD, &act, NULL)) abort(); >>> + } >>> +=20 >>> + int >>> + __cdecl >>> + ThreadPosix__value_of_SIGCHLD(void) >>> + { >>> + return SIGCHLD; >>> } >>> =20 >>> void From hosking at cs.purdue.edu Sun Feb 13 21:46:26 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Sun, 13 Feb 2011 15:46:26 -0500 Subject: [M3devel] More on threading In-Reply-To: <20110213181717.4F0E91A2078@async.async.caltech.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> Message-ID: <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> I know it worked properly at one point in time, when I stressed it very hard for some GC-related work I was doing. I need to see instances of failure (deadlock, crashes) captured in the debugger, running with the latest head code. I don't want to try and diagnose old versions of the system if at all possible. I may have some time to try your thread test program in the next couple of weeks. Please let me know workloads that reliably cause failure. On Feb 13, 2011, at 1:17 PM, Mika Nystrom wrote: > I don't think pthreads works properly anywhere at > the moment. My guess is that it never has, if you stress it hard enough. From mika at async.caltech.edu Sun Feb 13 22:15:51 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 13:15:51 -0800 Subject: [M3devel] More on threading In-Reply-To: <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> Message-ID: <20110213211551.2A60A1A2078@async.async.caltech.edu> I have bootstrapped so many versions of the compiler on so many systems in the past week I've probably lost track of where we are, yes. I'll do some testing on a bona fide head checkout and let you know what I find. Mika Tony Hosking writes: >I know it worked properly at one point in time, when I stressed it very = >hard for some GC-related work I was doing. >I need to see instances of failure (deadlock, crashes) captured in the = >debugger, running with the latest head code. >I don't want to try and diagnose old versions of the system if at all = >possible. >I may have some time to try your thread test program in the next couple = >of weeks. Please let me know workloads that reliably cause failure. > > >On Feb 13, 2011, at 1:17 PM, Mika Nystrom wrote: > >> I don't think pthreads works properly anywhere at >> the moment. My guess is that it never has, if you stress it hard = >enough. From mika at async.caltech.edu Sun Feb 13 23:08:04 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 14:08:04 -0800 Subject: [M3devel] SIGCHLD diff Message-ID: <20110213220804.7A9001A2078@async.async.caltech.edu> Hi Tony, I found a problem with the change. I would appreciate it if you and Jay could think about what I found and my solution (and anybody else is welcome to as well, of course). When fiddling with cm3 to get it to run in parallel, I pushed it up to fork up to 100 processes at once, just to see what happens. When I did that (with my SIGCHLD patch) I got a few errors such as illegal instruction and a bus error or two. I believe what was happening was that I was taking too many signals at once, because of the following code in ThreadPosix.m3: PROCEDURE switch_thread (sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); (* here *) INC(inCritical); (* mark signal as being delivered *) IF sig >= 0 AND sig < MaxSigs THEN gotSigs := gotSigs + SET OF Sig { sig } END; DEC(inCritical); IF inCritical = 0 AND heapState.inCritical = 0 THEN InternalYield () END; END switch_thread; I found it a bit odd that the first action of the signal handler is to enable signals. With VTALRM this is innocuous, but in my case, imagine what happens if you have a lot of exited child processes queued up. What happens is that your signal stack gets wound deeper and deeper, and each time you catch one SIGCHLD you ask for another. Eventually you run out of stack space. My change, which appears to fix this problem, is to split up the "allow" function as follows: PROCEDURE switch_thread (sig: int) RAISES {Alerted} = BEGIN allow_sigvtalrm (); (* VTALRM only *) INC(inCritical); IF inCritical = 1 THEN allow_othersigs (); END; (* SIGCHLD *) (* mark signal as being delivered *) IF sig >= 0 AND sig < MaxSigs THEN gotSigs := gotSigs + SET OF Sig { sig } END; DEC(inCritical); IF inCritical = 0 AND heapState.inCritical = 0 THEN InternalYield () END; END switch_thread; This way, SIGCHLD is not taken in nested signal handlers. But is there a problem here? Well, the fallback is that it takes 0.1 seconds to wake up on Process.Wait . Is there a chance that SIGCHLD signals are left pending? My argument is no, since in the "base" environment (i.e., not executing in a signal handler) SIGCHLD is enabled as a postcondition of allow_othersigs. All *other* occurrences of allow_sigvtalrm are of course replaced with allow_sigvtalrm(); allow_othersigs(); The goal is that SIGCHLD be enabled precisely when SIGVTALRM is except inside the signal handler itself. Mika Tony Hosking writes: >Sorry, I missed you e-mail describing this change. Looks good. > >Antony Hosking | Associate Professor | Computer Science | Purdue = >University >305 N. University Street | West Lafayette | IN 47907 | USA >Office +1 765 494 6001 | Mobile +1 765 427 5484 > > > > >On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: > >> Tony, >>=20 >> The old user threads has an old problem with Process.Wait. It has a=20= > >> Thread.Pause(0.1d0) in it. This causes the compiler to run three = >times >> slower than it should. The m3build I've been using for years has >> that Thread.Pause removed (so it uses 100% CPU instead, = >busy-waiting...) >>=20 >> But yeah user threads works the same as it used to (except for=20 >> now calling pthread_atfork, which I'll return to in another email...) >>=20 >> Mika >>=20 >> Tony Hosking writes: >>> Hi Mika, Why was this change needed (I haven't looked closely)? I = >don't =3D >>> think we diverged much from previous implementations of user threads, = >=3D >>> which presumably used to work for you. >>>=20 >>> On Feb 12, 2011, at 3:44 PM, Mika Nystrom wrote: >>>=20 >>>> =3D20 >>>> Index: ThreadPosix.i3 >>>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> =3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D= >3D=3D3D=3D3D=3D3D >>>> RCS file: =3D >>> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.i3,v >>>> retrieving revision 1.16 >>>> diff -c -r1.16 ThreadPosix.i3 >>>> *** ThreadPosix.i3 14 Apr 2010 09:53:34 -0000 1.16 >>>> --- ThreadPosix.i3 12 Feb 2011 20:43:26 -0000 >>>> *************** >>>> *** 39,42 **** >>>> --- 39,45 ---- >>>> =3D20 >>>> =3D >>> = >(*------------------------------------------------------------------------= >=3D >>> ---*) >>>> =3D20 >>>> + <*EXTERNAL ThreadPosix__value_of_SIGCHLD*> >>>> + PROCEDURE ValueOfSIGCHLD(): int; >>>> +=3D20 >>>> END ThreadPosix. >>>> Index: ThreadPosix.m3 >>>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> =3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D= >3D=3D3D=3D3D=3D3D >>>> RCS file: =3D >>> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosix.m3,v >>>> retrieving revision 1.73 >>>> diff -c -r1.73 ThreadPosix.m3 >>>> *** ThreadPosix.m3 28 Dec 2010 10:13:46 -0000 1.73 >>>> --- ThreadPosix.m3 12 Feb 2011 20:43:26 -0000 >>>> *************** >>>> *** 97,102 **** >>>> --- 97,105 ---- >>>> (* if state =3D3D pausing, the time at which we can restart *) >>>> waitingForTime: Time.T; >>>> =3D20 >>>> + (* if state =3D3D pausing, the signal that truncates the pause = >*) >>>> + waitingForSig: int :=3D3D -1; >>>> +=3D20 >>>> (* true if we are waiting during an AlertWait or AlertJoin=3D20 >>>> or AlertPause *) >>>> alertable: BOOLEAN :=3D3D FALSE; >>>> *************** >>>> *** 147,152 **** >>>> --- 150,160 ---- >>>> =3D20 >>>> defaultStackSize :=3D3D 3000; >>>> =3D20 >>>> + (* note that even though the "heavy machinery" is only used = >for=3D20 >>>> + multipleThreads, we still need to set up the signal handler = >so >>>> + that we can catch signals from other sources than thread =3D >>> switching. >>>> + e.g., we use SIGCHLD to speed up Process.Wait *) >>>> +=3D20 >>>> VAR >>>> stats: RECORD >>>> n_forks :=3D3D 0; >>>> *************** >>>> *** 511,527 **** >>>> XPause(until, FALSE); >>>> END Pause; >>>> =3D20 >>>> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D3D >>>> VAR until :=3D3D n + Time.Now (); >>>> BEGIN >>>> XPause(until, TRUE); >>>> END AlertPause; >>>> =3D20 >>>> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D3D FALSE) = >=3D >>> RAISES {Alerted} =3D3D >>>> BEGIN >>>> INC (inCritical); >>>> self.waitingForTime :=3D3D until; >>>> self.alertable :=3D3D alertable; >>>> ICannotRun (State.pausing); >>>> DEC (inCritical); >>>> InternalYield (); >>>> --- 519,546 ---- >>>> XPause(until, FALSE); >>>> END Pause; >>>> =3D20 >>>> + PROCEDURE SigPause(n: LONGREAL; sig: int)=3D3D >>>> + <*FATAL Alerted*> >>>> + VAR until :=3D3D n + Time.Now (); >>>> + BEGIN >>>> + XPause(until, FALSE, sig); >>>> + END SigPause; >>>> +=3D20 >>>> PROCEDURE AlertPause(n: LONGREAL) RAISES {Alerted}=3D3D >>>> VAR until :=3D3D n + Time.Now (); >>>> BEGIN >>>> XPause(until, TRUE); >>>> END AlertPause; >>>> =3D20 >>>> ! PROCEDURE XPause (READONLY until: Time.T; alertable :=3D3D FALSE; = >=3D >>> sig:int :=3D3D -1)=3D20 >>>> ! RAISES {Alerted} =3D3D >>>> BEGIN >>>> INC (inCritical); >>>> self.waitingForTime :=3D3D until; >>>> self.alertable :=3D3D alertable; >>>> + IF sig # -1 THEN >>>> + self.waitingForSig :=3D3D sig >>>> + END; >>>> ICannotRun (State.pausing); >>>> DEC (inCritical); >>>> InternalYield (); >>>> *************** >>>> *** 703,712 **** >>>> END; >>>> END StartSwitching; >>>> =3D20 >>>> ! PROCEDURE switch_thread (<*UNUSED*> sig: int) RAISES {Alerted} =3D3D= > >>>> BEGIN >>>> allow_sigvtalrm (); >>>> ! IF inCritical =3D3D 0 AND heapState.inCritical =3D3D 0 THEN =3D >>> InternalYield () END; >>>> END switch_thread; >>>> =3D20 >>>> (*------------------------------------------------------------- =3D >>> scheduler ---*) >>>> --- 722,750 ---- >>>> END; >>>> END StartSwitching; >>>> =3D20 >>>> ! CONST MaxSigs =3D3D 64; >>>> ! TYPE Sig =3D3D [ 0..MaxSigs-1 ]; >>>> !=3D20 >>>> ! (* in order to listen to other signals, they have to be enabled in >>>> ! allow_sigvtalrm as well *) >>>> ! VAR (*CONST*) SIGCHLD :=3D3D ValueOfSIGCHLD(); >>>> ! =3D20 >>>> ! gotSigs :=3D3D SET OF Sig { }; >>>> !=3D20 >>>> ! PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D3D >>>> BEGIN >>>> allow_sigvtalrm (); >>>> !=3D20 >>>> ! INC(inCritical); >>>> ! (* mark signal as being delivered *) >>>> ! IF sig >=3D3D 0 AND sig < MaxSigs THEN >>>> ! gotSigs :=3D3D gotSigs + SET OF Sig { sig } >>>> ! END; >>>> ! DEC(inCritical); >>>> !=3D20 >>>> ! IF inCritical =3D3D 0 AND heapState.inCritical =3D3D 0 THEN=3D20= > >>>> ! InternalYield ()=3D20 >>>> ! END; >>>> END switch_thread; >>>> =3D20 >>>> (*------------------------------------------------------------- =3D >>> scheduler ---*) >>>> *************** >>>> *** 782,792 **** >>>> IF t.alertable AND t.alertPending THEN >>>> CanRun (t); >>>> EXIT; >>>> =3D20 >>>> ELSIF t.waitingForTime <=3D3D now THEN >>>> CanRun (t); >>>> EXIT; >>>> ! =3D20 >>>> ELSIF NOT somePausing THEN >>>> earliest :=3D3D t.waitingForTime; >>>> somePausing :=3D3D TRUE; >>>> --- 820,835 ---- >>>> IF t.alertable AND t.alertPending THEN >>>> CanRun (t); >>>> EXIT; >>>> + =3D20 >>>> + ELSIF t.waitingForSig IN gotSigs THEN >>>> + t.waitingForSig :=3D3D -1; >>>> + CanRun(t); >>>> + EXIT; >>>> =3D20 >>>> ELSIF t.waitingForTime <=3D3D now THEN >>>> CanRun (t); >>>> EXIT; >>>> !=3D20 >>>> ELSIF NOT somePausing THEN >>>> earliest :=3D3D t.waitingForTime; >>>> somePausing :=3D3D TRUE; >>>> *************** >>>> *** 886,891 **** >>>> --- 929,936 ---- >>>> END; >>>> END; >>>> =3D20 >>>> + gotSigs :=3D3D SET OF Sig {}; >>>> +=3D20 >>>> IF t.state =3D3D State.alive AND (scanned OR NOT someBlocking) = >THEN >>>> IF perfOn THEN PerfRunning (t.id); END; >>>> (* At least one thread wants to run; transfer to it *) >>>> *************** >>>> *** 948,960 **** >>>> =3D20 >>>> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D3D >>>> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >>>> ! CONST Delay =3D3D 0.1D0; >>>> BEGIN >>>> LOOP >>>> WITH r =3D3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >>>> IF r # 0 THEN RETURN r END; >>>> END; >>>> ! Pause(Delay); >>>> END; >>>> END WaitProcess; >>>> =3D20 >>>> --- 993,1005 ---- >>>> =3D20 >>>> PROCEDURE WaitProcess (pid: int; VAR status: int): int =3D3D >>>> (* ThreadPThread.m3 and ThreadPosix.m3 are very similar. *) >>>> ! CONST Delay =3D3D 10.0D0; >>>> BEGIN >>>> LOOP >>>> WITH r =3D3D Uexec.waitpid(pid, ADR(status), Uexec.WNOHANG) DO >>>> IF r # 0 THEN RETURN r END; >>>> END; >>>> ! SigPause(Delay,SIGCHLD); >>>> END; >>>> END WaitProcess; >>>> =3D20 >>>> *************** >>>> *** 1361,1364 **** >>>> --- 1406,1412 ---- >>>> VAR debug :=3D3D RTParams.IsPresent ("debugthreads"); >>>> =3D20 >>>> BEGIN >>>> + (* we need to call set up the signal handler for other reasons =3D= > >>> than >>>> + just thread switching now *) >>>> + setup_sigvtalrm (switch_thread); >>>> END ThreadPosix. >>>> Index: ThreadPosixC.c >>>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> = >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3= >D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D >>> =3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D= >3D=3D3D=3D3D=3D3D >>>> RCS file: =3D >>> /usr/cvs/cm3/m3-libs/m3core/src/thread/POSIX/ThreadPosixC.c,v >>>> retrieving revision 1.59 >>>> diff -c -r1.59 ThreadPosixC.c >>>> *** ThreadPosixC.c 12 Feb 2011 00:56:32 -0000 1.59 >>>> --- ThreadPosixC.c 12 Feb 2011 20:43:26 -0000 >>>> *************** >>>> *** 78,88 **** >>>> --- 78,97 ---- >>>> =3D20 >>>> sigemptyset(&ThreadSwitchSignal); >>>> sigaddset(&ThreadSwitchSignal, SIG_TIMESLICE); >>>> + sigaddset(&ThreadSwitchSignal, SIGCHLD); >>>> =3D20 >>>> act.sa_handler =3D3D handler; >>>> act.sa_flags =3D3D SA_RESTART; >>>> sigemptyset(&(act.sa_mask)); >>>> if (sigaction (SIG_TIMESLICE, &act, NULL)) abort(); >>>> + if (sigaction (SIGCHLD, &act, NULL)) abort(); >>>> + } >>>> +=3D20 >>>> + int >>>> + __cdecl >>>> + ThreadPosix__value_of_SIGCHLD(void) >>>> + { >>>> + return SIGCHLD; >>>> } >>>> =3D20 >>>> void From mika at async.caltech.edu Sun Feb 13 23:27:51 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 14:27:51 -0800 Subject: [M3devel] More on threading In-Reply-To: <20110213181717.4F0E91A2078@async.async.caltech.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> Message-ID: <20110213222751.A98F31A2078@async.async.caltech.edu> All right, I have some more interesting observations on user threading. As well as on parallelizing CM3. They are connected because the latter is the source of the former. User threads on FreeBSD (FreeBSD7 /amd64) ======================= The fact that my process was getting stuck in "umtxn" (I think that is what top called it) turned out to be something inside fork() in libthr.so. Now, libthr.so is FreeBSD's name for pthreads, and this was a user threading program. This does not seem right. I tried the following: 1. remove the call to pthread_atfork No change in behavior. My parallelized CM3 still got stuck in umtxn. 2. remove the -pthread linking flag Success! Parallelized CM3 works! So, clearly -pthread is harmful to the health of Modula-3 programs running with user threading on FreeBSD. This is on FreeBSD 7.1, so it means pthread_atfork has to be removed as well. I'm not sure how to make the building of this as transparent as possible, but what's currently checked in is in any case wrong. Parallelizing CM3 ================= I found one bug in my earlier CM3 patch. Calls to Utils.* need to be monitored because there is a shared hash table in that module. New diff attached. With this diff the compiler runs correctly even when it is running 100 instances of the backend in parallel. Speedup is about 2.0 on a machine with four cores for a largish source directory. The max. number of subprocesses is set on line 985 of Builder.m3 . I am not claiming this patch is anywhere near "polished" enough to go into the CM3 tree at the moment. It's an experiment and starting point for further work... Mika Index: cm3/src/Builder.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/cm3/src/Builder.m3,v retrieving revision 1.36 diff -c -r1.36 Builder.m3 *** cm3/src/Builder.m3 24 Aug 2010 05:24:24 -0000 1.36 --- cm3/src/Builder.m3 13 Feb 2011 22:17:16 -0000 *************** *** 15,20 **** --- 15,22 ---- IMPORT QIdent; FROM Target IMPORT M3BackendMode_t, BackendAssembly, BackendModeStrings; FROM M3Path IMPORT OSKind, OSKindStrings; + IMPORT Pathname; + IMPORT RefSeq; TYPE UK = M3Unit.Kind; *************** *** 131,136 **** --- 133,139 ---- link_coverage : TEXT; (* coverage library *) m3_front_flags: Arg.List; (* configuration options for the front *) m3_options : Arg.List; (* misc. user options for the frontend *) + delayBackend := FALSE; END; TYPE *************** *** 932,937 **** --- 935,956 ---- (*------------------------------------------------------------ compilation --*) + TYPE MarkerPromise = QMachine.Promise BRANDED OBJECT OVERRIDES fulfil := FulfilNothing END; (* an empty promise, use as a marker *) + + PROCEDURE FulfilNothing(<*UNUSED*>mp : MarkerPromise) = BEGIN END FulfilNothing; + + TYPE SeqClosure = Thread.Closure OBJECT seq : RefSeq.T; OVERRIDES apply := SeqApply END; + + PROCEDURE SeqApply(cl : SeqClosure) : REFANY = + BEGIN (* force a Sequence of promises *) + FOR i := 0 TO cl.seq.size()-1 DO + WITH p = NARROW(cl.seq.get(i),QMachine.Promise) DO + p.fulfil() + END + END; + RETURN NIL + END SeqApply; + PROCEDURE CompileEverything (s: State; schedule: SourceList) = VAR u: M3Unit.T; BEGIN *************** *** 941,948 **** (* compile all the sources using the initial schedule *) FOR i := 0 TO LAST (schedule^) DO ! CompileOne (s, schedule[i]); END; FlushPending (s); (* recompile any interfaces where we goofed on the exports *) --- 960,1000 ---- (* compile all the sources using the initial schedule *) FOR i := 0 TO LAST (schedule^) DO ! s.delayBackend := TRUE; (* tell backend to promise instead of executing *) ! TRY ! CompileOne (s, schedule[i]); ! FINALLY ! s.delayBackend := FALSE; (* tell backend to go back to executing *) ! END; ! ! s.machine.promises.addhi(NEW(MarkerPromise)); (* promises, promises *) ! END; + + VAR + curSeq := NEW(RefSeq.T).init(); (* sequential chain *) + threads := NEW(RefSeq.T).init(); (* parallel chains *) + BEGIN + FOR i := 0 TO s.machine.promises.size()-1 DO + WITH p = s.machine.promises.get(i) DO + curSeq.addhi(p); + IF i = s.machine.promises.size()-1 OR ISTYPE(p,MarkerPromise) THEN (* mark as end of one sequential chain of promises *) + WITH cl = NEW(SeqClosure, seq := curSeq) DO + threads.addhi (Thread.Fork(cl)); (* start sequential chain *) + + IF threads.size() > 400 THEN EVAL Thread.Join(threads.remlo()) END; (* limit number of sequential chains executing in parallel *) + + curSeq := NEW(RefSeq.T).init() + END + END + END + END; + WHILE threads.size() > 0 DO EVAL Thread.Join(threads.remlo()) END; (* wait for all sequential chains to end *) + END; + + EVAL s.machine.promises.init(); (* clear out promises *) + + FlushPending (s); (* recompile any interfaces where we goofed on the exports *) *************** *** 1151,1156 **** --- 1203,1229 ---- END; END CompileM3; + TYPE + NotePromise = QMachine.Promise OBJECT + nam : Pathname.T; + OVERRIDES + fulfil := FulfilNP; + END; + + RemovePromise = QMachine.Promise OBJECT + nam : Pathname.T; + OVERRIDES + fulfil := FulfilRP; + END; + + VAR utilsMu := NEW(MUTEX); (* Utils.* fiddles with a table *) + + PROCEDURE FulfilNP(np : NotePromise) = + BEGIN LOCK utilsMu DO Utils.NoteTempFile(np.nam) END END FulfilNP; + + PROCEDURE FulfilRP(rp : RemovePromise) = + BEGIN LOCK utilsMu DO Utils.Remove(rp.nam) END END FulfilRP; + PROCEDURE PushOneM3 (s: State; u: M3Unit.T): BOOLEAN = VAR tmpC, tmpS: TEXT; *************** *** 1191,1208 **** | 3 => (* -bootstrap, +m3back, +asm *) tmpC := TempCName (u); tmpS := TempSName (u); IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; IF RunM3 (s, u, tmpC) THEN ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) ! AND RunAsm (s, tmpS, u.object) THEN END; need_merge := TRUE; END; IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; | 6, (* +bootstrap, +m3back, -asm *) 7 => (* +bootstrap, +m3back, +asm *) --- 1264,1320 ---- | 3 => (* -bootstrap, +m3back, +asm *) + IF s.delayBackend THEN tmpC := TempCName (u); tmpS := TempSName (u); + (* IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; + *) + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(NotePromise, nam := tmpC)) + END; + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(NotePromise, nam := tmpS)) + END; + IF RunM3 (s, u, tmpC) THEN ! s.machine.record(TRUE); ! TRY ! IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) ! AND RunAsm (s, tmpS, u.object) THEN ! END; ! FINALLY ! s.machine.record(FALSE) END; + need_merge := TRUE; END; + (* IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; + *) + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpC)) + END; + IF (NOT s.keep_files) THEN + s.machine.promises.addhi(NEW(RemovePromise, nam := tmpS)) + END; + + ELSE + tmpC := TempCName (u); + tmpS := TempSName (u); + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpC) END; + IF (NOT s.keep_files) THEN Utils.NoteTempFile (tmpS) END; + IF RunM3 (s, u, tmpC) THEN + IF RunM3Back (s, tmpC, tmpS, u.debug, u.optimize) + AND RunAsm (s, tmpS, u.object) THEN + END; + need_merge := TRUE; + END; + IF (NOT s.keep_files) THEN Utils.Remove (tmpC) END; + IF (NOT s.keep_files) THEN Utils.Remove (tmpS) END; + END | 6, (* +bootstrap, +m3back, -asm *) 7 => (* +bootstrap, +m3back, +asm *) Index: m3quake/src/QMachine.i3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.i3,v retrieving revision 1.6 diff -c -r1.6 QMachine.i3 *** m3quake/src/QMachine.i3 4 Sep 2009 10:24:07 -0000 1.6 --- m3quake/src/QMachine.i3 13 Feb 2011 22:17:18 -0000 *************** *** 8,13 **** --- 8,14 ---- IMPORT Thread, Wr, QValue, QCode; FROM Quake IMPORT Machine, Error, ID, IDMap; + IMPORT RefSeq; REVEAL T <: T_; *************** *** 15,20 **** --- 16,22 ---- T = Machine; T_ = OBJECT map: IDMap := NIL; (* READONLY *) + promises : RefSeq.T; METHODS init (map: IDMap): T; evaluate (s: QCode.Stream) RAISES {Error, Thread.Alerted}; *************** *** 37,42 **** --- 39,46 ---- set_wr (wr: Wr.T); exec_echo (b: BOOLEAN): BOOLEAN; trace (b: BOOLEAN); + + record(on : BOOLEAN); (* instead of performing certain acts, promise *) END; PROCEDURE PushBool (t: T; b: BOOLEAN); *************** *** 51,54 **** --- 55,63 ---- PROCEDURE GetEnv (default, v0, v1, v2, v3, v4: TEXT := NIL): TEXT; + TYPE Promise = OBJECT METHODS fulfil() RAISES { Error } END; + END QMachine. + + + Index: m3quake/src/QMachine.m3 =================================================================== RCS file: /usr/cvs/cm3/m3-sys/m3quake/src/QMachine.m3,v retrieving revision 1.35 diff -c -r1.35 QMachine.m3 *** m3quake/src/QMachine.m3 3 Aug 2010 09:40:04 -0000 1.35 --- m3quake/src/QMachine.m3 13 Feb 2011 22:17:18 -0000 *************** *** 16,21 **** --- 16,22 ---- IMPORT TextUtils, FSUtils, System, DirStack; (* sysutils *) IMPORT Compiler; IMPORT M3Path; + IMPORT RefSeq; CONST OnUnix = (Compiler.ThisOS = Compiler.OS.POSIX); *************** *** 44,49 **** --- 45,52 ---- shell : TEXT := NIL; sh_option : TEXT := NIL; tmp_dir : TEXT := NIL; + + doRecord := FALSE; OVERRIDES init := Init; evaluate := Evaluate; *************** *** 66,73 **** --- 69,81 ---- set_wr := SetWr; exec_echo := ExecEcho; trace := Trace; + + record := Record; END; + PROCEDURE Record(t : T; on : BOOLEAN) = + BEGIN t.doRecord := on END Record; + TYPE Registers = RECORD cp : QCode.Stream := NIL; (* code pointer *) *************** *** 139,144 **** --- 147,154 ---- t.globals := NEW (IntRefTbl.Default).init (); t.default_wr := Stdio.stdout; + t.promises := NEW(RefSeq.T).init(); + InitOSEnv (t); InitBuiltins (t); *************** *** 1555,1564 **** END; ELSE FlushIO (); ! Process.GetStandardFileHandles (stdin, stdout, stderr); ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), ! stdin := stdin, stdout := stdout, ! stderr := stderr); END; EXCEPT | Thread.Alerted => --- 1565,1594 ---- END; ELSE FlushIO (); ! IF t.doRecord THEN ! handle := NIL; ! WITH a = NEW(REF ARRAY OF TEXT, n_shell_args) DO ! a^ := SUBARRAY(args,0,n_shell_args); ! VAR wrx : Wr.T; BEGIN ! IF echo OR t.do_echo THEN ! wrx := wr ! ELSE ! wrx := NIL ! END; ! t.promises.addhi(NEW(ExecPromise, ! cmd := t.shell, ! wr := wrx, ! args := a, ! t := t, ! ignore_errors := ignore_errors)) ! END ! END ! ELSE ! Process.GetStandardFileHandles (stdin, stdout, stderr); ! handle := Process.Create (t.shell, SUBARRAY (args, 0, n_shell_args), ! stdin := stdin, stdout := stdout, ! stderr := stderr); ! END; END; EXCEPT | Thread.Alerted => *************** *** 1573,1579 **** END; (* wait for everything to shutdown... *) ! exit_code := Process.Wait (handle); END; IF onlyTry THEN --- 1603,1613 ---- END; (* wait for everything to shutdown... *) ! IF handle = NIL THEN ! exit_code := 0 ! ELSE ! exit_code := Process.Wait (handle); ! END; (* else we're only promising *) END; IF onlyTry THEN *************** *** 1589,1594 **** --- 1623,1664 ---- END ExecCommand; + TYPE + ExecPromise = Promise OBJECT + cmd : TEXT; + args : REF ARRAY OF TEXT; + t : T; + wr : Wr.T; + ignore_errors : BOOLEAN; + OVERRIDES + fulfil := FulfilExecPromise; + END; + + PROCEDURE FulfilExecPromise(ep : ExecPromise) RAISES { Error } = + VAR + stdin, stdout, stderr: File.T; + BEGIN + Process.GetStandardFileHandles (stdin, stdout, stderr); + TRY + IF ep.wr # NIL THEN + Wr.PutText (ep.wr, ep.args[1]); + Wr.PutText (ep.wr, Wr.EOL); + FlushIO (); + END; + WITH handle = Process.Create (ep.cmd, ep.args^, + stdin := stdin, stdout := stdout, + stderr := stderr), + exit_code = Process.Wait(handle) DO + IF exit_code # 0 AND NOT ep.ignore_errors THEN + Err (ep.t, Fmt.F("exit %s: %s", Fmt.Int(exit_code), ep.cmd)) + END + END + EXCEPT + OSError.E (ec) => + Err (ep.t, Fmt.F ("exec failed%s *** %s", OSErr (ec), ep.cmd)); + END + END FulfilExecPromise; + PROCEDURE KillProcess (handle: Process.T) = BEGIN IF (handle # NIL) THEN From mika at async.caltech.edu Mon Feb 14 00:15:02 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 15:15:02 -0800 Subject: [M3devel] bootstrapping problems In-Reply-To: References: <20110205215619.E65981A207A@async.async.caltech.edu> , <20110205234331.2F13A1A207A@async.async.caltech.edu> Message-ID: <20110213231502.7A5FA1A2078@async.async.caltech.edu> Jay sent me the following instructions for bootstrapping a while back: ./upgrade.py nocleangcc && ./do-cm3-all.py realclean skipgcc && ./do-cm3-all.py buildship skipgcc It seems reasonable enough for me. I unpacked the REL distribution, checked everything out, etc. However something is very wrong. The newer back-end does not seem compatible with the old m3front (not even backward-compatible), and it's getting called at the wrong time: ignoring ../src/m3overrides rm .M3SHIP rm .M3OVERRIDES rm .generic.tmp rm .generic.tmp inhale libsysutils.m3x inhale /usr/local/cm3/pkg/m3core/AMD64_LINUX/libm3core.a inhale /usr/local/cm3/pkg/libm3/AMD64_LINUX/libm3.a new source -> compiling EnvUtils.i3 m3front ../src/EnvUtils.i3 -w1 /home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 compilation terminated. m3_backend => 1 m3cc (aka cm3cg) failed compiling: EnvUtils.ic rm EnvUtils.is Why is it calling the new back end? This is the very first thing it does (it hasn't cleaned m3cc for some reason). The new back end doesn't like the output of the old compiler: (119)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>/home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 compilation terminated. But the old one in /usr/local/cm3/bin likes it just fine... (120)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is (121)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX> Is something going wrong with the bootstrapping process? I'm going to try to delete the newer back end and see if that fixes it. Mika From mika at async.caltech.edu Mon Feb 14 00:16:26 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 15:16:26 -0800 Subject: [M3devel] bootstrapping problems In-Reply-To: <20110213231502.7A5FA1A2078@async.async.caltech.edu> References: <20110205215619.E65981A207A@async.async.caltech.edu> , <20110205234331.2F13A1A207A@async.async.caltech.edu> <20110213231502.7A5FA1A2078@async.async.caltech.edu> Message-ID: <20110213231626.200DB1A2078@async.async.caltech.edu> I should probably not have "nocleangcc" and "skipgcc" in there huh... still a bit weird that the back-end isn't backward compatible? Mika Mika Nystrom writes: >Jay sent me the following instructions for bootstrapping a while back: > >./upgrade.py nocleangcc && > ./do-cm3-all.py realclean skipgcc && > ./do-cm3-all.py buildship skipgcc > >It seems reasonable enough for me. > >I unpacked the REL distribution, checked everything out, etc. > >However something is very wrong. The newer back-end does not seem >compatible with the old m3front (not even backward-compatible), and it's >getting called at the wrong time: > >ignoring ../src/m3overrides > >rm .M3SHIP >rm .M3OVERRIDES >rm .generic.tmp >rm .generic.tmp >inhale libsysutils.m3x >inhale /usr/local/cm3/pkg/m3core/AMD64_LINUX/libm3core.a >inhale /usr/local/cm3/pkg/libm3/AMD64_LINUX/libm3.a > >new source -> compiling EnvUtils.i3 >m3front ../src/EnvUtils.i3 -w1 >/home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-alig >n-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is >cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 >compilation terminated. > m3_backend => 1 >m3cc (aka cm3cg) failed compiling: EnvUtils.ic >rm EnvUtils.is > >Why is it calling the new back end? This is the very first thing it does (it >hasn't cleaned m3cc for some reason). > >The new back end doesn't like the output of the old compiler: > >(119)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>/home/mika/cm3/cm3/m3-sys >/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables >-quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is >cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 >compilation terminated. > >But the old one in /usr/local/cm3/bin likes it just fine... > >(120)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>cm3cg -gstabs+ -m64 -fPIC > -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o E >nvUtils.is >(121)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX> > >Is something going wrong with the bootstrapping process? I'm going to >try to delete the newer back end and see if that fixes it. > > Mika From jay.krell at cornell.edu Mon Feb 14 03:16:48 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 14 Feb 2011 02:16:48 +0000 Subject: [M3devel] bootstrapping problems In-Reply-To: <20110213231626.200DB1A2078@async.async.caltech.edu> References: <20110205215619.E65981A207A@async.async.caltech.edu>, , , <20110205234331.2F13A1A207A@async.async.caltech.edu>, , <20110213231502.7A5FA1A2078@async.async.caltech.edu>, <20110213231626.200DB1A2078@async.async.caltech.edu> Message-ID: /home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg That is the problem. For a long time, my config file were willing to run that. Instead of cm3cg in $PATH or cm3cg next to cm3. It bit me a bit too so I stopped doing it that way. The instructions are ok. Certainly the skipgcc. Nocleangcc depends on underlying correct incrementality, so might be a bit fragile. The backend has no need or obligation to be forward or backward compatible with any cm3/m3front. It need only be compatible with the one it goes with. The upgrading of cm3 and cm3cg is carefully coordinated to avoid any mismatch. Whenever the uprading has gone wrong, it has been fixed. Not any sort of compatibility restored. Granted, we could better use the version declaration and error right away clearly. I can make a change there. - Jay > To: m3devel at elegosoft.com > Date: Sun, 13 Feb 2011 15:16:26 -0800 > From: mika at async.caltech.edu > Subject: Re: [M3devel] bootstrapping problems > > I should probably not have "nocleangcc" and "skipgcc" in there huh... > > still a bit weird that the back-end isn't backward compatible? > > Mika > > Mika Nystrom writes: > >Jay sent me the following instructions for bootstrapping a while back: > > > >./upgrade.py nocleangcc && > > ./do-cm3-all.py realclean skipgcc && > > ./do-cm3-all.py buildship skipgcc > > > >It seems reasonable enough for me. > > > >I unpacked the REL distribution, checked everything out, etc. > > > >However something is very wrong. The newer back-end does not seem > >compatible with the old m3front (not even backward-compatible), and it's > >getting called at the wrong time: > > > >ignoring ../src/m3overrides > > > >rm .M3SHIP > >rm .M3OVERRIDES > >rm .generic.tmp > >rm .generic.tmp > >inhale libsysutils.m3x > >inhale /usr/local/cm3/pkg/m3core/AMD64_LINUX/libm3core.a > >inhale /usr/local/cm3/pkg/libm3/AMD64_LINUX/libm3.a > > > >new source -> compiling EnvUtils.i3 > >m3front ../src/EnvUtils.i3 -w1 > >/home/mika/cm3/cm3/m3-sys/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-alig > >n-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is > >cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 > >compilation terminated. > > m3_backend => 1 > >m3cc (aka cm3cg) failed compiling: EnvUtils.ic > >rm EnvUtils.is > > > >Why is it calling the new back end? This is the very first thing it does (it > >hasn't cleaned m3cc for some reason). > > > >The new back end doesn't like the output of the old compiler: > > > >(119)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>/home/mika/cm3/cm3/m3-sys > >/m3cc/AMD64_LINUX/cm3cg -gstabs+ -m64 -fPIC -mno-align-double -funwind-tables > >-quiet -fno-reorder-blocks EnvUtils.ic -o EnvUtils.is > >cm3cg: fatal error: *** illegal type: 0x17, at m3cg_lineno 4 > >compilation terminated. > > > >But the old one in /usr/local/cm3/bin likes it just fine... > > > >(120)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX>cm3cg -gstabs+ -m64 -fPIC > > -mno-align-double -funwind-tables -quiet -fno-reorder-blocks EnvUtils.ic -o E > >nvUtils.is > >(121)ukdsl204:~/cm3/cm3/m3-libs/sysutils/AMD64_LINUX> > > > >Is something going wrong with the bootstrapping process? I'm going to > >try to delete the newer back end and see if that fixes it. > > > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Mon Feb 14 03:55:41 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 13 Feb 2011 18:55:41 -0800 Subject: [M3devel] More on threading In-Reply-To: <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> <919139D0-3EF8-4758-84A5-74F1EA86C059@cs.purdue.edu> Message-ID: <20110214025541.76AC51A2078@async.async.caltech.edu> All right, I have bootstrapped from the current CVS head with the PTHREAD thread implementation and obtained the following behavior. I'm on AMD64_LINUX here, but can repeat on AMD64_FREEBSD, LINUXLIBC6, I386_DARWIN if either of those is of any special interest. The AMD64_LINUX system reports as follows: Linux ukdsl204 2.6.26-2-amd64 #1 SMP Thu Sep 16 15:56:38 UTC 2010 x86_64 GNU/Linux Thread test results: all : 1. assert failed, ThreadPThread.m3, line 966 2. SEGV pc = 0x430f40 = Move + 0x6b in ../src/runtime/common/RTCollector.m3 3. ThreadPThreadC.c:372: ThreadPThread__pthread_mutex_delete: Assertion `e == 0' failed. read : OK nxread [read w/o try-except] : OK tryexcept : OK fork : OK forktoomuch [Process.Create w/o Process.Wait] : OK alloc : deadlock creat [ make new threads ] : deadlock lock : OK Now we can also try combinations, but I won't go tediously through all possibilites, just a few: alloc,nxread: SEGV pc = 0x430f40 = Move + 0x6b in ../src/runtime/common/RTCollector.m3 The "simplest" test that shows problem is "-tests alloc -n 2" (just two instances of alloc). Mika Tony Hosking writes: >I know it worked properly at one point in time, when I stressed it very = >hard for some GC-related work I was doing. >I need to see instances of failure (deadlock, crashes) captured in the = >debugger, running with the latest head code. >I don't want to try and diagnose old versions of the system if at all = >possible. >I may have some time to try your thread test program in the next couple = >of weeks. Please let me know workloads that reliably cause failure. > > >On Feb 13, 2011, at 1:17 PM, Mika Nystrom wrote: > >> I don't think pthreads works properly anywhere at >> the moment. My guess is that it never has, if you stress it hard = >enough. From rodney_bates at lcwb.coop Mon Feb 14 16:53:19 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Mon, 14 Feb 2011 09:53:19 -0600 Subject: [M3devel] threading on Windows? In-Reply-To: <20110209220057.790671A2078@async.async.caltech.edu> References: <20110209220057.790671A2078@async.async.caltech.edu> Message-ID: <4D594FEF.1030309@lcwb.coop> On 02/09/2011 04:00 PM, Mika Nystrom wrote: > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... Hmm, I'll try to look at that. I have been using the release branch for work on other projects. Even there, I have been seeing some problems in m3gdb not handling 64-bit processors. > > Mika > From mika at async.caltech.edu Mon Feb 14 22:18:29 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Mon, 14 Feb 2011 13:18:29 -0800 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110213220804.7A9001A2078@async.async.caltech.edu> References: <20110213220804.7A9001A2078@async.async.caltech.edu> Message-ID: <20110214211829.BA3761A2078@async.async.caltech.edu> Hi again m3devel, Does anyone understand the code in ThreadPosix.m3? After looking through it again I think my attached solution (sent out yesterday afternoon) is right, but I'm still just a little bit fuzzy on how the code works (is supposed to work) but I have some hypotheses. I realized it's been a long, long, loooong time since I coded anything with signals and signal handlers and I've forgotten much of the semantics. After all since Concurrent Pascal came out in '76 signals has always been the "old" way of doing things, right? (And I quit programming C in favor of M3 in 1999...) In ThreadPosixC.c, there's a function called setup_sigvtalrm that registers the Modula-3 procedure switch_thread as a signal handler. My understanding is that in C, signals are masked atomically on the call into the signal handler and unmasked atomically when the handler returns (unless the signal mask is changed, etc., while in the handler). However, switch_thread... doesn't return? Or? Or it does a longjmp somewhere? Which means signals may not be unmasked(?) Which is the reason for the calling "allow_sigvtalrm" as the first statement of the signal handler? Now I can't just turn on ALL signals in the signal handler because then I may get very deeply nested signal handlers, so I only call allow_allsignals (or whatever we call it) from the first-level signal handler. Hmm. Maybe that is not quite right? Do we eventually get back there? Yes I think we must because certain things depend on inCritical being zero... and when inCritical is zero SIGCHLD is again enabled. No I still don't fully understand how the code is supposed to work, especially in the presence of nested signal handling. Does anyone understand better? (I am still going to try to commit my patch later today because I don't think it could possibly make things worse and it does make a significant real-world improvement to the compiler performance under user threads.) Mika Mika Nystrom writes: >Hi Tony, > >I found a problem with the change. I would appreciate it if you and >Jay could think about what I found and my solution (and anybody else >is welcome to as well, of course). > >When fiddling with cm3 to get it to run in parallel, I pushed it up to >fork up to 100 processes at once, just to see what happens. When I did >that (with my SIGCHLD patch) I got a few errors such as illegal instruction >and a bus error or two. I believe what was happening was that I was >taking too many signals at once, because of the following code in >ThreadPosix.m3: > >PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); (* here *) > > INC(inCritical); > (* mark signal as being delivered *) > IF sig >= 0 AND sig < MaxSigs THEN > gotSigs := gotSigs + SET OF Sig { sig } > END; > DEC(inCritical); > > IF inCritical = 0 AND heapState.inCritical = 0 THEN > InternalYield () > END; > END switch_thread; > >I found it a bit odd that the first action of the signal handler is to >enable signals. With VTALRM this is innocuous, but in my case, imagine >what happens if you have a lot of exited child processes queued up. >What happens is that your signal stack gets wound deeper and deeper, and >each time you catch one SIGCHLD you ask for another. Eventually you >run out of stack space. > >My change, which appears to fix this problem, is to split up the "allow" >function as follows: > >PROCEDURE switch_thread (sig: int) RAISES {Alerted} = > BEGIN > allow_sigvtalrm (); (* VTALRM only *) > > INC(inCritical); > IF inCritical = 1 THEN allow_othersigs (); END; (* SIGCHLD *) > > (* mark signal as being delivered *) > IF sig >= 0 AND sig < MaxSigs THEN > gotSigs := gotSigs + SET OF Sig { sig } > END; > DEC(inCritical); > > IF inCritical = 0 AND heapState.inCritical = 0 THEN > InternalYield () > END; > END switch_thread; > >This way, SIGCHLD is not taken in nested signal handlers. But is there a >problem here? Well, the fallback is that it takes 0.1 seconds to wake >up on Process.Wait . > >Is there a chance that SIGCHLD signals are left pending? My argument is >no, since in the "base" environment (i.e., not executing in a signal handler) >SIGCHLD is enabled as a postcondition of allow_othersigs. > >All *other* occurrences of allow_sigvtalrm are of course replaced with > >allow_sigvtalrm(); allow_othersigs(); > >The goal is that SIGCHLD be enabled precisely when SIGVTALRM is except inside >the signal handler itself. > > Mika > > >Tony Hosking writes: >>Sorry, I missed you e-mail describing this change. Looks good. >> >>Antony Hosking | Associate Professor | Computer Science | Purdue = >>University >>305 N. University Street | West Lafayette | IN 47907 | USA >>Office +1 765 494 6001 | Mobile +1 765 427 5484 >> >> >> >> >>On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: >> >>> Tony, >>>=20 >>> The old user threads has an old problem with Process.Wait. It has a=20= >> >>> Thread.Pause(0.1d0) in it. This causes the compiler to run three = >>times >>> slower than it should. The m3build I've been using for years has >>> that Thread.Pause removed (so it uses 100% CPU instead, = >>busy-waiting...) >>>=20 >>> But yeah user threads works the same as it used to (except for=20 >>> now calling pthread_atfork, which I'll return to in another email...) >>>=20 >>> Mika ... From rcolebur at SCIRES.COM Tue Feb 15 00:01:14 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Mon, 14 Feb 2011 18:01:14 -0500 Subject: [M3devel] More on threading In-Reply-To: <20110213181717.4F0E91A2078@async.async.caltech.edu> References: <20110213070414.DD0341A2078@async.async.caltech.edu> <20110213181717.4F0E91A2078@async.async.caltech.edu> Message-ID: Mika wrote... >But no I don't use Windows. I was asking on here how to get started with >Modula-3 on Windows once but my machine was/is a Windows 2000 system and >apparently CM3 no longer works on win2k..? I use CM3 on Windows 2000/XP/7 and it works on all of these versions. --Randy Coleburn From rcolebur at SCIRES.COM Tue Feb 15 00:54:50 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Mon, 14 Feb 2011 18:54:50 -0500 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, , , , , , , , , , , , , , , , <39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, , <9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, , , , <74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, , , , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu>, , Message-ID: Jay: Up to now, I've been using my Windows CMD scripts to "upgrade", namely: scripts\dev\windows\ RCC_upgradeCM3.CMD But, based on your message, I am trying to do as you suggest, i.e., use your "upgrade.py". I am running Windows 7. I have just installed Python 2.7.1 via http://www.python.org/ftp/python/2.7.1/python-2.7.1.msi Here is error message I get: C:\cm3\Sandbox\scripts\python>upgrade.py Traceback (most recent call last): File "C:\cm3\Sandbox\scripts\python\upgrade.py", line 4, in import pylib File "C:\cm3\Sandbox\scripts\python\pylib.py", line 612, in if Target.startswith("NT386"): AttributeError: 'NoneType' object has no attribute 'startswith' Please advise. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Saturday, February 12, 2011 5:39 AM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? TextUtils.mo : error LNK2001: unresolved external symbol __alloca TextUtils.mo : error LNK2001: unresolved external symbol _Csetjmp__Jumpbuf_size sysutils.dll : fatal error LNK1120: 2 unresolved externals Most likely you didn't build it correctly. Go back to a working cm3/m3core/libm3 and run upgrade.py. There should be no longer any references to alloca of Csetjmp__Jumpbuf_size. Really, Csetjmp__Jumpbuf_size should never have been a problem. That likely also indicates some sort of incorrect bootstrap. > Are you saying you do, or do not, want me to work further on the solving the "problem," regardless of how we describe it ? I still want alloca to work, either via a function call or a specific m3cg method. (I still don't see that RTHooks.Alloca makes better sense; I still don't see a way to avoid hacking the compiler, given the custom calling convention.) It will be difficult to test the fix until/unless m3front is changed back. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Fri, 11 Feb 2011 18:17:15 -0500 Subject: Re: [M3devel] threading on Windows? Jay: Ok, I don't mean to start a new issue or upset anyone. I see you checked in some new changes. I updated to latest HEAD and tried to build, but again I get unresolved symbols beginning with m3-libs\sysutils (see attached sysutils.lst) So, not sure what you mean by "it should be ok now". Are you saying you do, or do not, want me to work further on the solving the "problem," regardless of how we describe it ? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, February 11, 2011 4:52 PM To: Tony; Coleburn, Randy Cc: m3devel Subject: RE: [M3devel] threading on Windows? It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - Jay ________________________________ From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new "heap-based jmpbuf allocation" have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate "alloca" routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I "do" anything, I'll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the "heap based jumpbuf allocation scheme" was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am "in the dark", both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I'm seeing several different symbols that don't resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking > wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don't see any reference to "m3_alloca" or even "alloc" in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay ________________________________ From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say "special case" calls "m3_alloca", where do I go about finding this "special case"? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 15 02:12:09 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 15 Feb 2011 01:12:09 +0000 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , <20110213181717.4F0E91A2078@async.async.caltech.edu>, Message-ID: Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Tue Feb 15 02:34:59 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 15 Feb 2011 01:34:59 +0000 Subject: [M3devel] threading on Windows? In-Reply-To: References: <20110209220057.790671A2078@async.async.caltech.edu>, ,,, ,,, ,,, ,,, ,,, ,,, ,,, ,,<39996C38-B933-4272-91BA-C99EA2EADCA7@gmail.com>, ,,<9C2C9FFD-1CD2-46F6-8FF9-2D57176E8067@cs.purdue.edu>, ,,, ,,<74A36A39-2B99-4EA2-A4FD-A7D74A55BF8C@cs.purdue.edu>, ,,, , , <449A236C-1F38-4F1A-AA1E-574893E00511@cs.purdue.edu>, , , , , , Message-ID: I'll have to look into it. It is possible I dropped support for "NT386" in favor of "I386_NT". I've definitely used the Python a fair amount on Windows, but less lately. You can probably force it by first setting the environment variable CM3_TARGET to NT386 or I386_NT. Or by giving either of those parameters on the command line. But you shouldn't have to. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Mon, 14 Feb 2011 18:54:50 -0500 Subject: Re: [M3devel] threading on Windows? Jay: Up to now, I?ve been using my Windows CMD scripts to ?upgrade?, namely: scripts\dev\windows\ RCC_upgradeCM3.CMD But, based on your message, I am trying to do as you suggest, i.e., use your ?upgrade.py?. I am running Windows 7. I have just installed Python 2.7.1 via http://www.python.org/ftp/python/2.7.1/python-2.7.1.msi Here is error message I get: C:\cm3\Sandbox\scripts\python>upgrade.py Traceback (most recent call last): File "C:\cm3\Sandbox\scripts\python\upgrade.py", line 4, in import pylib File "C:\cm3\Sandbox\scripts\python\pylib.py", line 612, in if Target.startswith("NT386"): AttributeError: 'NoneType' object has no attribute 'startswith' Please advise. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Saturday, February 12, 2011 5:39 AM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? TextUtils.mo : error LNK2001: unresolved external symbol __alloca TextUtils.mo : error LNK2001: unresolved external symbol _Csetjmp__Jumpbuf_size sysutils.dll : fatal error LNK1120: 2 unresolved externals Most likely you didn't build it correctly. Go back to a working cm3/m3core/libm3 and run upgrade.py. There should be no longer any references to alloca of Csetjmp__Jumpbuf_size. Really, Csetjmp__Jumpbuf_size should never have been a problem. That likely also indicates some sort of incorrect bootstrap. > Are you saying you do, or do not, want me to work further on the solving the ?problem,? regardless of how we describe it ? I still want alloca to work, either via a function call or a specific m3cg method. (I still don't see that RTHooks.Alloca makes better sense; I still don't see a way to avoid hacking the compiler, given the custom calling convention.) It will be difficult to test the fix until/unless m3front is changed back. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Fri, 11 Feb 2011 18:17:15 -0500 Subject: Re: [M3devel] threading on Windows? Jay: Ok, I don?t mean to start a new issue or upset anyone. I see you checked in some new changes. I updated to latest HEAD and tried to build, but again I get unresolved symbols beginning with m3-libs\sysutils (see attached sysutils.lst) So, not sure what you mean by ?it should be ok now?. Are you saying you do, or do not, want me to work further on the solving the ?problem,? regardless of how we describe it ? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Friday, February 11, 2011 4:52 PM To: Tony; Coleburn, Randy Cc: m3devel Subject: RE: [M3devel] threading on Windows? It isn't heap-based! And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak". It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function. I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else. It should be ok now. But still only a temporary solution. We have to either move all the back to the old way, or make the new way work. As we have two implementations of many things, it is common for one or the other to break. The fix is really not difficult. We can't be stuck like this on everything. But granted, m3front remains confusing to me, so I am stuck for a bit doing this much better. Anyway, my time is now significantly reallocated, so it all matters less. - Jay From: hosking at cs.purdue.edu Date: Fri, 11 Feb 2011 16:29:43 -0500 To: rcolebur at SCIRES.COM CC: m3devel at elegosoft.com Subject: Re: [M3devel] threading on Windows? Randy, I am in complete agreement with you. Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote: Tony: Thank you very much for this explanation. So, it appears that the changes introduced to support the new ?heap-based jmpbuf allocation? have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely: 1. Space leak problem. 2. Binding to appropriate ?alloca? routine on Windows platform. Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform. I will begin looking at the code and see if I can make some progress on these 2 problems. Before I ?do? anything, I?ll run my proposed solution thru the m3devel group first. Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated. Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. Seems to me that if the ?heap based jumpbuf allocation scheme? was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms. Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken. If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE. Regards, Randy Coleburn From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 4:26 PM To: Coleburn, Randy Cc: m3devel Subject: Re: [M3devel] threading on Windows? Hi Randy, A far as I know, the changes are as follows. TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf. Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size. This meant having to declare the jmp_buf accurately for the target. What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage. The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately, so that targets need no longer declare their jmpbuf_size. This was intended to make porting easier. There are several problems here. First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case. The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak. The other problem was how to bind to the appropriate alloca routine. Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca. My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function. On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined. To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation. I don't think this will involve much work, but I don't currently have any spare cycles to spend on it. -- Tony Antony Hosking | Associate Professor | Computer Science | Purdue University 305 N. University Street | West Lafayette | IN 47907 | USA Office +1 765 494 6001 | Mobile +1 765 427 5484 On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote: Jay / Tony: I am seeing in this thread that the approach to fixing the problem is not yet agreed upon. I will be glad to help in any way I can, but right now, I am ?in the dark?, both as to what change was introduced that caused the problem, and the various potential solution paths. All I know is that it has been broken for a number of weeks now. I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions. If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future. Otherwise, I will have to wait for one of you to solve the problem. BTW, when I look at the linker error reports for the various packages, I?m seeing several different symbols that don?t resolve, not just the one dealing with memory allocation. Regards, Randy From: Tony Hosking [mailto:hosking at cs.purdue.edu] Sent: Thursday, February 10, 2011 3:33 PM To: jay.krell at cornell.edu Cc: Coleburn, Randy; m3devel Subject: Re: [M3devel] threading on Windows? Why is it better? Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler. On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote: 1. I never understood why this any better. 2. It how things are currently. 3. Why not m3cg.alloca? (but see #2) - Jay/iPad On Feb 9, 2011, at 9:48 PM, Tony Hosking wrote: What happened to fixing this with runtime hooks? On Feb 9, 2011, at 11:40 PM, Jay K wrote: Of course not. It is a made-up function that the frontend generates a call to. That it never previously did. Using the general function call interface. You need to add a special case, in the general function call code, to do something different and specific for this specific function. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Wed, 9 Feb 2011 22:28:45 -050 Subject: Re: [M3devel] threading on Windows? Jay: I looked thru the 4,618 lines of M3x86.m3, but I don?t see any reference to ?m3_alloca? or even ?alloc? in this file. Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 9:01 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] threading on Windows? In m3back/src/M3x86.m3. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Wed, 9 Feb 2011 18:22:56 -0500 Subject: Re: [M3devel] threading on Windows? I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve. All I know is that everything was working fine until I checked out the HEAD repository. When you say ?special case? calls ?m3_alloca?, where do I go about finding this ?special case?? Are we talking Modula-3 code, C code, Assembler, what? What package/module? Regards, Randy From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Wednesday, February 09, 2011 5:59 PM To: Coleburn, Randy; Mika Nystrom; m3devel Subject: RE: [M3devel] threading on Windows? Maybe someone else can do it? The fix is: special case calls to the function "m3_alloca". Change it to call "_chkstk" (or maybe "chkstk", whatever works). The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp. To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how. - Jay > From: rcolebur at SCIRES.COM > To: mika at async.caltech.edu; m3devel at elegosoft.com > Date: Wed, 9 Feb 2011 17:25:20 -0500 > Subject: Re: [M3devel] threading on Windows? > > Mika: > > Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in. > > I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link. > > As soon as I can get the build problem resolved, I'll try out your test program. > > Regards, > Randy Coleburn > > -----Original Message----- > From: Mika Nystrom [mailto:mika at async.caltech.edu] > Sent: Wednesday, February 09, 2011 5:01 PM > To: m3devel at elegosoft.com > Subject: [M3devel] threading on Windows? > > Hi m3devel, > > I'm just curious if anyone out there who's running CM3 on Windows has had > a chance to try my thread testing program yet. Current status on Unix > (Linux, FreeBSD) appears to be that there are problems in pthreads (but > I think only under heavy load) and that user threading works perfectly. > So I now wonder how things are on the third threading platform (Windows). > The thread testing program is at m3-libs/m3core/tests/thread . > > (Note however that there is a memory leak in TRY-EXCEPT in the current > head so don't try to update everything to the CVS head. The previous > release should be OK.) > > Also has anyone else noticed that debugging information has recently > broken? m3gdb is very unhappy on all platforms for me... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Tue Feb 15 02:56:57 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Mon, 14 Feb 2011 17:56:57 -0800 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20110215013018.B4DC12474005@birch.elegosoft.com> Message-ID: <20110215015657.38D541A2078@async.async.caltech.edu> Jay K writes: >--_0b68c717-b1e6-43b7-b824-79e05d0f3a43_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >You might want to throttle the number of outstanding promises. Yeah that's in a different module. QMachine doesn't control what's actually done with the promises (how many in parallel are run etc). The idea is: 1. client of QMachine sets "doRecord" flag 2. QMachine writes all the back-end promises to a sequence. 3. client clears doRecord flag After that it's up to the client to launch the promises as it desires, one at a time or 100 at a time, as the case may be. I'm looking at how to specify this now. You're right it's annoying there's no automatic way of doing it. Default will be serial as always, the impatient can specify a flag or a setting somewhere. I'm leaning towards making it both a config variable and a command-line option... The code is kind of ugly but it sped up my compiles so much I immediately felt I couldn't live without it....I'm taking care that it won't change the behavior for anyone else (unless they ask for it). Mika >If only Posix provided good functions for e.g. finding the number of proces= >sors. >Or if NT or Posix made it easy to discover the number of spindles backing a= > path=2C >or to easily measure system utilization and backing down when it gets near = >100%. >You know=2C an idle machine should be maxed out=2C but no more. >A machine with an interactive user should reserve CPU/memory/I/O for forgro= >und activity.. >I don't I've ever seen this stuff done well though. >=20 > - Jay > >=20 >> Date: Tue=2C 15 Feb 2011 02:30:18 +0000 >> To: m3commit at elegosoft.com >> From: mika at elego.de >> Subject: [M3commit] CVS Update: cm3 >>=20 >> CVSROOT: /usr/cvs >> Changes by: mika at birch. 11/02/15 02:30:18 >>=20 >> Modified files: >> cm3/m3-sys/m3quake/src/: QMachine.i3 QMachine.m3 m3makefile=20 >> Added files: >> cm3/m3-sys/m3quake/src/: QPromise.i3 QPromise.m3=20 >>=20 >> Log message: >> Add support for the QMachine to be able to record (certain) Process.Creat= >es as "promises" rather than executing them on the fly. >>=20 >> The purpose of this modification is to allow parallelization of the CM3 c= >ompiler. The vast majority of the runtime of the compilation of a typical M= >odula-3 program is spent in the part "promised" here (when using the GCC ba= >ckend). >>=20 > = > >--_0b68c717-b1e6-43b7-b824-79e05d0f3a43_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >You might want to throttle the number of outstanding promises.
>If only Posix provided good functions for e.g. finding the number of proces= >sors.
>Or if =3BNT or Posix made it easy to discover the number of spindles ba= >cking a path=2C
>or to easily measure system utilization and backing down when it =3Bget= >s near 100%.
>You know=2C an idle machine should be maxed out=2C but no more.
>A machine with an interactive user should reserve =3BCPU/memory/I/O for= > forground activity..
>I don't I've ever seen this stuff done well though.
> =3B
> =3B- Jay

 =3B
>>=3B Date: Tue=2C 15 Feb 2011 02:30:18 +0000
>=3B To: m3commit at elego= >soft.com
>=3B From: mika at elego.de
>=3B Subject: [M3commit] CVS Up= >date: cm3
>=3B
>=3B CVSROOT: /usr/cvs
>=3B Changes by: mika= >@birch. 11/02/15 02:30:18
>=3B
>=3B Modified files:
>=3B cm= >3/m3-sys/m3quake/src/: QMachine.i3 QMachine.m3 m3makefile
>=3B Added = >files:
>=3B cm3/m3-sys/m3quake/src/: QPromise.i3 QPromise.m3
>= >=3B
>=3B Log message:
>=3B Add support for the QMachine to be ab= >le to record (certain) Process.Creates as "promises" rather than executing = >them on the fly.
>=3B
>=3B The purpose of this modification is t= >o allow parallelization of the CM3 compiler. The vast majority of the runti= >me of the compilation of a typical Modula-3 program is spent in the part "p= >romised" here (when using the GCC backend).
>=3B
> y> >= > >--_0b68c717-b1e6-43b7-b824-79e05d0f3a43_-- From wagner at elegosoft.com Tue Feb 15 08:53:37 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Tue, 15 Feb 2011 08:53:37 +0100 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110214211829.BA3761A2078@async.async.caltech.edu> References: <20110213220804.7A9001A2078@async.async.caltech.edu> <20110214211829.BA3761A2078@async.async.caltech.edu> Message-ID: <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> Yes, please commit changes that improve matters, even if they are not perfect. Perhaps Tony will review your latest suggestion again? Your thread test program should be integrated into the Hudson regression test framework for the cm3 current line; I don't think it is already. That way we would get an overview of the thread state for all platforms at least once a day. It would be great if the tests would show up in the cm3-current-test-m3tests-TARGET jobs; it shouldn't be too difficult. Basically, only one script should need to be extended, and some XML result generation added. IIRC it's in cm3/scripts/regression/defs.sh in test_m3tests(), or (in quake) in the m3tests m3makefile. Can you do that? If not, I may have a look myself at the weekend... Olaf PS: We really should get all our code base into a state that you can run all your applications without any problems on current. After all there aren't that many active M3 users... PPS: I need to return to my offline testing of LDAP protocol handlers for online charging servers now :-/ Quoting Mika Nystrom : > Hi again m3devel, > > Does anyone understand the code in ThreadPosix.m3? > > After looking through it again I think my attached solution (sent out > yesterday afternoon) is right, but I'm still just a little bit fuzzy on > how the code works (is supposed to work) but I have some hypotheses. > > I realized it's been a long, long, loooong time since I coded anything > with signals and signal handlers and I've forgotten much of the semantics. > After all since Concurrent Pascal came out in '76 signals has always been > the "old" way of doing things, right? (And I quit programming C in favor > of M3 in 1999...) > > In ThreadPosixC.c, there's a function called setup_sigvtalrm that > registers the Modula-3 procedure switch_thread as a signal handler. > > My understanding is that in C, signals are masked atomically on the > call into the signal handler and unmasked atomically when the handler > returns (unless the signal mask is changed, etc., while in the handler). > > However, switch_thread... doesn't return? Or? Or it does a longjmp > somewhere? Which means signals may not be unmasked(?) Which is the reason > for the calling "allow_sigvtalrm" as the first statement of the signal > handler? > > Now I can't just turn on ALL signals in the signal handler because then > I may get very deeply nested signal handlers, so I only call allow_allsignals > (or whatever we call it) from the first-level signal handler. Hmm. > Maybe that is not quite right? Do we eventually get back there? Yes > I think we must because certain things depend on inCritical being zero... > and when inCritical is zero SIGCHLD is again enabled. > > No I still don't fully understand how the code is supposed to work, > especially in the presence of nested signal handling. Does anyone > understand better? > > (I am still going to try to commit my patch later today because I don't > think it could possibly make things worse and it does make a significant > real-world improvement to the compiler performance under user threads.) > > Mika > > Mika Nystrom writes: >> Hi Tony, >> >> I found a problem with the change. I would appreciate it if you and >> Jay could think about what I found and my solution (and anybody else >> is welcome to as well, of course). >> >> When fiddling with cm3 to get it to run in parallel, I pushed it up to >> fork up to 100 processes at once, just to see what happens. When I did >> that (with my SIGCHLD patch) I got a few errors such as illegal instruction >> and a bus error or two. I believe what was happening was that I was >> taking too many signals at once, because of the following code in >> ThreadPosix.m3: >> >> PROCEDURE switch_thread (sig: int) RAISES {Alerted} = >> BEGIN >> allow_sigvtalrm (); (* here *) >> >> INC(inCritical); >> (* mark signal as being delivered *) >> IF sig >= 0 AND sig < MaxSigs THEN >> gotSigs := gotSigs + SET OF Sig { sig } >> END; >> DEC(inCritical); >> >> IF inCritical = 0 AND heapState.inCritical = 0 THEN >> InternalYield () >> END; >> END switch_thread; >> >> I found it a bit odd that the first action of the signal handler is to >> enable signals. With VTALRM this is innocuous, but in my case, imagine >> what happens if you have a lot of exited child processes queued up. >> What happens is that your signal stack gets wound deeper and deeper, and >> each time you catch one SIGCHLD you ask for another. Eventually you >> run out of stack space. >> >> My change, which appears to fix this problem, is to split up the "allow" >> function as follows: >> >> PROCEDURE switch_thread (sig: int) RAISES {Alerted} = >> BEGIN >> allow_sigvtalrm (); (* VTALRM only *) >> >> INC(inCritical); >> IF inCritical = 1 THEN allow_othersigs (); END; (* SIGCHLD *) >> >> (* mark signal as being delivered *) >> IF sig >= 0 AND sig < MaxSigs THEN >> gotSigs := gotSigs + SET OF Sig { sig } >> END; >> DEC(inCritical); >> >> IF inCritical = 0 AND heapState.inCritical = 0 THEN >> InternalYield () >> END; >> END switch_thread; >> >> This way, SIGCHLD is not taken in nested signal handlers. But is there a >> problem here? Well, the fallback is that it takes 0.1 seconds to wake >> up on Process.Wait . >> >> Is there a chance that SIGCHLD signals are left pending? My argument is >> no, since in the "base" environment (i.e., not executing in a >> signal handler) >> SIGCHLD is enabled as a postcondition of allow_othersigs. >> >> All *other* occurrences of allow_sigvtalrm are of course replaced with >> >> allow_sigvtalrm(); allow_othersigs(); >> >> The goal is that SIGCHLD be enabled precisely when SIGVTALRM is >> except inside >> the signal handler itself. >> >> Mika >> >> >> Tony Hosking writes: >>> Sorry, I missed you e-mail describing this change. Looks good. >>> >>> Antony Hosking | Associate Professor | Computer Science | Purdue = >>> University >>> 305 N. University Street | West Lafayette | IN 47907 | USA >>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>> >>> >>> >>> >>> On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: >>> >>>> Tony, >>>> =20 >>>> The old user threads has an old problem with Process.Wait. It has a=20= >>> >>>> Thread.Pause(0.1d0) in it. This causes the compiler to run three = >>> times >>>> slower than it should. The m3build I've been using for years has >>>> that Thread.Pause removed (so it uses 100% CPU instead, = >>> busy-waiting...) >>>> =20 >>>> But yeah user threads works the same as it used to (except for=20 >>>> now calling pthread_atfork, which I'll return to in another email...) >>>> =20 >>>> Mika > ... > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From rcolebur at SCIRES.COM Tue Feb 15 23:05:37 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Tue, 15 Feb 2011 17:05:37 -0500 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , <20110213181717.4F0E91A2078@async.async.caltech.edu>, Message-ID: Jay, since I have a Windows2000 system, I can test the MAC address discovery (or anything else you need). Just let me know. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Monday, February 14, 2011 8:12 PM To: Coleburn, Randy; m3devel Subject: Re: [M3devel] More on threading Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Wed Feb 16 01:56:49 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 16 Feb 2011 00:56:49 +0000 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , , , <20110213181717.4F0E91A2078@async.async.caltech.edu>, , , , Message-ID: It should work. http://dcvs.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/uid/WIN32/MachineID.c?rev=1.14;content-type=text%2Fplain Bing for "MSDN GetIfTable" => http://msdn.microsoft.com/en-us/library/aa365943(VS.85).aspx Mika, what made you think cm3 doesn't work on Windows 2000?? I don't think anything has been remotely suggested. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Tue, 15 Feb 2011 17:05:37 -0500 Subject: Re: [M3devel] More on threading Jay, since I have a Windows2000 system, I can test the MAC address discovery (or anything else you need). Just let me know. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Monday, February 14, 2011 8:12 PM To: Coleburn, Randy; m3devel Subject: Re: [M3devel] More on threading Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Wed Feb 16 05:17:06 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Tue, 15 Feb 2011 20:17:06 -0800 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , , , <20110213181717.4F0E91A2078@async.async.caltech.edu>, , , , Message-ID: <20110216041706.87A2D1A2078@async.async.caltech.edu> Jay K writes: >--_a2f17729-cb11-4f2b-8c70-c3b583cda751_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >It should work. >=20 >=20 >http://dcvs.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/uid/WIN3= >2/MachineID.c?rev=3D1.14=3Bcontent-type=3Dtext%2Fplain >=20 >=20 >Bing for "MSDN GetIfTable" >=3D> http://msdn.microsoft.com/en-us/library/aa365943(VS.85).aspx >=20 >=20 >Mika=2C what made you think cm3 doesn't work on Windows 2000?? >I don't think anything has been remotely suggested. Hi Jay, I searched my email and I do indeed misremember---sort of. I had an email exchange with you on Friday 28 August 2009 where I ran into a bunch of problems that I didn't know how to solve, and I gave up there. And you asked me if I "planned to upgrade to XP or newer anytime soon". I may have taken that as a sign that I shouldn't expect things to work on win2k but you didn't actually say that anywhere. Mika From wagner at elegosoft.com Thu Feb 17 23:47:30 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Thu, 17 Feb 2011 23:47:30 +0100 Subject: [M3devel] cm3-ide In-Reply-To: References: Message-ID: <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com> Quoting ??? ????????? : > Hi, > I am a student of Software Engineering. > I want to learn the language modula-3 and I'm trying to install IDE on my > computer. > I have windows 7 operating system and I can not find any suitable > installation file. > Do you have appropriate file or instructions how to do it on another system? > I need it for exercise in school and I'd love it if you could help me. > Thank you. > zeev blumentzvaig What exactly have you tried already? The archive for Windows listed on http://www.opencm3.net/download.html, http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should work for Windows 7, too, as far as I know. You simply need to unpack it and install a suitable linker, as described for example in http://www.opencm3.net/installation-windows.html. In order to help you, we need to know exactly what you have done and what failed. I'm not an expert on Windows; others on the list may be able to help better, so I CC this to m3devel. Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Thu Feb 17 23:56:05 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Thu, 17 Feb 2011 14:56:05 -0800 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> References: <20110213220804.7A9001A2078@async.async.caltech.edu> <20110214211829.BA3761A2078@async.async.caltech.edu> <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> Message-ID: <20110217225605.644AC1A2078@async.async.caltech.edu> Hi Olaf, I have committed the SIGCHLD diff and run quite heavy jobs with it in the code, and so far I have not seen any problems. I am fairly sure there won't be any. Can you show me how to integrate the test program? Note that when it fails it quite often (usually) hangs rather than producing any specific error output. I have also definitely run into pthreads problems on I386_DARWIN now, not running the threadtest program. They show up when I enable the parallel back-end builds (I suspect they will do that on all architectures with pthreads---I've rebuilt all my CM3 installations with user threads). Mika Olaf Wagner writes: >Yes, please commit changes that improve matters, even if they are not >perfect. Perhaps Tony will review your latest suggestion again? > >Your thread test program should be integrated into the Hudson regression >test framework for the cm3 current line; I don't think it is already. >That way we would get an overview of the thread state for all platforms >at least once a day. It would be great if the tests would show up >in the cm3-current-test-m3tests-TARGET jobs; it shouldn't be too >difficult. Basically, only one script should need to be extended, and >some XML result generation added. IIRC it's in cm3/scripts/regression/defs.s= >h >in test_m3tests(), or (in quake) in the m3tests m3makefile. > >Can you do that? If not, I may have a look myself at the weekend... > >Olaf > >PS: We really should get all our code base into a state that you can >run all your applications without any problems on current. After all >there aren't that many active M3 users... > >PPS: I need to return to my offline testing of LDAP protocol handlers >for online charging servers now :-/ > >Quoting Mika Nystrom : > >> Hi again m3devel, >> >> Does anyone understand the code in ThreadPosix.m3? >> >> After looking through it again I think my attached solution (sent out >> yesterday afternoon) is right, but I'm still just a little bit fuzzy on >> how the code works (is supposed to work) but I have some hypotheses. >> >> I realized it's been a long, long, loooong time since I coded anything >> with signals and signal handlers and I've forgotten much of the semantics. >> After all since Concurrent Pascal came out in '76 signals has always been >> the "old" way of doing things, right? (And I quit programming C in favor >> of M3 in 1999...) >> >> In ThreadPosixC.c, there's a function called setup_sigvtalrm that >> registers the Modula-3 procedure switch_thread as a signal handler. >> >> My understanding is that in C, signals are masked atomically on the >> call into the signal handler and unmasked atomically when the handler >> returns (unless the signal mask is changed, etc., while in the handler). >> >> However, switch_thread... doesn't return? Or? Or it does a longjmp >> somewhere? Which means signals may not be unmasked(?) Which is the reaso= >n >> for the calling "allow_sigvtalrm" as the first statement of the signal >> handler? >> >> Now I can't just turn on ALL signals in the signal handler because then >> I may get very deeply nested signal handlers, so I only call allow_allsign= >als >> (or whatever we call it) from the first-level signal handler. Hmm. >> Maybe that is not quite right? Do we eventually get back there? Yes >> I think we must because certain things depend on inCritical being zero... >> and when inCritical is zero SIGCHLD is again enabled. >> >> No I still don't fully understand how the code is supposed to work, >> especially in the presence of nested signal handling. Does anyone >> understand better? >> >> (I am still going to try to commit my patch later today because I don't >> think it could possibly make things worse and it does make a significant >> real-world improvement to the compiler performance under user threads.) >> >> Mika >> >> Mika Nystrom writes: >>> Hi Tony, >>> >>> I found a problem with the change. I would appreciate it if you and >>> Jay could think about what I found and my solution (and anybody else >>> is welcome to as well, of course). >>> >>> When fiddling with cm3 to get it to run in parallel, I pushed it up to >>> fork up to 100 processes at once, just to see what happens. When I did >>> that (with my SIGCHLD patch) I got a few errors such as illegal instructi= >on >>> and a bus error or two. I believe what was happening was that I was >>> taking too many signals at once, because of the following code in >>> ThreadPosix.m3: >>> >>> PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>> BEGIN >>> allow_sigvtalrm (); (* here *) >>> >>> INC(inCritical); >>> (* mark signal as being delivered *) >>> IF sig >=3D 0 AND sig < MaxSigs THEN >>> gotSigs :=3D gotSigs + SET OF Sig { sig } >>> END; >>> DEC(inCritical); >>> >>> IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >>> InternalYield () >>> END; >>> END switch_thread; >>> >>> I found it a bit odd that the first action of the signal handler is to >>> enable signals. With VTALRM this is innocuous, but in my case, imagine >>> what happens if you have a lot of exited child processes queued up. >>> What happens is that your signal stack gets wound deeper and deeper, and >>> each time you catch one SIGCHLD you ask for another. Eventually you >>> run out of stack space. >>> >>> My change, which appears to fix this problem, is to split up the "allow" >>> function as follows: >>> >>> PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>> BEGIN >>> allow_sigvtalrm (); (* VTALRM only *) >>> >>> INC(inCritical); >>> IF inCritical =3D 1 THEN allow_othersigs (); END; (* SIGCHLD *) >>> >>> (* mark signal as being delivered *) >>> IF sig >=3D 0 AND sig < MaxSigs THEN >>> gotSigs :=3D gotSigs + SET OF Sig { sig } >>> END; >>> DEC(inCritical); >>> >>> IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >>> InternalYield () >>> END; >>> END switch_thread; >>> >>> This way, SIGCHLD is not taken in nested signal handlers. But is there a >>> problem here? Well, the fallback is that it takes 0.1 seconds to wake >>> up on Process.Wait . >>> >>> Is there a chance that SIGCHLD signals are left pending? My argument is >>> no, since in the "base" environment (i.e., not executing in a =20 >>> signal handler) >>> SIGCHLD is enabled as a postcondition of allow_othersigs. >>> >>> All *other* occurrences of allow_sigvtalrm are of course replaced with >>> >>> allow_sigvtalrm(); allow_othersigs(); >>> >>> The goal is that SIGCHLD be enabled precisely when SIGVTALRM is =20 >>> except inside >>> the signal handler itself. >>> >>> Mika >>> >>> >>> Tony Hosking writes: >>>> Sorry, I missed you e-mail describing this change. Looks good. >>>> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue =3D >>>> University >>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>> >>>> >>>> >>>> >>>> On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: >>>> >>>>> Tony, >>>>> =3D20 >>>>> The old user threads has an old problem with Process.Wait. It has a=3D= >20=3D >>>> >>>>> Thread.Pause(0.1d0) in it. This causes the compiler to run three =3D >>>> times >>>>> slower than it should. The m3build I've been using for years has >>>>> that Thread.Pause removed (so it uses 100% CPU instead, =3D >>>> busy-waiting...) >>>>> =3D20 >>>>> But yeah user threads works the same as it used to (except for=3D20 >>>>> now calling pthread_atfork, which I'll return to in another email...) >>>>> =3D20 >>>>> Mika >> ... >> > > > >--=20 >Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa= >ny >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz= >: Berlin >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Fri Feb 18 09:32:46 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 18 Feb 2011 09:32:46 +0100 Subject: [M3devel] Fwd: Re: cm3-ide Message-ID: <20110218093246.fzjch5luokkwgokk@mail.elegosoft.com> FYI ----- Forwarded message from zeev.blu at gmail.com ----- Date: Fri, 18 Feb 2011 09:03:38 +0200 From: ??? ????????? Reply-To: ??? ????????? Subject: Re: cm3-ide To: Olaf Wagner Cc: m3devel at elegosoft.com, m3-support at elego.de, moriah.moshe at gmail.com Hi, First, thanks for the quick reply. I have tried to use the file you gave me, and some other files from that site. I have some questions: 1. If I understand correctly, the method of working with this compiler is through the CMD in WINDOWS. Is not this language a IDE to work with? 2. When I tried to compile , I got a linking error: " link is not recognized as an internal or external command, operable program or batch file." And then another error: "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " And finally I received: " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" failed. " Thanks for your help zeev blumentzvaig 2011/2/18 Olaf Wagner > Quoting ??? ????????? : > > Hi, >> I am a student of Software Engineering. >> I want to learn the language modula-3 and I'm trying to install IDE on my >> computer. >> I have windows 7 operating system and I can not find any suitable >> installation file. >> Do you have appropriate file or instructions how to do it on another >> system? >> I need it for exercise in school and I'd love it if you could help me. >> Thank you. >> zeev blumentzvaig >> > > What exactly have you tried already? > > The archive for Windows listed on http://www.opencm3.net/download.html, > http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should > work for Windows 7, too, as far as I know. You simply need to > unpack it and install a suitable linker, as described for example in > http://www.opencm3.net/installation-windows.html. > > In order to help you, we need to know exactly what you have done and > what failed. I'm not an expert on Windows; others on the list may > be able to help better, so I CC this to m3devel. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 > 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: > DE163214194 > > ----- End forwarded message ----- -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 -------------- next part -------------- Hi, First, thanks for the quick reply. I have tried to use the file you gave me, and some other files from that site. I have some questions: 1. If I understand correctly, the method of working with this compiler is through the CMD in WINDOWS. Is not this language a IDE to work with? 2. When I tried to compile , I got a linking error: " link is not recognized as an internal or external command, operable program or batch file." And then another error: "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " And finally I received: " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" failed. " Thanks for your help zeev blumentzvaig 2011/2/18 Olaf Wagner > Quoting ??? ????????? : > > Hi, >> I am a student of Software Engineering. >> I want to learn the language modula-3 and I'm trying to install IDE on my >> computer. >> I have windows 7 operating system and I can not find any suitable >> installation file. >> Do you have appropriate file or instructions how to do it on another >> system? >> I need it for exercise in school and I'd love it if you could help me. >> Thank you. >> zeev blumentzvaig >> > > What exactly have you tried already? > > The archive for Windows listed on http://www.opencm3.net/download.html, > http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should > work for Windows 7, too, as far as I know. You simply need to > unpack it and install a suitable linker, as described for example in > http://www.opencm3.net/installation-windows.html. > > In order to help you, we need to know exactly what you have done and > what failed. I'm not an expert on Windows; others on the list may > be able to help better, so I CC this to m3devel. > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 > 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: > DE163214194 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Fri Feb 18 09:40:14 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 18 Feb 2011 09:40:14 +0100 Subject: [M3devel] cm3-ide In-Reply-To: References: <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com> Message-ID: <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> Quoting ??? ????????? : > Hi, > First, thanks for the quick reply. > > I have tried to use the file you gave me, and some other files from that > site. > I have some questions: > 1. If I understand correctly, the method of working with this compiler is > through the CMD in WINDOWS. Is not this language a IDE to work with? A language and its implementation in a compiler does not necessarily require a full-fledged intergrated development environment like Eclipse. My faourite IDE when I was actively using M3 was always XEmacs :-) There _is_ a web-based cm3-ide (which should be contained in that archive) that allows you to build and manage your projects with a standard web browser, but that might be quite different from your expectations of an IDE, too ;-) > 2. When I tried to compile , I got a linking error: > > " link is not recognized as an internal or external command, operable > program or batch file." > > And then another error: > > "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " > > And finally I received: > > " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" > failed. " You either missed the installation of a suitable system linker as described in the "Windows installation notes" referred to below or you didn't set the appropriate variables in your environment or in the cm3 configuration files. CM3 does not include a system linker, and on Windows you have got to install one yourself, while on all other systems it usually is installed by default. You can use the cm3 option "-commands" to actually view the external commands the builder is calling. I hope this helps, Olaf > Thanks for your help > > zeev blumentzvaig > > > 2011/2/18 Olaf Wagner > >> Quoting ??? ????????? : >> >> Hi, >>> I am a student of Software Engineering. >>> I want to learn the language modula-3 and I'm trying to install IDE on my >>> computer. >>> I have windows 7 operating system and I can not find any suitable >>> installation file. >>> Do you have appropriate file or instructions how to do it on another >>> system? >>> I need it for exercise in school and I'd love it if you could help me. >>> Thank you. >>> zeev blumentzvaig >>> >> >> What exactly have you tried already? >> >> The archive for Windows listed on http://www.opencm3.net/download.html, >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should >> work for Windows 7, too, as far as I know. You simply need to >> unpack it and install a suitable linker, as described for example in >> http://www.opencm3.net/installation-windows.html. >> >> In order to help you, we need to know exactly what you have done and >> what failed. I'm not an expert on Windows; others on the list may >> be able to help better, so I CC this to m3devel. >> >> Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From wagner at elegosoft.com Fri Feb 18 10:14:14 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 18 Feb 2011 10:14:14 +0100 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110217225605.644AC1A2078@async.async.caltech.edu> References: <20110213220804.7A9001A2078@async.async.caltech.edu> <20110214211829.BA3761A2078@async.async.caltech.edu> <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> <20110217225605.644AC1A2078@async.async.caltech.edu> Message-ID: <20110218101414.yu4y8iyg0kkw40cg@mail.elegosoft.com> Quoting Mika Nystrom : > Hi Olaf, > > I have committed the SIGCHLD diff and run quite heavy jobs with it > in the code, and so far I have not seen any problems. I am fairly sure > there won't be any. Sounds good. > Can you show me how to integrate the test program? I already gave some hints below. I think what you definitely could do is writing the required XML output, either in M3 or as shell script. There's a simple schema describing what is required; some time ago we've put a copy on our web server: http://www.elegosoft.com/misc/JUnitXSchema.xsd It may be easier to just look at an example, e.g. http://hudson.modula3.com:8080/job/cm3-current-test-m3tests-AMD64_FREEBSD/ws/cm3/m3-sys/m3tests/m3tests-results.xml > Note that when it fails > it quite often (usually) hangs rather than producing any specific error > output. If we cannot avoid non-termination on our own, we must rely on the Hudson job timeout. This would definitely mean that we run the program in their own jobs or at the end of the other tests. Still, I think the whole test job will fail if it doesn't terminate. A runtime supervision for regression testing is something that is missing in our M3 world as has been noted before :-( If you provide the XML output, I can care for the integration into Hudson this or the next weekend. Or alternatively, you can do that on your own if you like and I help as good as I can. If you haven't got an account on Hudson, I can quickly create one. > I have also definitely run into pthreads problems on I386_DARWIN now, > not running the threadtest program. They show up when I enable the > parallel back-end builds (I suspect they will do that on all architectures > with pthreads---I've rebuilt all my CM3 installations with user threads). Hm, sounds like the complete switch to pthreads might have been a little too early, but we haven't had any reliable data to decide on IIRC. All the world uses pthreads, so I assumed they would be OK for us, too :-/ However, we need to fix the problems now; I hope Antony can spare some time. He's been the main architect for that. And it's good to know that user threads still work. Olaf > Mika > > Olaf Wagner writes: >> Yes, please commit changes that improve matters, even if they are not >> perfect. Perhaps Tony will review your latest suggestion again? >> >> Your thread test program should be integrated into the Hudson regression >> test framework for the cm3 current line; I don't think it is already. >> That way we would get an overview of the thread state for all platforms >> at least once a day. It would be great if the tests would show up >> in the cm3-current-test-m3tests-TARGET jobs; it shouldn't be too >> difficult. Basically, only one script should need to be extended, and >> some XML result generation added. IIRC it's in >> cm3/scripts/regression/defs.s= >> h >> in test_m3tests(), or (in quake) in the m3tests m3makefile. >> >> Can you do that? If not, I may have a look myself at the weekend... >> >> Olaf >> >> PS: We really should get all our code base into a state that you can >> run all your applications without any problems on current. After all >> there aren't that many active M3 users... >> >> PPS: I need to return to my offline testing of LDAP protocol handlers >> for online charging servers now :-/ >> >> Quoting Mika Nystrom : >> >>> Hi again m3devel, >>> >>> Does anyone understand the code in ThreadPosix.m3? >>> >>> After looking through it again I think my attached solution (sent out >>> yesterday afternoon) is right, but I'm still just a little bit fuzzy on >>> how the code works (is supposed to work) but I have some hypotheses. >>> >>> I realized it's been a long, long, loooong time since I coded anything >>> with signals and signal handlers and I've forgotten much of the semantics. >>> After all since Concurrent Pascal came out in '76 signals has always been >>> the "old" way of doing things, right? (And I quit programming C in favor >>> of M3 in 1999...) >>> >>> In ThreadPosixC.c, there's a function called setup_sigvtalrm that >>> registers the Modula-3 procedure switch_thread as a signal handler. >>> >>> My understanding is that in C, signals are masked atomically on the >>> call into the signal handler and unmasked atomically when the handler >>> returns (unless the signal mask is changed, etc., while in the handler). >>> >>> However, switch_thread... doesn't return? Or? Or it does a longjmp >>> somewhere? Which means signals may not be unmasked(?) Which is the reaso= >> n >>> for the calling "allow_sigvtalrm" as the first statement of the signal >>> handler? >>> >>> Now I can't just turn on ALL signals in the signal handler because then >>> I may get very deeply nested signal handlers, so I only call allow_allsign= >> als >>> (or whatever we call it) from the first-level signal handler. Hmm. >>> Maybe that is not quite right? Do we eventually get back there? Yes >>> I think we must because certain things depend on inCritical being zero... >>> and when inCritical is zero SIGCHLD is again enabled. >>> >>> No I still don't fully understand how the code is supposed to work, >>> especially in the presence of nested signal handling. Does anyone >>> understand better? >>> >>> (I am still going to try to commit my patch later today because I don't >>> think it could possibly make things worse and it does make a significant >>> real-world improvement to the compiler performance under user threads.) >>> >>> Mika >>> >>> Mika Nystrom writes: >>>> Hi Tony, >>>> >>>> I found a problem with the change. I would appreciate it if you and >>>> Jay could think about what I found and my solution (and anybody else >>>> is welcome to as well, of course). >>>> >>>> When fiddling with cm3 to get it to run in parallel, I pushed it up to >>>> fork up to 100 processes at once, just to see what happens. When I did >>>> that (with my SIGCHLD patch) I got a few errors such as illegal instructi= >> on >>>> and a bus error or two. I believe what was happening was that I was >>>> taking too many signals at once, because of the following code in >>>> ThreadPosix.m3: >>>> >>>> PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>>> BEGIN >>>> allow_sigvtalrm (); (* here *) >>>> >>>> INC(inCritical); >>>> (* mark signal as being delivered *) >>>> IF sig >=3D 0 AND sig < MaxSigs THEN >>>> gotSigs :=3D gotSigs + SET OF Sig { sig } >>>> END; >>>> DEC(inCritical); >>>> >>>> IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >>>> InternalYield () >>>> END; >>>> END switch_thread; >>>> >>>> I found it a bit odd that the first action of the signal handler is to >>>> enable signals. With VTALRM this is innocuous, but in my case, imagine >>>> what happens if you have a lot of exited child processes queued up. >>>> What happens is that your signal stack gets wound deeper and deeper, and >>>> each time you catch one SIGCHLD you ask for another. Eventually you >>>> run out of stack space. >>>> >>>> My change, which appears to fix this problem, is to split up the "allow" >>>> function as follows: >>>> >>>> PROCEDURE switch_thread (sig: int) RAISES {Alerted} =3D >>>> BEGIN >>>> allow_sigvtalrm (); (* VTALRM only *) >>>> >>>> INC(inCritical); >>>> IF inCritical =3D 1 THEN allow_othersigs (); END; (* SIGCHLD *) >>>> >>>> (* mark signal as being delivered *) >>>> IF sig >=3D 0 AND sig < MaxSigs THEN >>>> gotSigs :=3D gotSigs + SET OF Sig { sig } >>>> END; >>>> DEC(inCritical); >>>> >>>> IF inCritical =3D 0 AND heapState.inCritical =3D 0 THEN >>>> InternalYield () >>>> END; >>>> END switch_thread; >>>> >>>> This way, SIGCHLD is not taken in nested signal handlers. But is there a >>>> problem here? Well, the fallback is that it takes 0.1 seconds to wake >>>> up on Process.Wait . >>>> >>>> Is there a chance that SIGCHLD signals are left pending? My argument is >>>> no, since in the "base" environment (i.e., not executing in a =20 >>>> signal handler) >>>> SIGCHLD is enabled as a postcondition of allow_othersigs. >>>> >>>> All *other* occurrences of allow_sigvtalrm are of course replaced with >>>> >>>> allow_sigvtalrm(); allow_othersigs(); >>>> >>>> The goal is that SIGCHLD be enabled precisely when SIGVTALRM is =20 >>>> except inside >>>> the signal handler itself. >>>> >>>> Mika >>>> >>>> >>>> Tony Hosking writes: >>>>> Sorry, I missed you e-mail describing this change. Looks good. >>>>> >>>>> Antony Hosking | Associate Professor | Computer Science | Purdue =3D >>>>> University >>>>> 305 N. University Street | West Lafayette | IN 47907 | USA >>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484 >>>>> >>>>> >>>>> >>>>> >>>>> On Feb 13, 2011, at 3:37 PM, Mika Nystrom wrote: >>>>> >>>>>> Tony, >>>>>> =3D20 >>>>>> The old user threads has an old problem with Process.Wait. It has a=3D= >> 20=3D >>>>> >>>>>> Thread.Pause(0.1d0) in it. This causes the compiler to run three =3D >>>>> times >>>>>> slower than it should. The m3build I've been using for years has >>>>>> that Thread.Pause removed (so it uses 100% CPU instead, =3D >>>>> busy-waiting...) >>>>>> =3D20 >>>>>> But yeah user threads works the same as it used to (except for=3D20 >>>>>> now calling pthread_atfork, which I'll return to in another email...) >>>>>> =3D20 >>>>>> Mika >>> ... >>> >> >> >> >> --=20 >> Olaf Wagner -- elego Software Solutions GmbH >> Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa= >> ny >> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 >> http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz= >> : Berlin >> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Fri Feb 18 13:55:32 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 18 Feb 2011 12:55:32 +0000 Subject: [M3devel] cm3-ide In-Reply-To: <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> References: , <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com>, , <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> Message-ID: > A language and its implementation in a compiler does not necessarily > require a full-fledged intergrated development environment like Eclipse. Programmers today are not used to using plain text editors. They are used to wizards and intellisense. It would take a lot of work I think to make Modula-3 at all usable to any programmer under 30 years old. Even older folks have largely become spoiled. - Jay > Date: Fri, 18 Feb 2011 09:40:14 +0100 > From: wagner at elegosoft.com > To: zeev.blu at gmail.com > CC: m3devel at elegosoft.com; m3-support at elego.de; moriah.moshe at gmail.com > Subject: Re: [M3devel] cm3-ide > > Quoting ??? ????????? : > > > Hi, > > First, thanks for the quick reply. > > > > I have tried to use the file you gave me, and some other files from that > > site. > > I have some questions: > > 1. If I understand correctly, the method of working with this compiler is > > through the CMD in WINDOWS. Is not this language a IDE to work with? > > A language and its implementation in a compiler does not necessarily > require a full-fledged intergrated development environment like Eclipse. > My faourite IDE when I was actively using M3 was always XEmacs :-) > There _is_ a web-based cm3-ide (which should be contained in that > archive) that allows you to build and manage your projects with a > standard web browser, but that might be quite different from your > expectations of an IDE, too ;-) > > > 2. When I tried to compile , I got a linking error: > > > > " link is not recognized as an internal or external command, operable > > program or batch file." > > > > And then another error: > > > > "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " > > > > And finally I received: > > > > " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" > > failed. " > > You either missed the installation of a suitable system linker as > described in the "Windows installation notes" referred to below or > you didn't set the appropriate variables in your environment or in > the cm3 configuration files. CM3 does not include a system linker, > and on Windows you have got to install one yourself, while on all > other systems it usually is installed by default. > > You can use the cm3 option "-commands" to actually view the external > commands the builder is calling. > > I hope this helps, > > Olaf > > > Thanks for your help > > > > zeev blumentzvaig > > > > > > 2011/2/18 Olaf Wagner > > > >> Quoting ??? ????????? : > >> > >> Hi, > >>> I am a student of Software Engineering. > >>> I want to learn the language modula-3 and I'm trying to install IDE on my > >>> computer. > >>> I have windows 7 operating system and I can not find any suitable > >>> installation file. > >>> Do you have appropriate file or instructions how to do it on another > >>> system? > >>> I need it for exercise in school and I'd love it if you could help me. > >>> Thank you. > >>> zeev blumentzvaig > >>> > >> > >> What exactly have you tried already? > >> > >> The archive for Windows listed on http://www.opencm3.net/download.html, > >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should > >> work for Windows 7, too, as far as I know. You simply need to > >> unpack it and install a suitable linker, as described for example in > >> http://www.opencm3.net/installation-windows.html. > >> > >> In order to help you, we need to know exactly what you have done and > >> what failed. I'm not an expert on Windows; others on the list may > >> be able to help better, so I CC this to m3devel. > >> > >> Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Fri Feb 18 14:12:21 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Fri, 18 Feb 2011 14:12:21 +0100 Subject: [M3devel] cm3-ide In-Reply-To: References: , <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com>, , <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> Message-ID: <20110218141221.075c2m16gwcw4s4g@mail.elegosoft.com> Quoting Jay K : > > A language and its implementation in a compiler does not necessarily > > require a full-fledged intergrated development environment like Eclipse. > > Programmers today are not used to using plain text editors. > They are used to wizards and intellisense. > It would take a lot of work I think to make Modula-3 > at all usable to any programmer under 30 years old. I don't think that an integration of M3 into Eclispe for example would be very difficult. As I'm not using it myself, someone else will have to provide it though. > Even older folks have largely become spoiled. I'm still fairly content with using the command line and text editors, even though that must seem old-fashioned ;-) Olaf > - Jay >> Date: Fri, 18 Feb 2011 09:40:14 +0100 >> From: wagner at elegosoft.com >> To: zeev.blu at gmail.com >> CC: m3devel at elegosoft.com; m3-support at elego.de; moriah.moshe at gmail.com >> Subject: Re: [M3devel] cm3-ide >> >> Quoting ??? ????????? : >> >> > Hi, >> > First, thanks for the quick reply. >> > >> > I have tried to use the file you gave me, and some other files from that >> > site. >> > I have some questions: >> > 1. If I understand correctly, the method of working with this compiler is >> > through the CMD in WINDOWS. Is not this language a IDE to work with? >> >> A language and its implementation in a compiler does not necessarily >> require a full-fledged intergrated development environment like Eclipse. >> My faourite IDE when I was actively using M3 was always XEmacs :-) >> There _is_ a web-based cm3-ide (which should be contained in that >> archive) that allows you to build and manage your projects with a >> standard web browser, but that might be quite different from your >> expectations of an IDE, too ;-) >> >> > 2. When I tried to compile , I got a linking error: >> > >> > " link is not recognized as an internal or external command, operable >> > program or batch file." >> > >> > And then another error: >> > >> > "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " >> > >> > And finally I received: >> > >> > " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" >> > failed. " >> >> You either missed the installation of a suitable system linker as >> described in the "Windows installation notes" referred to below or >> you didn't set the appropriate variables in your environment or in >> the cm3 configuration files. CM3 does not include a system linker, >> and on Windows you have got to install one yourself, while on all >> other systems it usually is installed by default. >> >> You can use the cm3 option "-commands" to actually view the external >> commands the builder is calling. >> >> I hope this helps, >> >> Olaf >> >> > Thanks for your help >> > >> > zeev blumentzvaig >> > >> > >> > 2011/2/18 Olaf Wagner >> > >> >> Quoting ??? ????????? : >> >> >> >> Hi, >> >>> I am a student of Software Engineering. >> >>> I want to learn the language modula-3 and I'm trying to install >> IDE on my >> >>> computer. >> >>> I have windows 7 operating system and I can not find any suitable >> >>> installation file. >> >>> Do you have appropriate file or instructions how to do it on another >> >>> system? >> >>> I need it for exercise in school and I'd love it if you could help me. >> >>> Thank you. >> >>> zeev blumentzvaig >> >>> >> >> >> >> What exactly have you tried already? >> >> >> >> The archive for Windows listed on http://www.opencm3.net/download.html, >> >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should >> >> work for Windows 7, too, as far as I know. You simply need to >> >> unpack it and install a suitable linker, as described for example in >> >> http://www.opencm3.net/installation-windows.html. >> >> >> >> In order to help you, we need to know exactly what you have done and >> >> what failed. I'm not an expert on Windows; others on the list may >> >> be able to help better, so I CC this to m3devel. >> >> >> >> Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From hosking at cs.purdue.edu Fri Feb 18 15:42:58 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 18 Feb 2011 09:42:58 -0500 Subject: [M3devel] SIGCHLD diff In-Reply-To: <20110218101414.yu4y8iyg0kkw40cg@mail.elegosoft.com> References: <20110213220804.7A9001A2078@async.async.caltech.edu> <20110214211829.BA3761A2078@async.async.caltech.edu> <20110215085337.pezwvp7m8c4c8kco@mail.elegosoft.com> <20110217225605.644AC1A2078@async.async.caltech.edu> <20110218101414.yu4y8iyg0kkw40cg@mail.elegosoft.com> Message-ID: <947FF939-07BD-42F5-9E0B-DBCA9A51E68B@cs.purdue.edu> There has been a lot of untested churn in the pthreads implementation since I last looked closely at it. I think we have seen some bitrot that needs attention. I hope to have some time soon. On Feb 18, 2011, at 4:14 AM, Olaf Wagner wrote: > Hm, sounds like the complete switch to pthreads might have been a little > too early, but we haven't had any reliable data to decide on IIRC. All > the world uses pthreads, so I assumed they would be OK for us, too :-/ > However, we need to fix the problems now; I hope Antony can spare some > time. He's been the main architect for that. From dabenavidesd at yahoo.es Fri Feb 18 15:37:24 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Fri, 18 Feb 2011 14:37:24 +0000 (GMT) Subject: [M3devel] cm3-ide In-Reply-To: <20110218141221.075c2m16gwcw4s4g@mail.elegosoft.com> Message-ID: <31891.31208.qm@web29708.mail.ird.yahoo.com> Hi all: yes, it's agood idea, in fact there is one still in preparation M3clipse, the syntactic analyzer, parser and abstract tree builder and has not yet its semantic analyzer, all are written in Java, so it helps to this purposes, perhaps you can can have a hand of it, for now it just colorize its syntax and so see it: http://sourceforge.net/projects/m3clipse/files (it doesn't have any files right now, I believe because of recent crashed project repository server, I can try to put it on the web if any interest its license is EPL) You have this feature in another integrated development environment in another good package, cooledit, I think it features what you are suggesting, even it has an integrated keyword highlighting and code completion but still though it has integrated gdb window support, you might want to add m3gdb support to it (ideally I would say): http://freshmeat.net/projects/cooledit If you need cooledit Modula-3 syntax files you can tell me or be put on the repository for everybody if any interested just tell me, this is written by Pino Zollo. Yet another "cool" option is an "intelligent" or simply advanced integrated programming environment, it has a graphical debugger (in an internal debugging language) and so you can take a look at it in its book: http://www.cs.brown.edu/~spr/research/envfield.html And finally about cm3ide, I think it makes sense to have an distributed environment for reading its hyperlinked source code in almost any web browser by free, i.e a browser plugin which has an speech synthesizer for blind or low vision people. Of course any other custom plugin for any type of checking is a wanted feature, i.e. static analysis or runtime failures could be detected and prompted by it, which is sort of good if have any failure is good to be reported about it, and just not only throw a program debug image file don't you think? Hope it helps, thanks in advance. --- El vie, 18/2/11, Olaf Wagner escribi?: > De: Olaf Wagner > Asunto: Re: [M3devel] cm3-ide > Para: "Jay K" > CC: "m3devel" , m3-support at elego.de, moriah.moshe at gmail.com, zeev.blu at gmail.com > Fecha: viernes, 18 de febrero, 2011 08:12 > Quoting Jay K : > > > > A language and its implementation in a > compiler does not necessarily > > > require a full-fledged intergrated > development environment like Eclipse. > > > > Programmers today are not used to using plain text > editors. > > They are used to wizards and intellisense. > > It would take a lot of work I think to make Modula-3 > > at all usable to any programmer under 30 years old. > > I don't think that an integration of M3 into Eclispe for > example > would be very difficult. As I'm not using it myself, > someone else > will have to provide it though. > > > Even older folks have largely become spoiled. > > I'm still fairly content with using the command line and > text > editors, even though that must seem old-fashioned ;-) > > Olaf > > > - Jay > >> Date: Fri, 18 Feb 2011 09:40:14 +0100 > >> From: wagner at elegosoft.com > >> To: zeev.blu at gmail.com > >> CC: m3devel at elegosoft.com; > m3-support at elego.de; > moriah.moshe at gmail.com > >> Subject: Re: [M3devel] cm3-ide > >> > >> Quoting ??? ????????? : > >> > >> > Hi, > >> > First, thanks for the quick reply. > >> > > >> > I have tried to use the file you gave me, and > some other files from that > >> > site. > >> > I have some questions: > >> > 1. If I understand correctly, the method of > working with this compiler is > >> > through the CMD in WINDOWS. Is not this > language a IDE to work with? > >> > >> A language and its implementation in a compiler > does not necessarily > >> require a full-fledged intergrated development > environment like Eclipse. > >> My faourite IDE when I was actively using M3 was > always XEmacs :-) > >> There _is_ a web-based cm3-ide (which should be > contained in that > >> archive) that allows you to build and manage your > projects with a > >> standard web browser, but that might be quite > different from your > >> expectations of an IDE, too ;-) > >> > >> > 2. When I tried to compile , I got a linking > error: > >> > > >> > " link is not recognized as an internal or > external command, operable > >> > program or batch file." > >> > > >> > And then another error: > >> > > >> > "... \cm3 \ bin \ cm3.cfg ", line 455: quake > runtime error: link failed " > >> > > >> > And finally I received: > >> > > >> > " Fatal Error: procedure "m3_link" defined in > "... \ cm3 \ bin \ cm3.cfg" > >> > failed. " > >> > >> You either missed the installation of a suitable > system linker as > >> described in the "Windows installation notes" > referred to below or > >> you didn't set the appropriate variables in your > environment or in > >> the cm3 configuration files. CM3 does not include > a system linker, > >> and on Windows you have got to install one > yourself, while on all > >> other systems it usually is installed by default. > >> > >> You can use the cm3 option "-commands" to actually > view the external > >> commands the builder is calling. > >> > >> I hope this helps, > >> > >> Olaf > >> > >> > Thanks for your help > >> > > >> > zeev blumentzvaig > >> > > >> > > >> > 2011/2/18 Olaf Wagner > >> > > >> >> Quoting ??? ????????? : > >> >> > >> >> Hi, > >> >>> I am a student of Software > Engineering. > >> >>> I want to learn the language modula-3 > and I'm trying to install > >> IDE on my > >> >>> computer. > >> >>> I have windows 7 operating system and > I can not find any suitable > >> >>> installation file. > >> >>> Do you have appropriate file or > instructions how to do it on another > >> >>> system? > >> >>> I need it for exercise in school and > I'd love it if you could help me. > >> >>> Thank you. > >> >>> zeev blumentzvaig > >> >>> > >> >> > >> >> What exactly have you tried already? > >> >> > >> >> The archive for Windows listed on http://www.opencm3.net/download.html, > >> >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, > should > >> >> work for Windows 7, too, as far as I > know. You simply need to > >> >> unpack it and install a suitable linker, > as described for example in > >> >> http://www.opencm3.net/installation-windows.html. > >> >> > >> >> In order to help you, we need to know > exactly what you have done and > >> >> what failed. I'm not an expert on > Windows; others on the list may > >> >> be able to help better, so I CC this to > m3devel. > >> >> > >> >> Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 > 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | > Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | > USt-IdNr: DE163214194 > > From hosking at cs.purdue.edu Fri Feb 18 15:55:40 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 18 Feb 2011 09:55:40 -0500 Subject: [M3devel] cm3-ide In-Reply-To: References: , <20110217234730.am1zyuyhc84okcoo@mail.elegosoft.com>, , <20110218094014.rgcwoes3k40oww0k@mail.elegosoft.com> Message-ID: Ergo: let's get M3 integrated with Eclipse. Unfortunately, adding new languages properly to Eclipse is a painful process. On Feb 18, 2011, at 7:55 AM, Jay K wrote: > > A language and its implementation in a compiler does not necessarily > > require a full-fledged intergrated development environment like Eclipse. > > > Programmers today are not used to using plain text editors. > They are used to wizards and intellisense. > It would take a lot of work I think to make Modula-3 > at all usable to any programmer under 30 years old. > Even older folks have largely become spoiled. > > - Jay > > Date: Fri, 18 Feb 2011 09:40:14 +0100 > > From: wagner at elegosoft.com > > To: zeev.blu at gmail.com > > CC: m3devel at elegosoft.com; m3-support at elego.de; moriah.moshe at gmail.com > > Subject: Re: [M3devel] cm3-ide > > > > Quoting ??? ????????? : > > > > > Hi, > > > First, thanks for the quick reply. > > > > > > I have tried to use the file you gave me, and some other files from that > > > site. > > > I have some questions: > > > 1. If I understand correctly, the method of working with this compiler is > > > through the CMD in WINDOWS. Is not this language a IDE to work with? > > > > A language and its implementation in a compiler does not necessarily > > require a full-fledged intergrated development environment like Eclipse. > > My faourite IDE when I was actively using M3 was always XEmacs :-) > > There _is_ a web-based cm3-ide (which should be contained in that > > archive) that allows you to build and manage your projects with a > > standard web browser, but that might be quite different from your > > expectations of an IDE, too ;-) > > > > > 2. When I tried to compile , I got a linking error: > > > > > > " link is not recognized as an internal or external command, operable > > > program or batch file." > > > > > > And then another error: > > > > > > "... \cm3 \ bin \ cm3.cfg ", line 455: quake runtime error: link failed " > > > > > > And finally I received: > > > > > > " Fatal Error: procedure "m3_link" defined in "... \ cm3 \ bin \ cm3.cfg" > > > failed. " > > > > You either missed the installation of a suitable system linker as > > described in the "Windows installation notes" referred to below or > > you didn't set the appropriate variables in your environment or in > > the cm3 configuration files. CM3 does not include a system linker, > > and on Windows you have got to install one yourself, while on all > > other systems it usually is installed by default. > > > > You can use the cm3 option "-commands" to actually view the external > > commands the builder is calling. > > > > I hope this helps, > > > > Olaf > > > > > Thanks for your help > > > > > > zeev blumentzvaig > > > > > > > > > 2011/2/18 Olaf Wagner > > > > > >> Quoting ??? ????????? : > > >> > > >> Hi, > > >>> I am a student of Software Engineering. > > >>> I want to learn the language modula-3 and I'm trying to install IDE on my > > >>> computer. > > >>> I have windows 7 operating system and I can not find any suitable > > >>> installation file. > > >>> Do you have appropriate file or instructions how to do it on another > > >>> system? > > >>> I need it for exercise in school and I'd love it if you could help me. > > >>> Thank you. > > >>> zeev blumentzvaig > > >>> > > >> > > >> What exactly have you tried already? > > >> > > >> The archive for Windows listed on http://www.opencm3.net/download.html, > > >> http://www.opencm3.net/cm3-min-WIN32-NT386-d5.5.0.zip, should > > >> work for Windows 7, too, as far as I know. You simply need to > > >> unpack it and install a suitable linker, as described for example in > > >> http://www.opencm3.net/installation-windows.html. > > >> > > >> In order to help you, we need to know exactly what you have done and > > >> what failed. I'm not an expert on Windows; others on the list may > > >> be able to help better, so I CC this to m3devel. > > >> > > >> Olaf > > -- > > Olaf Wagner -- elego Software Solutions GmbH > > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 19 01:22:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Fri, 18 Feb 2011 16:22:50 -0800 Subject: [M3devel] Text.Hash in CM3 vs PM3 Message-ID: <20110219002250.9E9EE1A2078@async.async.caltech.edu> Hi m3devel, Am I reading the code correctly if I say that Text.Hash (e.g., String8.Hash) is approximately four times slower (on i386 and other 32-bit systems) in CM3 than it was in PM3 (eight times on Alpha and amd64) (i.e., UnsafeHash.Hash)? More, I suppose, on Alpha 21164 and Alpha 21064, which don't even have byte operations... Does anyone out there depend on the specific hash values of existing strings? (I'm not sure the SRC/PM3 algorithm can be adapted to CM3, maybe it can...) Mika From rcolebur at SCIRES.COM Sat Feb 19 10:34:38 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sat, 19 Feb 2011 04:34:38 -0500 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, , , , <20110213181717.4F0E91A2078@async.async.caltech.edu>, , , , , Message-ID: Jay: For Windows 2000 SP4, using Visual C++ 2005 Express Edition: 1. I tried building the MachineID.c program in the first example link, but it runs into an unresolved symbol at link time. cl MachineID.c Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 Copyright (C) Microsoft Corporation. All rights reserved. MachineID.c Microsoft (R) Incremental Linker Version 8.00.50727.762 Copyright (C) Microsoft Corporation. All rights reserved. /out:MachineID.exe MachineID.obj LIBCMT.lib(crt0.obj) : error LNK2019: unresolved external symbol _main referenced in function ___tmainCRTStartup MachineID.exe : fatal error LNK1120: 1 unresolved externals 2. I was able to build the program in the 2nd link. Here is the output from running this program: Num Entries: 2 Index[0]: 16777219 InterfaceName[0]: osoft Visual Studio 8\Common7\Tools\ Description[0]: EL574ND4 Ethernet Adapter Type[0]: Ethernet Mtu[0]: 1500 Speed[0]: 100000000 Physical Addr: 00-00-86-5E-DF-A0 Admin Status[0]: 1 Oper Status[0]: Operational Index[1]: 1 InterfaceName[1]: Description[1]: MS TCP Loopback interface Type[1]: Software Lookback Mtu[1]: 1500 Speed[1]: 10000000 Physical Addr: Admin Status[1]: 1 Oper Status[1]: Operational Regards, Randy ________________________________ From: jayk123 at hotmail.com [jayk123 at hotmail.com] On Behalf Of Jay K [jay.krell at cornell.edu] Sent: Tuesday, February 15, 2011 7:56 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] More on threading It should work. http://dcvs.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/uid/WIN32/MachineID.c?rev=1.14;content-type=text%2Fplain Bing for "MSDN GetIfTable" => http://msdn.microsoft.com/en-us/library/aa365943(VS.85).aspx Mika, what made you think cm3 doesn't work on Windows 2000?? I don't think anything has been remotely suggested. - Jay ________________________________ From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Tue, 15 Feb 2011 17:05:37 -0500 Subject: Re: [M3devel] More on threading Jay, since I have a Windows2000 system, I can test the MAC address discovery (or anything else you need). Just let me know. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Monday, February 14, 2011 8:12 PM To: Coleburn, Randy; m3devel Subject: Re: [M3devel] More on threading Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 19 11:03:54 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 19 Feb 2011 10:03:54 +0000 Subject: [M3devel] More on threading In-Reply-To: References: <20110213070414.DD0341A2078@async.async.caltech.edu>, ,,, ,,<20110213181717.4F0E91A2078@async.async.caltech.edu>, , , , , , , , , , Message-ID: 1. That is the expected result. Please please look closer. 2. Yeah, the sample I derived from is a bit buggy but I think I fixed its bugs, I vaguely recall. I'm sure there's no problem here, anyway. At least on Windows 2000 and newer. Maybe on NT 3.x, NT 4.0, Win9x, maybe depending on the install of IE. But I doubt anyone cares. Heck, for a very long time, this code didn't really work for MacOSX, FreeBSD, NetBSD, OpenBSD, Solaris, only Windows, Linux, OSF/1, and nobody seemed to notice.. (it falls back to the local IPv4 address, which isn't very unique typically). I think it works everywhere now, through heavy #ifdef. (Given the level of #ifdef, I wonder if we should either combine Windows and Posix into one file, or split up Posix into multiple files, or just leave it alone..) Thanks, - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Sat, 19 Feb 2011 04:34:38 -0500 Subject: Re: [M3devel] More on threading Jay: For Windows 2000 SP4, using Visual C++ 2005 Express Edition: 1. I tried building the MachineID.c program in the first example link, but it runs into an unresolved symbol at link time. cl MachineID.c Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 Copyright (C) Microsoft Corporation. All rights reserved. MachineID.c Microsoft (R) Incremental Linker Version 8.00.50727.762 Copyright (C) Microsoft Corporation. All rights reserved. /out:MachineID.exe MachineID.obj LIBCMT.lib(crt0.obj) : error LNK2019: unresolved external symbol _main referenced in function ___tmainCRTStartup MachineID.exe : fatal error LNK1120: 1 unresolved externals 2. I was able to build the program in the 2nd link. Here is the output from running this program: Num Entries: 2 Index[0]: 16777219 InterfaceName[0]: osoft Visual Studio 8\Common7\Tools\ Description[0]: EL574ND4 Ethernet Adapter Type[0]: Ethernet Mtu[0]: 1500 Speed[0]: 100000000 Physical Addr: 00-00-86-5E-DF-A0 Admin Status[0]: 1 Oper Status[0]: Operational Index[1]: 1 InterfaceName[1]: Description[1]: MS TCP Loopback interface Type[1]: Software Lookback Mtu[1]: 1500 Speed[1]: 10000000 Physical Addr: Admin Status[1]: 1 Oper Status[1]: Operational Regards, Randy From: jayk123 at hotmail.com [jayk123 at hotmail.com] On Behalf Of Jay K [jay.krell at cornell.edu] Sent: Tuesday, February 15, 2011 7:56 PM To: Coleburn, Randy; m3devel Subject: RE: [M3devel] More on threading It should work. http://dcvs.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/uid/WIN32/MachineID.c?rev=1.14;content-type=text%2Fplain Bing for "MSDN GetIfTable" => http://msdn.microsoft.com/en-us/library/aa365943(VS.85).aspx Mika, what made you think cm3 doesn't work on Windows 2000?? I don't think anything has been remotely suggested. - Jay From: rcolebur at SCIRES.COM To: jay.krell at cornell.edu; m3devel at elegosoft.com Date: Tue, 15 Feb 2011 17:05:37 -0500 Subject: Re: [M3devel] More on threading Jay, since I have a Windows2000 system, I can test the MAC address discovery (or anything else you need). Just let me know. Regards, Randy Coleburn From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K Sent: Monday, February 14, 2011 8:12 PM To: Coleburn, Randy; m3devel Subject: Re: [M3devel] More on threading Agreed -- I don't know of any reason Windows 2000 wouldn't work. Or possibly even older. The main "upgrade driving" factor is what the C toolset/runtime supports. And I've tested building with quite old C toolsets. Hm. Maybe the new-ish MAC address discovery code doesn't work on older systems? That could be. I could make that use GetProcAddress and fallback to the older code. That is an area with the historical code seemed too non-ideal to stick with, and clearer more direct code might depend on newer systems. - Jay > From: rcolebur at SCIRES.COM > To: m3devel at elegosoft.com > Date: Mon, 14 Feb 2011 18:01:14 -0500 > Subject: Re: [M3devel] More on threading > > Mika wrote... > >But no I don't use Windows. I was asking on here how to get started with > >Modula-3 on Windows once but my machine was/is a Windows 2000 system and > >apparently CM3 no longer works on win2k..? > > I use CM3 on Windows 2000/XP/7 and it works on all of these versions. > --Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 19 19:29:30 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 10:29:30 -0800 Subject: [M3devel] SEGV mapping to RuntimeError Message-ID: <20110219182930.BDCAB1A2078@async.async.caltech.edu> Dear m3devel, For a while it has annoyed me that segmentation violations cause an unconditional program abort. I've changed that now so that (under user threads at least) we instead get a RuntimeError. Here's an example of the mechanism at work in an interactive Scheme environment. Consider the unhelpful interface and module Crash: INTERFACE Crash; PROCEDURE Me(); END Crash. MODULE Crash; PROCEDURE Me() = VAR ptr : REF INTEGER := NIL; BEGIN ptr^ := 0 END Me; BEGIN END Crash. Here's an example of what happens if you now call this from an interactive interpreter that catches the exception RuntimeError.E: M-Scheme Experimental LITHP ITH LITHENING. > (require-modules "m3") #t > (Crash.Me) EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > (+ 3 4) 7 > I just realized I may have broken pthreads, let me go back and double-check it. runtime/POSIX and thread/POSIX don't refer to the same thing do they... Mika From jay.krell at cornell.edu Sat Feb 19 20:53:49 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 19 Feb 2011 19:53:49 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110219182930.BDCAB1A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> Message-ID: Letting any code run after a SIGSEGV is dubious. Imagine the heap is corrupted. And then you run more code. And the code happens to call malloc. Or printf to log something. I suppose there might be an application that maps memory gradually, as pieces of a buffer are hit. Might. - Jay > To: m3devel at elegosoft.com > Date: Sat, 19 Feb 2011 10:29:30 -0800 > From: mika at async.caltech.edu > Subject: [M3devel] SEGV mapping to RuntimeError > > > Dear m3devel, > > For a while it has annoyed me that segmentation violations cause an > unconditional program abort. I've changed that now so that (under user > threads at least) we instead get a RuntimeError. Here's an example of > the mechanism at work in an interactive Scheme environment. Consider > the unhelpful interface and module Crash: > > INTERFACE Crash; PROCEDURE Me(); END Crash. > > MODULE Crash; > > PROCEDURE Me() = > VAR ptr : REF INTEGER := NIL; BEGIN > ptr^ := 0 > END Me; > > BEGIN END Crash. > > Here's an example of what happens if you now call this from an interactive > interpreter that catches the exception RuntimeError.E: > > M-Scheme Experimental > LITHP ITH LITHENING. > > (require-modules "m3") > #t > > (Crash.Me) > EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > > (+ 3 4) > 7 > > > > I just realized I may have broken pthreads, let me go back and double-check it. > runtime/POSIX and thread/POSIX don't refer to the same thing do they... > > Mika > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 19 21:38:21 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 12:38:21 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> Message-ID: <20110219203821.2016B1A2078@async.async.caltech.edu> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, you know! "Corrupting the heap" is something that can only happen as a result of an unchecked runtime error. Unchecked runtime errors cannot happen in modules not marked UNSAFE. SEGV is, however, used by the CM3 implementation (and its predecessors) to signal a certain kind of *checked* runtime error, namely, the dereferencing of a NIL reference. Correct me if I am wrong, but an attempt to dereference NIL is not going to leave the heap corrupted? And if you stick to safe code, the only SEGVs I think you get in the current CM3 are ones from NIL dereferences. Hence, as long as you stick with safe code, the only time the code I checked in earlier gets triggered is for NIL dereferences, which should never corrupt the heap. So SEGV is not sometimes, but in fact always recoverable. :-) Mika P.S. the bit above "if you stick to safe code": if you actually program in Modula-3 you almost never use UNSAFE. I went through my repository and I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, many of the UNSAFE modules are glue code to Fortran routines, which could relatively easily be verified to be safe in the Modula-3 sense. Almost all what remains is glue to some C library, which wouldn't be necessary if the rest of the world would wake up out of the dark ages, but I don't have the time to rewrite every single library from scratch myself. Jay K writes: >--_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Letting any code run after a SIGSEGV is dubious. >Imagine the heap is corrupted. >And then you run more code. >And the code happens to call malloc. >Or printf to log something. >=20 >I suppose there might be an application that maps memory >gradually=2C as pieces of a buffer are hit. Might. >=20 > - Jay >=20 >> To: m3devel at elegosoft.com >> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >> From: mika at async.caltech.edu >> Subject: [M3devel] SEGV mapping to RuntimeError >>=20 >>=20 >> Dear m3devel=2C >>=20 >> For a while it has annoyed me that segmentation violations cause an >> unconditional program abort. I've changed that now so that (under user >> threads at least) we instead get a RuntimeError. Here's an example of >> the mechanism at work in an interactive Scheme environment. Consider >> the unhelpful interface and module Crash: >>=20 >> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>=20 >> MODULE Crash=3B >>=20 >> PROCEDURE Me() =3D >> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >> ptr^ :=3D 0 >> END Me=3B >>=20 >> BEGIN END Crash. >>=20 >> Here's an example of what happens if you now call this from an interactiv= >e >> interpreter that catches the exception RuntimeError.E: >>=20 >> M-Scheme Experimental >> LITHP ITH LITHENING. >> > (require-modules "m3") >> #t >> > (Crash.Me) >> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >> > (+ 3 4)=20 >> 7 >> >=20 >>=20 >> I just realized I may have broken pthreads=2C let me go back and double-c= >heck it.=20 >> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>=20 >> Mika >>=20 > = > >--_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >Letting any code run after a SIGSEGV is dubious.
>Imagine the heap =3Bis corrupted.
>And then you run more code.
>And the code happens to call malloc.
>Or printf to log something.
> =3B
>I suppose there might be an application that maps memory
>gradually=2C as pieces of a buffer are hit. Might.
> =3B
> =3B- Jay
 =3B
>>=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >=2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >olations cause an
>=3B unconditional program abort. I've changed that = >now so that (under user
>=3B threads at least) we instead get a Runtim= >eError. Here's an example of
>=3B the mechanism at work in an interact= >ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>=3B Here's an example of what happens if you now call this from an inter= >active
>=3B interpreter that catches the exception RuntimeError.E:
= >>=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >gt=3B >=3B (require-modules "m3")
>=3B #t
>=3B >=3B (Crash.Me= >)
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > location.
>=3B >=3B (+ 3 4)
>=3B 7
>=3B >=3B
>= >=3B
>=3B I just realized I may have broken pthreads=2C let me go back= > and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >= > >--_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- From rodney_bates at lcwb.coop Sun Feb 20 00:17:55 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sat, 19 Feb 2011 17:17:55 -0600 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110219203821.2016B1A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> Message-ID: <4D604FA3.3060708@lcwb.coop> I know of one other place the compilers rely on hardware memory protection to detect a checked runtime error, and that is stack overflow. This won't corrupt anything, but is hard to distinguish from dereferencing NIL. This could probably be distinguished after the fact by some low-level, target-dependent code. I have found it by looking at assembly code at the point of failure--usually right after a stack pointer push. Detecting this via compiler-generated checks would probably be more extravagant than many other checks, as it is so frequent. I am not aware of any really good solution to this in any implementation of any language. On 02/19/2011 02:38 PM, Mika Nystrom wrote: > Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > you know! > > "Corrupting the heap" is something that can only happen as a result of > an unchecked runtime error. Unchecked runtime errors cannot happen in > modules not marked UNSAFE. > > SEGV is, however, used by the CM3 implementation (and its predecessors) > to signal a certain kind of *checked* runtime error, namely, the > dereferencing of a NIL reference. Correct me if I am wrong, but an > attempt to dereference NIL is not going to leave the heap corrupted? > > And if you stick to safe code, the only SEGVs I think you get in the > current CM3 are ones from NIL dereferences. > > Hence, as long as you stick with safe code, the only time the code I > checked in earlier gets triggered is for NIL dereferences, which should > never corrupt the heap. So SEGV is not sometimes, but in fact always > recoverable. > > :-) > > Mika > > P.S. the bit above "if you stick to safe code": if you actually program in > Modula-3 you almost never use UNSAFE. I went through my repository and > I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > many of the UNSAFE modules are glue code to Fortran routines, which > could relatively easily be verified to be safe in the Modula-3 sense. > Almost all what remains is glue to some C library, which wouldn't be > necessary if the rest of the world would wake up out of the dark ages, but > I don't have the time to rewrite every single library from scratch myself. > > > Jay K writes: >> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >> Content-Type: text/plain; charset="iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> >> Letting any code run after a SIGSEGV is dubious. >> Imagine the heap is corrupted. >> And then you run more code. >> And the code happens to call malloc. >> Or printf to log something. >> =20 >> I suppose there might be an application that maps memory >> gradually=2C as pieces of a buffer are hit. Might. >> =20 >> - Jay >> =20 >>> To: m3devel at elegosoft.com >>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>> From: mika at async.caltech.edu >>> Subject: [M3devel] SEGV mapping to RuntimeError >>> =20 >>> =20 >>> Dear m3devel=2C >>> =20 >>> For a while it has annoyed me that segmentation violations cause an >>> unconditional program abort. I've changed that now so that (under user >>> threads at least) we instead get a RuntimeError. Here's an example of >>> the mechanism at work in an interactive Scheme environment. Consider >>> the unhelpful interface and module Crash: >>> =20 >>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>> =20 >>> MODULE Crash=3B >>> =20 >>> PROCEDURE Me() =3D >>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>> ptr^ :=3D 0 >>> END Me=3B >>> =20 >>> BEGIN END Crash. >>> =20 >>> Here's an example of what happens if you now call this from an interactiv= >> e >>> interpreter that catches the exception RuntimeError.E: >>> =20 >>> M-Scheme Experimental >>> LITHP ITH LITHENING. >>>> (require-modules "m3") >>> #t >>>> (Crash.Me) >>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>> (+ 3 4)=20 >>> 7 >>>> =20 >>> =20 >>> I just realized I may have broken pthreads=2C let me go back and double-c= >> heck it.=20 >>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>> =20 >>> Mika >>> =20 >> = >> >> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >> Content-Type: text/html; charset="iso-8859-1" >> Content-Transfer-Encoding: quoted-printable >> >> >> >> >> >> >> Letting any code run after a SIGSEGV is dubious.
>> Imagine the heap =3Bis corrupted.
>> And then you run more code.
>> And the code happens to call malloc.
>> Or printf to log something.
>>  =3B
>> I suppose there might be an application that maps memory
>> gradually=2C as pieces of a buffer are hit. Might.
>>  =3B
>>  =3B- Jay
 =3B
>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >> olations cause an
>=3B unconditional program abort. I've changed that = >> now so that (under user
>=3B threads at least) we instead get a Runtim= >> eError. Here's an example of
>=3B the mechanism at work in an interact= >> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >> >=3B Here's an example of what happens if you now call this from an inter= >> active
>=3B interpreter that catches the exception RuntimeError.E:
= >> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >> >> = >> >> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > From mika at async.caltech.edu Sun Feb 20 00:27:29 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 15:27:29 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <4D604FA3.3060708@lcwb.coop> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> Message-ID: <20110219232729.53A991A2078@async.async.caltech.edu> Ah, yes, stack protection. Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. Hmm, I get SIGILL on AMD64_FREEBSD as well: time ../AMD64_FREEBSD/stubexample M-Scheme Experimental LITHP ITH LITHENING. > (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) f > (f 0) Illegal instruction 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w What absolutely must not happen, of course, is that the runtime hangs while executing only safe code... Mika "Rodney M. Bates" writes: >I know of one other place the compilers rely on hardware memory protection >to detect a checked runtime error, and that is stack overflow. This won't >corrupt anything, but is hard to distinguish from dereferencing NIL. >This could probably be distinguished after the fact by some low-level, >target-dependent code. I have found it by looking at assembly code at >the point of failure--usually right after a stack pointer push. > >Detecting this via compiler-generated checks would probably be more >extravagant than many other checks, as it is so frequent. I am not >aware of any really good solution to this in any implementation of any >language. > >On 02/19/2011 02:38 PM, Mika Nystrom wrote: >> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >> you know! >> >> "Corrupting the heap" is something that can only happen as a result of >> an unchecked runtime error. Unchecked runtime errors cannot happen in >> modules not marked UNSAFE. >> >> SEGV is, however, used by the CM3 implementation (and its predecessors) >> to signal a certain kind of *checked* runtime error, namely, the >> dereferencing of a NIL reference. Correct me if I am wrong, but an >> attempt to dereference NIL is not going to leave the heap corrupted? >> >> And if you stick to safe code, the only SEGVs I think you get in the >> current CM3 are ones from NIL dereferences. >> >> Hence, as long as you stick with safe code, the only time the code I >> checked in earlier gets triggered is for NIL dereferences, which should >> never corrupt the heap. So SEGV is not sometimes, but in fact always >> recoverable. >> >> :-) >> >> Mika >> >> P.S. the bit above "if you stick to safe code": if you actually program in >> Modula-3 you almost never use UNSAFE. I went through my repository and >> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >> many of the UNSAFE modules are glue code to Fortran routines, which >> could relatively easily be verified to be safe in the Modula-3 sense. >> Almost all what remains is glue to some C library, which wouldn't be >> necessary if the rest of the world would wake up out of the dark ages, but >> I don't have the time to rewrite every single library from scratch myself. >> >> >> Jay K writes: >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>> Content-Type: text/plain; charset="iso-8859-1" >>> Content-Transfer-Encoding: quoted-printable >>> >>> >>> Letting any code run after a SIGSEGV is dubious. >>> Imagine the heap is corrupted. >>> And then you run more code. >>> And the code happens to call malloc. >>> Or printf to log something. >>> =20 >>> I suppose there might be an application that maps memory >>> gradually=2C as pieces of a buffer are hit. Might. >>> =20 >>> - Jay >>> =20 >>>> To: m3devel at elegosoft.com >>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>> From: mika at async.caltech.edu >>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>> =20 >>>> =20 >>>> Dear m3devel=2C >>>> =20 >>>> For a while it has annoyed me that segmentation violations cause an >>>> unconditional program abort. I've changed that now so that (under user >>>> threads at least) we instead get a RuntimeError. Here's an example of >>>> the mechanism at work in an interactive Scheme environment. Consider >>>> the unhelpful interface and module Crash: >>>> =20 >>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>> =20 >>>> MODULE Crash=3B >>>> =20 >>>> PROCEDURE Me() =3D >>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>> ptr^ :=3D 0 >>>> END Me=3B >>>> =20 >>>> BEGIN END Crash. >>>> =20 >>>> Here's an example of what happens if you now call this from an interactiv= >>> e >>>> interpreter that catches the exception RuntimeError.E: >>>> =20 >>>> M-Scheme Experimental >>>> LITHP ITH LITHENING. >>>>> (require-modules "m3") >>>> #t >>>>> (Crash.Me) >>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>> (+ 3 4)=20 >>>> 7 >>>>> =20 >>>> =20 >>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>> heck it.=20 >>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>> =20 >>>> Mika >>>> =20 >>> = >>> >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>> Content-Type: text/html; charset="iso-8859-1" >>> Content-Transfer-Encoding: quoted-printable >>> >>> >>> >>> >>> >>> >>> Letting any code run after a SIGSEGV is dubious.
>>> Imagine the heap =3Bis corrupted.
>>> And then you run more code.
>>> And the code happens to call malloc.
>>> Or printf to log something.
>>>  =3B
>>> I suppose there might be an application that maps memory
>>> gradually=2C as pieces of a buffer are hit. Might.
>>>  =3B
>>>  =3B- Jay
 =3B
>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>> olations cause an
>=3B unconditional program abort. I've changed that = >>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>> >=3B Here's an example of what happens if you now call this from an inter= >>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>> >>> = >>> >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >> From jay.krell at cornell.edu Sun Feb 20 00:36:25 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 19 Feb 2011 23:36:25 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110219232729.53A991A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu> Message-ID: Modula-3 doesn't live in a vacuum. The vast majority of surrounding code is C and C++. There are bugs everywhere. - Jay > To: rodney_bates at lcwb.coop > Date: Sat, 19 Feb 2011 15:27:29 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > Ah, yes, stack protection. > > Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > > Hmm, I get SIGILL on AMD64_FREEBSD as well: > > time ../AMD64_FREEBSD/stubexample > M-Scheme Experimental > LITHP ITH LITHENING. > > (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > f > > (f 0) > Illegal instruction > 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > > What absolutely must not happen, of course, is that the runtime hangs > while executing only safe code... > > Mika > > "Rodney M. Bates" writes: > >I know of one other place the compilers rely on hardware memory protection > >to detect a checked runtime error, and that is stack overflow. This won't > >corrupt anything, but is hard to distinguish from dereferencing NIL. > >This could probably be distinguished after the fact by some low-level, > >target-dependent code. I have found it by looking at assembly code at > >the point of failure--usually right after a stack pointer push. > > > >Detecting this via compiler-generated checks would probably be more > >extravagant than many other checks, as it is so frequent. I am not > >aware of any really good solution to this in any implementation of any > >language. > > > >On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >> you know! > >> > >> "Corrupting the heap" is something that can only happen as a result of > >> an unchecked runtime error. Unchecked runtime errors cannot happen in > >> modules not marked UNSAFE. > >> > >> SEGV is, however, used by the CM3 implementation (and its predecessors) > >> to signal a certain kind of *checked* runtime error, namely, the > >> dereferencing of a NIL reference. Correct me if I am wrong, but an > >> attempt to dereference NIL is not going to leave the heap corrupted? > >> > >> And if you stick to safe code, the only SEGVs I think you get in the > >> current CM3 are ones from NIL dereferences. > >> > >> Hence, as long as you stick with safe code, the only time the code I > >> checked in earlier gets triggered is for NIL dereferences, which should > >> never corrupt the heap. So SEGV is not sometimes, but in fact always > >> recoverable. > >> > >> :-) > >> > >> Mika > >> > >> P.S. the bit above "if you stick to safe code": if you actually program in > >> Modula-3 you almost never use UNSAFE. I went through my repository and > >> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >> many of the UNSAFE modules are glue code to Fortran routines, which > >> could relatively easily be verified to be safe in the Modula-3 sense. > >> Almost all what remains is glue to some C library, which wouldn't be > >> necessary if the rest of the world would wake up out of the dark ages, but > >> I don't have the time to rewrite every single library from scratch myself. > >> > >> > >> Jay K writes: > >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>> Content-Type: text/plain; charset="iso-8859-1" > >>> Content-Transfer-Encoding: quoted-printable > >>> > >>> > >>> Letting any code run after a SIGSEGV is dubious. > >>> Imagine the heap is corrupted. > >>> And then you run more code. > >>> And the code happens to call malloc. > >>> Or printf to log something. > >>> =20 > >>> I suppose there might be an application that maps memory > >>> gradually=2C as pieces of a buffer are hit. Might. > >>> =20 > >>> - Jay > >>> =20 > >>>> To: m3devel at elegosoft.com > >>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>> From: mika at async.caltech.edu > >>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>> =20 > >>>> =20 > >>>> Dear m3devel=2C > >>>> =20 > >>>> For a while it has annoyed me that segmentation violations cause an > >>>> unconditional program abort. I've changed that now so that (under user > >>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>> the mechanism at work in an interactive Scheme environment. Consider > >>>> the unhelpful interface and module Crash: > >>>> =20 > >>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>> =20 > >>>> MODULE Crash=3B > >>>> =20 > >>>> PROCEDURE Me() =3D > >>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>> ptr^ :=3D 0 > >>>> END Me=3B > >>>> =20 > >>>> BEGIN END Crash. > >>>> =20 > >>>> Here's an example of what happens if you now call this from an interactiv= > >>> e > >>>> interpreter that catches the exception RuntimeError.E: > >>>> =20 > >>>> M-Scheme Experimental > >>>> LITHP ITH LITHENING. > >>>>> (require-modules "m3") > >>>> #t > >>>>> (Crash.Me) > >>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>> (+ 3 4)=20 > >>>> 7 > >>>>> =20 > >>>> =20 > >>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>> heck it.=20 > >>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>> =20 > >>>> Mika > >>>> =20 > >>> = > >>> > >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>> Content-Type: text/html; charset="iso-8859-1" > >>> Content-Transfer-Encoding: quoted-printable > >>> > >>> > >>> > >>> > >>> > >>> > >>> Letting any code run after a SIGSEGV is dubious.
> >>> Imagine the heap =3Bis corrupted.
> >>> And then you run more code.
> >>> And the code happens to call malloc.
> >>> Or printf to log something.
> >>>  =3B
> >>> I suppose there might be an application that maps memory
> >>> gradually=2C as pieces of a buffer are hit. Might.
> >>>  =3B
> >>>  =3B- Jay
 =3B
> >>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>> olations cause an
>=3B unconditional program abort. I've changed that = > >>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>> >=3B Here's an example of what happens if you now call this from an inter= > >>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>> > >>> = > >>> > >>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 20 01:13:14 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 16:13:14 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu> Message-ID: <20110220001314.7F2951A2078@async.async.caltech.edu> Jay K writes: >--_1802b75d-f353-43c9-bd6c-02a34e2b7ae8_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Modula-3 doesn't live in a vacuum. >The vast majority of surrounding code is C and C++. >There are bugs everywhere. > - Jay It's rather up to the programmer using it, isn't it? If you write your code in pure Modula-3, you should be able to rely on the guarantees of the Green Book (and therefore that SIGSEGV means something relatively specific, not that "the world just ended"). If you insist on linking with broken C and C++... well you get what you deserve, probably best not to try to handle RuntimeError.E in that case. In any case, to an interactive user of my Scheme interpreter it is very useful that a NIL dereference drops him at the read-eval-print loop rather than killing the world. E.g., if as I had yesterday, a 1-gigabyte datastructure parsed in memory, and I lost the world because I happened to try to set-car! on a null list somewhere. Mika From jay.krell at cornell.edu Sun Feb 20 01:26:30 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 20 Feb 2011 00:26:30 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220001314.7F2951A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , <4D604FA3.3060708@lcwb.coop>, , <20110219232729.53A991A2078@async.async.caltech.edu>, , <20110220001314.7F2951A2078@async.async.caltech.edu> Message-ID: NIL might be a special case. But by and large, continuing to do anything after a SIGSEGV is a bad idea. You think (you say) you are continuing on and preserving state, but the state of the world is now arbitrarily altered and unknown and all future computation based on it is suspect. Best to throw it away. And avoid incorrect results when there is a such a strong hint that things have gone awry. In the Windows C programming world, you can catch access violatios (SIGSEGV) very easily with __except. But prevailing wisdom now and for a long time is exactly what I'm saying -- that catching and continuing isn't "robust", it is letting the damage of bugs continue further. I'm undecided as to the level of guarantees of the safety guarantees, given the surrounding unsafe world. I believe safety depends on a lack of bugs in certain places, certain fairly large places, and therefore the safety guarantees should be considered significant and useful, but not absolute. There really are bugs everywhere. In the compiler (Modula-3, C, etc.), in the C runtime, in the kernels, in the CPUs. Recent AMD processors have a bug in integer division even..one of the most surprising things I've seen in a long time.. (really, recent processors, integer division, I'm not talking about the ancient Pentium FDIV bug). There is a published errata for it and it has been fixed. But I managed to hit it in my code.. - Jay > To: jay.krell at cornell.edu > Date: Sat, 19 Feb 2011 16:13:14 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > Jay K writes: > >--_1802b75d-f353-43c9-bd6c-02a34e2b7ae8_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >Modula-3 doesn't live in a vacuum. > >The vast majority of surrounding code is C and C++. > >There are bugs everywhere. > > - Jay > > It's rather up to the programmer using it, isn't it? If you write your > code in pure Modula-3, you should be able to rely on the guarantees of > the Green Book (and therefore that SIGSEGV means something relatively > specific, not that "the world just ended"). If you insist on linking > with broken C and C++... well you get what you deserve, probably best > not to try to handle RuntimeError.E in that case. > > In any case, to an interactive user of my Scheme interpreter it is very > useful that a NIL dereference drops him at the read-eval-print loop > rather than killing the world. E.g., if as I had yesterday, a 1-gigabyte > datastructure parsed in memory, and I lost the world because I happened > to try to set-car! on a null list somewhere. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 20 01:59:04 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 16:59:04 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , <4D604FA3.3060708@lcwb.coop>, , <20110219232729.53A991A2078@async.async.caltech.edu>, , <20110220001314.7F2951A2078@async.async.caltech.edu> Message-ID: <20110220005904.A15F21A2078@async.async.caltech.edu> Jay K writes: >--_e4a53a7c-4058-44e7-bcce-18b6ae35efbe_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >NIL might be a special case. >But by and large=2C continuing to do anything after a SIGSEGV is a bad idea= >. >You think (you say) you are continuing on and preserving state=2C but the s= >tate >of the world is now arbitrarily altered and unknown and all future computat= >ion >based on it is suspect. Best to throw it away. And avoid incorrect results >when there is a such a strong hint that things have gone awry. Let's try this again. The Green Book unequivocally says that non-UNSAFE code can't get the world into a "state arbitrarily altered and unknown." That doesn't depend on the special case NIL, that's inherent in the design of the language and runtime. Yet non-UNSAFE code can (for efficiency reasons of the implementation) cause a SIGSEGV. If the state of the world is somehow arbitrarily altered and unknown I would suggest tracking down the bugs (in UNSAFE code, compiler, C code, wherever) and fixing them rather than repeating the C programmer's mantra that SIGSEGV is or ought to be fatal to the point that you need to start from a clean slate. This sort of thing is precisely what Modula-3 was invented for!!!! Why do you think (crazy) people still use the programming language, fifteen years after it's been declared dead? By the way, every year that I teach a class, I always ask the students what programming languages they use/learn/want to learn. Over the years I've seen a steady progression of C++, Java, Python... until last year. The most popular answer was "C". I must say it felt a bit odd to be, technologically, 20 years ahead of people 20 years younger. Now, I always used to feel a bit bad about forcing students to study a "dead" programming language that would not help them in employment, research, etc. (Modular-3? What's that?) But better than a language that should be dead. Mika From jay.krell at cornell.edu Sun Feb 20 02:30:01 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 20 Feb 2011 01:30:01 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220005904.A15F21A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, ,,, ,,<20110219203821.2016B1A2078@async.async.caltech.edu>, ,,<4D604FA3.3060708@lcwb.coop>, , , <20110219232729.53A991A2078@async.async.caltech.edu>, , , , <20110220001314.7F2951A2078@async.async.caltech.edu>, , <20110220005904.A15F21A2078@async.async.caltech.edu> Message-ID: I just don't think because the Green Book says something, or because it is inherent in the design, that is necessarily true or inherent in the implementation. There are bugs *everywhere*, and they gradually chip away at all aspects of correctness. Much of this, there is nothing you can do about. The integer divide bug is an example that there is precious little you can do about. You can check that the errata is fixed, and not depend on the correctness until you have first check for it, but inevitably tons and tons of code will run before your correctness check (e.g. booting the OS) and so the state is already not guaranteed. I've also had my code fail to page in. That is another thing you can't do anything about, in usermode...except maybe keep your code to under a page in size. :) Simple progressing from one line of code to the next, or pieces within the line, can fail. But SIGSEGV I think is in another realm. It is a sign of problems that should generally be ignored. There is a term "fail fast" -- at a sign of problem, fail. Don't press on. There are arguments either way. The world, really, the larger human enterprise and setting, is not fail fast. We have lots of problems, yet almost everything almost always goes on. - Jay > To: jay.krell at cornell.edu > Date: Sat, 19 Feb 2011 16:59:04 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > Jay K writes: > >--_e4a53a7c-4058-44e7-bcce-18b6ae35efbe_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >NIL might be a special case. > >But by and large=2C continuing to do anything after a SIGSEGV is a bad idea= > >. > >You think (you say) you are continuing on and preserving state=2C but the s= > >tate > >of the world is now arbitrarily altered and unknown and all future computat= > >ion > >based on it is suspect. Best to throw it away. And avoid incorrect results > >when there is a such a strong hint that things have gone awry. > > Let's try this again. > > The Green Book unequivocally says that non-UNSAFE code can't get the > world into a "state arbitrarily altered and unknown." That doesn't > depend on the special case NIL, that's inherent in the design of the > language and runtime. > > Yet non-UNSAFE code can (for efficiency reasons of the implementation) > cause a SIGSEGV. > > If the state of the world is somehow arbitrarily altered and unknown I > would suggest tracking down the bugs (in UNSAFE code, compiler, C code, > wherever) and fixing them rather than repeating the C programmer's mantra > that SIGSEGV is or ought to be fatal to the point that you need to start > from a clean slate. > > This sort of thing is precisely what Modula-3 was invented for!!!! > Why do you think (crazy) people still use the programming language, > fifteen years after it's been declared dead? > > By the way, every year that I teach a class, I always ask the students > what programming languages they use/learn/want to learn. Over the years > I've seen a steady progression of C++, Java, Python... until last year. > The most popular answer was "C". I must say it felt a bit odd to be, > technologically, 20 years ahead of people 20 years younger. Now, > I always used to feel a bit bad about forcing students to study a > "dead" programming language that would not help them in employment, > research, etc. (Modular-3? What's that?) But better than a language > that should be dead. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 20 02:51:07 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 19 Feb 2011 17:51:07 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , , <4D604FA3.3060708@lcwb.coop>, , , <20110219232729.53A991A2078@async.async.caltech.edu>, , , , <20110220001314.7F2951A2078@async.async.caltech.edu>, , <20110220005904.A15F21A2078@async.async.caltech.edu> Message-ID: <20110220015107.110461A2078@async.async.caltech.edu> >But SIGSEGV I think is in another realm. It is a sign of problems that shou= >ld generally >be ignored. >There is a term "fail fast" -- at a sign of problem=2C fail. Don't press on= >. >There are arguments either way. >The world=2C really=2C the larger human enterprise and setting=2C is not fa= >il fast. >We have lots of problems=2C yet almost everything almost always goes on. Well, see, when you're using a Lisp interpreter, SIGSEGV in called code is not in another realm (this is my problem). It is also not in another realm if you are implementing an operating system without hardware memory protection (e.g., SPIN) or if you're running untrusted outside code in a "sandbox" (which you CAN build with Modula-3, again without memory protection). How would you like it if you had to reboot your computer every time you dereferenced a null pointer? Sounds a bit like the 1950s doesn't it? Why should I have to restart my Lisp interpreter if I slip on the keyboard? These are things you would never dream of doing in C but that Modula-3 is eminently suited for doing. For precisely these sorts of purposes, there's a big distinction designed into the language between failures in "safe" code (which mean just that the code failed) and failures in "unsafe" code (which mean that the world ended). If you're not doing any of the special things listed above you're free not to catch RuntimeError.E, in which case your program will crash on an uncaught exception if it receives a SIGSEGV. >I just don't think because the Green Book says something=2C or because it i= >s inherent in the >design=2C that is necessarily true or inherent in the implementation. >There are bugs *everywhere*=2C and they gradually chip away at all aspects = >of correctness. Dijkstra teaches that one should code to the specification. Coding to the implementation is a sure path to losing one's mind. Mika From jay.krell at cornell.edu Sun Feb 20 07:36:12 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 20 Feb 2011 06:36:12 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220015107.110461A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , ,,, , ,,<20110219203821.2016B1A2078@async.async.caltech.edu>, , ,,<4D604FA3.3060708@lcwb.coop>, , ,,<20110219232729.53A991A2078@async.async.caltech.edu>, ,,, , , <20110220001314.7F2951A2078@async.async.caltech.edu>, , , , <20110220005904.A15F21A2078@async.async.caltech.edu>, , <20110220015107.110461A2078@async.async.caltech.edu> Message-ID: - You can't get SIGSEGV w/o memory protection. SIGSEGV is generated by hardware. - Untrusted code should not be run in-process, at least unsafe untrusted code. - I don't have to reboot my computer for SIGSEGV, but individual processes are killed. The damage is contained as soon as it is detected. The nice thing about safe languages is the error is caught just before any damage is caused, i.e. array bounds checking. - Jay > To: jay.krell at cornell.edu > Date: Sat, 19 Feb 2011 17:51:07 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > >But SIGSEGV I think is in another realm. It is a sign of problems that shou= > >ld generally > >be ignored. > >There is a term "fail fast" -- at a sign of problem=2C fail. Don't press on= > >. > >There are arguments either way. > >The world=2C really=2C the larger human enterprise and setting=2C is not fa= > >il fast. > >We have lots of problems=2C yet almost everything almost always goes on. > > Well, see, when you're using a Lisp interpreter, SIGSEGV in called > code is not in another realm (this is my problem). It is also not > in another realm if you are implementing an operating system without > hardware memory protection (e.g., SPIN) or if you're running untrusted > outside code in a "sandbox" (which you CAN build with Modula-3, again > without memory protection). How would you like it if you had to reboot > your computer every time you dereferenced a null pointer? Sounds a bit > like the 1950s doesn't it? Why should I have to restart my Lisp > interpreter if I slip on the keyboard? > > These are things you would never dream of doing in C but that Modula-3 > is eminently suited for doing. For precisely these sorts of purposes, > there's a big distinction designed into the language between failures > in "safe" code (which mean just that the code failed) and failures in > "unsafe" code (which mean that the world ended). > > If you're not doing any of the special things listed above you're free > not to catch RuntimeError.E, in which case your program will crash on > an uncaught exception if it receives a SIGSEGV. > > > >I just don't think because the Green Book says something=2C or because it i= > >s inherent in the > >design=2C that is necessarily true or inherent in the implementation. > >There are bugs *everywhere*=2C and they gradually chip away at all aspects = > >of correctness. > > Dijkstra teaches that one should code to the specification. Coding to > the implementation is a sure path to losing one's mind. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagner at elegosoft.com Sun Feb 20 13:28:52 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Sun, 20 Feb 2011 13:28:52 +0100 Subject: [M3devel] platform-independent kill process Message-ID: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> I've been thinking a bit about the hanging tests problem in our regression test framework. I'd like to provide a simple utility to start a program with time supervision. But as far as I can see, Modula-3 does not provide a platform-independent way of terminating a process. Of course, on Unix I can send signals; but what would I do on Windows? A quick scan of the win32 interfaces in m3core hasn't brought up anything, but I might have looked for the wrong texts. Could we easily extend the Process interface e.g. by PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException ? Any opinions about that? Different interface suggestions? Rather create ProcessExtras than extending a `standard' interface? We currently only have to consider Unix and Windows, or are there other active flavours of OSs to care for? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Sun Feb 20 17:54:00 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 08:54:00 -0800 Subject: [M3devel] platform-independent kill process In-Reply-To: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> References: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> Message-ID: <20110220165400.384291A2078@async.async.caltech.edu> While poking around the compiler sources I found something called M3Process.Interrupt in m3middle---note the comment about Win32: PROCEDURE Interrupt (t: Process.T) RAISES {OSError.E}; (* Send an interrupt signal to process 't'. *) PROCEDURE Abort (t: Process.T) RAISES {OSError.E}; (* Make a best effort attempt to stop process 't'. Warning: aborting a process may leave it in a broken state or fail to release its OS resources (ie. Win32 is broken!) *) POSIX implementation: PROCEDURE Interrupt (t: Process.T) RAISES {OSError.E} = VAR pid := Process.GetID (t); BEGIN IF Usignal.kill (pid, Usignal.SIGINT) # 0 THEN OSErrorPosix.Raise (); END; END Interrupt; PROCEDURE Abort (t: Process.T) RAISES {OSError.E} = VAR pid := Process.GetID (t); BEGIN IF Usignal.kill (pid, Usignal.SIGKILL) # 0 THEN OSErrorPosix.Raise (); END; END Abort; WIN32: PROCEDURE Interrupt (t: Process.T) RAISES {OSError.E} = BEGIN Abort (t); (* can't do any better yet... *) END Interrupt; PROCEDURE Abort (t: Process.T) RAISES {OSError.E} = VAR hProcess := LOOPHOLE (Process.GetID (t), WinNT.HANDLE); BEGIN IF WinBase.TerminateProcess (hProcess, 1) = 0 THEN OSErrorWin32.Raise (); END; END Abort; I don't know enough about Windows to deciper the warning about releasing resources... Mika Olaf Wagner writes: >I've been thinking a bit about the hanging tests problem in our regression >test framework. I'd like to provide a simple utility to start a program >with time supervision. But as far as I can see, Modula-3 does not provide >a platform-independent way of terminating a process. Of course, on Unix >I can send signals; but what would I do on Windows? A quick scan of the >win32 interfaces in m3core hasn't brought up anything, but I might >have looked for the wrong texts. > >Could we easily extend the Process interface e.g. by > > PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException > >? >Any opinions about that? Different interface suggestions? >Rather create ProcessExtras than extending a `standard' interface? > >We currently only have to consider Unix and Windows, or are there >other active flavours of OSs to care for? > >Olaf >--=20 >Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb=E4ude 12, 13355 Berlin, Germany >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch=E4ftsf=FChrer: Olaf Wagner | Sitz: Berl= >in >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Sun Feb 20 18:12:19 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 09:12:19 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , , , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , , , <4D604FA3.3060708@lcwb.coop>, , , , <20110219232729.53A991A2078@async.async.caltech.edu>, , , , , , <20110220001314.7F2951A2078@async.async.caltech.edu>, , , , <20110220005904.A15F21A2078@async.async.caltech.edu>, , <20110220015107.110461A2078@async.async.caltech.edu> Message-ID: <20110220171219.9CE011A2078@async.async.caltech.edu> Jay K writes: > - You can't get SIGSEGV w/o memory protection. SIGSEGV is generated by har= >dware. >=20 I'm not interested in SIGSEGV per se. The point is to map whatever bad memory reference can be generated by non-UNSAFE code to a RuntimeError.E. If you have the hardware to do that, you use it. A Modula-3 implementation on a system without memory protection would have to generate RuntimeError.E using a different mechanism. > - Untrusted code should not be run in-process=2C at least unsafe untrusted= > code. Why not? If you can prove it can't cause damage, then it can't. Why should Java be able to run untrusted code and not we? >=20 > - I don't have to reboot my computer for SIGSEGV=2C but individual process= >es are killed. It's the same argument. Why do you trust your kernel? It's buggy too! If a program crashes, better to reboot the machine to be sure you didn't infect something else. Heck, unplug it for three minutes just to make sure... > The damage is contained as soon as it is detected. The nice thing about = >safe languages > is the error is caught just before any damage is caused=2C i.e. array b= >ounds checking. My point exactly. No reason to start from a clean slate, if you have some obvious earlier place to jump back to. All the examples I've given of situations where it's useful to catch the RuntimeError.E corresponding to SIGSEGV are systems that can be thought of as pseudo-OSes. You have a "job" or "process" doing something within the context of a larger system, or at least there are levels of trust. You're running "an untrusted module" or "a user command". I won't call them "processes" because the terminology would get confusing, but it's the same idea. If one of these modules crashes, there is really no need to abort the whole world, and it would be extremely annoying to user(s) to have to do so. Mika From rodney_bates at lcwb.coop Sun Feb 20 18:58:27 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 20 Feb 2011 11:58:27 -0600 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110219232729.53A991A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> <20110219232729.53A991A2078@async.async.caltech.edu> Message-ID: <4D615643.7060107@lcwb.coop> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. Probably a fully protected guard page at the end of the stack. This technique always worries me a bit because a procedure with a really big activation record could jump right past it. Probably it would almost always access the first page of the big area before storing anything into later pages. On 02/19/2011 05:27 PM, Mika Nystrom wrote: > Ah, yes, stack protection. > > Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > > Hmm, I get SIGILL on AMD64_FREEBSD as well: > > time ../AMD64_FREEBSD/stubexample > M-Scheme Experimental > LITHP ITH LITHENING. >> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > f >> (f 0) > Illegal instruction > 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > > What absolutely must not happen, of course, is that the runtime hangs > while executing only safe code... > > Mika > > "Rodney M. Bates" writes: >> I know of one other place the compilers rely on hardware memory protection >> to detect a checked runtime error, and that is stack overflow. This won't >> corrupt anything, but is hard to distinguish from dereferencing NIL. >> This could probably be distinguished after the fact by some low-level, >> target-dependent code. I have found it by looking at assembly code at >> the point of failure--usually right after a stack pointer push. >> >> Detecting this via compiler-generated checks would probably be more >> extravagant than many other checks, as it is so frequent. I am not >> aware of any really good solution to this in any implementation of any >> language. >> >> On 02/19/2011 02:38 PM, Mika Nystrom wrote: >>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >>> you know! >>> >>> "Corrupting the heap" is something that can only happen as a result of >>> an unchecked runtime error. Unchecked runtime errors cannot happen in >>> modules not marked UNSAFE. >>> >>> SEGV is, however, used by the CM3 implementation (and its predecessors) >>> to signal a certain kind of *checked* runtime error, namely, the >>> dereferencing of a NIL reference. Correct me if I am wrong, but an >>> attempt to dereference NIL is not going to leave the heap corrupted? >>> >>> And if you stick to safe code, the only SEGVs I think you get in the >>> current CM3 are ones from NIL dereferences. >>> >>> Hence, as long as you stick with safe code, the only time the code I >>> checked in earlier gets triggered is for NIL dereferences, which should >>> never corrupt the heap. So SEGV is not sometimes, but in fact always >>> recoverable. >>> >>> :-) >>> >>> Mika >>> >>> P.S. the bit above "if you stick to safe code": if you actually program in >>> Modula-3 you almost never use UNSAFE. I went through my repository and >>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >>> many of the UNSAFE modules are glue code to Fortran routines, which >>> could relatively easily be verified to be safe in the Modula-3 sense. >>> Almost all what remains is glue to some C library, which wouldn't be >>> necessary if the rest of the world would wake up out of the dark ages, but >>> I don't have the time to rewrite every single library from scratch myself. >>> >>> >>> Jay K writes: >>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>> Content-Type: text/plain; charset="iso-8859-1" >>>> Content-Transfer-Encoding: quoted-printable >>>> >>>> >>>> Letting any code run after a SIGSEGV is dubious. >>>> Imagine the heap is corrupted. >>>> And then you run more code. >>>> And the code happens to call malloc. >>>> Or printf to log something. >>>> =20 >>>> I suppose there might be an application that maps memory >>>> gradually=2C as pieces of a buffer are hit. Might. >>>> =20 >>>> - Jay >>>> =20 >>>>> To: m3devel at elegosoft.com >>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>>> From: mika at async.caltech.edu >>>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>>> =20 >>>>> =20 >>>>> Dear m3devel=2C >>>>> =20 >>>>> For a while it has annoyed me that segmentation violations cause an >>>>> unconditional program abort. I've changed that now so that (under user >>>>> threads at least) we instead get a RuntimeError. Here's an example of >>>>> the mechanism at work in an interactive Scheme environment. Consider >>>>> the unhelpful interface and module Crash: >>>>> =20 >>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>>> =20 >>>>> MODULE Crash=3B >>>>> =20 >>>>> PROCEDURE Me() =3D >>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>>> ptr^ :=3D 0 >>>>> END Me=3B >>>>> =20 >>>>> BEGIN END Crash. >>>>> =20 >>>>> Here's an example of what happens if you now call this from an interactiv= >>>> e >>>>> interpreter that catches the exception RuntimeError.E: >>>>> =20 >>>>> M-Scheme Experimental >>>>> LITHP ITH LITHENING. >>>>>> (require-modules "m3") >>>>> #t >>>>>> (Crash.Me) >>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>>> (+ 3 4)=20 >>>>> 7 >>>>>> =20 >>>>> =20 >>>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>>> heck it.=20 >>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>>> =20 >>>>> Mika >>>>> =20 >>>> = >>>> >>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>> Content-Type: text/html; charset="iso-8859-1" >>>> Content-Transfer-Encoding: quoted-printable >>>> >>>> >>>> >>>> >>>> >>>> >>>> Letting any code run after a SIGSEGV is dubious.
>>>> Imagine the heap =3Bis corrupted.
>>>> And then you run more code.
>>>> And the code happens to call malloc.
>>>> Or printf to log something.
>>>>  =3B
>>>> I suppose there might be an application that maps memory
>>>> gradually=2C as pieces of a buffer are hit. Might.
>>>>  =3B
>>>>  =3B- Jay
 =3B
>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>>> olations cause an
>=3B unconditional program abort. I've changed that = >>>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>>> >=3B Here's an example of what happens if you now call this from an inter= >>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>>> >>>> = >>>> >>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >>> > From mika at async.caltech.edu Sun Feb 20 19:37:46 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 10:37:46 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <4D615643.7060107@lcwb.coop> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> <20110219232729.53A991A2078@async.async.caltech.edu> <4D615643.7060107@lcwb.coop> Message-ID: <20110220183746.195C11A2078@async.async.caltech.edu> On a 64-bit machine, at least, there ought to be enough virtual memory that you could just have a gap between thread stacks big enough to allow for a protection area larger than the largest possible (implementation-defined) activation record, no? I know I've run into trouble with very large activation records in the past (and not because I was running out of stack space, either). Or at least a procedure with a very large activation record (or a procedure calling it) could be required to call some sort of check routine "EnoughStackSpaceRemaining()" before starting to scribble on the activation record? Also the end of the activation record must be written to at least once, or else the memory protection won't be triggered. In any case if this is done properly the same mechanism I proposed for SIGSEGV ought to be able to catch stack overflow, no? Well, as long as signals are delivered on a separate stack. If signals are delivered on the same stack, the signal handler would get nastier, it would have to make space through some manipulations (maybe temporarily unporotecting the redzone page?) for its own purposes... but I don't see why it couldn't be done. Not sure why I'm getting SIGILL... maybe I am getting my signal handler activated inside the redzone page because of a difference in signal handling..? I remember reading something about sigaltstack... I would of course love to be able to recover from stack overflow, too. In some sense, since it's a generally unknown limit, it's even less of a fatal error than a NIL dereference (hence makes even more sense to catch it). Mika "Rodney M. Bates" writes: >I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. >Probably a fully protected guard page at the end of the stack. This technique >always worries me a bit because a procedure with a really big activation record >could jump right past it. Probably it would almost always access the first page >of the big area before storing anything into later pages. > >On 02/19/2011 05:27 PM, Mika Nystrom wrote: >> Ah, yes, stack protection. >> >> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. >> >> Hmm, I get SIGILL on AMD64_FREEBSD as well: >> >> time ../AMD64_FREEBSD/stubexample >> M-Scheme Experimental >> LITHP ITH LITHENING. >>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) >> f >>> (f 0) >> Illegal instruction >> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w >> >> What absolutely must not happen, of course, is that the runtime hangs >> while executing only safe code... >> >> Mika >> >> "Rodney M. Bates" writes: >>> I know of one other place the compilers rely on hardware memory protection >>> to detect a checked runtime error, and that is stack overflow. This won't >>> corrupt anything, but is hard to distinguish from dereferencing NIL. >>> This could probably be distinguished after the fact by some low-level, >>> target-dependent code. I have found it by looking at assembly code at >>> the point of failure--usually right after a stack pointer push. >>> >>> Detecting this via compiler-generated checks would probably be more >>> extravagant than many other checks, as it is so frequent. I am not >>> aware of any really good solution to this in any implementation of any >>> language. >>> >>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: >>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >>>> you know! >>>> >>>> "Corrupting the heap" is something that can only happen as a result of >>>> an unchecked runtime error. Unchecked runtime errors cannot happen in >>>> modules not marked UNSAFE. >>>> >>>> SEGV is, however, used by the CM3 implementation (and its predecessors) >>>> to signal a certain kind of *checked* runtime error, namely, the >>>> dereferencing of a NIL reference. Correct me if I am wrong, but an >>>> attempt to dereference NIL is not going to leave the heap corrupted? >>>> >>>> And if you stick to safe code, the only SEGVs I think you get in the >>>> current CM3 are ones from NIL dereferences. >>>> >>>> Hence, as long as you stick with safe code, the only time the code I >>>> checked in earlier gets triggered is for NIL dereferences, which should >>>> never corrupt the heap. So SEGV is not sometimes, but in fact always >>>> recoverable. >>>> >>>> :-) >>>> >>>> Mika >>>> >>>> P.S. the bit above "if you stick to safe code": if you actually program in >>>> Modula-3 you almost never use UNSAFE. I went through my repository and >>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >>>> many of the UNSAFE modules are glue code to Fortran routines, which >>>> could relatively easily be verified to be safe in the Modula-3 sense. >>>> Almost all what remains is glue to some C library, which wouldn't be >>>> necessary if the rest of the world would wake up out of the dark ages, but >>>> I don't have the time to rewrite every single library from scratch myself. >>>> >>>> >>>> Jay K writes: >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>> Content-Type: text/plain; charset="iso-8859-1" >>>>> Content-Transfer-Encoding: quoted-printable >>>>> >>>>> >>>>> Letting any code run after a SIGSEGV is dubious. >>>>> Imagine the heap is corrupted. >>>>> And then you run more code. >>>>> And the code happens to call malloc. >>>>> Or printf to log something. >>>>> =20 >>>>> I suppose there might be an application that maps memory >>>>> gradually=2C as pieces of a buffer are hit. Might. >>>>> =20 >>>>> - Jay >>>>> =20 >>>>>> To: m3devel at elegosoft.com >>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>>>> From: mika at async.caltech.edu >>>>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>>>> =20 >>>>>> =20 >>>>>> Dear m3devel=2C >>>>>> =20 >>>>>> For a while it has annoyed me that segmentation violations cause an >>>>>> unconditional program abort. I've changed that now so that (under user >>>>>> threads at least) we instead get a RuntimeError. Here's an example of >>>>>> the mechanism at work in an interactive Scheme environment. Consider >>>>>> the unhelpful interface and module Crash: >>>>>> =20 >>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>>>> =20 >>>>>> MODULE Crash=3B >>>>>> =20 >>>>>> PROCEDURE Me() =3D >>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>>>> ptr^ :=3D 0 >>>>>> END Me=3B >>>>>> =20 >>>>>> BEGIN END Crash. >>>>>> =20 >>>>>> Here's an example of what happens if you now call this from an interactiv= >>>>> e >>>>>> interpreter that catches the exception RuntimeError.E: >>>>>> =20 >>>>>> M-Scheme Experimental >>>>>> LITHP ITH LITHENING. >>>>>>> (require-modules "m3") >>>>>> #t >>>>>>> (Crash.Me) >>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>>>> (+ 3 4)=20 >>>>>> 7 >>>>>>> =20 >>>>>> =20 >>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>>>> heck it.=20 >>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>>>> =20 >>>>>> Mika >>>>>> =20 >>>>> = >>>>> >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>> Content-Type: text/html; charset="iso-8859-1" >>>>> Content-Transfer-Encoding: quoted-printable >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Letting any code run after a SIGSEGV is dubious.
>>>>> Imagine the heap =3Bis corrupted.
>>>>> And then you run more code.
>>>>> And the code happens to call malloc.
>>>>> Or printf to log something.
>>>>>  =3B
>>>>> I suppose there might be an application that maps memory
>>>>> gradually=2C as pieces of a buffer are hit. Might.
>>>>>  =3B
>>>>>  =3B- Jay
 =3B
>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>>>> olations cause an
>=3B unconditional program abort. I've changed that = >>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>>>> >=3B Here's an example of what happens if you now call this from an inter= >>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>>>> >>>>> = >>>>> >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >>>> >> From jay.krell at cornell.edu Sun Feb 20 23:38:19 2011 From: jay.krell at cornell.edu (Jay K) Date: Sun, 20 Feb 2011 22:38:19 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220183746.195C11A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu>, <4D615643.7060107@lcwb.coop>, <20110220183746.195C11A2078@async.async.caltech.edu> Message-ID: Probably for this reason: There is a requirement on NT that stack pages be touched in order. Functions with locals totally more than 4K call _chkstk (aka _alloca) to allocate their stack, instead of the usual register subtraction. It contains a loop that touches a byte every 4K. Otherwise, if you have lots of functions with small frames, the stack is touched by virtue of the call function pushing the return address. Modula-3 has long failed to uphold this contract, and still does. I've never seen it clearly documented, but you can see it is what the C compiler does. I had thought there were other reasons for this behavior. I thought you actually get an exception of the stack is touched out of order ("the first time"). But I think I did an experiment long ago with Modula-3 and there was no exception. I don't know about other platforms. I've also seen evidence in gcc and/or Target.i3 that compiler writers know well about such mechanisms -- there being a constant to set as to what size locals trigger special behavior. But m3back doesn't do anything here. Probably we could use _chkstk unconditonally as well -- need to see what it does for numbers smaller than 4K. But that'd be a deoptimization in the common case, and deoptimizing only as necessary should be easy enough. - Jay > To: rodney_bates at lcwb.coop > Date: Sun, 20 Feb 2011 10:37:46 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > On a 64-bit machine, at least, there ought to be enough virtual > memory that you could just have a gap between thread stacks big > enough to allow for a protection area larger than the largest possible > (implementation-defined) activation record, no? I know I've run into > trouble with very large activation records in the past (and not because > I was running out of stack space, either). > > Or at least a procedure with a very large activation record (or > a procedure calling it) could be required to call some sort of check > routine "EnoughStackSpaceRemaining()" before starting to scribble > on the activation record? > > Also the end of the activation record must be written to at least once, > or else the memory protection won't be triggered. > > In any case if this is done properly the same mechanism I proposed for > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > signals are delivered on a separate stack. If signals are delivered on > the same stack, the signal handler would get nastier, it would have to > make space through some manipulations (maybe temporarily unporotecting > the redzone page?) for its own purposes... but I don't see why it > couldn't be done. > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > activated inside the redzone page because of a difference in signal > handling..? I remember reading something about sigaltstack... > > I would of course love to be able to recover from stack overflow, too. > In some sense, since it's a generally unknown limit, it's even less of > a fatal error than a NIL dereference (hence makes even more sense to > catch it). > > Mika > > "Rodney M. Bates" writes: > >I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >Probably a fully protected guard page at the end of the stack. This technique > >always worries me a bit because a procedure with a really big activation record > >could jump right past it. Probably it would almost always access the first page > >of the big area before storing anything into later pages. > > > >On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >> Ah, yes, stack protection. > >> > >> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >> > >> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >> > >> time ../AMD64_FREEBSD/stubexample > >> M-Scheme Experimental > >> LITHP ITH LITHENING. > >>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >> f > >>> (f 0) > >> Illegal instruction > >> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >> > >> What absolutely must not happen, of course, is that the runtime hangs > >> while executing only safe code... > >> > >> Mika > >> > >> "Rodney M. Bates" writes: > >>> I know of one other place the compilers rely on hardware memory protection > >>> to detect a checked runtime error, and that is stack overflow. This won't > >>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>> This could probably be distinguished after the fact by some low-level, > >>> target-dependent code. I have found it by looking at assembly code at > >>> the point of failure--usually right after a stack pointer push. > >>> > >>> Detecting this via compiler-generated checks would probably be more > >>> extravagant than many other checks, as it is so frequent. I am not > >>> aware of any really good solution to this in any implementation of any > >>> language. > >>> > >>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>> you know! > >>>> > >>>> "Corrupting the heap" is something that can only happen as a result of > >>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>> modules not marked UNSAFE. > >>>> > >>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>> to signal a certain kind of *checked* runtime error, namely, the > >>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>> > >>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>> current CM3 are ones from NIL dereferences. > >>>> > >>>> Hence, as long as you stick with safe code, the only time the code I > >>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>> recoverable. > >>>> > >>>> :-) > >>>> > >>>> Mika > >>>> > >>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>> Almost all what remains is glue to some C library, which wouldn't be > >>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>> I don't have the time to rewrite every single library from scratch myself. > >>>> > >>>> > >>>> Jay K writes: > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious. > >>>>> Imagine the heap is corrupted. > >>>>> And then you run more code. > >>>>> And the code happens to call malloc. > >>>>> Or printf to log something. > >>>>> =20 > >>>>> I suppose there might be an application that maps memory > >>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>> =20 > >>>>> - Jay > >>>>> =20 > >>>>>> To: m3devel at elegosoft.com > >>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>> From: mika at async.caltech.edu > >>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>> =20 > >>>>>> =20 > >>>>>> Dear m3devel=2C > >>>>>> =20 > >>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>> the unhelpful interface and module Crash: > >>>>>> =20 > >>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>> =20 > >>>>>> MODULE Crash=3B > >>>>>> =20 > >>>>>> PROCEDURE Me() =3D > >>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>> ptr^ :=3D 0 > >>>>>> END Me=3B > >>>>>> =20 > >>>>>> BEGIN END Crash. > >>>>>> =20 > >>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>> e > >>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>> =20 > >>>>>> M-Scheme Experimental > >>>>>> LITHP ITH LITHENING. > >>>>>>> (require-modules "m3") > >>>>>> #t > >>>>>>> (Crash.Me) > >>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>> (+ 3 4)=20 > >>>>>> 7 > >>>>>>> =20 > >>>>>> =20 > >>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>> heck it.=20 > >>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>> =20 > >>>>>> Mika > >>>>>> =20 > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>> Imagine the heap =3Bis corrupted.
> >>>>> And then you run more code.
> >>>>> And the code happens to call malloc.
> >>>>> Or printf to log something.
> >>>>>  =3B
> >>>>> I suppose there might be an application that maps memory
> >>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>  =3B
> >>>>>  =3B- Jay
 =3B
> >>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>> > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>> > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 01:41:28 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 00:41:28 +0000 Subject: [M3devel] libm3/Process Message-ID: Process.Wait frees up resources. Is that the right way to go? Shouldn't it register a cleanup function for the garbage collector to call, like we do for mutexes? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Mon Feb 21 02:15:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 17:15:50 -0800 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20110221003248.11C9C2474005@birch.elegosoft.com>, Message-ID: <20110221011550.0D2791A2078@async.async.caltech.edu> Jay I didn't quite follow your previous messages. What is the correct solution, and what did you mean when you were talking about the garbage collector? It does seem like---if a Process handle goes out of scope and gets collected then yes the runtime should do the equivalent of "EVAL Process.Wait" at some future time. But that seems unrelated/orthogonal to the issues with the Windows process killing... The process killing in m3middle/M3Process I think is only called in a few places in the compiler and then under what already are dubious circumstances. For instance I see it done where Process.Create has failed. But if Process.Create has failed, then there's no process to kill, so it all seems rather pointless. It is also called as one of the last actions before exiting the parent process, so there's another reason for its being pointless. In other words I think it's safe to say that if you have an idea for how to implement process-killing correctly on Windows you can do it however you like without fear of breaking any existing code... Mika But I can Jay K writes: >--_d36c70d3-e71b-433d-abdc-06703a929f89_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >The code here confuses hProcess and dwProcessId. >Can we introduce a public function here in libm3 instead of m3middle? >ie: to avoid answering the question as to how to expose hProcess? >And to avoid the loophole as well? > >Thanks=2C > - Jay > >From: jay.krell at cornell.edu >To: jkrell at elego.de=3B m3commit at elegosoft.com >Subject: RE: [M3commit] CVS Update: cm3 >Date: Mon=2C 21 Feb 2011 00:35:38 +0000 > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= >rect here. > > - Jay > >> Date: Mon=2C 21 Feb 2011 01:32:47 +0000 >> To: m3commit at elegosoft.com >> From: jkrell at elego.de >> Subject: [M3commit] CVS Update: cm3 >>=20 >> CVSROOT: /usr/cvs >> Changes by: jkrell at birch. 11/02/21 01:32:47 >>=20 >> Modified files: >> cm3/m3-sys/m3middle/src/: M3Process.i3=20 >>=20 >> Log message: >> remove incorrect comments=3B Win32 is not broken >>=20 > = > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >The code here confuses hProcess and dwProcessId.
Can we introduce a publ= >ic function here in libm3 instead of m3middle?
ie: to avoid answering th= >e question as to how to expose hProcess?
And to avoid the loophole as we= >ll?

Thanks=2C
 =3B- Jay

From: = >jay.krell at cornell.edu
To: jkrell at elego.de=3B m3commit at elegosoft.com
S= >ubject: RE: [M3commit] CVS Update: cm3
Date: Mon=2C 21 Feb 2011 00:35:38= > +0000

> >"> > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= >rect here.

 =3B- Jay

>=3B Date: Mon=2C 21 Feb 2011 01:3= >2:47 +0000
>=3B To: m3commit at elegosoft.com
>=3B From: jkrell at eleg= >o.de
>=3B Subject: [M3commit] CVS Update: cm3
>=3B
>=3B CVS= >ROOT: /usr/cvs
>=3B Changes by: jkrell at birch. 11/02/21 01:32:47
>= >=3B
>=3B Modified files:
>=3B cm3/m3-sys/m3middle/src/: M3Proce= >ss.i3
>=3B
>=3B Log message:
>=3B remove incorrect commen= >ts=3B Win32 is not broken
>=3B
>= > >--_d36c70d3-e71b-433d-abdc-06703a929f89_-- From rodney_bates at lcwb.coop Mon Feb 21 03:21:27 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Sun, 20 Feb 2011 20:21:27 -0600 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110220183746.195C11A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> <20110219232729.53A991A2078@async.async.caltech.edu> <4D615643.7060107@lcwb.coop> <20110220183746.195C11A2078@async.async.caltech.edu> Message-ID: <4D61CC27.7070704@lcwb.coop> On 02/20/2011 12:37 PM, Mika Nystrom wrote: > On a 64-bit machine, at least, there ought to be enough virtual > memory that you could just have a gap between thread stacks big > enough to allow for a protection area larger than the largest possible > (implementation-defined) activation record, no? I know I've run into > trouble with very large activation records in the past (and not because > I was running out of stack space, either). > > Or at least a procedure with a very large activation record (or > a procedure calling it) could be required to call some sort of check > routine "EnoughStackSpaceRemaining()" before starting to scribble > on the activation record? Hmm, I like this idea. It would introduce normal-case runtime overhead only for such procedures, and these are likely rare. Also, assuming the procedure actually uses very much of its large AR, it should also have enough computation time to wash out the stack check overhead. > > Also the end of the activation record must be written to at least once, > or else the memory protection won't be triggered. > I was thinking (as an alternative mechanism) of having the compiler intentionally add enough artificial write(s) as necessary to ensure storing within the red zone, and not just beyond it. This seems trickier to get right and harder to distinguish after the fact from a NIL dereference. > In any case if this is done properly the same mechanism I proposed for > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > signals are delivered on a separate stack. If signals are delivered on > the same stack, the signal handler would get nastier, it would have to > make space through some manipulations (maybe temporarily unporotecting > the redzone page?) for its own purposes... but I don't see why it > couldn't be done. > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > activated inside the redzone page because of a difference in signal > handling..? I remember reading something about sigaltstack... > > I would of course love to be able to recover from stack overflow, too. > In some sense, since it's a generally unknown limit, it's even less of > a fatal error than a NIL dereference (hence makes even more sense to > catch it). I think this would be a nice mechanism to have available. It would have to be used with some care. In any case, it would be really nice and more frequently so, to at least have runtime error messages that distinguished stack overflow from NIL deref. > > Mika > > "Rodney M. Bates" writes: >> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. >> Probably a fully protected guard page at the end of the stack. This technique >> always worries me a bit because a procedure with a really big activation record >> could jump right past it. Probably it would almost always access the first page >> of the big area before storing anything into later pages. >> >> On 02/19/2011 05:27 PM, Mika Nystrom wrote: >>> Ah, yes, stack protection. >>> >>> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. >>> >>> Hmm, I get SIGILL on AMD64_FREEBSD as well: >>> >>> time ../AMD64_FREEBSD/stubexample >>> M-Scheme Experimental >>> LITHP ITH LITHENING. >>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) >>> f >>>> (f 0) >>> Illegal instruction >>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w >>> >>> What absolutely must not happen, of course, is that the runtime hangs >>> while executing only safe code... >>> >>> Mika >>> >>> "Rodney M. Bates" writes: >>>> I know of one other place the compilers rely on hardware memory protection >>>> to detect a checked runtime error, and that is stack overflow. This won't >>>> corrupt anything, but is hard to distinguish from dereferencing NIL. >>>> This could probably be distinguished after the fact by some low-level, >>>> target-dependent code. I have found it by looking at assembly code at >>>> the point of failure--usually right after a stack pointer push. >>>> >>>> Detecting this via compiler-generated checks would probably be more >>>> extravagant than many other checks, as it is so frequent. I am not >>>> aware of any really good solution to this in any implementation of any >>>> language. >>>> >>>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: >>>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >>>>> you know! >>>>> >>>>> "Corrupting the heap" is something that can only happen as a result of >>>>> an unchecked runtime error. Unchecked runtime errors cannot happen in >>>>> modules not marked UNSAFE. >>>>> >>>>> SEGV is, however, used by the CM3 implementation (and its predecessors) >>>>> to signal a certain kind of *checked* runtime error, namely, the >>>>> dereferencing of a NIL reference. Correct me if I am wrong, but an >>>>> attempt to dereference NIL is not going to leave the heap corrupted? >>>>> >>>>> And if you stick to safe code, the only SEGVs I think you get in the >>>>> current CM3 are ones from NIL dereferences. >>>>> >>>>> Hence, as long as you stick with safe code, the only time the code I >>>>> checked in earlier gets triggered is for NIL dereferences, which should >>>>> never corrupt the heap. So SEGV is not sometimes, but in fact always >>>>> recoverable. >>>>> >>>>> :-) >>>>> >>>>> Mika >>>>> >>>>> P.S. the bit above "if you stick to safe code": if you actually program in >>>>> Modula-3 you almost never use UNSAFE. I went through my repository and >>>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >>>>> many of the UNSAFE modules are glue code to Fortran routines, which >>>>> could relatively easily be verified to be safe in the Modula-3 sense. >>>>> Almost all what remains is glue to some C library, which wouldn't be >>>>> necessary if the rest of the world would wake up out of the dark ages, but >>>>> I don't have the time to rewrite every single library from scratch myself. >>>>> >>>>> >>>>> Jay K writes: >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>>> Content-Type: text/plain; charset="iso-8859-1" >>>>>> Content-Transfer-Encoding: quoted-printable >>>>>> >>>>>> >>>>>> Letting any code run after a SIGSEGV is dubious. >>>>>> Imagine the heap is corrupted. >>>>>> And then you run more code. >>>>>> And the code happens to call malloc. >>>>>> Or printf to log something. >>>>>> =20 >>>>>> I suppose there might be an application that maps memory >>>>>> gradually=2C as pieces of a buffer are hit. Might. >>>>>> =20 >>>>>> - Jay >>>>>> =20 >>>>>>> To: m3devel at elegosoft.com >>>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>>>>> From: mika at async.caltech.edu >>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>>>>> =20 >>>>>>> =20 >>>>>>> Dear m3devel=2C >>>>>>> =20 >>>>>>> For a while it has annoyed me that segmentation violations cause an >>>>>>> unconditional program abort. I've changed that now so that (under user >>>>>>> threads at least) we instead get a RuntimeError. Here's an example of >>>>>>> the mechanism at work in an interactive Scheme environment. Consider >>>>>>> the unhelpful interface and module Crash: >>>>>>> =20 >>>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>>>>> =20 >>>>>>> MODULE Crash=3B >>>>>>> =20 >>>>>>> PROCEDURE Me() =3D >>>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>>>>> ptr^ :=3D 0 >>>>>>> END Me=3B >>>>>>> =20 >>>>>>> BEGIN END Crash. >>>>>>> =20 >>>>>>> Here's an example of what happens if you now call this from an interactiv= >>>>>> e >>>>>>> interpreter that catches the exception RuntimeError.E: >>>>>>> =20 >>>>>>> M-Scheme Experimental >>>>>>> LITHP ITH LITHENING. >>>>>>>> (require-modules "m3") >>>>>>> #t >>>>>>>> (Crash.Me) >>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>>>>> (+ 3 4)=20 >>>>>>> 7 >>>>>>>> =20 >>>>>>> =20 >>>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>>>>> heck it.=20 >>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>>>>> =20 >>>>>>> Mika >>>>>>> =20 >>>>>> = >>>>>> >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>>> Content-Type: text/html; charset="iso-8859-1" >>>>>> Content-Transfer-Encoding: quoted-printable >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Letting any code run after a SIGSEGV is dubious.
>>>>>> Imagine the heap =3Bis corrupted.
>>>>>> And then you run more code.
>>>>>> And the code happens to call malloc.
>>>>>> Or printf to log something.
>>>>>>  =3B
>>>>>> I suppose there might be an application that maps memory
>>>>>> gradually=2C as pieces of a buffer are hit. Might.
>>>>>>  =3B
>>>>>>  =3B- Jay
 =3B
>>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>>>>> olations cause an
>=3B unconditional program abort. I've changed that = >>>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>>>>> >=3B Here's an example of what happens if you now call this from an inter= >>>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>>>>> >>>>>> = >>>>>> >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >>>>> >>> > From jay.krell at cornell.edu Mon Feb 21 04:24:53 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 03:24:53 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <4D61CC27.7070704@lcwb.coop> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu>, <4D615643.7060107@lcwb.coop>, <20110220183746.195C11A2078@async.async.caltech.edu>, <4D61CC27.7070704@lcwb.coop> Message-ID: We should look at what C does on the various targets. NT already has a specific easy simple mechanism/policy here, that Modula-3 should follow but doesn't. Specifically, C code in NT can overflow its stack, but it is caught "right away" and not via random corruption of neighboring memory. Perhaps, hopefully, C code in other targets is this good. If not, well, the NT mechanism makes a lot of sense for all targets: if function has less than a page of locals, do nothing special. if function has more than a page of locals, touch each page, in order, at the start of the function. "page" is hardware specific, however I believe all of our targets have 4K or 8K pages, and if you just hardcode 4K, that works, with a slight slight pessimization on targets with 8K pages (e.g. I think Sparc and IA64, but I'd have to dig around). Actually NT reserves like two pages at the end of the stack. One to trigger stack overflow exception, one to give an exception handler a little bit of room to deal with it, such as by capturing a dump of some sort, and exiting, and then if the exception handler uses more than its page, the process is I think terminated. FURTHERMORE, this is a detail that SHOULD be handled by the existing gcc backend. I don't know if it does, but it ought to. But the NT backend does not, and should. It's possible this problem has regressed with the 1K jmpbuf I put in. (I'm very very inclined to shrink that down either to precise levels or approximate but usually smaller levels. m3core did have an assert at startup that the size was large enough. Still, we can't get the 128 bit alignment some targets prefer (powerpc) or require (hppa64). Still, the alloca solution is better, if done right.) - Jay > Date: Sun, 20 Feb 2011 20:21:27 -0600 > From: rodney_bates at lcwb.coop > To: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > > On 02/20/2011 12:37 PM, Mika Nystrom wrote: > > On a 64-bit machine, at least, there ought to be enough virtual > > memory that you could just have a gap between thread stacks big > > enough to allow for a protection area larger than the largest possible > > (implementation-defined) activation record, no? I know I've run into > > trouble with very large activation records in the past (and not because > > I was running out of stack space, either). > > > > Or at least a procedure with a very large activation record (or > > a procedure calling it) could be required to call some sort of check > > routine "EnoughStackSpaceRemaining()" before starting to scribble > > on the activation record? > > Hmm, I like this idea. It would introduce normal-case runtime overhead > only for such procedures, and these are likely rare. Also, assuming the procedure > actually uses very much of its large AR, it should also have enough computation > time to wash out the stack check overhead. > > > > > Also the end of the activation record must be written to at least once, > > or else the memory protection won't be triggered. > > > > I was thinking (as an alternative mechanism) of having the compiler intentionally > add enough artificial write(s) as necessary to ensure storing within the > red zone, and not just beyond it. This seems trickier to get right and > harder to distinguish after the fact from a NIL dereference. > > > In any case if this is done properly the same mechanism I proposed for > > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > > signals are delivered on a separate stack. If signals are delivered on > > the same stack, the signal handler would get nastier, it would have to > > make space through some manipulations (maybe temporarily unporotecting > > the redzone page?) for its own purposes... but I don't see why it > > couldn't be done. > > > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > > activated inside the redzone page because of a difference in signal > > handling..? I remember reading something about sigaltstack... > > > > I would of course love to be able to recover from stack overflow, too. > > In some sense, since it's a generally unknown limit, it's even less of > > a fatal error than a NIL dereference (hence makes even more sense to > > catch it). > > I think this would be a nice mechanism to have available. It would have to > be used with some care. In any case, it would be really nice and more > frequently so, to at least have runtime error messages that distinguished > stack overflow from NIL deref. > > > > > Mika > > > > "Rodney M. Bates" writes: > >> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >> Probably a fully protected guard page at the end of the stack. This technique > >> always worries me a bit because a procedure with a really big activation record > >> could jump right past it. Probably it would almost always access the first page > >> of the big area before storing anything into later pages. > >> > >> On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >>> Ah, yes, stack protection. > >>> > >>> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >>> > >>> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >>> > >>> time ../AMD64_FREEBSD/stubexample > >>> M-Scheme Experimental > >>> LITHP ITH LITHENING. > >>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >>> f > >>>> (f 0) > >>> Illegal instruction > >>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >>> > >>> What absolutely must not happen, of course, is that the runtime hangs > >>> while executing only safe code... > >>> > >>> Mika > >>> > >>> "Rodney M. Bates" writes: > >>>> I know of one other place the compilers rely on hardware memory protection > >>>> to detect a checked runtime error, and that is stack overflow. This won't > >>>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>>> This could probably be distinguished after the fact by some low-level, > >>>> target-dependent code. I have found it by looking at assembly code at > >>>> the point of failure--usually right after a stack pointer push. > >>>> > >>>> Detecting this via compiler-generated checks would probably be more > >>>> extravagant than many other checks, as it is so frequent. I am not > >>>> aware of any really good solution to this in any implementation of any > >>>> language. > >>>> > >>>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>>> you know! > >>>>> > >>>>> "Corrupting the heap" is something that can only happen as a result of > >>>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>>> modules not marked UNSAFE. > >>>>> > >>>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>>> to signal a certain kind of *checked* runtime error, namely, the > >>>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>>> > >>>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>>> current CM3 are ones from NIL dereferences. > >>>>> > >>>>> Hence, as long as you stick with safe code, the only time the code I > >>>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>>> recoverable. > >>>>> > >>>>> :-) > >>>>> > >>>>> Mika > >>>>> > >>>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>>> Almost all what remains is glue to some C library, which wouldn't be > >>>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>>> I don't have the time to rewrite every single library from scratch myself. > >>>>> > >>>>> > >>>>> Jay K writes: > >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>>> Content-Transfer-Encoding: quoted-printable > >>>>>> > >>>>>> > >>>>>> Letting any code run after a SIGSEGV is dubious. > >>>>>> Imagine the heap is corrupted. > >>>>>> And then you run more code. > >>>>>> And the code happens to call malloc. > >>>>>> Or printf to log something. > >>>>>> =20 > >>>>>> I suppose there might be an application that maps memory > >>>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>>> =20 > >>>>>> - Jay > >>>>>> =20 > >>>>>>> To: m3devel at elegosoft.com > >>>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>>> From: mika at async.caltech.edu > >>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>>> =20 > >>>>>>> =20 > >>>>>>> Dear m3devel=2C > >>>>>>> =20 > >>>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>>> the unhelpful interface and module Crash: > >>>>>>> =20 > >>>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>>> =20 > >>>>>>> MODULE Crash=3B > >>>>>>> =20 > >>>>>>> PROCEDURE Me() =3D > >>>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>>> ptr^ :=3D 0 > >>>>>>> END Me=3B > >>>>>>> =20 > >>>>>>> BEGIN END Crash. > >>>>>>> =20 > >>>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>>> e > >>>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>>> =20 > >>>>>>> M-Scheme Experimental > >>>>>>> LITHP ITH LITHENING. > >>>>>>>> (require-modules "m3") > >>>>>>> #t > >>>>>>>> (Crash.Me) > >>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>>> (+ 3 4)=20 > >>>>>>> 7 > >>>>>>>> =20 > >>>>>>> =20 > >>>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>>> heck it.=20 > >>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>>> =20 > >>>>>>> Mika > >>>>>>> =20 > >>>>>> = > >>>>>> > >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>>> Content-Transfer-Encoding: quoted-printable > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>>> Imagine the heap =3Bis corrupted.
> >>>>>> And then you run more code.
> >>>>>> And the code happens to call malloc.
> >>>>>> Or printf to log something.
> >>>>>>  =3B
> >>>>>> I suppose there might be an application that maps memory
> >>>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>>  =3B
> >>>>>>  =3B- Jay
 =3B
> >>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>>> > >>>>>> = > >>>>>> > >>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>>> > >>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 04:28:10 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 03:28:10 +0000 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: <20110221011550.0D2791A2078@async.async.caltech.edu> References: <20110221003248.11C9C2474005@birch.elegosoft.com>, , , , <20110221011550.0D2791A2078@async.async.caltech.edu> Message-ID: Wait? No, that is probably just bogus posix thinking. You can just close your handle to the process. You don't have to wait for it to exit. It is somewhat orthogonal. The thing is, the request for Process.Kill, got me looking at when does libm3 close the Process handle and when does it use the handle correctly (i.e. not m3middle's Process.Abort!) It does the CloseHandle in Process.Wait, so if that is never called, there is a leak. Anyway, this should be easy to make correct. (So easy, though, it boggles the mind as to why it was done this way in the first place. I do have to check somethig -- it is SLIGHTLY possible I broke it. I know I fiddled around with the definition of process ids, made them match the underlying platform. I'll check the history. m3middle uses a LOOPHOLE on some "id" to get the "handle". Anyway, I'll check and make it correct.) - Jay > To: jay.krell at cornell.edu > Date: Sun, 20 Feb 2011 17:15:50 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] [M3commit] CVS Update: cm3 > > > Jay I didn't quite follow your previous messages. What is the correct > solution, and what did you mean when you were talking about the garbage > collector? It does seem like---if a Process handle goes out of scope > and gets collected then yes the runtime should do the equivalent of "EVAL > Process.Wait" at some future time. But that seems unrelated/orthogonal > to the issues with the Windows process killing... > > The process killing in m3middle/M3Process I think is only called in > a few places in the compiler and then under what already are dubious > circumstances. For instance I see it done where Process.Create has > failed. But if Process.Create has failed, then there's no process to > kill, so it all seems rather pointless. It is also called as one of > the last actions before exiting the parent process, so there's another > reason for its being pointless. > > In other words I think it's safe to say that if you have an idea for > how to implement process-killing correctly on Windows you can do it > however you like without fear of breaking any existing code... > > Mika > > But I can > > Jay K writes: > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >The code here confuses hProcess and dwProcessId. > >Can we introduce a public function here in libm3 instead of m3middle? > >ie: to avoid answering the question as to how to expose hProcess? > >And to avoid the loophole as well? > > > >Thanks=2C > > - Jay > > > >From: jay.krell at cornell.edu > >To: jkrell at elego.de=3B m3commit at elegosoft.com > >Subject: RE: [M3commit] CVS Update: cm3 > >Date: Mon=2C 21 Feb 2011 00:35:38 +0000 > > > > > > > > > > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= > >rect here. > > > > - Jay > > > >> Date: Mon=2C 21 Feb 2011 01:32:47 +0000 > >> To: m3commit at elegosoft.com > >> From: jkrell at elego.de > >> Subject: [M3commit] CVS Update: cm3 > >>=20 > >> CVSROOT: /usr/cvs > >> Changes by: jkrell at birch. 11/02/21 01:32:47 > >>=20 > >> Modified files: > >> cm3/m3-sys/m3middle/src/: M3Process.i3=20 > >>=20 > >> Log message: > >> remove incorrect comments=3B Win32 is not broken > >>=20 > > = > > > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ > >Content-Type: text/html; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > > > > > > > >The code here confuses hProcess and dwProcessId.
Can we introduce a publ= > >ic function here in libm3 instead of m3middle?
ie: to avoid answering th= > >e question as to how to expose hProcess?
And to avoid the loophole as we= > >ll?

Thanks=2C
 =3B- Jay

From: = > >jay.krell at cornell.edu
To: jkrell at elego.de=3B m3commit at elegosoft.com
S= > >ubject: RE: [M3commit] CVS Update: cm3
Date: Mon=2C 21 Feb 2011 00:35:38= > > +0000

> > > > >"> > > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= > >rect here.

 =3B- Jay

>=3B Date: Mon=2C 21 Feb 2011 01:3= > >2:47 +0000
>=3B To: m3commit at elegosoft.com
>=3B From: jkrell at eleg= > >o.de
>=3B Subject: [M3commit] CVS Update: cm3
>=3B
>=3B CVS= > >ROOT: /usr/cvs
>=3B Changes by: jkrell at birch. 11/02/21 01:32:47
>= > >=3B
>=3B Modified files:
>=3B cm3/m3-sys/m3middle/src/: M3Proce= > >ss.i3
>=3B
>=3B Log message:
>=3B remove incorrect commen= > >ts=3B Win32 is not broken
>=3B
> >= > > > >--_d36c70d3-e71b-433d-abdc-06703a929f89_-- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 04:46:04 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 03:46:04 +0000 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20110221003248.11C9C2474005@birch.elegosoft.com>, , , , , , , <20110221011550.0D2791A2078@async.async.caltech.edu>, Message-ID: Ok, it looks like I did break it. http://modula3.elegosoft.com/cgi-bin/cvsweb.cgi/cm3/m3-libs/libm3/src/os/WIN32/ProcessWin32.m3.diff?r1=1.6;r2=1.7 GetID used to return the handle. I changed it to return the id. m3middle/Process.Abort LOOPHOLEs GetID to a handle. I think the GetID change should probably stay. But certainly one or the other must change. The handle is not unique across the system, just the process, whereas the id is unique across the system. If Process.i3 only mandates within the process, then maybe the old way. Maybe. - Jay From: jay.krell at cornell.edu To: mika at async.caltech.edu Date: Mon, 21 Feb 2011 03:28:10 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] [M3commit] CVS Update: cm3 Wait? No, that is probably just bogus posix thinking. You can just close your handle to the process. You don't have to wait for it to exit. It is somewhat orthogonal. The thing is, the request for Process.Kill, got me looking at when does libm3 close the Process handle and when does it use the handle correctly (i.e. not m3middle's Process.Abort!) It does the CloseHandle in Process.Wait, so if that is never called, there is a leak. Anyway, this should be easy to make correct. (So easy, though, it boggles the mind as to why it was done this way in the first place. I do have to check somethig -- it is SLIGHTLY possible I broke it. I know I fiddled around with the definition of process ids, made them match the underlying platform. I'll check the history. m3middle uses a LOOPHOLE on some "id" to get the "handle". Anyway, I'll check and make it correct.) - Jay > To: jay.krell at cornell.edu > Date: Sun, 20 Feb 2011 17:15:50 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] [M3commit] CVS Update: cm3 > > > Jay I didn't quite follow your previous messages. What is the correct > solution, and what did you mean when you were talking about the garbage > collector? It does seem like---if a Process handle goes out of scope > and gets collected then yes the runtime should do the equivalent of "EVAL > Process.Wait" at some future time. But that seems unrelated/orthogonal > to the issues with the Windows process killing... > > The process killing in m3middle/M3Process I think is only called in > a few places in the compiler and then under what already are dubious > circumstances. For instance I see it done where Process.Create has > failed. But if Process.Create has failed, then there's no process to > kill, so it all seems rather pointless. It is also called as one of > the last actions before exiting the parent process, so there's another > reason for its being pointless. > > In other words I think it's safe to say that if you have an idea for > how to implement process-killing correctly on Windows you can do it > however you like without fear of breaking any existing code... > > Mika > > But I can > > Jay K writes: > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >The code here confuses hProcess and dwProcessId. > >Can we introduce a public function here in libm3 instead of m3middle? > >ie: to avoid answering the question as to how to expose hProcess? > >And to avoid the loophole as well? > > > >Thanks=2C > > - Jay > > > >From: jay.krell at cornell.edu > >To: jkrell at elego.de=3B m3commit at elegosoft.com > >Subject: RE: [M3commit] CVS Update: cm3 > >Date: Mon=2C 21 Feb 2011 00:35:38 +0000 > > > > > > > > > > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= > >rect here. > > > > - Jay > > > >> Date: Mon=2C 21 Feb 2011 01:32:47 +0000 > >> To: m3commit at elegosoft.com > >> From: jkrell at elego.de > >> Subject: [M3commit] CVS Update: cm3 > >>=20 > >> CVSROOT: /usr/cvs > >> Changes by: jkrell at birch. 11/02/21 01:32:47 > >>=20 > >> Modified files: > >> cm3/m3-sys/m3middle/src/: M3Process.i3=20 > >>=20 > >> Log message: > >> remove incorrect comments=3B Win32 is not broken > >>=20 > > = > > > >--_d36c70d3-e71b-433d-abdc-06703a929f89_ > >Content-Type: text/html; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > > > > > > > >The code here confuses hProcess and dwProcessId.
Can we introduce a publ= > >ic function here in libm3 instead of m3middle?
ie: to avoid answering th= > >e question as to how to expose hProcess?
And to avoid the loophole as we= > >ll?

Thanks=2C
 =3B- Jay

From: = > >jay.krell at cornell.edu
To: jkrell at elego.de=3B m3commit at elegosoft.com
S= > >ubject: RE: [M3commit] CVS Update: cm3
Date: Mon=2C 21 Feb 2011 00:35:38= > > +0000

> > > > >"> > > > > > > > > > >Hm. Actually=2C Win32 is not broken=2C but libm3/m3middle are clearly incor= > >rect here.

 =3B- Jay

>=3B Date: Mon=2C 21 Feb 2011 01:3= > >2:47 +0000
>=3B To: m3commit at elegosoft.com
>=3B From: jkrell at eleg= > >o.de
>=3B Subject: [M3commit] CVS Update: cm3
>=3B
>=3B CVS= > >ROOT: /usr/cvs
>=3B Changes by: jkrell at birch. 11/02/21 01:32:47
>= > >=3B
>=3B Modified files:
>=3B cm3/m3-sys/m3middle/src/: M3Proce= > >ss.i3
>=3B
>=3B Log message:
>=3B remove incorrect commen= > >ts=3B Win32 is not broken
>=3B
> >= > > > >--_d36c70d3-e71b-433d-abdc-06703a929f89_-- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 04:49:34 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 03:49:34 +0000 Subject: [M3devel] platform-independent kill process In-Reply-To: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> References: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> Message-ID: Yes, please. I'm not sure Terminate should return anything though. In Win32 you would use TerminateProcess(). If you must have the exit code, that would be GetExitCodeProcess. Though the process might still be running and not have an orderly exit code. Oh, and TerminateProcess does have to set the exit code. In m3middle they set it to 1. > We currently only have to consider Unix and Windows, or are there > other active flavours of OSs to care for? I don't think so. There was OS/2, BeOS, Amiga, MacOS < 10.... - Jay > Date: Sun, 20 Feb 2011 13:28:52 +0100 > From: wagner at elegosoft.com > To: m3devel at elegosoft.com > Subject: [M3devel] platform-independent kill process > > I've been thinking a bit about the hanging tests problem in our regression > test framework. I'd like to provide a simple utility to start a program > with time supervision. But as far as I can see, Modula-3 does not provide > a platform-independent way of terminating a process. Of course, on Unix > I can send signals; but what would I do on Windows? A quick scan of the > win32 interfaces in m3core hasn't brought up anything, but I might > have looked for the wrong texts. > > Could we easily extend the Process interface e.g. by > > PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException > > ? > Any opinions about that? Different interface suggestions? > Rather create ProcessExtras than extending a `standard' interface? > > We currently only have to consider Unix and Windows, or are there > other active flavours of OSs to care for? > > Olaf > -- > Olaf Wagner -- elego Software Solutions GmbH > Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 > http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcolebur at SCIRES.COM Mon Feb 21 05:13:54 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sun, 20 Feb 2011 23:13:54 -0500 Subject: [M3devel] platform-independent kill process In-Reply-To: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> References: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> Message-ID: Olaf: If the regression tests are written in Modula-3, why not use Network Objects to provide an interface that can be called to tell the program to stop? This would be a "graceful" shutdown mechanism, rather than just killing the process at the OS level. I've used this technique before with other types of programs. Regards, Randy -----Original Message----- From: Olaf Wagner [mailto:wagner at elegosoft.com] Sent: Sunday, February 20, 2011 7:29 AM To: m3devel at elegosoft.com Subject: [M3devel] platform-independent kill process I've been thinking a bit about the hanging tests problem in our regression test framework. I'd like to provide a simple utility to start a program with time supervision. But as far as I can see, Modula-3 does not provide a platform-independent way of terminating a process. Of course, on Unix I can send signals; but what would I do on Windows? A quick scan of the win32 interfaces in m3core hasn't brought up anything, but I might have looked for the wrong texts. Could we easily extend the Process interface e.g. by PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException ? Any opinions about that? Different interface suggestions? Rather create ProcessExtras than extending a `standard' interface? We currently only have to consider Unix and Windows, or are there other active flavours of OSs to care for? Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From mika at async.caltech.edu Mon Feb 21 05:50:31 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 20 Feb 2011 20:50:31 -0800 Subject: [M3devel] [M3commit] CVS Update: cm3 In-Reply-To: References: <20110221003248.11C9C2474005@birch.elegosoft.com>, , , , <20110221011550.0D2791A2078@async.async.caltech.edu> Message-ID: <20110221045031.1D5D81A2078@async.async.caltech.edu> Jay K writes: >--_0b0e48b4-5c54-4fbb-8adf-2eb48374df09_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Wait? No=2C that is probably just bogus posix thinking. >You can just close your handle to the process. >You don't have to wait for it to exit. Well on Unix I don't think you can do that, can you? You can't tell the system that you're not going to be calling "wait4" on a particular PID... or can you? If not, the portable thing to do is to do what I said... register a GC cleanup such that when a Process.T goes out of scope the runtime does a Process.Wait in the background. It'd work on both Unix and Windows and use existing code rather than introducing a new abstraction. One issue is that with my SIGCHLD modification you will get zombies if you don't call wait4 on Unix. Process.i3 does mention this possibility so I don't think I "broke" anything. (Coding to the specification cuts both ways...) Mika From wagner at elegosoft.com Mon Feb 21 07:49:08 2011 From: wagner at elegosoft.com (Olaf Wagner) Date: Mon, 21 Feb 2011 07:49:08 +0100 Subject: [M3devel] platform-independent kill process In-Reply-To: References: <20110220132852.2n2ikasp8gokk0ws@mail.elegosoft.com> Message-ID: <20110221074908.jndwjzx2bqcoo880@mail.elegosoft.com> Quoting "Coleburn, Randy" : > Olaf: > > If the regression tests are written in Modula-3, why not use Network > Objects to provide an interface that can be called to tell the > program to stop? Baucse that involves too many other sub-systems and components. I'd like to keep the requirements for tests limited. When we test such basic things like threading, we shouldn't have to assume that the network subsystem and RPC are working well, too. > This would be a "graceful" shutdown mechanism, rather than just > killing the process at the OS level. I'm not sure that it would work in the presence of threading problems, but I may be mistaken there. > I've used this technique before with other types of programs. Nothing speaking against that, if your application needs all the functionality anyway. However, I think M3 should support a basic mechanism on process level. Olaf > Regards, > Randy > > -----Original Message----- > From: Olaf Wagner [mailto:wagner at elegosoft.com] > Sent: Sunday, February 20, 2011 7:29 AM > To: m3devel at elegosoft.com > Subject: [M3devel] platform-independent kill process > > I've been thinking a bit about the hanging tests problem in our regression > test framework. I'd like to provide a simple utility to start a program > with time supervision. But as far as I can see, Modula-3 does not provide > a platform-independent way of terminating a process. Of course, on Unix > I can send signals; but what would I do on Windows? A quick scan of the > win32 interfaces in m3core hasn't brought up anything, but I might > have looked for the wrong texts. > > Could we easily extend the Process interface e.g. by > > PROCEDURE Terminate( p: T ): ExitCode RAISES TerminationException > > ? > Any opinions about that? Different interface suggestions? > Rather create ProcessExtras than extending a `standard' interface? > > We currently only have to consider Unix and Windows, or are there > other active flavours of OSs to care for? > > Olaf -- Olaf Wagner -- elego Software Solutions GmbH Gustav-Meyer-Allee 25 / Geb?ude 12, 13355 Berlin, Germany phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95 http://www.elegosoft.com | Gesch?ftsf?hrer: Olaf Wagner | Sitz: Berlin Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194 From jay.krell at cornell.edu Mon Feb 21 12:12:10 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 11:12:10 +0000 Subject: [M3devel] libm3/Process.ExitCode? Message-ID: TYPE ExitCode = [0 .. 16_7FFFFFFF]; (* An exit code (or status) of zero normally means successful termination, and a non-zero value normally indicates an error, but the exact conventions vary between systems and programs. *) Is there a good reason this isn't a full INTEGER? Or at least a full Cstdint.int32_t? In the Win32 case, we mask the actual exit code with this, which will lead to the non-zero 0x80000000 becoming zero. I'm inclined to change it like: before: RETURN Word.And(status, LAST(ExitCode)) after: (* for some reason, cut off the high bit, but * if that turns non-zero into zero, then still return non-zero *) maskedStatus := Word.And(status, LAST(ExitCode)); IF status <> 0 and maskedStatus = 0 THEN maskedStatus := 1; END; RETURN maskedStatus Or change ExitCode to plain lossless INTEGER. But maybe negative values were avoided for some reason? I understand that on Posix, I think, you just get an 8 bit unsigned value. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 13:00:14 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 12:00:14 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , <4D604FA3.3060708@lcwb.coop>, , <20110219232729.53A991A2078@async.async.caltech.edu>, , <4D615643.7060107@lcwb.coop>, , <20110220183746.195C11A2078@async.async.caltech.edu>, Message-ID: Ok, I'm disappointed to report that I did some quick checking on Darwin/x86, Linux/x86, Solaris/sparc32/x86 and none of them seem to be smart about this. Er, less they were being really smart. int F1(void) { volatile char a[4000]; return a[3999]; } int F2(void) { volatile char a[6000]; return a[5999]; } int F4(void) { volatile char a[60000]; return a[59999]; } I guess I should redo all the tests but accessing [0] also/instead. So, yes, we should definitely consider doing something. However...however, the frontend doesn't know which locals wiil be optimized away. The backend is setup to do this...but it is only used for NT systems, darn. jbook2:gcc jay$ grep CHECK_STACK_LIMIT */*/*/* gcc/config/i386/cygming.h:#define CHECK_STACK_LIMIT 4000 gcc/config/i386/cygwin.asm: than CHECK_STACK_LIMIT bytes in one go. Touching the stack at 4K gcc/config/i386/i386-interix.h:#define CHECK_STACK_LIMIT 0x1000 gcc/config/i386/i386.c:#ifndef CHECK_STACK_LIMIT gcc/config/i386/i386.c:#define CHECK_STACK_LIMIT (-1) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT)) gcc/config/i386/i386.c: else if (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT))) gcc/config/i386/i386.md:#ifndef CHECK_STACK_LIMIT gcc/config/i386/i386.md:#define CHECK_STACK_LIMIT 0 gcc/config/i386/i386.md: if (CHECK_STACK_LIMIT && CONST_INT_P (operands[1]) gcc/config/i386/i386.md: && INTVAL (operands[1]) < CHECK_STACK_LIMIT) jbook2:gcc jay$ grep CHECK_STACK_LIMIT */*/*/*/* jbook2:gcc jay$ grep TARGET_STACK_PROBE * jbook2:gcc jay$ grep TARGET_STACK_PROBE */* gcc/ChangeLog: * config/i386/i386.c (ix86_expand_prologue) [TARGET_STACK_PROBE]: gcc/ChangeLog-2005: (TARGET_TLS_DIRECT_SEG_REFS, TARGET_STACK_PROBE) jbook2:gcc jay$ grep TARGET_STACK_PROBE */*/* jbook2:gcc jay$ grep TARGET_STACK_PROBE */*/*/* gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT)) gcc/config/i386/i386.c: else if (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT))) gcc/config/i386/i386.md: "!TARGET_64BIT && TARGET_STACK_PROBE" gcc/config/i386/i386.md: "TARGET_64BIT && TARGET_STACK_PROBE" gcc/config/i386/i386.md: "TARGET_STACK_PROBE" - Jay From: jay.krell at cornell.edu To: mika at async.caltech.edu; rodney_bates at lcwb.coop Date: Sun, 20 Feb 2011 22:38:19 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] SEGV mapping to RuntimeError Probably for this reason: There is a requirement on NT that stack pages be touched in order. Functions with locals totally more than 4K call _chkstk (aka _alloca) to allocate their stack, instead of the usual register subtraction. It contains a loop that touches a byte every 4K. Otherwise, if you have lots of functions with small frames, the stack is touched by virtue of the call function pushing the return address. Modula-3 has long failed to uphold this contract, and still does. I've never seen it clearly documented, but you can see it is what the C compiler does. I had thought there were other reasons for this behavior. I thought you actually get an exception of the stack is touched out of order ("the first time"). But I think I did an experiment long ago with Modula-3 and there was no exception. I don't know about other platforms. I've also seen evidence in gcc and/or Target.i3 that compiler writers know well about such mechanisms -- there being a constant to set as to what size locals trigger special behavior. But m3back doesn't do anything here. Probably we could use _chkstk unconditonally as well -- need to see what it does for numbers smaller than 4K. But that'd be a deoptimization in the common case, and deoptimizing only as necessary should be easy enough. - Jay > To: rodney_bates at lcwb.coop > Date: Sun, 20 Feb 2011 10:37:46 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > On a 64-bit machine, at least, there ought to be enough virtual > memory that you could just have a gap between thread stacks big > enough to allow for a protection area larger than the largest possible > (implementation-defined) activation record, no? I know I've run into > trouble with very large activation records in the past (and not because > I was running out of stack space, either). > > Or at least a procedure with a very large activation record (or > a procedure calling it) could be required to call some sort of check > routine "EnoughStackSpaceRemaining()" before starting to scribble > on the activation record? > > Also the end of the activation record must be written to at least once, > or else the memory protection won't be triggered. > > In any case if this is done properly the same mechanism I proposed for > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > signals are delivered on a separate stack. If signals are delivered on > the same stack, the signal handler would get nastier, it would have to > make space through some manipulations (maybe temporarily unporotecting > the redzone page?) for its own purposes... but I don't see why it > couldn't be done. > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > activated inside the redzone page because of a difference in signal > handling..? I remember reading something about sigaltstack... > > I would of course love to be able to recover from stack overflow, too. > In some sense, since it's a generally unknown limit, it's even less of > a fatal error than a NIL dereference (hence makes even more sense to > catch it). > > Mika > > "Rodney M. Bates" writes: > >I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >Probably a fully protected guard page at the end of the stack. This technique > >always worries me a bit because a procedure with a really big activation record > >could jump right past it. Probably it would almost always access the first page > >of the big area before storing anything into later pages. > > > >On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >> Ah, yes, stack protection. > >> > >> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >> > >> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >> > >> time ../AMD64_FREEBSD/stubexample > >> M-Scheme Experimental > >> LITHP ITH LITHENING. > >>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >> f > >>> (f 0) > >> Illegal instruction > >> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >> > >> What absolutely must not happen, of course, is that the runtime hangs > >> while executing only safe code... > >> > >> Mika > >> > >> "Rodney M. Bates" writes: > >>> I know of one other place the compilers rely on hardware memory protection > >>> to detect a checked runtime error, and that is stack overflow. This won't > >>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>> This could probably be distinguished after the fact by some low-level, > >>> target-dependent code. I have found it by looking at assembly code at > >>> the point of failure--usually right after a stack pointer push. > >>> > >>> Detecting this via compiler-generated checks would probably be more > >>> extravagant than many other checks, as it is so frequent. I am not > >>> aware of any really good solution to this in any implementation of any > >>> language. > >>> > >>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>> you know! > >>>> > >>>> "Corrupting the heap" is something that can only happen as a result of > >>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>> modules not marked UNSAFE. > >>>> > >>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>> to signal a certain kind of *checked* runtime error, namely, the > >>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>> > >>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>> current CM3 are ones from NIL dereferences. > >>>> > >>>> Hence, as long as you stick with safe code, the only time the code I > >>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>> recoverable. > >>>> > >>>> :-) > >>>> > >>>> Mika > >>>> > >>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>> Almost all what remains is glue to some C library, which wouldn't be > >>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>> I don't have the time to rewrite every single library from scratch myself. > >>>> > >>>> > >>>> Jay K writes: > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious. > >>>>> Imagine the heap is corrupted. > >>>>> And then you run more code. > >>>>> And the code happens to call malloc. > >>>>> Or printf to log something. > >>>>> =20 > >>>>> I suppose there might be an application that maps memory > >>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>> =20 > >>>>> - Jay > >>>>> =20 > >>>>>> To: m3devel at elegosoft.com > >>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>> From: mika at async.caltech.edu > >>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>> =20 > >>>>>> =20 > >>>>>> Dear m3devel=2C > >>>>>> =20 > >>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>> the unhelpful interface and module Crash: > >>>>>> =20 > >>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>> =20 > >>>>>> MODULE Crash=3B > >>>>>> =20 > >>>>>> PROCEDURE Me() =3D > >>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>> ptr^ :=3D 0 > >>>>>> END Me=3B > >>>>>> =20 > >>>>>> BEGIN END Crash. > >>>>>> =20 > >>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>> e > >>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>> =20 > >>>>>> M-Scheme Experimental > >>>>>> LITHP ITH LITHENING. > >>>>>>> (require-modules "m3") > >>>>>> #t > >>>>>>> (Crash.Me) > >>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>> (+ 3 4)=20 > >>>>>> 7 > >>>>>>> =20 > >>>>>> =20 > >>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>> heck it.=20 > >>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>> =20 > >>>>>> Mika > >>>>>> =20 > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>> Imagine the heap =3Bis corrupted.
> >>>>> And then you run more code.
> >>>>> And the code happens to call malloc.
> >>>>> Or printf to log something.
> >>>>>  =3B
> >>>>> I suppose there might be an application that maps memory
> >>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>  =3B
> >>>>>  =3B- Jay
 =3B
> >>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>> > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>> > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Mon Feb 21 13:05:33 2011 From: jay.krell at cornell.edu (Jay K) Date: Mon, 21 Feb 2011 12:05:33 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, ,,, ,,<20110219203821.2016B1A2078@async.async.caltech.edu>, ,,<4D604FA3.3060708@lcwb.coop>, ,,<20110219232729.53A991A2078@async.async.caltech.edu>, ,,<4D615643.7060107@lcwb.coop>, , , <20110220183746.195C11A2078@async.async.caltech.edu>, , , Message-ID: Hm. Maybe we just need to pass -fstack-check to the backend. Needs experimentation... - Jay From: jay.krell at cornell.edu To: mika at async.caltech.edu; rodney_bates at lcwb.coop Date: Mon, 21 Feb 2011 12:00:14 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] SEGV mapping to RuntimeError Ok, I'm disappointed to report that I did some quick checking on Darwin/x86, Linux/x86, Solaris/sparc32/x86 and none of them seem to be smart about this. Er, less they were being really smart. int F1(void) { volatile char a[4000]; return a[3999]; } int F2(void) { volatile char a[6000]; return a[5999]; } int F4(void) { volatile char a[60000]; return a[59999]; } I guess I should redo all the tests but accessing [0] also/instead. So, yes, we should definitely consider doing something. However...however, the frontend doesn't know which locals wiil be optimized away. The backend is setup to do this...but it is only used for NT systems, darn. jbook2:gcc jay$ grep CHECK_STACK_LIMIT */*/*/* gcc/config/i386/cygming.h:#define CHECK_STACK_LIMIT 4000 gcc/config/i386/cygwin.asm: than CHECK_STACK_LIMIT bytes in one go. Touching the stack at 4K gcc/config/i386/i386-interix.h:#define CHECK_STACK_LIMIT 0x1000 gcc/config/i386/i386.c:#ifndef CHECK_STACK_LIMIT gcc/config/i386/i386.c:#define CHECK_STACK_LIMIT (-1) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT)) gcc/config/i386/i386.c: else if (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT))) gcc/config/i386/i386.md:#ifndef CHECK_STACK_LIMIT gcc/config/i386/i386.md:#define CHECK_STACK_LIMIT 0 gcc/config/i386/i386.md: if (CHECK_STACK_LIMIT && CONST_INT_P (operands[1]) gcc/config/i386/i386.md: && INTVAL (operands[1]) < CHECK_STACK_LIMIT) jbook2:gcc jay$ grep CHECK_STACK_LIMIT */*/*/*/* jbook2:gcc jay$ grep TARGET_STACK_PROBE * jbook2:gcc jay$ grep TARGET_STACK_PROBE */* gcc/ChangeLog: * config/i386/i386.c (ix86_expand_prologue) [TARGET_STACK_PROBE]: gcc/ChangeLog-2005: (TARGET_TLS_DIRECT_SEG_REFS, TARGET_STACK_PROBE) jbook2:gcc jay$ grep TARGET_STACK_PROBE */*/* jbook2:gcc jay$ grep TARGET_STACK_PROBE */*/*/* gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT)) gcc/config/i386/i386.c: else if (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT) gcc/config/i386/i386.c: && (! TARGET_STACK_PROBE || allocate < CHECK_STACK_LIMIT))) gcc/config/i386/i386.md: "!TARGET_64BIT && TARGET_STACK_PROBE" gcc/config/i386/i386.md: "TARGET_64BIT && TARGET_STACK_PROBE" gcc/config/i386/i386.md: "TARGET_STACK_PROBE" - Jay From: jay.krell at cornell.edu To: mika at async.caltech.edu; rodney_bates at lcwb.coop Date: Sun, 20 Feb 2011 22:38:19 +0000 CC: m3devel at elegosoft.com Subject: Re: [M3devel] SEGV mapping to RuntimeError Probably for this reason: There is a requirement on NT that stack pages be touched in order. Functions with locals totally more than 4K call _chkstk (aka _alloca) to allocate their stack, instead of the usual register subtraction. It contains a loop that touches a byte every 4K. Otherwise, if you have lots of functions with small frames, the stack is touched by virtue of the call function pushing the return address. Modula-3 has long failed to uphold this contract, and still does. I've never seen it clearly documented, but you can see it is what the C compiler does. I had thought there were other reasons for this behavior. I thought you actually get an exception of the stack is touched out of order ("the first time"). But I think I did an experiment long ago with Modula-3 and there was no exception. I don't know about other platforms. I've also seen evidence in gcc and/or Target.i3 that compiler writers know well about such mechanisms -- there being a constant to set as to what size locals trigger special behavior. But m3back doesn't do anything here. Probably we could use _chkstk unconditonally as well -- need to see what it does for numbers smaller than 4K. But that'd be a deoptimization in the common case, and deoptimizing only as necessary should be easy enough. - Jay > To: rodney_bates at lcwb.coop > Date: Sun, 20 Feb 2011 10:37:46 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > On a 64-bit machine, at least, there ought to be enough virtual > memory that you could just have a gap between thread stacks big > enough to allow for a protection area larger than the largest possible > (implementation-defined) activation record, no? I know I've run into > trouble with very large activation records in the past (and not because > I was running out of stack space, either). > > Or at least a procedure with a very large activation record (or > a procedure calling it) could be required to call some sort of check > routine "EnoughStackSpaceRemaining()" before starting to scribble > on the activation record? > > Also the end of the activation record must be written to at least once, > or else the memory protection won't be triggered. > > In any case if this is done properly the same mechanism I proposed for > SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > signals are delivered on a separate stack. If signals are delivered on > the same stack, the signal handler would get nastier, it would have to > make space through some manipulations (maybe temporarily unporotecting > the redzone page?) for its own purposes... but I don't see why it > couldn't be done. > > Not sure why I'm getting SIGILL... maybe I am getting my signal handler > activated inside the redzone page because of a difference in signal > handling..? I remember reading something about sigaltstack... > > I would of course love to be able to recover from stack overflow, too. > In some sense, since it's a generally unknown limit, it's even less of > a fatal error than a NIL dereference (hence makes even more sense to > catch it). > > Mika > > "Rodney M. Bates" writes: > >I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >Probably a fully protected guard page at the end of the stack. This technique > >always worries me a bit because a procedure with a really big activation record > >could jump right past it. Probably it would almost always access the first page > >of the big area before storing anything into later pages. > > > >On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >> Ah, yes, stack protection. > >> > >> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >> > >> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >> > >> time ../AMD64_FREEBSD/stubexample > >> M-Scheme Experimental > >> LITHP ITH LITHENING. > >>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >> f > >>> (f 0) > >> Illegal instruction > >> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >> > >> What absolutely must not happen, of course, is that the runtime hangs > >> while executing only safe code... > >> > >> Mika > >> > >> "Rodney M. Bates" writes: > >>> I know of one other place the compilers rely on hardware memory protection > >>> to detect a checked runtime error, and that is stack overflow. This won't > >>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>> This could probably be distinguished after the fact by some low-level, > >>> target-dependent code. I have found it by looking at assembly code at > >>> the point of failure--usually right after a stack pointer push. > >>> > >>> Detecting this via compiler-generated checks would probably be more > >>> extravagant than many other checks, as it is so frequent. I am not > >>> aware of any really good solution to this in any implementation of any > >>> language. > >>> > >>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>> you know! > >>>> > >>>> "Corrupting the heap" is something that can only happen as a result of > >>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>> modules not marked UNSAFE. > >>>> > >>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>> to signal a certain kind of *checked* runtime error, namely, the > >>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>> > >>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>> current CM3 are ones from NIL dereferences. > >>>> > >>>> Hence, as long as you stick with safe code, the only time the code I > >>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>> recoverable. > >>>> > >>>> :-) > >>>> > >>>> Mika > >>>> > >>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>> Almost all what remains is glue to some C library, which wouldn't be > >>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>> I don't have the time to rewrite every single library from scratch myself. > >>>> > >>>> > >>>> Jay K writes: > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious. > >>>>> Imagine the heap is corrupted. > >>>>> And then you run more code. > >>>>> And the code happens to call malloc. > >>>>> Or printf to log something. > >>>>> =20 > >>>>> I suppose there might be an application that maps memory > >>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>> =20 > >>>>> - Jay > >>>>> =20 > >>>>>> To: m3devel at elegosoft.com > >>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>> From: mika at async.caltech.edu > >>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>> =20 > >>>>>> =20 > >>>>>> Dear m3devel=2C > >>>>>> =20 > >>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>> the unhelpful interface and module Crash: > >>>>>> =20 > >>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>> =20 > >>>>>> MODULE Crash=3B > >>>>>> =20 > >>>>>> PROCEDURE Me() =3D > >>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>> ptr^ :=3D 0 > >>>>>> END Me=3B > >>>>>> =20 > >>>>>> BEGIN END Crash. > >>>>>> =20 > >>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>> e > >>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>> =20 > >>>>>> M-Scheme Experimental > >>>>>> LITHP ITH LITHENING. > >>>>>>> (require-modules "m3") > >>>>>> #t > >>>>>>> (Crash.Me) > >>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>> (+ 3 4)=20 > >>>>>> 7 > >>>>>>> =20 > >>>>>> =20 > >>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>> heck it.=20 > >>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>> =20 > >>>>>> Mika > >>>>>> =20 > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>> Content-Transfer-Encoding: quoted-printable > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>> Imagine the heap =3Bis corrupted.
> >>>>> And then you run more code.
> >>>>> And the code happens to call malloc.
> >>>>> Or printf to log something.
> >>>>>  =3B
> >>>>> I suppose there might be an application that maps memory
> >>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>  =3B
> >>>>>  =3B- Jay
 =3B
> >>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>> > >>>>> = > >>>>> > >>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>> > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Tue Feb 22 01:01:12 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Mon, 21 Feb 2011 16:01:12 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> Message-ID: <20110222000112.5A7131A2078@async.async.caltech.edu> There is in fact a small problem with my change to the way SIGSEGV is handled. It now appears that bad memory references are all happening in ThreadPosix.m3 (i.e., we've lost some of the debugging help of the error message). I'm not quite sure how to fix this although I'm sure all the bits are around. My guess is that instead of performing RAISE RuntimeError.E in the thread switcher, the code ought to call RTHooks.Raise with an argument derived from the faulting address as reported to the signal handler. This is probably quite easy, but does anyone know the right routines to call? One would need to track down the routine that faulted as well as the RT0 record corresponding to the exception RuntimeError.E. Mika Jay K writes: >Letting any code run after a SIGSEGV is dubious. >Imagine the heap is corrupted. >And then you run more code. >And the code happens to call malloc. >Or printf to log something. >=20 >I suppose there might be an application that maps memory >gradually=2C as pieces of a buffer are hit. Might. From dabenavidesd at yahoo.es Tue Feb 22 02:07:45 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Tue, 22 Feb 2011 01:07:45 +0000 (GMT) Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222000112.5A7131A2078@async.async.caltech.edu> Message-ID: <533385.35225.qm@web29704.mail.ird.yahoo.com> Hi all: although this is not of my full understanding, CM3 JVM was supposed to handle in such a way the exceptions of jvm ?-op fault script programs, given that: there was a runtime above the CM3 runtime and besides having its own spaces for stack, thread stacks and runtime machinery, loader, etc. Please just to advert this is and I guess was the case for CM3 JVM, if so, let the system handle it in that way, why do you need to fully manage to handle that as in Modula-3 runtime per se, my believe is that you could do that for the language environment routines separately, that's why pragmas are really useful. I don't believe it allowed to signal like in Modula-3 terms threads but you could for instance lock a thread until you have time in the scheduler from Modula-3, still this is more flexible nowadays I believe with multiprocessors, etc. Similarly SPIN used or performed protection mechanism, like domain level, and at language level the heap freezes when handling untrusted code (even extra checked array bounds violation faults in non-Modula-3 code), so you have same runtime I believe for doing that too in Modula-3, again, is just another time of problem arises, perhaps this can be tried for thread stacks, etc and in kernel threads well as something alike, I really think could be done. Thanks in advance --- El lun, 21/2/11, Mika Nystrom escribi?: > De: Mika Nystrom > Asunto: Re: [M3devel] SEGV mapping to RuntimeError > Para: "Jay K" , m3devel at elegosoft.com > Fecha: lunes, 21 de febrero, 2011 19:01 > > There is in fact a small problem with my change to the way > SIGSEGV is > handled. It now appears that bad memory references > are all happening > in ThreadPosix.m3 (i.e., we've lost some of the debugging > help of the > error message). > > I'm not quite sure how to fix this although I'm sure all > the bits are > around. My guess is that instead of performing RAISE > RuntimeError.E > in the thread switcher, the code ought to call > RTHooks.Raise with an > argument derived from the faulting address as reported to > the signal > handler. This is probably quite easy, but does anyone > know the > right routines to call? One would need to track down > the routine that > faulted as well as the RT0 record corresponding to the > exception > RuntimeError.E. > > Mika > > Jay K writes: > >Letting any code run after a SIGSEGV is dubious. > >Imagine the heap is corrupted. > >And then you run more code. > >And the code happens to call malloc. > >Or printf to log something. > >=20 > >I suppose there might be an application that maps > memory > >gradually=2C as pieces of a buffer are hit. Might. > From mika at async.caltech.edu Tue Feb 22 03:12:00 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Mon, 21 Feb 2011 18:12:00 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <533385.35225.qm@web29704.mail.ird.yahoo.com> References: <533385.35225.qm@web29704.mail.ird.yahoo.com> Message-ID: <20110222021200.9F0EE1A2078@async.async.caltech.edu> There are several issues here and yes this the RuntimeError.E exception is taken either from SPIN or the CM3 JVM, I don't know which. I only understand the user threads code so I have made that raise the RuntimeError.E instead of dumping core on a segfault. I think this is what the CM JVM was supposed to do as well---but honestly I can't tell if that work was ever completed. They do support mapping various other runtime errors (subrange mismatch, e.g.) into exceptions, and that works well. However the segmentation fault mapping was not functional: there was no other way of handling it than dumping core on the whole M3 environment and runtime before my changes last week. My last email was about a bug in my changes---not in the actual functionality but the fact that if you *don't* catch the RuntimeError.E exception that results from a bad memory reference, the runtime quits with a slightly confusing error message, apparently implicating the threading system when the bug is actually in normal, non-UNSAFE code. Mika "Daniel Alejandro Benavides D." writes: >Hi all: >although this is not of my full understanding, CM3 JVM was supposed to hand= >le in such a way the exceptions of jvm =B5-op fault script programs, given = >that: there was a runtime above the CM3 runtime and besides having its own = >spaces for stack, thread stacks and runtime machinery, loader, etc. >Please just to advert this is and I guess was the case for CM3 JVM, if so, = >let the system handle it in that way, why do you need to fully manage to ha= >ndle that as in Modula-3 runtime per se, my believe is that you could do th= >at for the language environment routines separately, that's why pragmas are= > really useful. I don't believe it allowed to signal like in Modula-3 terms= > threads but you could for instance lock a thread until you have time in th= >e scheduler from Modula-3, still this is more flexible nowadays I believe w= >ith multiprocessors, etc. >Similarly SPIN used or performed protection mechanism, like domain level, a= >nd at language level the heap freezes when handling untrusted code (even ex= >tra checked array bounds violation faults in non-Modula-3 code), so you hav= >e same runtime I believe for doing that too in Modula-3, again, is just ano= >ther time of problem arises, perhaps this can be tried for thread stacks, e= >tc and in kernel threads well as something alike, I really think could be d= >one. >Thanks in advance > >--- El lun, 21/2/11, Mika Nystrom escribi=F3: > >> De: Mika Nystrom >> Asunto: Re: [M3devel] SEGV mapping to RuntimeError >> Para: "Jay K" , m3devel at elegosoft.com >> Fecha: lunes, 21 de febrero, 2011 19:01 >>=20 >> There is in fact a small problem with my change to the way >> SIGSEGV is >> handled. It now appears that bad memory references >> are all happening >> in ThreadPosix.m3 (i.e., we've lost some of the debugging >> help of the >> error message). >>=20 >> I'm not quite sure how to fix this although I'm sure all >> the bits are >> around. My guess is that instead of performing RAISE >> RuntimeError.E >> in the thread switcher, the code ought to call >> RTHooks.Raise with an >> argument derived from the faulting address as reported to >> the signal >> handler. This is probably quite easy, but does anyone >> know the=20 >> right routines to call? One would need to track down >> the routine that >> faulted as well as the RT0 record corresponding to the >> exception >> RuntimeError.E. >>=20 >> Mika >>=20 >> Jay K writes: >> >Letting any code run after a SIGSEGV is dubious. >> >Imagine the heap is corrupted. >> >And then you run more code. >> >And the code happens to call malloc. >> >Or printf to log something. >> >=3D20 >> >I suppose there might be an application that maps >> memory >> >gradually=3D2C as pieces of a buffer are hit. Might. >> =0A=0A=0A From dabenavidesd at yahoo.es Tue Feb 22 05:31:58 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Tue, 22 Feb 2011 04:31:58 +0000 (GMT) Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222021200.9F0EE1A2078@async.async.caltech.edu> Message-ID: <806751.73807.qm@web29712.mail.ird.yahoo.com> Hi all: yes I see, things are harder than I thought here but still seems the problem is not directly in the runtime environment system, perhaps this is more just another time of System native restrictions: according to this guys: http://grsecurity.net/confighelp.php/ (see for Automatically emulate sigreturn trampolines) that's why perhaps the SPIN could do that without requiring the process to terminate which is otherwise is very odd I believe for such restrictions, still need to check more details: Interesting this emails go into the details of more advanced programs to analyze ine ach platform the runtime behavior many years ago, look for Hp-UX implementation "bug". Again grsecurity OS doers talk again even of (see CVSup): http://pax.grsecurity.net/docs/randexec.txt I hope it helps a bit, thanks in advance --- El lun, 21/2/11, Mika Nystrom escribi?: > De: Mika Nystrom > Asunto: Re: [M3devel] SEGV mapping to RuntimeError > Para: "Daniel Alejandro Benavides D." > CC: m3devel at elegosoft.com > Fecha: lunes, 21 de febrero, 2011 21:12 > > There are several issues here and yes this the > RuntimeError.E exception > is taken either from SPIN or the CM3 JVM, I don't know > which. > > I only understand the user threads code so I have made that > raise the > RuntimeError.E instead of dumping core on a segfault. > I think this is what > the CM JVM was supposed to do as well---but honestly I > can't tell if that > work was ever completed. They do support mapping > various other runtime > errors (subrange mismatch, e.g.) into exceptions, and that > works well. > However the segmentation fault mapping was not functional: > there was no > other way of handling it than dumping core on the whole M3 > environment > and runtime before my changes last week. > > My last email was about a bug in my changes---not in the > actual functionality > but the fact that if you *don't* catch the RuntimeError.E > exception that > results from a bad memory reference, the runtime quits with > a slightly > confusing error message, apparently implicating the > threading system when > the bug is actually in normal, non-UNSAFE code. > > Mika > > "Daniel Alejandro Benavides D." writes: > >Hi all: > >although this is not of my full understanding, CM3 JVM > was supposed to hand= > >le in such a way the exceptions of jvm =B5-op fault > script programs, given = > >that: there was a runtime above the CM3 runtime and > besides having its own = > >spaces for stack, thread stacks and runtime machinery, > loader, etc. > >Please just to advert this is and I guess was the case > for CM3 JVM, if so, = > >let the system handle it in that way, why do you need > to fully manage to ha= > >ndle that as in Modula-3 runtime per se, my believe is > that you could do th= > >at for the language environment routines separately, > that's why pragmas are= > > really useful. I don't believe it allowed to signal > like in Modula-3 terms= > > threads but you could for instance lock a thread until > you have time in th= > >e scheduler from Modula-3, still this is more flexible > nowadays I believe w= > >ith multiprocessors, etc. > >Similarly SPIN used or performed protection mechanism, > like domain level, a= > >nd at language level the heap freezes when handling > untrusted code (even ex= > >tra checked array bounds violation faults in > non-Modula-3 code), so you hav= > >e same runtime I believe for doing that too in > Modula-3, again, is just ano= > >ther time of problem arises, perhaps this can be tried > for thread stacks, e= > >tc and in kernel threads well as something alike, I > really think could be d= > >one. > >Thanks in advance > > > >--- El lun, 21/2/11, Mika Nystrom > escribi=F3: > > > >> De: Mika Nystrom > >> Asunto: Re: [M3devel] SEGV mapping to > RuntimeError > >> Para: "Jay K" , > m3devel at elegosoft.com > >> Fecha: lunes, 21 de febrero, 2011 19:01 > >>=20 > >> There is in fact a small problem with my change to > the way > >> SIGSEGV is > >> handled. It now appears that bad memory > references > >> are all happening > >> in ThreadPosix.m3 (i.e., we've lost some of the > debugging > >> help of the > >> error message). > >>=20 > >> I'm not quite sure how to fix this although I'm > sure all > >> the bits are > >> around. My guess is that instead of > performing RAISE > >> RuntimeError.E > >> in the thread switcher, the code ought to call > >> RTHooks.Raise with an > >> argument derived from the faulting address as > reported to > >> the signal > >> handler. This is probably quite easy, but > does anyone > >> know the=20 > >> right routines to call? One would need to > track down > >> the routine that > >> faulted as well as the RT0 record corresponding to > the > >> exception > >> RuntimeError.E. > >>=20 > >> Mika > >>=20 > >> Jay K writes: > >> >Letting any code run after a SIGSEGV is > dubious. > >> >Imagine the heap is corrupted. > >> >And then you run more code. > >> >And the code happens to call malloc. > >> >Or printf to log something. > >> >=3D20 > >> >I suppose there might be an application that > maps > >> memory > >> >gradually=3D2C as pieces of a buffer are hit. > Might. > >> =0A=0A=0A > From dabenavidesd at yahoo.es Tue Feb 22 05:40:34 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Tue, 22 Feb 2011 04:40:34 +0000 (GMT) Subject: [M3devel] SEGV mapping to RuntimeError Message-ID: <748970.47663.qm@web29719.mail.ird.yahoo.com> Hi all: sorry missed the second link in the comments about HP--UX and Nil dereferences: http://www.usenix.org/publications/library/proceedings/sd96/full_papers/small.txt Still they seemed to be some option in the compiler to check where platform needed to catch the RuntimeError either from compiler inserted checks for NIL and from trapped OS call or just ignore it in some circunstances like HP-UX case, see for that it. Thanks in advance --- El lun, 21/2/11, Daniel Alejandro Benavides D. escribi?: > De: Daniel Alejandro Benavides D. > Asunto: Re: [M3devel] SEGV mapping to RuntimeError > Para: "Mika Nystrom" > CC: m3devel at elegosoft.com > Fecha: lunes, 21 de febrero, 2011 23:31 > Hi all: > yes I see, things are harder than I thought here but still > seems the problem is not directly in the runtime environment > system, perhaps this is more just another time of System > native restrictions: > according to this guys: > http://grsecurity.net/confighelp.php/ > (see for Automatically emulate sigreturn trampolines) > that's why perhaps the SPIN could do that without requiring > the process to terminate which is otherwise is very odd I > believe for such restrictions, still need to check more > details: > Interesting this emails go into the details of more > advanced programs to analyze ine ach platform the runtime > behavior many years ago, look for Hp-UX implementation > "bug". > Again grsecurity OS doers talk again even of (see CVSup): > http://pax.grsecurity.net/docs/randexec.txt > I hope it helps a bit, thanks in advance > > --- El lun, 21/2/11, Mika Nystrom > escribi?: > > > De: Mika Nystrom > > Asunto: Re: [M3devel] SEGV mapping to RuntimeError > > Para: "Daniel Alejandro Benavides D." > > CC: m3devel at elegosoft.com > > Fecha: lunes, 21 de febrero, 2011 21:12 > > > > There are several issues here and yes this the > > RuntimeError.E exception > > is taken either from SPIN or the CM3 JVM, I don't > know > > which. > > > > I only understand the user threads code so I have made > that > > raise the > > RuntimeError.E instead of dumping core on a segfault. > > > I think this is what > > the CM JVM was supposed to do as well---but honestly > I > > can't tell if that > > work was ever completed. They do support > mapping > > various other runtime > > errors (subrange mismatch, e.g.) into exceptions, and > that > > works well. > > However the segmentation fault mapping was not > functional: > > there was no > > other way of handling it than dumping core on the > whole M3 > > environment > > and runtime before my changes last week. > > > > My last email was about a bug in my changes---not in > the > > actual functionality > > but the fact that if you *don't* catch the > RuntimeError.E > > exception that > > results from a bad memory reference, the runtime quits > with > > a slightly > > confusing error message, apparently implicating the > > threading system when > > the bug is actually in normal, non-UNSAFE code. > > > > Mika > > > > "Daniel Alejandro Benavides D." writes: > > >Hi all: > > >although this is not of my full understanding, CM3 > JVM > > was supposed to hand= > > >le in such a way the exceptions of jvm =B5-op > fault > > script programs, given = > > >that: there was a runtime above the CM3 runtime > and > > besides having its own = > > >spaces for stack, thread stacks and runtime > machinery, > > loader, etc. > > >Please just to advert this is and I guess was the > case > > for CM3 JVM, if so, = > > >let the system handle it in that way, why do you > need > > to fully manage to ha= > > >ndle that as in Modula-3 runtime per se, my > believe is > > that you could do th= > > >at for the language environment routines > separately, > > that's why pragmas are= > > > really useful. I don't believe it allowed to > signal > > like in Modula-3 terms= > > > threads but you could for instance lock a thread > until > > you have time in th= > > >e scheduler from Modula-3, still this is more > flexible > > nowadays I believe w= > > >ith multiprocessors, etc. > > >Similarly SPIN used or performed protection > mechanism, > > like domain level, a= > > >nd at language level the heap freezes when > handling > > untrusted code (even ex= > > >tra checked array bounds violation faults in > > non-Modula-3 code), so you hav= > > >e same runtime I believe for doing that too in > > Modula-3, again, is just ano= > > >ther time of problem arises, perhaps this can be > tried > > for thread stacks, e= > > >tc and in kernel threads well as something alike, > I > > really think could be d= > > >one. > > >Thanks in advance > > > > > >--- El lun, 21/2/11, Mika Nystrom > > escribi=F3: > > > > > >> De: Mika Nystrom > > >> Asunto: Re: [M3devel] SEGV mapping to > > RuntimeError > > >> Para: "Jay K" , > > m3devel at elegosoft.com > > >> Fecha: lunes, 21 de febrero, 2011 19:01 > > >>=20 > > >> There is in fact a small problem with my > change to > > the way > > >> SIGSEGV is > > >> handled. It now appears that bad > memory > > references > > >> are all happening > > >> in ThreadPosix.m3 (i.e., we've lost some of > the > > debugging > > >> help of the > > >> error message). > > >>=20 > > >> I'm not quite sure how to fix this although > I'm > > sure all > > >> the bits are > > >> around. My guess is that instead of > > performing RAISE > > >> RuntimeError.E > > >> in the thread switcher, the code ought to > call > > >> RTHooks.Raise with an > > >> argument derived from the faulting address > as > > reported to > > >> the signal > > >> handler. This is probably quite easy, > but > > does anyone > > >> know the=20 > > >> right routines to call? One would need > to > > track down > > >> the routine that > > >> faulted as well as the RT0 record > corresponding to > > the > > >> exception > > >> RuntimeError.E. > > >>=20 > > >> Mika > > >>=20 > > >> Jay K writes: > > >> >Letting any code run after a SIGSEGV is > > dubious. > > >> >Imagine the heap is corrupted. > > >> >And then you run more code. > > >> >And the code happens to call malloc. > > >> >Or printf to log something. > > >> >=3D20 > > >> >I suppose there might be an application > that > > maps > > >> memory > > >> >gradually=3D2C as pieces of a buffer are > hit. > > Might. > > >> =0A=0A=0A > > > > > > From mika at async.caltech.edu Tue Feb 22 18:44:05 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Tue, 22 Feb 2011 09:44:05 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <4D61CC27.7070704@lcwb.coop> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu> <20110219203821.2016B1A2078@async.async.caltech.edu> <4D604FA3.3060708@lcwb.coop> <20110219232729.53A991A2078@async.async.caltech.edu> <4D615643.7060107@lcwb.coop> <20110220183746.195C11A2078@async.async.caltech.edu> <4D61CC27.7070704@lcwb.coop> Message-ID: <20110222174405.E15481A2078@async.async.caltech.edu> Ok so I was thinking about this. Why on earth is stack overflow even a problem? Consider the following procedure call (in my code, stack grows upwards): (* sp at x, pc at y *) y: P(args) z: next_statement decompose as follows: (* sp at x, pc at y *) y: Push(args etc. and ret. address z) Jump(P) z: next_statement Now, we say: y: ok := check_stack(size of frame) IF NOT ok THEN abort() END; Push(args etc. and ret. address z) Jump(P) z: next_statement (note check_stack and the following IF can be implemented by hardware, need not actually be an instruction) Let me change the code a tad: y: ok := check_stack(size of frame) y':IF NOT ok THEN WITH new_stack_bottom = malloc(stack_size) huge_amount = new_stack_bottom - sp DO create_redzone_at(new_stack_bottom+stack_size-redzone_size) EVAL alloca(huge_amount) END END; Push(args etc. and ret. address z) Jump(P) z: IF NOT ok THEN destroy_redzone(...); free(new_stack_bottom) END Note 1. cleanup of redzone could be postponed to return of caller....when alloca in any case has to be cleaned up. Note 2. the test IF NOT ok at z is more expensive to implement than the one at y because you can't really use hardware for it. A hardware callback can be arranged though: VAR ptr := sp; y: ok := check_stack(size of frame) y':IF NOT ok THEN ptr := 0; (* illegal address *) fault_address := z; WITH new_stack_bottom = malloc(stack_size) huge_amount = new_stack_bottom - sp DO create_redzone_at(new_stack_bottom+stack_size-redzone_size) EVAL alloca(huge_amount) END END; Push(args etc. and ret. address z) Jump(P) z: EVAL ptr^ (* [ NOT ok -> hardware callback to SEGV: ] *) SEGV(signalpc): IF NOT ok AND signalpc = fault_address THEN destroy_redzone(...); free(new_stack_bottom) END Mika "Rodney M. Bates" writes: > > >On 02/20/2011 12:37 PM, Mika Nystrom wrote: >> On a 64-bit machine, at least, there ought to be enough virtual >> memory that you could just have a gap between thread stacks big >> enough to allow for a protection area larger than the largest possible >> (implementation-defined) activation record, no? I know I've run into >> trouble with very large activation records in the past (and not because >> I was running out of stack space, either). >> >> Or at least a procedure with a very large activation record (or >> a procedure calling it) could be required to call some sort of check >> routine "EnoughStackSpaceRemaining()" before starting to scribble >> on the activation record? > >Hmm, I like this idea. It would introduce normal-case runtime overhead >only for such procedures, and these are likely rare. Also, assuming the procedure >actually uses very much of its large AR, it should also have enough computation >time to wash out the stack check overhead. > >> >> Also the end of the activation record must be written to at least once, >> or else the memory protection won't be triggered. >> > >I was thinking (as an alternative mechanism) of having the compiler intentionally >add enough artificial write(s) as necessary to ensure storing within the >red zone, and not just beyond it. This seems trickier to get right and >harder to distinguish after the fact from a NIL dereference. > >> In any case if this is done properly the same mechanism I proposed for >> SIGSEGV ought to be able to catch stack overflow, no? Well, as long as >> signals are delivered on a separate stack. If signals are delivered on >> the same stack, the signal handler would get nastier, it would have to >> make space through some manipulations (maybe temporarily unporotecting >> the redzone page?) for its own purposes... but I don't see why it >> couldn't be done. >> >> Not sure why I'm getting SIGILL... maybe I am getting my signal handler >> activated inside the redzone page because of a difference in signal >> handling..? I remember reading something about sigaltstack... >> >> I would of course love to be able to recover from stack overflow, too. >> In some sense, since it's a generally unknown limit, it's even less of >> a fatal error than a NIL dereference (hence makes even more sense to >> catch it). > >I think this would be a nice mechanism to have available. It would have to >be used with some care. In any case, it would be really nice and more >frequently so, to at least have runtime error messages that distinguished >stack overflow from NIL deref. > >> >> Mika >> >> "Rodney M. Bates" writes: >>> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. >>> Probably a fully protected guard page at the end of the stack. This technique >>> always worries me a bit because a procedure with a really big activation record >>> could jump right past it. Probably it would almost always access the first page >>> of the big area before storing anything into later pages. >>> >>> On 02/19/2011 05:27 PM, Mika Nystrom wrote: >>>> Ah, yes, stack protection. >>>> >>>> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. >>>> >>>> Hmm, I get SIGILL on AMD64_FREEBSD as well: >>>> >>>> time ../AMD64_FREEBSD/stubexample >>>> M-Scheme Experimental >>>> LITHP ITH LITHENING. >>>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) >>>> f >>>>> (f 0) >>>> Illegal instruction >>>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w >>>> >>>> What absolutely must not happen, of course, is that the runtime hangs >>>> while executing only safe code... >>>> >>>> Mika >>>> >>>> "Rodney M. Bates" writes: >>>>> I know of one other place the compilers rely on hardware memory protection >>>>> to detect a checked runtime error, and that is stack overflow. This won't >>>>> corrupt anything, but is hard to distinguish from dereferencing NIL. >>>>> This could probably be distinguished after the fact by some low-level, >>>>> target-dependent code. I have found it by looking at assembly code at >>>>> the point of failure--usually right after a stack pointer push. >>>>> >>>>> Detecting this via compiler-generated checks would probably be more >>>>> extravagant than many other checks, as it is so frequent. I am not >>>>> aware of any really good solution to this in any implementation of any >>>>> language. >>>>> >>>>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: >>>>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, >>>>>> you know! >>>>>> >>>>>> "Corrupting the heap" is something that can only happen as a result of >>>>>> an unchecked runtime error. Unchecked runtime errors cannot happen in >>>>>> modules not marked UNSAFE. >>>>>> >>>>>> SEGV is, however, used by the CM3 implementation (and its predecessors) >>>>>> to signal a certain kind of *checked* runtime error, namely, the >>>>>> dereferencing of a NIL reference. Correct me if I am wrong, but an >>>>>> attempt to dereference NIL is not going to leave the heap corrupted? >>>>>> >>>>>> And if you stick to safe code, the only SEGVs I think you get in the >>>>>> current CM3 are ones from NIL dereferences. >>>>>> >>>>>> Hence, as long as you stick with safe code, the only time the code I >>>>>> checked in earlier gets triggered is for NIL dereferences, which should >>>>>> never corrupt the heap. So SEGV is not sometimes, but in fact always >>>>>> recoverable. >>>>>> >>>>>> :-) >>>>>> >>>>>> Mika >>>>>> >>>>>> P.S. the bit above "if you stick to safe code": if you actually program in >>>>>> Modula-3 you almost never use UNSAFE. I went through my repository and >>>>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, >>>>>> many of the UNSAFE modules are glue code to Fortran routines, which >>>>>> could relatively easily be verified to be safe in the Modula-3 sense. >>>>>> Almost all what remains is glue to some C library, which wouldn't be >>>>>> necessary if the rest of the world would wake up out of the dark ages, but >>>>>> I don't have the time to rewrite every single library from scratch myself. >>>>>> >>>>>> >>>>>> Jay K writes: >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>>>> Content-Type: text/plain; charset="iso-8859-1" >>>>>>> Content-Transfer-Encoding: quoted-printable >>>>>>> >>>>>>> >>>>>>> Letting any code run after a SIGSEGV is dubious. >>>>>>> Imagine the heap is corrupted. >>>>>>> And then you run more code. >>>>>>> And the code happens to call malloc. >>>>>>> Or printf to log something. >>>>>>> =20 >>>>>>> I suppose there might be an application that maps memory >>>>>>> gradually=2C as pieces of a buffer are hit. Might. >>>>>>> =20 >>>>>>> - Jay >>>>>>> =20 >>>>>>>> To: m3devel at elegosoft.com >>>>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 >>>>>>>> From: mika at async.caltech.edu >>>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError >>>>>>>> =20 >>>>>>>> =20 >>>>>>>> Dear m3devel=2C >>>>>>>> =20 >>>>>>>> For a while it has annoyed me that segmentation violations cause an >>>>>>>> unconditional program abort. I've changed that now so that (under user >>>>>>>> threads at least) we instead get a RuntimeError. Here's an example of >>>>>>>> the mechanism at work in an interactive Scheme environment. Consider >>>>>>>> the unhelpful interface and module Crash: >>>>>>>> =20 >>>>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. >>>>>>>> =20 >>>>>>>> MODULE Crash=3B >>>>>>>> =20 >>>>>>>> PROCEDURE Me() =3D >>>>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN >>>>>>>> ptr^ :=3D 0 >>>>>>>> END Me=3B >>>>>>>> =20 >>>>>>>> BEGIN END Crash. >>>>>>>> =20 >>>>>>>> Here's an example of what happens if you now call this from an interactiv= >>>>>>> e >>>>>>>> interpreter that catches the exception RuntimeError.E: >>>>>>>> =20 >>>>>>>> M-Scheme Experimental >>>>>>>> LITHP ITH LITHENING. >>>>>>>>> (require-modules "m3") >>>>>>>> #t >>>>>>>>> (Crash.Me) >>>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. >>>>>>>>> (+ 3 4)=20 >>>>>>>> 7 >>>>>>>>> =20 >>>>>>>> =20 >>>>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= >>>>>>> heck it.=20 >>>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... >>>>>>>> =20 >>>>>>>> Mika >>>>>>>> =20 >>>>>>> = >>>>>>> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >>>>>>> Content-Type: text/html; charset="iso-8859-1" >>>>>>> Content-Transfer-Encoding: quoted-printable >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Letting any code run after a SIGSEGV is dubious.
>>>>>>> Imagine the heap =3Bis corrupted.
>>>>>>> And then you run more code.
>>>>>>> And the code happens to call malloc.
>>>>>>> Or printf to log something.
>>>>>>>  =3B
>>>>>>> I suppose there might be an application that maps memory
>>>>>>> gradually=2C as pieces of a buffer are hit. Might.
>>>>>>>  =3B
>>>>>>>  =3B- Jay
 =3B
>>>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= >>>>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= >>>>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= >>>>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= >>>>>>> olations cause an
>=3B unconditional program abort. I've changed that = >>>>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= >>>>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= >>>>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= >>>>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= >>>>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= >>>>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= >>>>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= >>>>>>> >=3B Here's an example of what happens if you now call this from an inter= >>>>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= >>>>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= >>>>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= >>>>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= >>>>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= >>>>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= >>>>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= >>>>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= >>>>>>> >>>>>>> = >>>>>>> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >>>>>> >>>> >> From rodney_bates at lcwb.coop Tue Feb 22 18:59:08 2011 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Tue, 22 Feb 2011 11:59:08 -0600 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222021200.9F0EE1A2078@async.async.caltech.edu> References: <533385.35225.qm@web29704.mail.ird.yahoo.com> <20110222021200.9F0EE1A2078@async.async.caltech.edu> Message-ID: <4D63F96C.7020500@lcwb.coop> I don't know how relevant this is, but FWIW, I played with some patches in an interactive program I have that catches its own assertion failures and recovers from some of them. I was already using RTProcess.RegisterExitor with my own procedure. I made the exitor just call RTException.Raise. Then I deliberately did a NIL deref inside a try block catching RTException.E and ignoring it. It did prevent termination of my program. But there were two other problems: 1) The SIGSEGV messages came out on the command line before the exitor was invoked. 2) After the exception was caught, the program was catatonic, without using CPU. This is with pthreads. I haven't figured out why the wait. m3gdb seems to be useless in looking at other threads than the one it happens to get into, when using pthreads. On 02/21/2011 08:12 PM, Mika Nystrom wrote: > There are several issues here and yes this the RuntimeError.E exception > is taken either from SPIN or the CM3 JVM, I don't know which. > > I only understand the user threads code so I have made that raise the > RuntimeError.E instead of dumping core on a segfault. I think this is what > the CM JVM was supposed to do as well---but honestly I can't tell if that > work was ever completed. They do support mapping various other runtime > errors (subrange mismatch, e.g.) into exceptions, and that works well. > However the segmentation fault mapping was not functional: there was no > other way of handling it than dumping core on the whole M3 environment > and runtime before my changes last week. > > My last email was about a bug in my changes---not in the actual functionality > but the fact that if you *don't* catch the RuntimeError.E exception that > results from a bad memory reference, the runtime quits with a slightly > confusing error message, apparently implicating the threading system when > the bug is actually in normal, non-UNSAFE code. > > Mika > From jay.krell at cornell.edu Tue Feb 22 19:06:42 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 22 Feb 2011 18:06:42 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222174405.E15481A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu>, <4D615643.7060107@lcwb.coop>, <20110220183746.195C11A2078@async.async.caltech.edu>, <4D61CC27.7070704@lcwb.coop>, <20110222174405.E15481A2078@async.async.caltech.edu> Message-ID: (aside, and possible agreement: right -- an interpreter should consider NOT recursing on the machine stack whenever code it is interpreting recurses, but definitely some do) - Jay > To: rodney_bates at lcwb.coop > Date: Tue, 22 Feb 2011 09:44:05 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > > Ok so I was thinking about this. > > Why on earth is stack overflow even a problem? > > Consider the following procedure call (in my code, stack grows upwards): > > (* sp at x, pc at y *) > y: P(args) > z: next_statement > > decompose as follows: > > (* sp at x, pc at y *) > y: Push(args etc. and ret. address z) > Jump(P) > z: next_statement > > Now, we say: > > y: ok := check_stack(size of frame) > IF NOT ok THEN abort() END; > Push(args etc. and ret. address z) > Jump(P) > z: next_statement > > (note check_stack and the following IF can be implemented by hardware, > need not actually be an instruction) > > Let me change the code a tad: > > y: ok := check_stack(size of frame) > y':IF NOT ok THEN > WITH new_stack_bottom = malloc(stack_size) > huge_amount = new_stack_bottom - sp DO > create_redzone_at(new_stack_bottom+stack_size-redzone_size) > EVAL alloca(huge_amount) > END > END; > Push(args etc. and ret. address z) > Jump(P) > z: IF NOT ok THEN destroy_redzone(...); free(new_stack_bottom) END > > Note 1. cleanup of redzone could be postponed to return of caller....when > alloca in any case has to be cleaned up. > > Note 2. the test IF NOT ok at z is more expensive to implement than the > one at y because you can't really use hardware for it. A hardware callback > can be arranged though: > > VAR ptr := sp; > y: ok := check_stack(size of frame) > y':IF NOT ok THEN > ptr := 0; (* illegal address *) > fault_address := z; > WITH new_stack_bottom = malloc(stack_size) > huge_amount = new_stack_bottom - sp DO > create_redzone_at(new_stack_bottom+stack_size-redzone_size) > EVAL alloca(huge_amount) > END > END; > Push(args etc. and ret. address z) > Jump(P) > z: EVAL ptr^ (* [ NOT ok -> hardware callback to SEGV: ] *) > > SEGV(signalpc): IF NOT ok AND signalpc = fault_address THEN destroy_redzone(...); free(new_stack_bottom) END > > Mika > > > > > > "Rodney M. Bates" writes: > > > > > >On 02/20/2011 12:37 PM, Mika Nystrom wrote: > >> On a 64-bit machine, at least, there ought to be enough virtual > >> memory that you could just have a gap between thread stacks big > >> enough to allow for a protection area larger than the largest possible > >> (implementation-defined) activation record, no? I know I've run into > >> trouble with very large activation records in the past (and not because > >> I was running out of stack space, either). > >> > >> Or at least a procedure with a very large activation record (or > >> a procedure calling it) could be required to call some sort of check > >> routine "EnoughStackSpaceRemaining()" before starting to scribble > >> on the activation record? > > > >Hmm, I like this idea. It would introduce normal-case runtime overhead > >only for such procedures, and these are likely rare. Also, assuming the procedure > >actually uses very much of its large AR, it should also have enough computation > >time to wash out the stack check overhead. > > > >> > >> Also the end of the activation record must be written to at least once, > >> or else the memory protection won't be triggered. > >> > > > >I was thinking (as an alternative mechanism) of having the compiler intentionally > >add enough artificial write(s) as necessary to ensure storing within the > >red zone, and not just beyond it. This seems trickier to get right and > >harder to distinguish after the fact from a NIL dereference. > > > >> In any case if this is done properly the same mechanism I proposed for > >> SIGSEGV ought to be able to catch stack overflow, no? Well, as long as > >> signals are delivered on a separate stack. If signals are delivered on > >> the same stack, the signal handler would get nastier, it would have to > >> make space through some manipulations (maybe temporarily unporotecting > >> the redzone page?) for its own purposes... but I don't see why it > >> couldn't be done. > >> > >> Not sure why I'm getting SIGILL... maybe I am getting my signal handler > >> activated inside the redzone page because of a difference in signal > >> handling..? I remember reading something about sigaltstack... > >> > >> I would of course love to be able to recover from stack overflow, too. > >> In some sense, since it's a generally unknown limit, it's even less of > >> a fatal error than a NIL dereference (hence makes even more sense to > >> catch it). > > > >I think this would be a nice mechanism to have available. It would have to > >be used with some care. In any case, it would be really nice and more > >frequently so, to at least have runtime error messages that distinguished > >stack overflow from NIL deref. > > > >> > >> Mika > >> > >> "Rodney M. Bates" writes: > >>> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX. > >>> Probably a fully protected guard page at the end of the stack. This technique > >>> always worries me a bit because a procedure with a really big activation record > >>> could jump right past it. Probably it would almost always access the first page > >>> of the big area before storing anything into later pages. > >>> > >>> On 02/19/2011 05:27 PM, Mika Nystrom wrote: > >>>> Ah, yes, stack protection. > >>>> > >>>> Do you know if it's a SIGSEGV, not a SIGBUS? I know I have seen SIGILL on Macs. > >>>> > >>>> Hmm, I get SIGILL on AMD64_FREEBSD as well: > >>>> > >>>> time ../AMD64_FREEBSD/stubexample > >>>> M-Scheme Experimental > >>>> LITHP ITH LITHENING. > >>>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >>>> f > >>>>> (f 0) > >>>> Illegal instruction > >>>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >>>> > >>>> What absolutely must not happen, of course, is that the runtime hangs > >>>> while executing only safe code... > >>>> > >>>> Mika > >>>> > >>>> "Rodney M. Bates" writes: > >>>>> I know of one other place the compilers rely on hardware memory protection > >>>>> to detect a checked runtime error, and that is stack overflow. This won't > >>>>> corrupt anything, but is hard to distinguish from dereferencing NIL. > >>>>> This could probably be distinguished after the fact by some low-level, > >>>>> target-dependent code. I have found it by looking at assembly code at > >>>>> the point of failure--usually right after a stack pointer push. > >>>>> > >>>>> Detecting this via compiler-generated checks would probably be more > >>>>> extravagant than many other checks, as it is so frequent. I am not > >>>>> aware of any really good solution to this in any implementation of any > >>>>> language. > >>>>> > >>>>> On 02/19/2011 02:38 PM, Mika Nystrom wrote: > >>>>>> Jay, sometimes I wonder about you: this is a Modula-3 mailing list, > >>>>>> you know! > >>>>>> > >>>>>> "Corrupting the heap" is something that can only happen as a result of > >>>>>> an unchecked runtime error. Unchecked runtime errors cannot happen in > >>>>>> modules not marked UNSAFE. > >>>>>> > >>>>>> SEGV is, however, used by the CM3 implementation (and its predecessors) > >>>>>> to signal a certain kind of *checked* runtime error, namely, the > >>>>>> dereferencing of a NIL reference. Correct me if I am wrong, but an > >>>>>> attempt to dereference NIL is not going to leave the heap corrupted? > >>>>>> > >>>>>> And if you stick to safe code, the only SEGVs I think you get in the > >>>>>> current CM3 are ones from NIL dereferences. > >>>>>> > >>>>>> Hence, as long as you stick with safe code, the only time the code I > >>>>>> checked in earlier gets triggered is for NIL dereferences, which should > >>>>>> never corrupt the heap. So SEGV is not sometimes, but in fact always > >>>>>> recoverable. > >>>>>> > >>>>>> :-) > >>>>>> > >>>>>> Mika > >>>>>> > >>>>>> P.S. the bit above "if you stick to safe code": if you actually program in > >>>>>> Modula-3 you almost never use UNSAFE. I went through my repository and > >>>>>> I have 40 modules using UNSAFE out of a total of 4,559. Furthermore, > >>>>>> many of the UNSAFE modules are glue code to Fortran routines, which > >>>>>> could relatively easily be verified to be safe in the Modula-3 sense. > >>>>>> Almost all what remains is glue to some C library, which wouldn't be > >>>>>> necessary if the rest of the world would wake up out of the dark ages, but > >>>>>> I don't have the time to rewrite every single library from scratch myself. > >>>>>> > >>>>>> > >>>>>> Jay K writes: > >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>>>> Content-Type: text/plain; charset="iso-8859-1" > >>>>>>> Content-Transfer-Encoding: quoted-printable > >>>>>>> > >>>>>>> > >>>>>>> Letting any code run after a SIGSEGV is dubious. > >>>>>>> Imagine the heap is corrupted. > >>>>>>> And then you run more code. > >>>>>>> And the code happens to call malloc. > >>>>>>> Or printf to log something. > >>>>>>> =20 > >>>>>>> I suppose there might be an application that maps memory > >>>>>>> gradually=2C as pieces of a buffer are hit. Might. > >>>>>>> =20 > >>>>>>> - Jay > >>>>>>> =20 > >>>>>>>> To: m3devel at elegosoft.com > >>>>>>>> Date: Sat=2C 19 Feb 2011 10:29:30 -0800 > >>>>>>>> From: mika at async.caltech.edu > >>>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >>>>>>>> =20 > >>>>>>>> =20 > >>>>>>>> Dear m3devel=2C > >>>>>>>> =20 > >>>>>>>> For a while it has annoyed me that segmentation violations cause an > >>>>>>>> unconditional program abort. I've changed that now so that (under user > >>>>>>>> threads at least) we instead get a RuntimeError. Here's an example of > >>>>>>>> the mechanism at work in an interactive Scheme environment. Consider > >>>>>>>> the unhelpful interface and module Crash: > >>>>>>>> =20 > >>>>>>>> INTERFACE Crash=3B PROCEDURE Me()=3B END Crash. > >>>>>>>> =20 > >>>>>>>> MODULE Crash=3B > >>>>>>>> =20 > >>>>>>>> PROCEDURE Me() =3D > >>>>>>>> VAR ptr : REF INTEGER :=3D NIL=3B BEGIN > >>>>>>>> ptr^ :=3D 0 > >>>>>>>> END Me=3B > >>>>>>>> =20 > >>>>>>>> BEGIN END Crash. > >>>>>>>> =20 > >>>>>>>> Here's an example of what happens if you now call this from an interactiv= > >>>>>>> e > >>>>>>>> interpreter that catches the exception RuntimeError.E: > >>>>>>>> =20 > >>>>>>>> M-Scheme Experimental > >>>>>>>> LITHP ITH LITHENING. > >>>>>>>>> (require-modules "m3") > >>>>>>>> #t > >>>>>>>>> (Crash.Me) > >>>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory location. > >>>>>>>>> (+ 3 4)=20 > >>>>>>>> 7 > >>>>>>>>> =20 > >>>>>>>> =20 > >>>>>>>> I just realized I may have broken pthreads=2C let me go back and double-c= > >>>>>>> heck it.=20 > >>>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do they... > >>>>>>>> =20 > >>>>>>>> Mika > >>>>>>>> =20 > >>>>>>> = > >>>>>>> > >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >>>>>>> Content-Type: text/html; charset="iso-8859-1" > >>>>>>> Content-Transfer-Encoding: quoted-printable > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Letting any code run after a SIGSEGV is dubious.
> >>>>>>> Imagine the heap =3Bis corrupted.
> >>>>>>> And then you run more code.
> >>>>>>> And the code happens to call malloc.
> >>>>>>> Or printf to log something.
> >>>>>>>  =3B
> >>>>>>> I suppose there might be an application that maps memory
> >>>>>>> gradually=2C as pieces of a buffer are hit. Might.
> >>>>>>>  =3B
> >>>>>>>  =3B- Jay
 =3B
> >>>>>>> >=3B To: m3devel at elegosoft.com
>=3B Date: Sat=2C 19 Feb 2011 10:29:3= > >>>>>>> 0 -0800
>=3B From: mika at async.caltech.edu
>=3B Subject: [M3devel]= > >>>>>>> SEGV mapping to RuntimeError
>=3B
>=3B
>=3B Dear m3devel= > >>>>>>> =2C
>=3B
>=3B For a while it has annoyed me that segmentation vi= > >>>>>>> olations cause an
>=3B unconditional program abort. I've changed that = > >>>>>>> now so that (under user
>=3B threads at least) we instead get a Runtim= > >>>>>>> eError. Here's an example of
>=3B the mechanism at work in an interact= > >>>>>>> ive Scheme environment. Consider
>=3B the unhelpful interface and modu= > >>>>>>> le Crash:
>=3B
>=3B INTERFACE Crash=3B PROCEDURE Me()=3B END Cra= > >>>>>>> sh.
>=3B
>=3B MODULE Crash=3B
>=3B
>=3B PROCEDURE Me(= > >>>>>>> ) =3D
>=3B VAR ptr : REF INTEGER :=3D NIL=3B BEGIN
>=3B ptr^ :=3D= > >>>>>>> 0
>=3B END Me=3B
>=3B
>=3B BEGIN END Crash.
>=3B
= > >>>>>>> >=3B Here's an example of what happens if you now call this from an inter= > >>>>>>> active
>=3B interpreter that catches the exception RuntimeError.E:
= > >>>>>>> >=3B
>=3B M-Scheme Experimental
>=3B LITHP ITH LITHENING.
&= > >>>>>>> gt=3B>=3B (require-modules "m3")
>=3B #t
>=3B>=3B (Crash.Me= > >>>>>>> )
>=3B EXCEPTION! RuntimeError! Attempt to reference an illegal memory= > >>>>>>> location.
>=3B>=3B (+ 3 4)
>=3B 7
>=3B>=3B
>= > >>>>>>> =3B
>=3B I just realized I may have broken pthreads=2C let me go back= > >>>>>>> and double-check it.
>=3B runtime/POSIX and thread/POSIX don't refer= > >>>>>>> to the same thing do they...
>=3B
>=3B Mika
>=3B
= > >>>>>>> > >>>>>>> = > >>>>>>> > >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >>>>>> > >>>> > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Tue Feb 22 19:45:02 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Tue, 22 Feb 2011 10:45:02 -0800 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , <20110219203821.2016B1A2078@async.async.caltech.edu>, <4D604FA3.3060708@lcwb.coop>, <20110219232729.53A991A2078@async.async.caltech.edu>, <4D615643.7060107@lcwb.coop>, <20110220183746.195C11A2078@async.async.caltech.edu>, <4D61CC27.7070704@lcwb.coop>, <20110222174405.E15481A2078@async.async.caltech.edu> Message-ID: <20110222184502.88B921A2078@async.async.caltech.edu> Jay K writes: >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >(aside=2C and possible agreement: right -- an interpreter should consider N= >OT recursing on the machine >stack whenever code it is interpreting recurses=2C but definitely some do) >=20 > - Jay Jay, yes I agree---but: 1. of course it's easier to code the interpreter to recurse on the machine stack 2. code called out from the interpreter will in any case recurse on the (a) machine stack 3. again if one can "fix" the stack mechanism so that running out of stack space is non-fatal, there's no reason not to use the machine stack I've found that Modula-3's garbage collector is brutally slow and that converting code to using stack-allocated memory whenever possible is a huge performance win under both PM3 and CM3. Perhaps this is a problem with the garbage collector. I also have a partial design for a Scheme compiler (emitting Modula-3) that would use the machine stack whenever possible rather than heap allocation and do a "tricky thing" with local procedures to copy the stack whenever a reference to the stack escapes. (Most of the time you can implement Scheme on a stack but sometimes you need to keep part of it live after you return; hence implementations (including mine) heap-allocate local variables...) I realize my code is perhaps difficult to read. The gist of it is this: when you run out of stack space, allocate a new stack anywhere else in memory. Make it appear that you have alloca'd all the space between the old and new stacks (into a variable that you of course never touch), then continue. An issue is what to do when you return, you have to make sure that you restore your old redzone.. perhaps you don't need to free the stack eagerly: you could instead remember you have that page allocated and re-use it next time you hit the redzone, or put it in a pool to allow it to be used by another thread. The arbitrary choice of stack sizes has always bothered me greatly. Some of my programs have thousands of threads and they use stack space irregularly... Mika >> To: rodney_bates at lcwb.coop >> Date: Tue=2C 22 Feb 2011 09:44:05 -0800 >> From: mika at async.caltech.edu >> CC: m3devel at elegosoft.com >> Subject: Re: [M3devel] SEGV mapping to RuntimeError >>=20 >>=20 >> Ok so I was thinking about this. >>=20 >> Why on earth is stack overflow even a problem? >>=20 >> Consider the following procedure call (in my code=2C stack grows upwards)= >: >>=20 >> (* sp at x=2C pc at y *) >> y: P(args) >> z: next_statement >>=20 >> decompose as follows: >>=20 >> (* sp at x=2C pc at y *) >> y: Push(args etc. and ret. address z) >> Jump(P) >> z: next_statement >>=20 >> Now=2C we say: >>=20 >> y: ok :=3D check_stack(size of frame) >> IF NOT ok THEN abort() END=3B >> Push(args etc. and ret. address z) >> Jump(P) >> z: next_statement >>=20 >> (note check_stack and the following IF can be implemented by hardware=2C >> need not actually be an instruction) >>=20 >> Let me change the code a tad: >>=20 >> y: ok :=3D check_stack(size of frame) >> y':IF NOT ok THEN=20 >> WITH new_stack_bottom =3D malloc(stack_size) >> huge_amount =3D new_stack_bottom - sp DO >> create_redzone_at(new_stack_bottom+stack_size-redzone_size) >> EVAL alloca(huge_amount)=20 >> END >> END=3B >> Push(args etc. and ret. address z) >> Jump(P) >> z: IF NOT ok THEN destroy_redzone(...)=3B free(new_stack_bottom) END >>=20 >> Note 1. cleanup of redzone could be postponed to return of caller....when >> alloca in any case has to be cleaned up. >>=20 >> Note 2. the test IF NOT ok at z is more expensive to implement than the >> one at y because you can't really use hardware for it. A hardware callbac= >k >> can be arranged though: >>=20 >> VAR ptr :=3D sp=3B >> y: ok :=3D check_stack(size of frame) >> y':IF NOT ok THEN=20 >> ptr :=3D 0=3B (* illegal address *) >> fault_address :=3D z=3B >> WITH new_stack_bottom =3D malloc(stack_size) >> huge_amount =3D new_stack_bottom - sp DO >> create_redzone_at(new_stack_bottom+stack_size-redzone_size) >> EVAL alloca(huge_amount)=20 >> END >> END=3B >> Push(args etc. and ret. address z) >> Jump(P) >> z: EVAL ptr^ (* [ NOT ok -> hardware callback to SEGV: ] *) >>=20 >> SEGV(signalpc): IF NOT ok AND signalpc =3D fault_address THEN destroy_red= >zone(...)=3B free(new_stack_bottom) END >>=20 >> Mika >>=20 >>=20 >>=20 >>=20 >>=20 >> "Rodney M. Bates" writes: >> > >> > >> >On 02/20/2011 12:37 PM=2C Mika Nystrom wrote: >> >> On a 64-bit machine=2C at least=2C there ought to be enough virtual >> >> memory that you could just have a gap between thread stacks big >> >> enough to allow for a protection area larger than the largest possible >> >> (implementation-defined) activation record=2C no? I know I've run into >> >> trouble with very large activation records in the past (and not becaus= >e >> >> I was running out of stack space=2C either). >> >> >> >> Or at least a procedure with a very large activation record (or >> >> a procedure calling it) could be required to call some sort of check >> >> routine "EnoughStackSpaceRemaining()" before starting to scribble >> >> on the activation record? >> > >> >Hmm=2C I like this idea. It would introduce normal-case runtime overhead >> >only for such procedures=2C and these are likely rare. Also=2C assuming = >the procedure >> >actually uses very much of its large AR=2C it should also have enough co= >mputation >> >time to wash out the stack check overhead. >> > >> >> >> >> Also the end of the activation record must be written to at least once= >=2C >> >> or else the memory protection won't be triggered. >> >> >> > >> >I was thinking (as an alternative mechanism) of having the compiler inte= >ntionally >> >add enough artificial write(s) as necessary to ensure storing within the >> >red zone=2C and not just beyond it. This seems trickier to get right and >> >harder to distinguish after the fact from a NIL dereference. >> > >> >> In any case if this is done properly the same mechanism I proposed for >> >> SIGSEGV ought to be able to catch stack overflow=2C no? Well=2C as lon= >g as >> >> signals are delivered on a separate stack. If signals are delivered on >> >> the same stack=2C the signal handler would get nastier=2C it would hav= >e to >> >> make space through some manipulations (maybe temporarily unporotecting >> >> the redzone page?) for its own purposes... but I don't see why it >> >> couldn't be done. >> >> >> >> Not sure why I'm getting SIGILL... maybe I am getting my signal handle= >r >> >> activated inside the redzone page because of a difference in signal >> >> handling..? I remember reading something about sigaltstack... >> >> >> >> I would of course love to be able to recover from stack overflow=2C to= >o. >> >> In some sense=2C since it's a generally unknown limit=2C it's even les= >s of >> >> a fatal error than a NIL dereference (hence makes even more sense to >> >> catch it). >> > >> >I think this would be a nice mechanism to have available. It would have = >to >> >be used with some care. In any case=2C it would be really nice and more >> >frequently so=2C to at least have runtime error messages that distinguis= >hed >> >stack overflow from NIL deref. >> > >> >> >> >> Mika >> >> >> >> "Rodney M. Bates" writes: >> >>> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AM= >D64_LINUX. >> >>> Probably a fully protected guard page at the end of the stack. This t= >echnique >> >>> always worries me a bit because a procedure with a really big activat= >ion record >> >>> could jump right past it. Probably it would almost always access the = >first page >> >>> of the big area before storing anything into later pages. >> >>> >> >>> On 02/19/2011 05:27 PM=2C Mika Nystrom wrote: >> >>>> Ah=2C yes=2C stack protection. >> >>>> >> >>>> Do you know if it's a SIGSEGV=2C not a SIGBUS? I know I have seen SI= >GILL on Macs. >> >>>> >> >>>> Hmm=2C I get SIGILL on AMD64_FREEBSD as well: >> >>>> >> >>>> time ../AMD64_FREEBSD/stubexample >> >>>> M-Scheme Experimental >> >>>> LITHP ITH LITHENING. >> >>>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) >> >>>> f >> >>>>> (f 0) >> >>>> Illegal instruction >> >>>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w >> >>>> >> >>>> What absolutely must not happen=2C of course=2C is that the runtime = >hangs >> >>>> while executing only safe code... >> >>>> >> >>>> Mika >> >>>> >> >>>> "Rodney M. Bates" writes: >> >>>>> I know of one other place the compilers rely on hardware memory pro= >tection >> >>>>> to detect a checked runtime error=2C and that is stack overflow. Th= >is won't >> >>>>> corrupt anything=2C but is hard to distinguish from dereferencing N= >IL. >> >>>>> This could probably be distinguished after the fact by some low-lev= >el=2C >> >>>>> target-dependent code. I have found it by looking at assembly code = >at >> >>>>> the point of failure--usually right after a stack pointer push. >> >>>>> >> >>>>> Detecting this via compiler-generated checks would probably be more >> >>>>> extravagant than many other checks=2C as it is so frequent. I am no= >t >> >>>>> aware of any really good solution to this in any implementation of = >any >> >>>>> language. >> >>>>> >> >>>>> On 02/19/2011 02:38 PM=2C Mika Nystrom wrote: >> >>>>>> Jay=2C sometimes I wonder about you: this is a Modula-3 mailing li= >st=2C >> >>>>>> you know! >> >>>>>> >> >>>>>> "Corrupting the heap" is something that can only happen as a resul= >t of >> >>>>>> an unchecked runtime error. Unchecked runtime errors cannot happen= > in >> >>>>>> modules not marked UNSAFE. >> >>>>>> >> >>>>>> SEGV is=2C however=2C used by the CM3 implementation (and its pred= >ecessors) >> >>>>>> to signal a certain kind of *checked* runtime error=2C namely=2C t= >he >> >>>>>> dereferencing of a NIL reference. Correct me if I am wrong=2C but = >an >> >>>>>> attempt to dereference NIL is not going to leave the heap corrupte= >d? >> >>>>>> >> >>>>>> And if you stick to safe code=2C the only SEGVs I think you get in= > the >> >>>>>> current CM3 are ones from NIL dereferences. >> >>>>>> >> >>>>>> Hence=2C as long as you stick with safe code=2C the only time the = >code I >> >>>>>> checked in earlier gets triggered is for NIL dereferences=2C which= > should >> >>>>>> never corrupt the heap. So SEGV is not sometimes=2C but in fact al= >ways >> >>>>>> recoverable. >> >>>>>> >> >>>>>> :-) >> >>>>>> >> >>>>>> Mika >> >>>>>> >> >>>>>> P.S. the bit above "if you stick to safe code": if you actually pr= >ogram in >> >>>>>> Modula-3 you almost never use UNSAFE. I went through my repository= > and >> >>>>>> I have 40 modules using UNSAFE out of a total of 4=2C559. Furtherm= >ore=2C >> >>>>>> many of the UNSAFE modules are glue code to Fortran routines=2C wh= >ich >> >>>>>> could relatively easily be verified to be safe in the Modula-3 sen= >se. >> >>>>>> Almost all what remains is glue to some C library=2C which wouldn'= >t be >> >>>>>> necessary if the rest of the world would wake up out of the dark a= >ges=2C but >> >>>>>> I don't have the time to rewrite every single library from scratch= > myself. >> >>>>>> >> >>>>>> >> >>>>>> Jay K writes: >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >> >>>>>>> Content-Type: text/plain=3B charset=3D"iso-8859-1" >> >>>>>>> Content-Transfer-Encoding: quoted-printable >> >>>>>>> >> >>>>>>> >> >>>>>>> Letting any code run after a SIGSEGV is dubious. >> >>>>>>> Imagine the heap is corrupted. >> >>>>>>> And then you run more code. >> >>>>>>> And the code happens to call malloc. >> >>>>>>> Or printf to log something. >> >>>>>>> =3D20 >> >>>>>>> I suppose there might be an application that maps memory >> >>>>>>> gradually=3D2C as pieces of a buffer are hit. Might. >> >>>>>>> =3D20 >> >>>>>>> - Jay >> >>>>>>> =3D20 >> >>>>>>>> To: m3devel at elegosoft.com >> >>>>>>>> Date: Sat=3D2C 19 Feb 2011 10:29:30 -0800 >> >>>>>>>> From: mika at async.caltech.edu >> >>>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError >> >>>>>>>> =3D20 >> >>>>>>>> =3D20 >> >>>>>>>> Dear m3devel=3D2C >> >>>>>>>> =3D20 >> >>>>>>>> For a while it has annoyed me that segmentation violations cause= > an >> >>>>>>>> unconditional program abort. I've changed that now so that (unde= >r user >> >>>>>>>> threads at least) we instead get a RuntimeError. Here's an examp= >le of >> >>>>>>>> the mechanism at work in an interactive Scheme environment. Cons= >ider >> >>>>>>>> the unhelpful interface and module Crash: >> >>>>>>>> =3D20 >> >>>>>>>> INTERFACE Crash=3D3B PROCEDURE Me()=3D3B END Crash. >> >>>>>>>> =3D20 >> >>>>>>>> MODULE Crash=3D3B >> >>>>>>>> =3D20 >> >>>>>>>> PROCEDURE Me() =3D3D >> >>>>>>>> VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN >> >>>>>>>> ptr^ :=3D3D 0 >> >>>>>>>> END Me=3D3B >> >>>>>>>> =3D20 >> >>>>>>>> BEGIN END Crash. >> >>>>>>>> =3D20 >> >>>>>>>> Here's an example of what happens if you now call this from an i= >nteractiv=3D >> >>>>>>> e >> >>>>>>>> interpreter that catches the exception RuntimeError.E: >> >>>>>>>> =3D20 >> >>>>>>>> M-Scheme Experimental >> >>>>>>>> LITHP ITH LITHENING. >> >>>>>>>>> (require-modules "m3") >> >>>>>>>> #t >> >>>>>>>>> (Crash.Me) >> >>>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory = >location. >> >>>>>>>>> (+ 3 4)=3D20 >> >>>>>>>> 7 >> >>>>>>>>> =3D20 >> >>>>>>>> =3D20 >> >>>>>>>> I just realized I may have broken pthreads=3D2C let me go back a= >nd double-c=3D >> >>>>>>> heck it.=3D20 >> >>>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do = >they... >> >>>>>>>> =3D20 >> >>>>>>>> Mika >> >>>>>>>> =3D20 >> >>>>>>> =3D >> >>>>>>> >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ >> >>>>>>> Content-Type: text/html=3B charset=3D"iso-8859-1" >> >>>>>>> Content-Transfer-Encoding: quoted-printable >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> Letting any code run after a SIGSEGV is dubious.
>> >>>>>>> Imagine the heap =3D3Bis corrupted.
>> >>>>>>> And then you run more code.
>> >>>>>>> And the code happens to call malloc.
>> >>>>>>> Or printf to log something.
>> >>>>>>>  =3D3B
>> >>>>>>> I suppose there might be an application that maps memory
>> >>>>>>> gradually=3D2C as pieces of a buffer are hit. Might.
>> >>>>>>>  =3D3B
>> >>>>>>>  =3D3B- Jay
 =3D3B
>> >>>>>>> >=3D3B To: m3devel at elegosoft.com
>=3D3B Date: Sat=3D2C 19 = >Feb 2011 10:29:3=3D >> >>>>>>> 0 -0800
>=3D3B From: mika at async.caltech.edu
>=3D3B Subj= >ect: [M3devel]=3D >> >>>>>>> SEGV mapping to RuntimeError
>=3D3B
>=3D3B
>=3D3B = >Dear m3devel=3D >> >>>>>>> =3D2C
>=3D3B
>=3D3B For a while it has annoyed me that = >segmentation vi=3D >> >>>>>>> olations cause an
>=3D3B unconditional program abort. I've c= >hanged that =3D >> >>>>>>> now so that (under user
>=3D3B threads at least) we instead = >get a Runtim=3D >> >>>>>>> eError. Here's an example of
>=3D3B the mechanism at work in= > an interact=3D >> >>>>>>> ive Scheme environment. Consider
>=3D3B the unhelpful interf= >ace and modu=3D >> >>>>>>> le Crash:
>=3D3B
>=3D3B INTERFACE Crash=3D3B PROCEDURE = >Me()=3D3B END Cra=3D >> >>>>>>> sh.
>=3D3B
>=3D3B MODULE Crash=3D3B
>=3D3B
>= >=3D3B PROCEDURE Me(=3D >> >>>>>>> ) =3D3D
>=3D3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGINR>>=3D3B ptr^ :=3D3D=3D >> >>>>>>> 0
>=3D3B END Me=3D3B
>=3D3B
>=3D3B BEGIN END Crash= >.
>=3D3B
=3D >> >>>>>>> >=3D3B Here's an example of what happens if you now call this f= >rom an inter=3D >> >>>>>>> active
>=3D3B interpreter that catches the exception Runtime= >Error.E:
=3D >> >>>>>>> >=3D3B
>=3D3B M-Scheme Experimental
>=3D3B LITHP ITH = >LITHENING.
&=3D >> >>>>>>> gt=3D3B>=3D3B (require-modules "m3")
>=3D3B #t
>=3D3B= >>=3D3B (Crash.Me=3D >> >>>>>>> )
>=3D3B EXCEPTION! RuntimeError! Attempt to reference an il= >legal memory=3D >> >>>>>>> location.
>=3D3B>=3D3B (+ 3 4)
>=3D3B 7
>=3D3B&g= >t=3D3B
>=3D >> >>>>>>> =3D3B
>=3D3B I just realized I may have broken pthreads=3D2C= > let me go back=3D >> >>>>>>> and double-check it.
>=3D3B runtime/POSIX and thread/POSIX d= >on't refer=3D >> >>>>>>> to the same thing do they...
>=3D3B
>=3D3B Mika
>= >=3D3B
=3D >> >>>>>>> >> >>>>>>> =3D >> >>>>>>> >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- >> >>>>>> >> >>>> >> >> > = > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_ >Content-Type: text/html; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > > > >(aside=2C and possible agreement: right -- an interpreter should consider N= >OT recursing on the machine
>stack whenever code it is interpreting recurses=2C but definitely some do)<= >BR> > =3B
> =3B- Jay
 =3B
>>=3B To: rodney_bates at lcwb.coop
>=3B Date: Tue=2C 22 Feb 2011 09:44:= >05 -0800
>=3B From: mika at async.caltech.edu
>=3B CC: m3devel at elego= >soft.com
>=3B Subject: Re: [M3devel] SEGV mapping to RuntimeError
&= >gt=3B
>=3B
>=3B Ok so I was thinking about this.
>=3B
= >>=3B Why on earth is stack overflow even a problem?
>=3B
>=3B = >Consider the following procedure call (in my code=2C stack grows upwards):<= >BR>>=3B
>=3B (* sp at x=2C pc at y *)
>=3B y: P(args)
>= >=3B z: next_statement
>=3B
>=3B decompose as follows:
>=3B = >
>=3B (* sp at x=2C pc at y *)
>=3B y: Push(args etc. and ret. ad= >dress z)
>=3B Jump(P)
>=3B z: next_statement
>=3B
>=3B= > Now=2C we say:
>=3B
>=3B y: ok :=3D check_stack(size of frame)<= >BR>>=3B IF NOT ok THEN abort() END=3B
>=3B Push(args etc. and ret. a= >ddress z)
>=3B Jump(P)
>=3B z: next_statement
>=3B
>= >=3B (note check_stack and the following IF can be implemented by hardware= >=2C
>=3B need not actually be an instruction)
>=3B
>=3B Let= > me change the code a tad:
>=3B
>=3B y: ok :=3D check_stack(size= > of frame)
>=3B y':IF NOT ok THEN
>=3B WITH new_stack_bottom =3D= > malloc(stack_size)
>=3B huge_amount =3D new_stack_bottom - sp DO
&= >gt=3B create_redzone_at(new_stack_bottom+stack_size-redzone_size)
>=3B= > EVAL alloca(huge_amount)
>=3B END
>=3B END=3B
>=3B Push(ar= >gs etc. and ret. address z)
>=3B Jump(P)
>=3B z: IF NOT ok THEN d= >estroy_redzone(...)=3B free(new_stack_bottom) END
>=3B
>=3B Note= > 1. cleanup of redzone could be postponed to return of caller....when
&g= >t=3B alloca in any case has to be cleaned up.
>=3B
>=3B Note 2. = >the test IF NOT ok at z is more expensive to implement than the
>=3B o= >ne at y because you can't really use hardware for it. A hardware callbackR>>=3B can be arranged though:
>=3B
>=3B VAR ptr :=3D sp=3B>>=3B y: ok :=3D check_stack(size of frame)
>=3B y':IF NOT ok THEN <= >BR>>=3B ptr :=3D 0=3B (* illegal address *)
>=3B fault_address :=3D = >z=3B
>=3B WITH new_stack_bottom =3D malloc(stack_size)
>=3B huge_= >amount =3D new_stack_bottom - sp DO
>=3B create_redzone_at(new_stack_b= >ottom+stack_size-redzone_size)
>=3B EVAL alloca(huge_amount)
>= >=3B END
>=3B END=3B
>=3B Push(args etc. and ret. address z)
&g= >t=3B Jump(P)
>=3B z: EVAL ptr^ (* [ NOT ok ->=3B hardware callback t= >o SEGV: ] *)
>=3B
>=3B SEGV(signalpc): IF NOT ok AND signalpc = >=3D fault_address THEN destroy_redzone(...)=3B free(new_stack_bottom) ENDR>>=3B
>=3B Mika
>=3B
>=3B
>=3B
>=3B
>= >=3B
>=3B "Rodney M. Bates" writes:
>=3B >=3B
>=3B >=3B<= >BR>>=3B >=3BOn 02/20/2011 12:37 PM=2C Mika Nystrom wrote:
>=3B >= >=3B>=3B On a 64-bit machine=2C at least=2C there ought to be enough virtu= >al
>=3B >=3B>=3B memory that you could just have a gap between thr= >ead stacks big
>=3B >=3B>=3B enough to allow for a protection area= > larger than the largest possible
>=3B >=3B>=3B (implementation-de= >fined) activation record=2C no? I know I've run into
>=3B >=3B>=3B= > trouble with very large activation records in the past (and not because>>=3B >=3B>=3B I was running out of stack space=2C either).
>=3B= > >=3B>=3B
>=3B >=3B>=3B Or at least a procedure with a very la= >rge activation record (or
>=3B >=3B>=3B a procedure calling it) co= >uld be required to call some sort of check
>=3B >=3B>=3B routine "= >EnoughStackSpaceRemaining()" before starting to scribble
>=3B >=3B&g= >t=3B on the activation record?
>=3B >=3B
>=3B >=3BHmm=2C I li= >ke this idea. It would introduce normal-case runtime overhead
>=3B >= >=3Bonly for such procedures=2C and these are likely rare. Also=2C assuming = >the procedure
>=3B >=3Bactually uses very much of its large AR=2C it= > should also have enough computation
>=3B >=3Btime to wash out the s= >tack check overhead.
>=3B >=3B
>=3B >=3B>=3B
>=3B >= >=3B>=3B Also the end of the activation record must be written to at least= > once=2C
>=3B >=3B>=3B or else the memory protection won't be trig= >gered.
>=3B >=3B>=3B
>=3B >=3B
>=3B >=3BI was thinki= >ng (as an alternative mechanism) of having the compiler intentionally
&g= >t=3B >=3Badd enough artificial write(s) as necessary to ensure storing wi= >thin the
>=3B >=3Bred zone=2C and not just beyond it. This seems tri= >ckier to get right and
>=3B >=3Bharder to distinguish after the fact= > from a NIL dereference.
>=3B >=3B
>=3B >=3B>=3B In any cas= >e if this is done properly the same mechanism I proposed for
>=3B >= >=3B>=3B SIGSEGV ought to be able to catch stack overflow=2C no? Well=2C a= >s long as
>=3B >=3B>=3B signals are delivered on a separate stack.= > If signals are delivered on
>=3B >=3B>=3B the same stack=2C the s= >ignal handler would get nastier=2C it would have to
>=3B >=3B>=3B = >make space through some manipulations (maybe temporarily unporotecting
&= >gt=3B >=3B>=3B the redzone page?) for its own purposes... but I don't s= >ee why it
>=3B >=3B>=3B couldn't be done.
>=3B >=3B>=3BR>>=3B >=3B>=3B Not sure why I'm getting SIGILL... maybe I am getting= > my signal handler
>=3B >=3B>=3B activated inside the redzone page= > because of a difference in signal
>=3B >=3B>=3B handling..? I rem= >ember reading something about sigaltstack...
>=3B >=3B>=3B
>= >=3B >=3B>=3B I would of course love to be able to recover from stack ov= >erflow=2C too.
>=3B >=3B>=3B In some sense=2C since it's a general= >ly unknown limit=2C it's even less of
>=3B >=3B>=3B a fatal error = >than a NIL dereference (hence makes even more sense to
>=3B >=3B>= >=3B catch it).
>=3B >=3B
>=3B >=3BI think this would be a nic= >e mechanism to have available. It would have to
>=3B >=3Bbe used wit= >h some care. In any case=2C it would be really nice and more
>=3B >= >=3Bfrequently so=2C to at least have runtime error messages that distinguis= >hed
>=3B >=3Bstack overflow from NIL deref.
>=3B >=3B
>= >=3B >=3B>=3B
>=3B >=3B>=3B Mika
>=3B >=3B>=3B
>= >=3B >=3B>=3B "Rodney M. Bates" writes:
>=3B >=3B>=3B>=3B I a= >m pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX= >.
>=3B >=3B>=3B>=3B Probably a fully protected guard page at the= > end of the stack. This technique
>=3B >=3B>=3B>=3B always worri= >es me a bit because a procedure with a really big activation record
>= >=3B >=3B>=3B>=3B could jump right past it. Probably it would almost a= >lways access the first page
>=3B >=3B>=3B>=3B of the big area be= >fore storing anything into later pages.
>=3B >=3B>=3B>=3B
>= >=3B >=3B>=3B>=3B On 02/19/2011 05:27 PM=2C Mika Nystrom wrote:
>= >=3B >=3B>=3B>=3B>=3B Ah=2C yes=2C stack protection.
>=3B >= >=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B Do you know if it'= >s a SIGSEGV=2C not a SIGBUS? I know I have seen SIGILL on Macs.
>=3B &= >gt=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B Hmm=2C I get SIG= >ILL on AMD64_FREEBSD as well:
>=3B >=3B>=3B>=3B>=3B
>=3B = >>=3B>=3B>=3B>=3B time ../AMD64_FREEBSD/stubexample
>=3B >=3B= >>=3B>=3B>=3B M-Scheme Experimental
>=3B >=3B>=3B>=3B>=3B= > LITHP ITH LITHENING.
>=3B >=3B>=3B>=3B>=3B>=3B (define (f a= >) (+ (f (+ a 1)) (f (+ a 2))))
>=3B >=3B>=3B>=3B>=3B f
>= >=3B >=3B>=3B>=3B>=3B>=3B (f 0)
>=3B >=3B>=3B>=3B>=3B= > Illegal instruction
>=3B >=3B>=3B>=3B>=3B 3.847u 0.368s 0:13.= >32 31.5% 2160+284478k 0+0io 0pf+0w
>=3B >=3B>=3B>=3B>=3B
&g= >t=3B >=3B>=3B>=3B>=3B What absolutely must not happen=2C of course= >=2C is that the runtime hangs
>=3B >=3B>=3B>=3B>=3B while exec= >uting only safe code...
>=3B >=3B>=3B>=3B>=3B
>=3B >=3B= >>=3B>=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B
>=3B >=3B= >>=3B>=3B>=3B "Rodney M. Bates" writes:
>=3B >=3B>=3B>=3B&g= >t=3B>=3B I know of one other place the compilers rely on hardware memory = >protection
>=3B >=3B>=3B>=3B>=3B>=3B to detect a checked run= >time error=2C and that is stack overflow. This won't
>=3B >=3B>=3B= >>=3B>=3B>=3B corrupt anything=2C but is hard to distinguish from dere= >ferencing NIL.
>=3B >=3B>=3B>=3B>=3B>=3B This could probably= > be distinguished after the fact by some low-level=2C
>=3B >=3B>= >=3B>=3B>=3B>=3B target-dependent code. I have found it by looking at = >assembly code at
>=3B >=3B>=3B>=3B>=3B>=3B the point of fail= >ure--usually right after a stack pointer push.
>=3B >=3B>=3B>=3B= >>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B Detecting this via co= >mpiler-generated checks would probably be more
>=3B >=3B>=3B>=3B= >>=3B>=3B extravagant than many other checks=2C as it is so frequent. I = >am not
>=3B >=3B>=3B>=3B>=3B>=3B aware of any really good so= >lution to this in any implementation of any
>=3B >=3B>=3B>=3B>= >=3B>=3B language.
>=3B >=3B>=3B>=3B>=3B>=3B
>=3B >= >=3B>=3B>=3B>=3B>=3B On 02/19/2011 02:38 PM=2C Mika Nystrom wrote:R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B Jay=2C sometimes I wonder abo= >ut you: this is a Modula-3 mailing list=2C
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B you know!
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>>=3B >=3B>=3B>=3B>=3B>=3B>=3B "Corrupting the heap" is somet= hing that can only happen as a result of
>=3B >=3B>=3B>=3B>=3B= >>=3B>=3B an unchecked runtime error. Unchecked runtime errors cannot ha= >ppen in
>=3B >=3B>=3B>=3B>=3B>=3B>=3B modules not marked U= >NSAFE.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B SEGV is=2C however=2C used by the CM3 implement= >ation (and its predecessors)
>=3B >=3B>=3B>=3B>=3B>=3B>=3B= > to signal a certain kind of *checked* runtime error=2C namely=2C the
&g= >t=3B >=3B>=3B>=3B>=3B>=3B>=3B dereferencing of a NIL reference.= > Correct me if I am wrong=2C but an
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B attempt to dereference NIL is not going to leave the heap corrupt= >ed?
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B&g= >t=3B>=3B>=3B>=3B And if you stick to safe code=2C the only SEGVs I th= >ink you get in the
>=3B >=3B>=3B>=3B>=3B>=3B>=3B current C= >M3 are ones from NIL dereferences.
>=3B >=3B>=3B>=3B>=3B>=3B= >>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B Hence=2C as long as y= >ou stick with safe code=2C the only time the code I
>=3B >=3B>=3B&= >gt=3B>=3B>=3B>=3B checked in earlier gets triggered is for NIL derefe= >rences=2C which should
>=3B >=3B>=3B>=3B>=3B>=3B>=3B never= > corrupt the heap. So SEGV is not sometimes=2C but in fact always
>=3B= > >=3B>=3B>=3B>=3B>=3B>=3B recoverable.
>=3B >=3B>=3B&g= >t=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B :-)R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B&= >gt=3B>=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
&g= >t=3B >=3B>=3B>=3B>=3B>=3B>=3B P.S. the bit above "if you stick = >to safe code": if you actually program in
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B Modula-3 you almost never use UNSAFE. I went through my rep= >ository and
>=3B >=3B>=3B>=3B>=3B>=3B>=3B I have 40 module= >s using UNSAFE out of a total of 4=2C559. Furthermore=2C
>=3B >=3B&g= >t=3B>=3B>=3B>=3B>=3B many of the UNSAFE modules are glue code to Fo= >rtran routines=2C which
>=3B >=3B>=3B>=3B>=3B>=3B>=3B coul= >d relatively easily be verified to be safe in the Modula-3 sense.
>=3B= > >=3B>=3B>=3B>=3B>=3B>=3B Almost all what remains is glue to so= >me C library=2C which wouldn't be
>=3B >=3B>=3B>=3B>=3B>=3B&= >gt=3B necessary if the rest of the world would wake up out of the dark ages= >=2C but
>=3B >=3B>=3B>=3B>=3B>=3B>=3B I don't have the tim= >e to rewrite every single library from scratch myself.
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B<= >BR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B Jay K writes:
>=3B >= >=3B>=3B>=3B>=3B>=3B>=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912= >854f_
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Content-Type: te= >xt/plain=3B charset=3D"iso-8859-1"
>=3B >=3B>=3B>=3B>=3B>=3B= >>=3B>=3B Content-Transfer-Encoding: quoted-printable
>=3B >=3B&g= >t=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B= >>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Letting a= >ny code run after a SIGSEGV is dubious.
>=3B >=3B>=3B>=3B>=3B&= >gt=3B>=3B>=3B Imagine the heap is corrupted.
>=3B >=3B>=3B>= >=3B>=3B>=3B>=3B>=3B And then you run more code.
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B And the code happens to call malloc.
&= >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Or printf to log something= >.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>=3B >= >=3B>=3B>=3B>=3B>=3B>=3B>=3B I suppose there might be an applica= >tion that maps memory
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B = >gradually=3D2C as pieces of a buffer are hit. Might.
>=3B >=3B>=3B= >>=3B>=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B - Jay
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B = >=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B To: m3deve= >l at elegosoft.com
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = >Date: Sat=3D2C 19 Feb 2011 10:29:30 -0800
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B>=3B>=3B From: mika at async.caltech.edu
>=3B >=3B&g= >t=3B>=3B>=3B>=3B>=3B>=3B>=3B Subject: [M3devel] SEGV mapping to= > RuntimeError
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = >=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
&= >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B Dear m3devel=3D2C>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>=3B &g= >t=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B For a while it has annoyed m= >e that segmentation violations cause an
>=3B >=3B>=3B>=3B>=3B&= >gt=3B>=3B>=3B>=3B unconditional program abort. I've changed that now = >so that (under user
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>= >=3B threads at least) we instead get a RuntimeError. Here's an example ofR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B the mechanism at = >work in an interactive Scheme environment. Consider
>=3B >=3B>=3B&= >gt=3B>=3B>=3B>=3B>=3B>=3B the unhelpful interface and module Cras= >h:
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>= >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B INTERFACE Crash=3D3B P= >ROCEDURE Me()=3D3B END Crash.
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B MODULE Crash=3D3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= >=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = >PROCEDURE Me() =3D3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&g= >t=3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN
>=3B >=3B>=3B>= >=3B>=3B>=3B>=3B>=3B>=3B ptr^ :=3D3D 0
>=3B >=3B>=3B>= >=3B>=3B>=3B>=3B>=3B>=3B END Me=3D3B
>=3B >=3B>=3B>=3B&= >gt=3B>=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B>=3B BEGIN END Crash.
>=3B >=3B>=3B>=3B>=3B&g= >t=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B= >>=3B>=3B Here's an example of what happens if you now call this from an= > interactiv=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B e
&g= >t=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B interpreter that catc= >hes the exception RuntimeError.E:
>=3B >=3B>=3B>=3B>=3B>=3B&= >gt=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= >=3B>=3B M-Scheme Experimental
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B>=3B LITHP ITH LITHENING.
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B>=3B>=3B (require-modules "m3")
>=3B >=3B>=3B&g= >t=3B>=3B>=3B>=3B>=3B>=3B #t
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B>=3B>=3B (Crash.Me)
>=3B >=3B>=3B>=3B>=3B&g= >t=3B>=3B>=3B>=3B EXCEPTION! RuntimeError! Attempt to reference an ill= >egal memory location.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B>=3B (+ 3 4)=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= >=3B>=3B 7
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B>= >=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20<= >BR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B I just realized = >I may have broken pthreads=3D2C let me go back and double-c=3D
>=3B &g= >t=3B>=3B>=3B>=3B>=3B>=3B>=3B heck it.=3D20
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B>=3B runtime/POSIX and thread/POSIX don'= >t refer to the same thing do they...
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B= > =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D
>=3B &= >gt=3B>=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_
>=3B &g= >t=3B>=3B>=3B>=3B>=3B>=3B>=3B Content-Type: text/html=3B charset= >=3D"iso-8859-1"
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Conten= >t-Transfer-Encoding: quoted-printable
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3B= >html>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3Bhead&= >gt=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3Bstyle>= >=3B<=3B!--
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B .hmmessag= >e P
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B {
>=3B >=3B= >>=3B>=3B>=3B>=3B>=3B>=3B margin:0px=3D3B
>=3B >=3B>=3B= >>=3B>=3B>=3B>=3B>=3B padding:0px
>=3B >=3B>=3B>=3B>= >=3B>=3B>=3B>=3B }
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= >=3B body.hmmessage
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B {R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B font-size: 10pt=3D3B>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B font-family:Tahoma
&g= >t=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B }
>=3B >=3B>=3B>= >=3B>=3B>=3B>=3B>=3B -->=3B<=3B/style>=3B
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B <=3B/head>=3B
>=3B >=3B>=3B&= >gt=3B>=3B>=3B>=3B>=3B <=3Bbody class=3D3D'hmmessage'>=3B
>= >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Letting any code run after a= > SIGSEGV is dubious.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B= >>=3B>=3B Imagine the heap&=3Bnbsp=3D3Bis corrupted.<=3BBR>=3B>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B And then you run more co= >de.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B And = >the code happens to call malloc.<=3BBR>=3B
>=3B >=3B>=3B>=3B= >>=3B>=3B>=3B>=3B Or printf to log something.<=3BBR>=3B
>= >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B &=3Bnbsp=3D3B<=3BBR>= >=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B I suppose there mi= >ght be an application that maps memory<=3BBR>=3B
>=3B >=3B>=3B= >>=3B>=3B>=3B>=3B>=3B gradually=3D2C as pieces of a buffer are hit= >. Might.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B= > &=3Bnbsp=3D3B<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B &=3Bnbsp=3D3B- Jay<=3BBR>=3B&=3Bnbsp=3D3B<=3BBR>=3B= >
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B &=3Bgt=3D3B To: m3= >devel at elegosoft.com<=3BBR>=3B&=3Bgt=3D3B Date: Sat=3D2C 19 Feb 2011 = >10:29:3=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B 0 -0800<= >=3BBR>=3B&=3Bgt=3D3B From: mika at async.caltech.edu<=3BBR>=3B&=3B= >gt=3D3B Subject: [M3devel]=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B SEGV mapping to RuntimeError<=3BBR>=3B&=3Bgt=3D3B<=3BBR&= >gt=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B Dear m3devel=3D
>=3B &= >gt=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D2C<=3BBR>=3B&=3Bgt=3D3B= ><=3BBR>=3B&=3Bgt=3D3B For a while it has annoyed me that segmentatio= >n vi=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B olations cause= > an<=3BBR>=3B&=3Bgt=3D3B unconditional program abort. I've changed t= >hat =3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B now so that (u= >nder user<=3BBR>=3B&=3Bgt=3D3B threads at least) we instead get a Ru= >ntim=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B eError. Here's= > an example of<=3BBR>=3B&=3Bgt=3D3B the mechanism at work in an inte= >ract=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B ive Scheme env= >ironment. Consider<=3BBR>=3B&=3Bgt=3D3B the unhelpful interface and = >modu=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B le Crash:<= >=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B INTERFACE Crash=3D3B = >PROCEDURE Me()=3D3B END Cra=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B sh.<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B MODUL= >E Crash=3D3B<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B PROCED= >URE Me(=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B ) =3D3D<= >=3BBR>=3B&=3Bgt=3D3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN<=3B= >BR>=3B&=3Bgt=3D3B ptr^ :=3D3D=3D
>=3B >=3B>=3B>=3B>=3B>= >=3B>=3B>=3B 0<=3BBR>=3B&=3Bgt=3D3B END Me=3D3B<=3BBR>=3B&= >=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B BEGIN END Crash.<=3BBR>=3B&= >=3Bgt=3D3B<=3BBR>=3B=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B &=3Bgt=3D3B Here's an example of what happens if you now call this= > from an inter=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B acti= >ve<=3BBR>=3B&=3Bgt=3D3B interpreter that catches the exception Runti= >meError.E:<=3BBR>=3B=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= >gt=3B &=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B M-Scheme Experimental<= >=3BBR>=3B&=3Bgt=3D3B LITHP ITH LITHENING.<=3BBR>=3B&=3B=3D
&= >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B gt=3D3B&=3Bgt=3D3B (req= >uire-modules "m3")<=3BBR>=3B&=3Bgt=3D3B #t<=3BBR>=3B&=3Bgt=3D= >3B&=3Bgt=3D3B (Crash.Me=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B )<=3BBR>=3B&=3Bgt=3D3B EXCEPTION! RuntimeError! Attempt to= > reference an illegal memory=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= >=3B>=3B location.<=3BBR>=3B&=3Bgt=3D3B&=3Bgt=3D3B (+ 3 4)<=3B= >BR>=3B&=3Bgt=3D3B 7<=3BBR>=3B&=3Bgt=3D3B&=3Bgt=3D3B<=3BBR&= >gt=3B&=3Bgt=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D3= >B<=3BBR>=3B&=3Bgt=3D3B I just realized I may have broken pthreads=3D= >2C let me go back=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B a= >nd double-check it.<=3BBR>=3B&=3Bgt=3D3B runtime/POSIX and thread/PO= >SIX don't refer=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B to = >the same thing do they...<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bg= >t=3D3B Mika<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B =3D
>=3B >=3B&= >gt=3B>=3B>=3B>=3B>=3B>=3B <=3B/body>=3B
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B <=3B/html>=3B=3D
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B&= >gt=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_--
>=3B >=3B>= >=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B
>=3B &g= >t=3B>=3B
>= > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_-- From jay.krell at cornell.edu Tue Feb 22 23:08:58 2011 From: jay.krell at cornell.edu (Jay K) Date: Tue, 22 Feb 2011 22:08:58 +0000 Subject: [M3devel] SEGV mapping to RuntimeError In-Reply-To: <20110222184502.88B921A2078@async.async.caltech.edu> References: <20110219182930.BDCAB1A2078@async.async.caltech.edu>, , , , <20110219203821.2016B1A2078@async.async.caltech.edu>, , <4D604FA3.3060708@lcwb.coop>, , <20110219232729.53A991A2078@async.async.caltech.edu>, , <4D615643.7060107@lcwb.coop>, , <20110220183746.195C11A2078@async.async.caltech.edu>, , <4D61CC27.7070704@lcwb.coop>, , <20110222174405.E15481A2078@async.async.caltech.edu>, , <20110222184502.88B921A2078@async.async.caltech.edu> Message-ID: Heap vs. stack is indeed an unsolvable dilemna. Stack is much faster, but some arbitrary size must be chosen, that bothers me too, and detection and recovery from out of stack is tricky and not portable. Heap is slow, but portable, and generally limited to address space (but subject to fragmentation), easy to detect exhaustion. GC heap is usually fast to allocate, competitive with stack allocation, but the more you allocate, the more work for the GC to do. There is no free lunch -- not even as cheap as it might seem. :) Some systems..which I have never worked on..statically allocate everything. They need to know they will never run out of memory. > I realize my code is perhaps difficult to read. Not necessarily. I didn't even try yet. :) For many programmers, it is difficult to get them to read any code. :) - Jay > To: jay.krell at cornell.edu > Date: Tue, 22 Feb 2011 10:45:02 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] SEGV mapping to RuntimeError > > Jay K writes: > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >(aside=2C and possible agreement: right -- an interpreter should consider N= > >OT recursing on the machine > >stack whenever code it is interpreting recurses=2C but definitely some do) > >=20 > > - Jay > > > Jay, yes I agree---but: > > 1. of course it's easier to code the interpreter to recurse on the machine > stack > > 2. code called out from the interpreter will in any case recurse on the (a) > machine stack > > 3. again if one can "fix" the stack mechanism so that running out of > stack space is non-fatal, there's no reason not to use the machine stack > > I've found that Modula-3's garbage collector is brutally slow and that > converting code to using stack-allocated memory whenever possible is a > huge performance win under both PM3 and CM3. Perhaps this is a problem > with the garbage collector. > > I also have a partial design for a Scheme compiler (emitting Modula-3) > that would use the machine stack whenever possible rather than heap > allocation and do a "tricky thing" with local procedures to copy the > stack whenever a reference to the stack escapes. (Most of the time you > can implement Scheme on a stack but sometimes you need to keep part > of it live after you return; hence implementations (including mine) > heap-allocate local variables...) > > I realize my code is perhaps difficult to read. The gist of it is this: > when you run out of stack space, allocate a new stack anywhere else > in memory. Make it appear that you have alloca'd all the space between > the old and new stacks (into a variable that you of course never touch), > then continue. An issue is what to do when you return, you have to make > sure that you restore your old redzone.. perhaps you don't need to free > the stack eagerly: you could instead remember you have that page allocated > and re-use it next time you hit the redzone, or put it in a pool to allow > it to be used by another thread. > > The arbitrary choice of stack sizes has always bothered me greatly. > Some of my programs have thousands of threads and they use stack space > irregularly... > > Mika > > > >> To: rodney_bates at lcwb.coop > >> Date: Tue=2C 22 Feb 2011 09:44:05 -0800 > >> From: mika at async.caltech.edu > >> CC: m3devel at elegosoft.com > >> Subject: Re: [M3devel] SEGV mapping to RuntimeError > >>=20 > >>=20 > >> Ok so I was thinking about this. > >>=20 > >> Why on earth is stack overflow even a problem? > >>=20 > >> Consider the following procedure call (in my code=2C stack grows upwards)= > >: > >>=20 > >> (* sp at x=2C pc at y *) > >> y: P(args) > >> z: next_statement > >>=20 > >> decompose as follows: > >>=20 > >> (* sp at x=2C pc at y *) > >> y: Push(args etc. and ret. address z) > >> Jump(P) > >> z: next_statement > >>=20 > >> Now=2C we say: > >>=20 > >> y: ok :=3D check_stack(size of frame) > >> IF NOT ok THEN abort() END=3B > >> Push(args etc. and ret. address z) > >> Jump(P) > >> z: next_statement > >>=20 > >> (note check_stack and the following IF can be implemented by hardware=2C > >> need not actually be an instruction) > >>=20 > >> Let me change the code a tad: > >>=20 > >> y: ok :=3D check_stack(size of frame) > >> y':IF NOT ok THEN=20 > >> WITH new_stack_bottom =3D malloc(stack_size) > >> huge_amount =3D new_stack_bottom - sp DO > >> create_redzone_at(new_stack_bottom+stack_size-redzone_size) > >> EVAL alloca(huge_amount)=20 > >> END > >> END=3B > >> Push(args etc. and ret. address z) > >> Jump(P) > >> z: IF NOT ok THEN destroy_redzone(...)=3B free(new_stack_bottom) END > >>=20 > >> Note 1. cleanup of redzone could be postponed to return of caller....when > >> alloca in any case has to be cleaned up. > >>=20 > >> Note 2. the test IF NOT ok at z is more expensive to implement than the > >> one at y because you can't really use hardware for it. A hardware callbac= > >k > >> can be arranged though: > >>=20 > >> VAR ptr :=3D sp=3B > >> y: ok :=3D check_stack(size of frame) > >> y':IF NOT ok THEN=20 > >> ptr :=3D 0=3B (* illegal address *) > >> fault_address :=3D z=3B > >> WITH new_stack_bottom =3D malloc(stack_size) > >> huge_amount =3D new_stack_bottom - sp DO > >> create_redzone_at(new_stack_bottom+stack_size-redzone_size) > >> EVAL alloca(huge_amount)=20 > >> END > >> END=3B > >> Push(args etc. and ret. address z) > >> Jump(P) > >> z: EVAL ptr^ (* [ NOT ok -> hardware callback to SEGV: ] *) > >>=20 > >> SEGV(signalpc): IF NOT ok AND signalpc =3D fault_address THEN destroy_red= > >zone(...)=3B free(new_stack_bottom) END > >>=20 > >> Mika > >>=20 > >>=20 > >>=20 > >>=20 > >>=20 > >> "Rodney M. Bates" writes: > >> > > >> > > >> >On 02/20/2011 12:37 PM=2C Mika Nystrom wrote: > >> >> On a 64-bit machine=2C at least=2C there ought to be enough virtual > >> >> memory that you could just have a gap between thread stacks big > >> >> enough to allow for a protection area larger than the largest possible > >> >> (implementation-defined) activation record=2C no? I know I've run into > >> >> trouble with very large activation records in the past (and not becaus= > >e > >> >> I was running out of stack space=2C either). > >> >> > >> >> Or at least a procedure with a very large activation record (or > >> >> a procedure calling it) could be required to call some sort of check > >> >> routine "EnoughStackSpaceRemaining()" before starting to scribble > >> >> on the activation record? > >> > > >> >Hmm=2C I like this idea. It would introduce normal-case runtime overhead > >> >only for such procedures=2C and these are likely rare. Also=2C assuming = > >the procedure > >> >actually uses very much of its large AR=2C it should also have enough co= > >mputation > >> >time to wash out the stack check overhead. > >> > > >> >> > >> >> Also the end of the activation record must be written to at least once= > >=2C > >> >> or else the memory protection won't be triggered. > >> >> > >> > > >> >I was thinking (as an alternative mechanism) of having the compiler inte= > >ntionally > >> >add enough artificial write(s) as necessary to ensure storing within the > >> >red zone=2C and not just beyond it. This seems trickier to get right and > >> >harder to distinguish after the fact from a NIL dereference. > >> > > >> >> In any case if this is done properly the same mechanism I proposed for > >> >> SIGSEGV ought to be able to catch stack overflow=2C no? Well=2C as lon= > >g as > >> >> signals are delivered on a separate stack. If signals are delivered on > >> >> the same stack=2C the signal handler would get nastier=2C it would hav= > >e to > >> >> make space through some manipulations (maybe temporarily unporotecting > >> >> the redzone page?) for its own purposes... but I don't see why it > >> >> couldn't be done. > >> >> > >> >> Not sure why I'm getting SIGILL... maybe I am getting my signal handle= > >r > >> >> activated inside the redzone page because of a difference in signal > >> >> handling..? I remember reading something about sigaltstack... > >> >> > >> >> I would of course love to be able to recover from stack overflow=2C to= > >o. > >> >> In some sense=2C since it's a generally unknown limit=2C it's even les= > >s of > >> >> a fatal error than a NIL dereference (hence makes even more sense to > >> >> catch it). > >> > > >> >I think this would be a nice mechanism to have available. It would have = > >to > >> >be used with some care. In any case=2C it would be really nice and more > >> >frequently so=2C to at least have runtime error messages that distinguis= > >hed > >> >stack overflow from NIL deref. > >> > > >> >> > >> >> Mika > >> >> > >> >> "Rodney M. Bates" writes: > >> >>> I am pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AM= > >D64_LINUX. > >> >>> Probably a fully protected guard page at the end of the stack. This t= > >echnique > >> >>> always worries me a bit because a procedure with a really big activat= > >ion record > >> >>> could jump right past it. Probably it would almost always access the = > >first page > >> >>> of the big area before storing anything into later pages. > >> >>> > >> >>> On 02/19/2011 05:27 PM=2C Mika Nystrom wrote: > >> >>>> Ah=2C yes=2C stack protection. > >> >>>> > >> >>>> Do you know if it's a SIGSEGV=2C not a SIGBUS? I know I have seen SI= > >GILL on Macs. > >> >>>> > >> >>>> Hmm=2C I get SIGILL on AMD64_FREEBSD as well: > >> >>>> > >> >>>> time ../AMD64_FREEBSD/stubexample > >> >>>> M-Scheme Experimental > >> >>>> LITHP ITH LITHENING. > >> >>>>> (define (f a) (+ (f (+ a 1)) (f (+ a 2)))) > >> >>>> f > >> >>>>> (f 0) > >> >>>> Illegal instruction > >> >>>> 3.847u 0.368s 0:13.32 31.5% 2160+284478k 0+0io 0pf+0w > >> >>>> > >> >>>> What absolutely must not happen=2C of course=2C is that the runtime = > >hangs > >> >>>> while executing only safe code... > >> >>>> > >> >>>> Mika > >> >>>> > >> >>>> "Rodney M. Bates" writes: > >> >>>>> I know of one other place the compilers rely on hardware memory pro= > >tection > >> >>>>> to detect a checked runtime error=2C and that is stack overflow. Th= > >is won't > >> >>>>> corrupt anything=2C but is hard to distinguish from dereferencing N= > >IL. > >> >>>>> This could probably be distinguished after the fact by some low-lev= > >el=2C > >> >>>>> target-dependent code. I have found it by looking at assembly code = > >at > >> >>>>> the point of failure--usually right after a stack pointer push. > >> >>>>> > >> >>>>> Detecting this via compiler-generated checks would probably be more > >> >>>>> extravagant than many other checks=2C as it is so frequent. I am no= > >t > >> >>>>> aware of any really good solution to this in any implementation of = > >any > >> >>>>> language. > >> >>>>> > >> >>>>> On 02/19/2011 02:38 PM=2C Mika Nystrom wrote: > >> >>>>>> Jay=2C sometimes I wonder about you: this is a Modula-3 mailing li= > >st=2C > >> >>>>>> you know! > >> >>>>>> > >> >>>>>> "Corrupting the heap" is something that can only happen as a resul= > >t of > >> >>>>>> an unchecked runtime error. Unchecked runtime errors cannot happen= > > in > >> >>>>>> modules not marked UNSAFE. > >> >>>>>> > >> >>>>>> SEGV is=2C however=2C used by the CM3 implementation (and its pred= > >ecessors) > >> >>>>>> to signal a certain kind of *checked* runtime error=2C namely=2C t= > >he > >> >>>>>> dereferencing of a NIL reference. Correct me if I am wrong=2C but = > >an > >> >>>>>> attempt to dereference NIL is not going to leave the heap corrupte= > >d? > >> >>>>>> > >> >>>>>> And if you stick to safe code=2C the only SEGVs I think you get in= > > the > >> >>>>>> current CM3 are ones from NIL dereferences. > >> >>>>>> > >> >>>>>> Hence=2C as long as you stick with safe code=2C the only time the = > >code I > >> >>>>>> checked in earlier gets triggered is for NIL dereferences=2C which= > > should > >> >>>>>> never corrupt the heap. So SEGV is not sometimes=2C but in fact al= > >ways > >> >>>>>> recoverable. > >> >>>>>> > >> >>>>>> :-) > >> >>>>>> > >> >>>>>> Mika > >> >>>>>> > >> >>>>>> P.S. the bit above "if you stick to safe code": if you actually pr= > >ogram in > >> >>>>>> Modula-3 you almost never use UNSAFE. I went through my repository= > > and > >> >>>>>> I have 40 modules using UNSAFE out of a total of 4=2C559. Furtherm= > >ore=2C > >> >>>>>> many of the UNSAFE modules are glue code to Fortran routines=2C wh= > >ich > >> >>>>>> could relatively easily be verified to be safe in the Modula-3 sen= > >se. > >> >>>>>> Almost all what remains is glue to some C library=2C which wouldn'= > >t be > >> >>>>>> necessary if the rest of the world would wake up out of the dark a= > >ges=2C but > >> >>>>>> I don't have the time to rewrite every single library from scratch= > > myself. > >> >>>>>> > >> >>>>>> > >> >>>>>> Jay K writes: > >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >> >>>>>>> Content-Type: text/plain=3B charset=3D"iso-8859-1" > >> >>>>>>> Content-Transfer-Encoding: quoted-printable > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> Letting any code run after a SIGSEGV is dubious. > >> >>>>>>> Imagine the heap is corrupted. > >> >>>>>>> And then you run more code. > >> >>>>>>> And the code happens to call malloc. > >> >>>>>>> Or printf to log something. > >> >>>>>>> =3D20 > >> >>>>>>> I suppose there might be an application that maps memory > >> >>>>>>> gradually=3D2C as pieces of a buffer are hit. Might. > >> >>>>>>> =3D20 > >> >>>>>>> - Jay > >> >>>>>>> =3D20 > >> >>>>>>>> To: m3devel at elegosoft.com > >> >>>>>>>> Date: Sat=3D2C 19 Feb 2011 10:29:30 -0800 > >> >>>>>>>> From: mika at async.caltech.edu > >> >>>>>>>> Subject: [M3devel] SEGV mapping to RuntimeError > >> >>>>>>>> =3D20 > >> >>>>>>>> =3D20 > >> >>>>>>>> Dear m3devel=3D2C > >> >>>>>>>> =3D20 > >> >>>>>>>> For a while it has annoyed me that segmentation violations cause= > > an > >> >>>>>>>> unconditional program abort. I've changed that now so that (unde= > >r user > >> >>>>>>>> threads at least) we instead get a RuntimeError. Here's an examp= > >le of > >> >>>>>>>> the mechanism at work in an interactive Scheme environment. Cons= > >ider > >> >>>>>>>> the unhelpful interface and module Crash: > >> >>>>>>>> =3D20 > >> >>>>>>>> INTERFACE Crash=3D3B PROCEDURE Me()=3D3B END Crash. > >> >>>>>>>> =3D20 > >> >>>>>>>> MODULE Crash=3D3B > >> >>>>>>>> =3D20 > >> >>>>>>>> PROCEDURE Me() =3D3D > >> >>>>>>>> VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN > >> >>>>>>>> ptr^ :=3D3D 0 > >> >>>>>>>> END Me=3D3B > >> >>>>>>>> =3D20 > >> >>>>>>>> BEGIN END Crash. > >> >>>>>>>> =3D20 > >> >>>>>>>> Here's an example of what happens if you now call this from an i= > >nteractiv=3D > >> >>>>>>> e > >> >>>>>>>> interpreter that catches the exception RuntimeError.E: > >> >>>>>>>> =3D20 > >> >>>>>>>> M-Scheme Experimental > >> >>>>>>>> LITHP ITH LITHENING. > >> >>>>>>>>> (require-modules "m3") > >> >>>>>>>> #t > >> >>>>>>>>> (Crash.Me) > >> >>>>>>>> EXCEPTION! RuntimeError! Attempt to reference an illegal memory = > >location. > >> >>>>>>>>> (+ 3 4)=3D20 > >> >>>>>>>> 7 > >> >>>>>>>>> =3D20 > >> >>>>>>>> =3D20 > >> >>>>>>>> I just realized I may have broken pthreads=3D2C let me go back a= > >nd double-c=3D > >> >>>>>>> heck it.=3D20 > >> >>>>>>>> runtime/POSIX and thread/POSIX don't refer to the same thing do = > >they... > >> >>>>>>>> =3D20 > >> >>>>>>>> Mika > >> >>>>>>>> =3D20 > >> >>>>>>> =3D > >> >>>>>>> > >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_ > >> >>>>>>> Content-Type: text/html=3B charset=3D"iso-8859-1" > >> >>>>>>> Content-Transfer-Encoding: quoted-printable > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> Letting any code run after a SIGSEGV is dubious.
> >> >>>>>>> Imagine the heap =3D3Bis corrupted.
> >> >>>>>>> And then you run more code.
> >> >>>>>>> And the code happens to call malloc.
> >> >>>>>>> Or printf to log something.
> >> >>>>>>>  =3D3B
> >> >>>>>>> I suppose there might be an application that maps memory
> >> >>>>>>> gradually=3D2C as pieces of a buffer are hit. Might.
> >> >>>>>>>  =3D3B
> >> >>>>>>>  =3D3B- Jay
 =3D3B
> >> >>>>>>> >=3D3B To: m3devel at elegosoft.com
>=3D3B Date: Sat=3D2C 19 = > >Feb 2011 10:29:3=3D > >> >>>>>>> 0 -0800
>=3D3B From: mika at async.caltech.edu
>=3D3B Subj= > >ect: [M3devel]=3D > >> >>>>>>> SEGV mapping to RuntimeError
>=3D3B
>=3D3B
>=3D3B = > >Dear m3devel=3D > >> >>>>>>> =3D2C
>=3D3B
>=3D3B For a while it has annoyed me that = > >segmentation vi=3D > >> >>>>>>> olations cause an
>=3D3B unconditional program abort. I've c= > >hanged that =3D > >> >>>>>>> now so that (under user
>=3D3B threads at least) we instead = > >get a Runtim=3D > >> >>>>>>> eError. Here's an example of
>=3D3B the mechanism at work in= > > an interact=3D > >> >>>>>>> ive Scheme environment. Consider
>=3D3B the unhelpful interf= > >ace and modu=3D > >> >>>>>>> le Crash:
>=3D3B
>=3D3B INTERFACE Crash=3D3B PROCEDURE = > >Me()=3D3B END Cra=3D > >> >>>>>>> sh.
>=3D3B
>=3D3B MODULE Crash=3D3B
>=3D3B
>= > >=3D3B PROCEDURE Me(=3D > >> >>>>>>> ) =3D3D
>=3D3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN >R>>=3D3B ptr^ :=3D3D=3D > >> >>>>>>> 0
>=3D3B END Me=3D3B
>=3D3B
>=3D3B BEGIN END Crash= > >.
>=3D3B
=3D > >> >>>>>>> >=3D3B Here's an example of what happens if you now call this f= > >rom an inter=3D > >> >>>>>>> active
>=3D3B interpreter that catches the exception Runtime= > >Error.E:
=3D > >> >>>>>>> >=3D3B
>=3D3B M-Scheme Experimental
>=3D3B LITHP ITH = > >LITHENING.
&=3D > >> >>>>>>> gt=3D3B>=3D3B (require-modules "m3")
>=3D3B #t
>=3D3B= > >>=3D3B (Crash.Me=3D > >> >>>>>>> )
>=3D3B EXCEPTION! RuntimeError! Attempt to reference an il= > >legal memory=3D > >> >>>>>>> location.
>=3D3B>=3D3B (+ 3 4)
>=3D3B 7
>=3D3B&g= > >t=3D3B
>=3D > >> >>>>>>> =3D3B
>=3D3B I just realized I may have broken pthreads=3D2C= > > let me go back=3D > >> >>>>>>> and double-check it.
>=3D3B runtime/POSIX and thread/POSIX d= > >on't refer=3D > >> >>>>>>> to the same thing do they...
>=3D3B
>=3D3B Mika
>= > >=3D3B
=3D > >> >>>>>>> > >> >>>>>>> =3D > >> >>>>>>> > >> >>>>>>> --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_-- > >> >>>>>> > >> >>>> > >> >> > > = > > > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_ > >Content-Type: text/html; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > > > > > > > >(aside=2C and possible agreement: right -- an interpreter should consider N= > >OT recursing on the machine
> >stack whenever code it is interpreting recurses=2C but definitely some do)<= > >BR> > > =3B
> > =3B- Jay
 =3B
> >>=3B To: rodney_bates at lcwb.coop
>=3B Date: Tue=2C 22 Feb 2011 09:44:= > >05 -0800
>=3B From: mika at async.caltech.edu
>=3B CC: m3devel at elego= > >soft.com
>=3B Subject: Re: [M3devel] SEGV mapping to RuntimeError
&= > >gt=3B
>=3B
>=3B Ok so I was thinking about this.
>=3B
= > >>=3B Why on earth is stack overflow even a problem?
>=3B
>=3B = > >Consider the following procedure call (in my code=2C stack grows upwards):<= > >BR>>=3B
>=3B (* sp at x=2C pc at y *)
>=3B y: P(args)
>= > >=3B z: next_statement
>=3B
>=3B decompose as follows:
>=3B = > >
>=3B (* sp at x=2C pc at y *)
>=3B y: Push(args etc. and ret. ad= > >dress z)
>=3B Jump(P)
>=3B z: next_statement
>=3B
>=3B= > > Now=2C we say:
>=3B
>=3B y: ok :=3D check_stack(size of frame)<= > >BR>>=3B IF NOT ok THEN abort() END=3B
>=3B Push(args etc. and ret. a= > >ddress z)
>=3B Jump(P)
>=3B z: next_statement
>=3B
>= > >=3B (note check_stack and the following IF can be implemented by hardware= > >=2C
>=3B need not actually be an instruction)
>=3B
>=3B Let= > > me change the code a tad:
>=3B
>=3B y: ok :=3D check_stack(size= > > of frame)
>=3B y':IF NOT ok THEN
>=3B WITH new_stack_bottom =3D= > > malloc(stack_size)
>=3B huge_amount =3D new_stack_bottom - sp DO
&= > >gt=3B create_redzone_at(new_stack_bottom+stack_size-redzone_size)
>=3B= > > EVAL alloca(huge_amount)
>=3B END
>=3B END=3B
>=3B Push(ar= > >gs etc. and ret. address z)
>=3B Jump(P)
>=3B z: IF NOT ok THEN d= > >estroy_redzone(...)=3B free(new_stack_bottom) END
>=3B
>=3B Note= > > 1. cleanup of redzone could be postponed to return of caller....when
&g= > >t=3B alloca in any case has to be cleaned up.
>=3B
>=3B Note 2. = > >the test IF NOT ok at z is more expensive to implement than the
>=3B o= > >ne at y because you can't really use hardware for it. A hardware callback >R>>=3B can be arranged though:
>=3B
>=3B VAR ptr :=3D sp=3B >>>=3B y: ok :=3D check_stack(size of frame)
>=3B y':IF NOT ok THEN <= > >BR>>=3B ptr :=3D 0=3B (* illegal address *)
>=3B fault_address :=3D = > >z=3B
>=3B WITH new_stack_bottom =3D malloc(stack_size)
>=3B huge_= > >amount =3D new_stack_bottom - sp DO
>=3B create_redzone_at(new_stack_b= > >ottom+stack_size-redzone_size)
>=3B EVAL alloca(huge_amount)
>= > >=3B END
>=3B END=3B
>=3B Push(args etc. and ret. address z)
&g= > >t=3B Jump(P)
>=3B z: EVAL ptr^ (* [ NOT ok ->=3B hardware callback t= > >o SEGV: ] *)
>=3B
>=3B SEGV(signalpc): IF NOT ok AND signalpc = > >=3D fault_address THEN destroy_redzone(...)=3B free(new_stack_bottom) END >R>>=3B
>=3B Mika
>=3B
>=3B
>=3B
>=3B
>= > >=3B
>=3B "Rodney M. Bates" writes:
>=3B >=3B
>=3B >=3B<= > >BR>>=3B >=3BOn 02/20/2011 12:37 PM=2C Mika Nystrom wrote:
>=3B >= > >=3B>=3B On a 64-bit machine=2C at least=2C there ought to be enough virtu= > >al
>=3B >=3B>=3B memory that you could just have a gap between thr= > >ead stacks big
>=3B >=3B>=3B enough to allow for a protection area= > > larger than the largest possible
>=3B >=3B>=3B (implementation-de= > >fined) activation record=2C no? I know I've run into
>=3B >=3B>=3B= > > trouble with very large activation records in the past (and not because >>>=3B >=3B>=3B I was running out of stack space=2C either).
>=3B= > > >=3B>=3B
>=3B >=3B>=3B Or at least a procedure with a very la= > >rge activation record (or
>=3B >=3B>=3B a procedure calling it) co= > >uld be required to call some sort of check
>=3B >=3B>=3B routine "= > >EnoughStackSpaceRemaining()" before starting to scribble
>=3B >=3B&g= > >t=3B on the activation record?
>=3B >=3B
>=3B >=3BHmm=2C I li= > >ke this idea. It would introduce normal-case runtime overhead
>=3B >= > >=3Bonly for such procedures=2C and these are likely rare. Also=2C assuming = > >the procedure
>=3B >=3Bactually uses very much of its large AR=2C it= > > should also have enough computation
>=3B >=3Btime to wash out the s= > >tack check overhead.
>=3B >=3B
>=3B >=3B>=3B
>=3B >= > >=3B>=3B Also the end of the activation record must be written to at least= > > once=2C
>=3B >=3B>=3B or else the memory protection won't be trig= > >gered.
>=3B >=3B>=3B
>=3B >=3B
>=3B >=3BI was thinki= > >ng (as an alternative mechanism) of having the compiler intentionally
&g= > >t=3B >=3Badd enough artificial write(s) as necessary to ensure storing wi= > >thin the
>=3B >=3Bred zone=2C and not just beyond it. This seems tri= > >ckier to get right and
>=3B >=3Bharder to distinguish after the fact= > > from a NIL dereference.
>=3B >=3B
>=3B >=3B>=3B In any cas= > >e if this is done properly the same mechanism I proposed for
>=3B >= > >=3B>=3B SIGSEGV ought to be able to catch stack overflow=2C no? Well=2C a= > >s long as
>=3B >=3B>=3B signals are delivered on a separate stack.= > > If signals are delivered on
>=3B >=3B>=3B the same stack=2C the s= > >ignal handler would get nastier=2C it would have to
>=3B >=3B>=3B = > >make space through some manipulations (maybe temporarily unporotecting
&= > >gt=3B >=3B>=3B the redzone page?) for its own purposes... but I don't s= > >ee why it
>=3B >=3B>=3B couldn't be done.
>=3B >=3B>=3B >R>>=3B >=3B>=3B Not sure why I'm getting SIGILL... maybe I am getting= > > my signal handler
>=3B >=3B>=3B activated inside the redzone page= > > because of a difference in signal
>=3B >=3B>=3B handling..? I rem= > >ember reading something about sigaltstack...
>=3B >=3B>=3B
>= > >=3B >=3B>=3B I would of course love to be able to recover from stack ov= > >erflow=2C too.
>=3B >=3B>=3B In some sense=2C since it's a general= > >ly unknown limit=2C it's even less of
>=3B >=3B>=3B a fatal error = > >than a NIL dereference (hence makes even more sense to
>=3B >=3B>= > >=3B catch it).
>=3B >=3B
>=3B >=3BI think this would be a nic= > >e mechanism to have available. It would have to
>=3B >=3Bbe used wit= > >h some care. In any case=2C it would be really nice and more
>=3B >= > >=3Bfrequently so=2C to at least have runtime error messages that distinguis= > >hed
>=3B >=3Bstack overflow from NIL deref.
>=3B >=3B
>= > >=3B >=3B>=3B
>=3B >=3B>=3B Mika
>=3B >=3B>=3B
>= > >=3B >=3B>=3B "Rodney M. Bates" writes:
>=3B >=3B>=3B>=3B I a= > >m pretty sure the cases I've seen are SIGSEGV on LINUXLIBC6 and AMD64_LINUX= > >.
>=3B >=3B>=3B>=3B Probably a fully protected guard page at the= > > end of the stack. This technique
>=3B >=3B>=3B>=3B always worri= > >es me a bit because a procedure with a really big activation record
>= > >=3B >=3B>=3B>=3B could jump right past it. Probably it would almost a= > >lways access the first page
>=3B >=3B>=3B>=3B of the big area be= > >fore storing anything into later pages.
>=3B >=3B>=3B>=3B
>= > >=3B >=3B>=3B>=3B On 02/19/2011 05:27 PM=2C Mika Nystrom wrote:
>= > >=3B >=3B>=3B>=3B>=3B Ah=2C yes=2C stack protection.
>=3B >= > >=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B Do you know if it'= > >s a SIGSEGV=2C not a SIGBUS? I know I have seen SIGILL on Macs.
>=3B &= > >gt=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B Hmm=2C I get SIG= > >ILL on AMD64_FREEBSD as well:
>=3B >=3B>=3B>=3B>=3B
>=3B = > >>=3B>=3B>=3B>=3B time ../AMD64_FREEBSD/stubexample
>=3B >=3B= > >>=3B>=3B>=3B M-Scheme Experimental
>=3B >=3B>=3B>=3B>=3B= > > LITHP ITH LITHENING.
>=3B >=3B>=3B>=3B>=3B>=3B (define (f a= > >) (+ (f (+ a 1)) (f (+ a 2))))
>=3B >=3B>=3B>=3B>=3B f
>= > >=3B >=3B>=3B>=3B>=3B>=3B (f 0)
>=3B >=3B>=3B>=3B>=3B= > > Illegal instruction
>=3B >=3B>=3B>=3B>=3B 3.847u 0.368s 0:13.= > >32 31.5% 2160+284478k 0+0io 0pf+0w
>=3B >=3B>=3B>=3B>=3B
&g= > >t=3B >=3B>=3B>=3B>=3B What absolutely must not happen=2C of course= > >=2C is that the runtime hangs
>=3B >=3B>=3B>=3B>=3B while exec= > >uting only safe code...
>=3B >=3B>=3B>=3B>=3B
>=3B >=3B= > >>=3B>=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B
>=3B >=3B= > >>=3B>=3B>=3B "Rodney M. Bates" writes:
>=3B >=3B>=3B>=3B&g= > >t=3B>=3B I know of one other place the compilers rely on hardware memory = > >protection
>=3B >=3B>=3B>=3B>=3B>=3B to detect a checked run= > >time error=2C and that is stack overflow. This won't
>=3B >=3B>=3B= > >>=3B>=3B>=3B corrupt anything=2C but is hard to distinguish from dere= > >ferencing NIL.
>=3B >=3B>=3B>=3B>=3B>=3B This could probably= > > be distinguished after the fact by some low-level=2C
>=3B >=3B>= > >=3B>=3B>=3B>=3B target-dependent code. I have found it by looking at = > >assembly code at
>=3B >=3B>=3B>=3B>=3B>=3B the point of fail= > >ure--usually right after a stack pointer push.
>=3B >=3B>=3B>=3B= > >>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B Detecting this via co= > >mpiler-generated checks would probably be more
>=3B >=3B>=3B>=3B= > >>=3B>=3B extravagant than many other checks=2C as it is so frequent. I = > >am not
>=3B >=3B>=3B>=3B>=3B>=3B aware of any really good so= > >lution to this in any implementation of any
>=3B >=3B>=3B>=3B>= > >=3B>=3B language.
>=3B >=3B>=3B>=3B>=3B>=3B
>=3B >= > >=3B>=3B>=3B>=3B>=3B On 02/19/2011 02:38 PM=2C Mika Nystrom wrote: >R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B Jay=2C sometimes I wonder abo= > >ut you: this is a Modula-3 mailing list=2C
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B you know!
>=3B >=3B>=3B>=3B>=3B>=3B>=3B >>>=3B >=3B>=3B>=3B>=3B>=3B>=3B "Corrupting the heap" is somet= > hing that can only happen as a result of
>=3B >=3B>=3B>=3B>=3B= > >>=3B>=3B an unchecked runtime error. Unchecked runtime errors cannot ha= > >ppen in
>=3B >=3B>=3B>=3B>=3B>=3B>=3B modules not marked U= > >NSAFE.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B SEGV is=2C however=2C used by the CM3 implement= > >ation (and its predecessors)
>=3B >=3B>=3B>=3B>=3B>=3B>=3B= > > to signal a certain kind of *checked* runtime error=2C namely=2C the
&g= > >t=3B >=3B>=3B>=3B>=3B>=3B>=3B dereferencing of a NIL reference.= > > Correct me if I am wrong=2C but an
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B attempt to dereference NIL is not going to leave the heap corrupt= > >ed?
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B&g= > >t=3B>=3B>=3B>=3B And if you stick to safe code=2C the only SEGVs I th= > >ink you get in the
>=3B >=3B>=3B>=3B>=3B>=3B>=3B current C= > >M3 are ones from NIL dereferences.
>=3B >=3B>=3B>=3B>=3B>=3B= > >>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B Hence=2C as long as y= > >ou stick with safe code=2C the only time the code I
>=3B >=3B>=3B&= > >gt=3B>=3B>=3B>=3B checked in earlier gets triggered is for NIL derefe= > >rences=2C which should
>=3B >=3B>=3B>=3B>=3B>=3B>=3B never= > > corrupt the heap. So SEGV is not sometimes=2C but in fact always
>=3B= > > >=3B>=3B>=3B>=3B>=3B>=3B recoverable.
>=3B >=3B>=3B&g= > >t=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B :-) >R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B&= > >gt=3B>=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B>=3B>=3B
&g= > >t=3B >=3B>=3B>=3B>=3B>=3B>=3B P.S. the bit above "if you stick = > >to safe code": if you actually program in
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B Modula-3 you almost never use UNSAFE. I went through my rep= > >ository and
>=3B >=3B>=3B>=3B>=3B>=3B>=3B I have 40 module= > >s using UNSAFE out of a total of 4=2C559. Furthermore=2C
>=3B >=3B&g= > >t=3B>=3B>=3B>=3B>=3B many of the UNSAFE modules are glue code to Fo= > >rtran routines=2C which
>=3B >=3B>=3B>=3B>=3B>=3B>=3B coul= > >d relatively easily be verified to be safe in the Modula-3 sense.
>=3B= > > >=3B>=3B>=3B>=3B>=3B>=3B Almost all what remains is glue to so= > >me C library=2C which wouldn't be
>=3B >=3B>=3B>=3B>=3B>=3B&= > >gt=3B necessary if the rest of the world would wake up out of the dark ages= > >=2C but
>=3B >=3B>=3B>=3B>=3B>=3B>=3B I don't have the tim= > >e to rewrite every single library from scratch myself.
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B<= > >BR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B Jay K writes:
>=3B >= > >=3B>=3B>=3B>=3B>=3B>=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912= > >854f_
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Content-Type: te= > >xt/plain=3B charset=3D"iso-8859-1"
>=3B >=3B>=3B>=3B>=3B>=3B= > >>=3B>=3B Content-Transfer-Encoding: quoted-printable
>=3B >=3B&g= > >t=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B= > >>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Letting a= > >ny code run after a SIGSEGV is dubious.
>=3B >=3B>=3B>=3B>=3B&= > >gt=3B>=3B>=3B Imagine the heap is corrupted.
>=3B >=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B And then you run more code.
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B And the code happens to call malloc.
&= > >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Or printf to log something= > >.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>=3B >= > >=3B>=3B>=3B>=3B>=3B>=3B>=3B I suppose there might be an applica= > >tion that maps memory
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >gradually=3D2C as pieces of a buffer are hit. Might.
>=3B >=3B>=3B= > >>=3B>=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B - Jay
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B To: m3deve= > >l at elegosoft.com
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >Date: Sat=3D2C 19 Feb 2011 10:29:30 -0800
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B From: mika at async.caltech.edu
>=3B >=3B&g= > >t=3B>=3B>=3B>=3B>=3B>=3B>=3B Subject: [M3devel] SEGV mapping to= > > RuntimeError
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
&= > >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B Dear m3devel=3D2C >>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>=3B &g= > >t=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B For a while it has annoyed m= > >e that segmentation violations cause an
>=3B >=3B>=3B>=3B>=3B&= > >gt=3B>=3B>=3B>=3B unconditional program abort. I've changed that now = > >so that (under user
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B threads at least) we instead get a RuntimeError. Here's an example of >R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B the mechanism at = > >work in an interactive Scheme environment. Consider
>=3B >=3B>=3B&= > >gt=3B>=3B>=3B>=3B>=3B>=3B the unhelpful interface and module Cras= > >h:
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20
>= > >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B INTERFACE Crash=3D3B P= > >ROCEDURE Me()=3D3B END Crash.
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B MODULE Crash=3D3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B = > >PROCEDURE Me() =3D3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&g= > >t=3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN
>=3B >=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B ptr^ :=3D3D 0
>=3B >=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B END Me=3D3B
>=3B >=3B>=3B>=3B&= > >gt=3B>=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B BEGIN END Crash.
>=3B >=3B>=3B>=3B>=3B&g= > >t=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B= > >>=3B>=3B Here's an example of what happens if you now call this from an= > > interactiv=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B e
&g= > >t=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B interpreter that catc= > >hes the exception RuntimeError.E:
>=3B >=3B>=3B>=3B>=3B>=3B&= > >gt=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B M-Scheme Experimental
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B LITHP ITH LITHENING.
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B (require-modules "m3")
>=3B >=3B>=3B&g= > >t=3B>=3B>=3B>=3B>=3B>=3B #t
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B (Crash.Me)
>=3B >=3B>=3B>=3B>=3B&g= > >t=3B>=3B>=3B>=3B EXCEPTION! RuntimeError! Attempt to reference an ill= > >egal memory location.
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B>=3B (+ 3 4)=3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B 7
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D20<= > >BR>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B I just realized = > >I may have broken pthreads=3D2C let me go back and double-c=3D
>=3B &g= > >t=3B>=3B>=3B>=3B>=3B>=3B>=3B heck it.=3D20
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B>=3B runtime/POSIX and thread/POSIX don'= > >t refer to the same thing do they...
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B>=3B Mika
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B>=3B= > > =3D20
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D
>=3B &= > >gt=3B>=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_
>=3B &g= > >t=3B>=3B>=3B>=3B>=3B>=3B>=3B Content-Type: text/html=3B charset= > >=3D"iso-8859-1"
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Conten= > >t-Transfer-Encoding: quoted-printable
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3B= > >html>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3Bhead&= > >gt=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B <=3Bstyle>= > >=3B<=3B!--
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B .hmmessag= > >e P
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B {
>=3B >=3B= > >>=3B>=3B>=3B>=3B>=3B>=3B margin:0px=3D3B
>=3B >=3B>=3B= > >>=3B>=3B>=3B>=3B>=3B padding:0px
>=3B >=3B>=3B>=3B>= > >=3B>=3B>=3B>=3B }
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>= > >=3B body.hmmessage
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B { >R>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B font-size: 10pt=3D3B >>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B font-family:Tahoma
&g= > >t=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B }
>=3B >=3B>=3B>= > >=3B>=3B>=3B>=3B>=3B -->=3B<=3B/style>=3B
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B <=3B/head>=3B
>=3B >=3B>=3B&= > >gt=3B>=3B>=3B>=3B>=3B <=3Bbody class=3D3D'hmmessage'>=3B
>= > >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B Letting any code run after a= > > SIGSEGV is dubious.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B= > >>=3B>=3B Imagine the heap&=3Bnbsp=3D3Bis corrupted.<=3BBR>=3B >>>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B And then you run more co= > >de.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B And = > >the code happens to call malloc.<=3BBR>=3B
>=3B >=3B>=3B>=3B= > >>=3B>=3B>=3B>=3B Or printf to log something.<=3BBR>=3B
>= > >=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B &=3Bnbsp=3D3B<=3BBR>= > >=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B I suppose there mi= > >ght be an application that maps memory<=3BBR>=3B
>=3B >=3B>=3B= > >>=3B>=3B>=3B>=3B>=3B gradually=3D2C as pieces of a buffer are hit= > >. Might.<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B= > > &=3Bnbsp=3D3B<=3BBR>=3B
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B &=3Bnbsp=3D3B- Jay<=3BBR>=3B&=3Bnbsp=3D3B<=3BBR>=3B= > >
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B &=3Bgt=3D3B To: m3= > >devel at elegosoft.com<=3BBR>=3B&=3Bgt=3D3B Date: Sat=3D2C 19 Feb 2011 = > >10:29:3=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B 0 -0800<= > >=3BBR>=3B&=3Bgt=3D3B From: mika at async.caltech.edu<=3BBR>=3B&=3B= > >gt=3D3B Subject: [M3devel]=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B SEGV mapping to RuntimeError<=3BBR>=3B&=3Bgt=3D3B<=3BBR&= > >gt=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B Dear m3devel=3D
>=3B &= > >gt=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D2C<=3BBR>=3B&=3Bgt=3D3B= > ><=3BBR>=3B&=3Bgt=3D3B For a while it has annoyed me that segmentatio= > >n vi=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B olations cause= > > an<=3BBR>=3B&=3Bgt=3D3B unconditional program abort. I've changed t= > >hat =3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B now so that (u= > >nder user<=3BBR>=3B&=3Bgt=3D3B threads at least) we instead get a Ru= > >ntim=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B eError. Here's= > > an example of<=3BBR>=3B&=3Bgt=3D3B the mechanism at work in an inte= > >ract=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B ive Scheme env= > >ironment. Consider<=3BBR>=3B&=3Bgt=3D3B the unhelpful interface and = > >modu=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B le Crash:<= > >=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B INTERFACE Crash=3D3B = > >PROCEDURE Me()=3D3B END Cra=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B sh.<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B MODUL= > >E Crash=3D3B<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B PROCED= > >URE Me(=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B ) =3D3D<= > >=3BBR>=3B&=3Bgt=3D3B VAR ptr : REF INTEGER :=3D3D NIL=3D3B BEGIN<=3B= > >BR>=3B&=3Bgt=3D3B ptr^ :=3D3D=3D
>=3B >=3B>=3B>=3B>=3B>= > >=3B>=3B>=3B 0<=3BBR>=3B&=3Bgt=3D3B END Me=3D3B<=3BBR>=3B&= > >=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B BEGIN END Crash.<=3BBR>=3B&= > >=3Bgt=3D3B<=3BBR>=3B=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B &=3Bgt=3D3B Here's an example of what happens if you now call this= > > from an inter=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B acti= > >ve<=3BBR>=3B&=3Bgt=3D3B interpreter that catches the exception Runti= > >meError.E:<=3BBR>=3B=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B&= > >gt=3B &=3Bgt=3D3B<=3BBR>=3B&=3Bgt=3D3B M-Scheme Experimental<= > >=3BBR>=3B&=3Bgt=3D3B LITHP ITH LITHENING.<=3BBR>=3B&=3B=3D
&= > >gt=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B gt=3D3B&=3Bgt=3D3B (req= > >uire-modules "m3")<=3BBR>=3B&=3Bgt=3D3B #t<=3BBR>=3B&=3Bgt=3D= > >3B&=3Bgt=3D3B (Crash.Me=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B )<=3BBR>=3B&=3Bgt=3D3B EXCEPTION! RuntimeError! Attempt to= > > reference an illegal memory=3D
>=3B >=3B>=3B>=3B>=3B>=3B>= > >=3B>=3B location.<=3BBR>=3B&=3Bgt=3D3B&=3Bgt=3D3B (+ 3 4)<=3B= > >BR>=3B&=3Bgt=3D3B 7<=3BBR>=3B&=3Bgt=3D3B&=3Bgt=3D3B<=3BBR&= > >gt=3B&=3Bgt=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B =3D3= > >B<=3BBR>=3B&=3Bgt=3D3B I just realized I may have broken pthreads=3D= > >2C let me go back=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B a= > >nd double-check it.<=3BBR>=3B&=3Bgt=3D3B runtime/POSIX and thread/PO= > >SIX don't refer=3D
>=3B >=3B>=3B>=3B>=3B>=3B>=3B>=3B to = > >the same thing do they...<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B&=3Bg= > >t=3D3B Mika<=3BBR>=3B&=3Bgt=3D3B<=3BBR>=3B =3D
>=3B >=3B&= > >gt=3B>=3B>=3B>=3B>=3B>=3B <=3B/body>=3B
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B <=3B/html>=3B=3D
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B>=3B&= > >gt=3B>=3B --_a2a24b92-3b4c-456e-ab1b-c3f5e912854f_--
>=3B >=3B>= > >=3B>=3B>=3B>=3B>=3B
>=3B >=3B>=3B>=3B>=3B
>=3B &g= > >t=3B>=3B
> >= > > > >--_ab466c75-f74a-4983-8fec-4f513f45fe0b_-- -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Wed Feb 23 17:42:06 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Wed, 23 Feb 2011 08:42:06 -0800 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" Message-ID: <20110223164206.BE6931A2078@async.async.caltech.edu> Notes: 1. the Modula-3 garbage collector gets confused (shoulda thought of that), so I have to turn it off 2. however if the GC were fixed, there's no need to "fix up" the stack after the return---you can just allocate the new stacks as GC-traced: even though the collector is compacting they are "pinned" because they are referenced from the thread stack! 3. alloca is built into gcc nowadays, so I had to track down an assembly version: I suspect that the compiler front-end could be modified to emit a call to alloca and thus get gcc to generate the same code, obviating the need for any assembly code at all. (* Unsafe interface ***************************************************) INTERFACE Unsafe; FROM Ctypes IMPORT int; PROCEDURE DoIt(); <*EXTERNAL alloca*> PROCEDURE alloca(size : int) : INTEGER; END Unsafe. (* Unsafe implementation **********************************************) UNSAFE MODULE Unsafe; IMPORT Thread, IO, Fmt; IMPORT RTCollector, Random; EXCEPTION Quit; VAR rand := NEW(Random.Default).init(); PROCEDURE P(cl : Closure; arg : ARRAY[0..100] OF INTEGER) RAISES { Quit } = CONST Fudge = 1024; VAR stack : REF ARRAY OF INTEGER := NIL; BEGIN IO.Put(Fmt.F("Thread %s GetSP()=0x%s\n", Fmt.Int(cl.id), Fmt.Int(GetSP(), base := 16)) ); (*Thread.Pause(0.1d0);*) IF GetSP() < cl.lo + Fudge THEN IF cl.stacks >= rand.integer(3,10) THEN IO.Put(Fmt.F("Thread %s quitting after %s stacks\n", Fmt.Int(cl.id), Fmt.Int(cl.stacks))); RAISE Quit END; stack := NEW(REF ARRAY OF INTEGER, 8*1024); WITH lo = LOOPHOLE(ADR(stack[0]),INTEGER), hi = LOOPHOLE(ADR(stack[LAST(stack^)]),INTEGER), sp = GetSP() DO IO.Put(Fmt.F("Thread %s sp %s new stack lo %s hi %s\n", Fmt.Int(cl.id), Fmt.Int(sp, base := 16), Fmt.Int(lo, base := 16), Fmt.Int(hi, base:= 16))); (* new stack starts sp - hi away *) EVAL alloca(sp-hi); IO.Put(Fmt.F("After alloca, sp %s\n", Fmt.Int(GetSP(),base := 16))); cl.lo := lo; INC(cl.stacks) END END; P(cl, arg) END P; TYPE Closure = Thread.Closure OBJECT id : CARDINAL; lo : INTEGER; stacks : CARDINAL; OVERRIDES apply := Apply END; PROCEDURE Apply(cl : Closure) : REFANY = BEGIN Thread.Pause(1.0d0); LOOP TRY cl.lo := GetSP() - 1024*8; (* pretend we have 8k left *) cl.stacks := 0; P(cl, ARRAY [0..100] OF INTEGER { 0, .. }); EXCEPT Quit => (* start over *) END END END Apply; PROCEDURE DoIt() = BEGIN RTCollector.Disable(); FOR i := 1 TO 2 DO EVAL Thread.Fork(NEW(Closure, id := i)) END; LOOP Thread.Pause(1.0d0) END END DoIt; PROCEDURE GetSP() : INTEGER = BEGIN RETURN alloca(1) END GetSP; BEGIN END Unsafe. (* Main can't be UNSAFE in Modula-3 ***********************************) MODULE Main; IMPORT Unsafe; BEGIN Unsafe.DoIt() END Main. (* alloca.s from FreeBSD (old code from BSD/386) **********************) /*- * Copyright (c) 1990 The Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ .asciz "@(#)alloca.s 5.2 (Berkeley) 5/14/90" #include /* like alloc, but automatic automatic free in return */ .globl alloca .type alloca, @function alloca: popl %edx /* pop return addr */ popl %eax /* pop amount to allocate */ movl %esp,%ecx addl $3,%eax /* round up to next word */ andl $0xfffffffc,%eax subl %eax,%esp movl %esp,%eax /* base of newly allocated space */ pushl 8(%ecx) /* copy possible saved registers */ pushl 4(%ecx) pushl 0(%ecx) pushl %eax /* dummy to pop at callsite */ jmp *%edx /* "return" */ From jay.krell at cornell.edu Wed Feb 23 20:53:42 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 23 Feb 2011 19:53:42 +0000 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: <20110223164206.BE6931A2078@async.async.caltech.edu> References: <20110223164206.BE6931A2078@async.async.caltech.edu> Message-ID: Stack must be contiguous on NT. Some platforms -- NT -- actually know how big the current stack is -- and won't let you alloca past it. Best way to let the target handle making stacks is with Posix makecontext, for the many-but-not-all platforms that have it, or CreateFiber on NT. (OpenBSD and I think Darwin/amd64 lack makecontext. It is in Posix, but deprecated; heck Darwin doesn't even have Posix semaphores seemingly..) To avoid the assembly you could use a hack like: void* Unsafe__alloca(size_t n) { return alloca(n); } Beware. Dangerous platform/compiler-specific stuff here. - Jay > To: m3devel at elegosoft.com > Date: Wed, 23 Feb 2011 08:42:06 -0800 > From: mika at async.caltech.edu > Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" > > Notes: > > 1. the Modula-3 garbage collector gets confused (shoulda thought of that), > so I have to turn it off > > 2. however if the GC were fixed, there's no need to "fix up" the stack > after the return---you can just allocate the new stacks as GC-traced: > even though the collector is compacting they are "pinned" because > they are referenced from the thread stack! > > 3. alloca is built into gcc nowadays, so I had to track down an assembly > version: I suspect that the compiler front-end could be modified to > emit a call to alloca and thus get gcc to generate the same code, > obviating the need for any assembly code at all. > > > (* Unsafe interface ***************************************************) > > INTERFACE Unsafe; > FROM Ctypes IMPORT int; > > PROCEDURE DoIt(); > > <*EXTERNAL alloca*> > PROCEDURE alloca(size : int) : INTEGER; > > END Unsafe. > > (* Unsafe implementation **********************************************) > > UNSAFE MODULE Unsafe; > > IMPORT Thread, IO, Fmt; > IMPORT RTCollector, Random; > > EXCEPTION Quit; > > VAR rand := NEW(Random.Default).init(); > > PROCEDURE P(cl : Closure; arg : ARRAY[0..100] OF INTEGER) RAISES { Quit } = > CONST Fudge = 1024; > VAR stack : REF ARRAY OF INTEGER := NIL; > BEGIN > IO.Put(Fmt.F("Thread %s GetSP()=0x%s\n", > Fmt.Int(cl.id), > Fmt.Int(GetSP(), base := 16)) > ); > (*Thread.Pause(0.1d0);*) > > IF GetSP() < cl.lo + Fudge THEN > IF cl.stacks >= rand.integer(3,10) THEN > IO.Put(Fmt.F("Thread %s quitting after %s stacks\n", > Fmt.Int(cl.id), Fmt.Int(cl.stacks))); > RAISE Quit > END; > > stack := NEW(REF ARRAY OF INTEGER, 8*1024); > WITH lo = LOOPHOLE(ADR(stack[0]),INTEGER), > hi = LOOPHOLE(ADR(stack[LAST(stack^)]),INTEGER), > sp = GetSP() DO > IO.Put(Fmt.F("Thread %s sp %s new stack lo %s hi %s\n", > Fmt.Int(cl.id), > Fmt.Int(sp, base := 16), > Fmt.Int(lo, base := 16), > Fmt.Int(hi, base:= 16))); > > (* new stack starts sp - hi away *) > EVAL alloca(sp-hi); > IO.Put(Fmt.F("After alloca, sp %s\n", > Fmt.Int(GetSP(),base := 16))); > cl.lo := lo; > INC(cl.stacks) > END > END; > > P(cl, arg) > END P; > > TYPE > Closure = Thread.Closure OBJECT > id : CARDINAL; > lo : INTEGER; > stacks : CARDINAL; > OVERRIDES > apply := Apply > END; > > PROCEDURE Apply(cl : Closure) : REFANY = > BEGIN > Thread.Pause(1.0d0); > > LOOP > TRY > cl.lo := GetSP() - 1024*8; (* pretend we have 8k left *) > cl.stacks := 0; > P(cl, ARRAY [0..100] OF INTEGER { 0, .. }); > EXCEPT > Quit => (* start over *) > END > END > END Apply; > > PROCEDURE DoIt() = > BEGIN > RTCollector.Disable(); > > FOR i := 1 TO 2 DO > EVAL Thread.Fork(NEW(Closure, id := i)) > END; > > LOOP Thread.Pause(1.0d0) END > END DoIt; > > PROCEDURE GetSP() : INTEGER = BEGIN RETURN alloca(1) END GetSP; > > BEGIN END Unsafe. > > (* Main can't be UNSAFE in Modula-3 ***********************************) > > MODULE Main; > IMPORT Unsafe; > > BEGIN Unsafe.DoIt() END Main. > > (* alloca.s from FreeBSD (old code from BSD/386) **********************) > > /*- > * Copyright (c) 1990 The Regents of the University of California. > * All rights reserved. > * > * This code is derived from software contributed to Berkeley by > * William Jolitz. > * > * Redistribution and use in source and binary forms, with or without > * modification, are permitted provided that the following conditions > * are met: > * 1. Redistributions of source code must retain the above copyright > * notice, this list of conditions and the following disclaimer. > * 2. Redistributions in binary form must reproduce the above copyright > * notice, this list of conditions and the following disclaimer in the > * documentation and/or other materials provided with the distribution. > * 3. All advertising materials mentioning features or use of this software > * must display the following acknowledgement: > * This product includes software developed by the University of > * California, Berkeley and its contributors. > * 4. Neither the name of the University nor the names of its contributors > * may be used to endorse or promote products derived from this software > * without specific prior written permission. > * > * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND > * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE > * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE > * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL > * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS > * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) > * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT > * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY > * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > * SUCH DAMAGE. > */ > > .asciz "@(#)alloca.s 5.2 (Berkeley) 5/14/90" > #include > > > /* like alloc, but automatic automatic free in return */ > > .globl alloca > .type alloca, @function > alloca: > popl %edx /* pop return addr */ > popl %eax /* pop amount to allocate */ > movl %esp,%ecx > addl $3,%eax /* round up to next word */ > andl $0xfffffffc,%eax > subl %eax,%esp > movl %esp,%eax /* base of newly allocated space */ > pushl 8(%ecx) /* copy possible saved registers */ > pushl 4(%ecx) > pushl 0(%ecx) > pushl %eax /* dummy to pop at callsite */ > jmp *%edx /* "return" */ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Wed Feb 23 21:39:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Wed, 23 Feb 2011 12:39:50 -0800 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: References: <20110223164206.BE6931A2078@async.async.caltech.edu> Message-ID: <20110223203950.30ABF1A2078@async.async.caltech.edu> Jay K writes: >--_60ddb4ae-bef3-432c-a5fd-a46c878ce951_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Stack must be contiguous on NT. >Some platforms -- NT -- actually know how big the current stack is -- and w= >on't let you >alloca past it. Best way to let the target handle making stacks is with Pos= >ix makecontext=2C >for the many-but-not-all platforms that have it=2C or CreateFiber on NT. >(OpenBSD and I think Darwin/amd64 lack makecontext. It is in Posix=2C but d= >eprecated=3B >heck Darwin doesn't even have Posix semaphores seemingly..) The problem with making stacks with makecontext, etc., to me seems to be: what do you do when you want to return to the original stack. Can it be made to "just work"? Note that in my design, the stacks are all contiguous---but overlapping! There are no "new" stacks and no switching of stacks. You just pretend you have a huge data structure sitting where the other guys' stacks (and data, etc.) are. Cleanup is also easy, it's handled by the ordinary GC. > >To avoid the assembly you could use a hack like: >void* Unsafe__alloca(size_t n) { return alloca(n)=3B } Won't work. alloca releases memory at the }. What would avoid it is generating the call to alloca in the front end. > >Beware. >Dangerous platform/compiler-specific stuff here. Not really that specific. It relies on precisely one special property of the underlying system: that alloca doesn't do any error checking. The only changes that need to be made to CM3 are coming up with a mechanism for detecting the need for expanding the stack (could be done by bumping the redzone before winding the stack and handling this in a SEGV handler) and modifying the garbage collector to understand the scheme. I suggest adding a back pointer at the beginning of the newly allocated area that would let the GC go back to the previous one while scanning (or however it works). Users/platforms that don't need or can't support "leapfrog stacks" would not change except that their garbage collectors would have some code that's never executed... I've discussed the above with people who've been doing concurrent programming since before the glaciers came thru (implementations of Concurrent Pascal in the 1970s) and they said that they used linked (noncontiguous) stacks in those days. The state of the art of 2011 seems rather unimpressive... My feeling is that the stack overflow issue is one of the things hindering the acceptance of truly multi-threaded programming (thousands, tens of thousands, millions of threads). Mika From jay.krell at cornell.edu Wed Feb 23 21:51:35 2011 From: jay.krell at cornell.edu (Jay K) Date: Wed, 23 Feb 2011 20:51:35 +0000 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: <20110223203950.30ABF1A2078@async.async.caltech.edu> References: <20110223164206.BE6931A2078@async.async.caltech.edu>, , <20110223203950.30ABF1A2078@async.async.caltech.edu> Message-ID: > that alloca doesn't do any error checking No. There is also native exception handling to consider. NT essentially does error checking in both. > thousands, tens of thousands, millions of threads - 64bit address spaces - multiple machines ("cloud") > Won't work. alloca releases memory at the }. Only sort of. You can still return that point and still use the memory. You do risk, e.g. if you make any function calls, then the alloca memory and the function parameters might overlap. You could..platform-dependent..probably skip the first "few" bytes of the alloca memory. > Note that in my design, the stacks are all contiguous---but overlapping! Are your stacks also traversing heap and what-not, or are you just chopping up the current stack into lots of smaller stacks? i.e. alloca stack size times n and then create n threads? That's less bad. I had the impression you were skipping over other memory. You know, like, malloc your stack, and then alloca to get the stack pointer to hit the malloced memory. > What would avoid it is generating the call to alloca in the front end. We could w/o much difficulty provide alloca in the frontend. Probably limit it to unsafe modules. At least to support your experimenting and somewhat platform-dependent code. > in the 1970s.. I think the right place to experiment with this stuff is at a lower level. Or with more complete control of the system. ie. in the kernel, or on only one specific kernel, that you build/modify/write-from-scratch yourself. Gotta go, - Jay > To: jay.krell at cornell.edu > Date: Wed, 23 Feb 2011 12:39:50 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" > > Jay K writes: > >--_60ddb4ae-bef3-432c-a5fd-a46c878ce951_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > >Stack must be contiguous on NT. > >Some platforms -- NT -- actually know how big the current stack is -- and w= > >on't let you > >alloca past it. Best way to let the target handle making stacks is with Pos= > >ix makecontext=2C > >for the many-but-not-all platforms that have it=2C or CreateFiber on NT. > >(OpenBSD and I think Darwin/amd64 lack makecontext. It is in Posix=2C but d= > >eprecated=3B > >heck Darwin doesn't even have Posix semaphores seemingly..) > > The problem with making stacks with makecontext, etc., to me seems > to be: what do you do when you want to return to the original stack. > Can it be made to "just work"? > > Note that in my design, the stacks are all contiguous---but overlapping! > There are no "new" stacks and no switching of stacks. You just pretend > you have a huge data structure sitting where the other guys' stacks > (and data, etc.) are. > > Cleanup is also easy, it's handled by the ordinary GC. > > > > >To avoid the assembly you could use a hack like: > >void* Unsafe__alloca(size_t n) { return alloca(n)=3B } > > Won't work. alloca releases memory at the }. > > What would avoid it is generating the call to alloca in the front end. > > > > >Beware. > >Dangerous platform/compiler-specific stuff here. > > Not really that specific. It relies on precisely one special property > of the underlying system: that alloca doesn't do any error checking. > > The only changes that need to be made to CM3 are coming up with a > mechanism for detecting the need for expanding the stack (could be done by > bumping the redzone before winding the stack and handling this in a SEGV > handler) and modifying the garbage collector to understand the scheme. > I suggest adding a back pointer at the beginning of the newly allocated > area that would let the GC go back to the previous one while scanning > (or however it works). > > Users/platforms that don't need or can't support "leapfrog stacks" > would not change except that their garbage collectors would have some > code that's never executed... > > I've discussed the above with people who've been doing concurrent > programming since before the glaciers came thru (implementations of > Concurrent Pascal in the 1970s) and they said that they used linked > (noncontiguous) stacks in those days. The state of the art of 2011 > seems rather unimpressive... > > My feeling is that the stack overflow issue is one of the things hindering > the acceptance of truly multi-threaded programming (thousands, tens of > thousands, millions of threads). > > Mika > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Thu Feb 24 03:20:29 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Wed, 23 Feb 2011 18:20:29 -0800 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: References: <20110223164206.BE6931A2078@async.async.caltech.edu>, , <20110223203950.30ABF1A2078@async.async.caltech.edu> Message-ID: <20110224022029.B016E1A2078@async.async.caltech.edu> Jay K writes: >--_6d651072-194c-4975-9e55-045a636fc1ed_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > > > that alloca doesn't do any error checking > > >No. There is also native exception handling to consider. >NT essentially does error checking in both. Native exception handling? What do you mean? > > >> thousands=2C tens of thousands=2C millions of threads > > - 64bit address spaces > - multiple machines ("cloud") >=20 > > > Won't work. alloca releases memory at the }. >=20 >=20 > Only sort of. > You can still return that point and still use the memory. > You do risk=2C e.g. if you make any function calls=2C then the > alloca memory and the function parameters might overlap. > You could..platform-dependent..probably skip the first > "few" bytes of the alloca memory. Well that's the whole point of the alloca, that it moves the stack pointer beyond the danger region...(and keeps it there till you return) if you let it come back ... it defeats the purpose. > > > > Note that in my design=2C the stacks are all contiguous---but overlappin= >g! >=20 > Are your stacks also traversing heap and what-not=2C > or are you just chopping up the current stack into lots > of smaller stacks? Yes traverses heap, protected pages, I/O buffers, text segments,... whatever junk is in between the old stack and the newly allocated area. >=20 > i.e. alloca stack size times n and then create n threads? > That's less bad. >=20 > I had the impression you were skipping over other memory. >=20 > You know=2C like=2C malloc your stack=2C and then alloca to get > the stack pointer to hit the malloced memory. That's exactly right. malloc a new stack and use alloca to fiddle the stack pointer so it lands at the start of the new stack. Now, I don't know a darn thing about Windows, but I've been looking at Unix code and I see the following: * all alloca does is it modifies the stack pointer. It does this from user code without calling any sort of operating system or other runtime routines. * nothing special is done to "stacks" in Modula-3's "POSIX" user threading library---threads are allocated with a NEW(UNTRACED REF...) using the normal Modula-3 memory management routines. * the concept of "stack" (another one of those EWD inventions, incidentally) itself is a bit of a misnomer. There is only the stack pointer + conventions on how you use it. The "stack" is simply the area between where the thread began execution and the current value of the stack pointer. It's a matter of definition, simply. > > > What would avoid it is generating the call to alloca in the front end. >=20 >We could w/o much difficulty provide alloca in the frontend. >Probably limit it to unsafe modules. >At least to support your experimenting and somewhat platform-dependent code= >. Sure that would be fun :) > > > in the 1970s.. >=20 >=20 >I think the right place to experiment with this stuff is at a lower level. >Or with more complete control of the system. >ie. in the kernel=2C or on only one specific kernel=2C that you >build/modify/write-from-scratch yourself. Let me put it like this: if Modula-3's "user threads" work on architecture/OS combination X, I think you can just about (perhaps you can actually) prove that "leapfrog stacks" would work on the same architecture/OS combination, just based on an analysis of how the user threads allocate and use their stacks. But if user threads don't work (say Windows?) then it probably won't work. We do know, however, that user threads work just about everywhere... Mika > > >Gotta go=2C > - Jay > > > > > > From jay.krell at cornell.edu Thu Feb 24 11:09:33 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 24 Feb 2011 10:09:33 +0000 Subject: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" In-Reply-To: <20110224022029.B016E1A2078@async.async.caltech.edu> References: <20110223164206.BE6931A2078@async.async.caltech.edu>, , , , <20110223203950.30ABF1A2078@async.async.caltech.edu>, , <20110224022029.B016E1A2078@async.async.caltech.edu> Message-ID: > But if user threads don't work (say Windows?) then it probably won't > work. We do know, however, that user threads work just about everywhere. They don't. One could use fibers if one really wanted to do one's own scheduling, and locking. (user threads implies writing your own locks). On NT, alloca touches every page between the current and the result. (not every byte, that'd be wasteful, just every page) Touching the end of the stack generates stack overflow exceptions. So you can't go past it. That is, the very problem people were worried about, is solved on NT. "Native exception handling" is what e.g. C++ uses. I believe it validates the stack to be in range. In particular, on NT, the stack range is available somewhere, and occasionally used. - Jay > To: jay.krell at cornell.edu > Date: Wed, 23 Feb 2011 18:20:29 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] Avoding thread-stack overflow, proof of concept of "leapfrog stacks" > > Jay K writes: > >--_6d651072-194c-4975-9e55-045a636fc1ed_ > >Content-Type: text/plain; charset="iso-8859-1" > >Content-Transfer-Encoding: quoted-printable > > > > > > > that alloca doesn't do any error checking > > > > > >No. There is also native exception handling to consider. > >NT essentially does error checking in both. > > Native exception handling? What do you mean? > > > > > > >> thousands=2C tens of thousands=2C millions of threads > > > > - 64bit address spaces > > - multiple machines ("cloud") > >=20 > > > > > Won't work. alloca releases memory at the }. > >=20 > >=20 > > Only sort of. > > You can still return that point and still use the memory. > > You do risk=2C e.g. if you make any function calls=2C then the > > alloca memory and the function parameters might overlap. > > You could..platform-dependent..probably skip the first > > "few" bytes of the alloca memory. > > Well that's the whole point of the alloca, that it moves the stack > pointer beyond the danger region...(and keeps it there till you return) > if you let it come back ... it defeats the purpose. > > > > > > > > Note that in my design=2C the stacks are all contiguous---but overlappin= > >g! > >=20 > > Are your stacks also traversing heap and what-not=2C > > or are you just chopping up the current stack into lots > > of smaller stacks? > > > Yes traverses heap, protected pages, I/O buffers, text segments,... whatever > junk is in between the old stack and the newly allocated area. > > > >=20 > > i.e. alloca stack size times n and then create n threads? > > That's less bad. > >=20 > > I had the impression you were skipping over other memory. > >=20 > > You know=2C like=2C malloc your stack=2C and then alloca to get > > the stack pointer to hit the malloced memory. > > That's exactly right. malloc a new stack and use alloca to fiddle > the stack pointer so it lands at the start of the new stack. > > Now, I don't know a darn thing about Windows, but I've been looking at > Unix code and I see the following: > > * all alloca does is it modifies the stack pointer. It does this from > user code without calling any sort of operating system or other runtime > routines. > > * nothing special is done to "stacks" in Modula-3's "POSIX" user threading > library---threads are allocated with a NEW(UNTRACED REF...) using the > normal Modula-3 memory management routines. > > * the concept of "stack" (another one of those EWD inventions, > incidentally) itself is a bit of a misnomer. There is only the > stack pointer + conventions on how you use it. The "stack" is simply > the area between where the thread began execution and the current > value of the stack pointer. It's a matter of definition, simply. > > > > > > What would avoid it is generating the call to alloca in the front end. > >=20 > >We could w/o much difficulty provide alloca in the frontend. > >Probably limit it to unsafe modules. > >At least to support your experimenting and somewhat platform-dependent code= > >. > > Sure that would be fun :) > > > > > > in the 1970s.. > >=20 > >=20 > >I think the right place to experiment with this stuff is at a lower level. > >Or with more complete control of the system. > >ie. in the kernel=2C or on only one specific kernel=2C that you > >build/modify/write-from-scratch yourself. > > Let me put it like this: if Modula-3's "user threads" work on > architecture/OS combination X, I think you can just about (perhaps > you can actually) prove that "leapfrog stacks" would work on the same > architecture/OS combination, just based on an analysis of how the user > threads allocate and use their stacks. But if user threads don't work > (say Windows?) then it probably won't work. We do know, however, that > user threads work just about everywhere... > > Mika > > > > > > >Gotta go=2C > > - Jay > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Thu Feb 24 20:16:58 2011 From: dragisha at m3w.org (=?utf-8?Q?Dragi=C5=A1a_Duri=C4=87?=) Date: Thu, 24 Feb 2011 20:16:58 +0100 Subject: [M3devel] new target? :) Message-ID: http://code.google.com/chrome/nativeclient/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Thu Feb 24 22:46:17 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Thu, 24 Feb 2011 21:46:17 +0000 (GMT) Subject: [M3devel] new target? :) In-Reply-To: Message-ID: <758561.47161.qm@web29710.mail.ird.yahoo.com> Hi all: is interesting anyway but to write the web apps in c and good friends, hum ..., let me think if I can write a web server in less than 100 loc of Obliq and indeed one is written as an os extension, why would need any web page that renders c code, I think I would gain much more time writing myself rather than porting C (of course with its flavors) in Modula-3 or better if so in Obliq which is a mobile computing environment indeed, it makes me think, though the idea is good it makes little sense to make everything a client of a web browser rather than an instance of a structured platform, if they need only C why it has to be an instance of a web browser, it must be the browser makes better work for quick dynamic pages with C, or maybe another way of keeping it alive artificially, I mean what is the gain with that, if better runtime but, hum ...? but? it needs javascript, hum ... another loss of time (much time in inefficient checking), Nokia had the same idea with Qt and JS in WRT, (very little RT information), but still Obliq scripts can be type inferenced in much more smoothly way. I think this battle of JavaScript and Obliq and Modula-3 vs others could be fight yet one time again but needs more time for tools to make that a reality, still the others think they can do it, it's very interesting. Thanks in advance --- El jue, 24/2/11, Dragi?a Duri? escribi?: De: Dragi?a Duri? Asunto: [M3devel] new target? :) Para: "m3devel" Fecha: jueves, 24 de febrero, 2011 14:16 http://code.google.com/chrome/nativeclient/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Thu Feb 24 22:58:17 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 24 Feb 2011 21:58:17 +0000 Subject: [M3devel] new target? :) In-Reply-To: <758561.47161.qm@web29710.mail.ird.yahoo.com> References: , <758561.47161.qm@web29710.mail.ird.yahoo.com> Message-ID: I believe nativeclient depends on a fork of gcc. And I'm not sure if it always 32bit x86 or not. Part of what they do is limit the output such as to be easily disassembled and analyzed at load/runtime. If that is all, we could probably just take their patches. There is a paper about it.. I don't know what you link to though. Ultimately..you can tell what I'm going to say..a good solution is Modula-3-to-C, and then use whatever C compiler you want. Including the nativeclient gcc. An iPhone/iPad/iPod cross compiler. Older stuff like Irix/AIX/HPUX that we have no Hudson nodes for. NT/AMD64, NT/IA64. Android. Newer stuff that hasn't come along yet. Basically, generating C or C++ is a great portability mechanism, to old, existing, and new platforms. A bit better than depending on gcc. On Daniel's point, there's an ongoing race to provide the best new development environments that do or do not allow much code reuse. That are or are not "safe". not safe: iPhone, iPad, iPod, native for Android safe: Java on Android, Silverlight in web browser and Windows Phone 7, Go, JavaScript Those platforms have way more mindshare, support, tooling, libraries, than Modula-3 and I'm sure they always will. > let me think if I can write a web server in less than 100 loc It takes 5 lines in Python, as it is already builtin to the libraries -- confusing language and libraries, granted. - Jay Date: Thu, 24 Feb 2011 21:46:17 +0000 From: dabenavidesd at yahoo.es To: m3devel at elegosoft.com; dragisha at m3w.org Subject: Re: [M3devel] new target? :) Hi all: is interesting anyway but to write the web apps in c and good friends, hum ..., let me think if I can write a web server in less than 100 loc of Obliq and indeed one is written as an os extension, why would need any web page that renders c code, I think I would gain much more time writing myself rather than porting C (of course with its flavors) in Modula-3 or better if so in Obliq which is a mobile computing environment indeed, it makes me think, though the idea is good it makes little sense to make everything a client of a web browser rather than an instance of a structured platform, if they need only C why it has to be an instance of a web browser, it must be the browser makes better work for quick dynamic pages with C, or maybe another way of keeping it alive artificially, I mean what is the gain with that, if better runtime but, hum ...? but it needs javascript, hum ... another loss of time (much time in inefficient checking), Nokia had the same idea with Qt and JS in WRT, (very little RT information), but still Obliq scripts can be type inferenced in much more smoothly way. I think this battle of JavaScript and Obliq and Modula-3 vs others could be fight yet one time again but needs more time for tools to make that a reality, still the others think they can do it, it's very interesting. Thanks in advance --- El jue, 24/2/11, Dragi?a Duri? escribi?: De: Dragi?a Duri? Asunto: [M3devel] new target? :) Para: "m3devel" Fecha: jueves, 24 de febrero, 2011 14:16 http://code.google.com/chrome/nativeclient/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Thu Feb 24 23:15:51 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Thu, 24 Feb 2011 14:15:51 -0800 Subject: [M3devel] new target? :) In-Reply-To: References: , <758561.47161.qm@web29710.mail.ird.yahoo.com> Message-ID: <20110224221551.8B9271A2078@async.async.caltech.edu> Jay K writes: ... >Ultimately..you can tell what I'm going to say..a good solution is Modula-3= >-to-C=2C and then use whatever C compiler you want. >Including the nativeclient gcc. An iPhone/iPad/iPod cross compiler. Older s= I'm still curious why they abandoned this path. The first M3 compilers all generated C code. I read something about a Modula-2 compiler that was generating C code and therefore had problems with.. something. Also isn't the back-end m3cg actually quite an interesting program in its own right? You could imagine targeting other language front-ends to that back-end. I understand it is a good enough idea that RMS was very annoyed that m3cg even exists. Mika From jay.krell at cornell.edu Fri Feb 25 00:09:39 2011 From: jay.krell at cornell.edu (Jay K) Date: Thu, 24 Feb 2011 23:09:39 +0000 Subject: [M3devel] new target? :) In-Reply-To: <20110224221551.8B9271A2078@async.async.caltech.edu> References: , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , <20110224221551.8B9271A2078@async.async.caltech.edu> Message-ID: I never used the system, but one can imagine and it has been said here: slower to compile less good debugging -- and also better debugging! However I still think the large gain in portability, and the actual *gain* in debuggability, would be very worthwhile. (There are pluses and minuses for debugging; you'd get to use gdb instead of m3gdb, and have C syntax). You'd gain portability and ports to various real, old, existing, new systems. e.g. NT/IA64, NT/AMD64, Android, NativeClient. You'd gain a "standard" distribution format of just source.tar.gz + possibly ./configure + make + make install. The that's a little more work. I think there's some issue with the metadata for garbage collection. I think we maybe store frame offsets for locals? We'd maybe have to put all locals in a struct or such? Impeding optimization? Similarly, nested functions would need some handling. You would also gain tremendously better exeption handling implementation. You'd generate C on NT/VMS/OSF1, and use __try/__except/__finally there. You'd generate C++ for all others and use try/catch/finally. You'd get stack walking on essentially all platforms. Except for the small number of platforms that use setjmp/longjmp for C++ exception handling. In some cases you'd get better codegen, as gcc is often not the best. RMS's problem is that normally the gcc frontends and backends are statically linked. frontends: C, C++, Java, Ada Therefore the frontends get infected by the GPL. Good in RMS opinion. m3cg is, technically, a gcc frontend, and therefore GPL. However its input is not human authored text. Its input is the output of a "real" front end -- m3front/cm3. By making the "call mechanism" be write to a file and run a process, the GPL doesn't cross the barrier, leaving cm3/m3front, and more importantly, m3core/libm3, not GPL. I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. - Jay > To: jay.krell at cornell.edu > Date: Thu, 24 Feb 2011 14:15:51 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] new target? :) > > Jay K writes: > ... > >Ultimately..you can tell what I'm going to say..a good solution is Modula-3= > >-to-C=2C and then use whatever C compiler you want. > >Including the nativeclient gcc. An iPhone/iPad/iPod cross compiler. Older s= > > I'm still curious why they abandoned this path. The first M3 compilers > all generated C code. I read something about a Modula-2 compiler that > was generating C code and therefore had problems with.. something. > > Also isn't the back-end m3cg actually quite an interesting program in > its own right? You could imagine targeting other language front-ends > to that back-end. I understand it is a good enough idea that RMS was > very annoyed that m3cg even exists. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 25 00:12:21 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Thu, 24 Feb 2011 23:12:21 +0000 (GMT) Subject: [M3devel] new target? :) In-Reply-To: <20110224221551.8B9271A2078@async.async.caltech.edu> Message-ID: <281816.90569.qm@web29709.mail.ird.yahoo.com> Hi all: yeah, but others had very distinct opinions in tis own right with two experimental projects: http://sandbox.parc.com/parctab/csl9501/node6.html and same house: http://www.cl.cam.ac.uk/research/origami/Origami1997c/ and prior another one which had experience with Trestle http://www.usenix.org/publications/library/proceedings/mobile93/kantarjiev.html So even with the space limitations the systems run in Modula-3 technically better than any other implementation for its computing environment. Perhaps that is the point he computing paradigm and specially performance is vastly superior than those days to run something like Modula-3, but maybe more easy for Obliq nowadays. Thanks in advance --- El jue, 24/2/11, Mika Nystrom escribi?: > De: Mika Nystrom > Asunto: Re: [M3devel] new target? :) > Para: "Jay K" > CC: m3devel at elegosoft.com > Fecha: jueves, 24 de febrero, 2011 17:15 > Jay K writes: > ... > >Ultimately..you can tell what I'm going to say..a good > solution is Modula-3= > >-to-C=2C and then use whatever C compiler you want. > >Including the nativeclient gcc. An iPhone/iPad/iPod > cross compiler. Older s= > > I'm still curious why they abandoned this path. The > first M3 compilers > all generated C code. I read something about a > Modula-2 compiler that > was generating C code and therefore had problems with.. > something. > > Also isn't the back-end m3cg actually quite an interesting > program in > its own right? You could imagine targeting other > language front-ends > to that back-end. I understand it is a good enough > idea that RMS was > very annoyed that m3cg even exists. > > Mika > From mika at async.caltech.edu Fri Feb 25 02:31:00 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Thu, 24 Feb 2011 17:31:00 -0800 Subject: [M3devel] new target? :) In-Reply-To: References: , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , <20110224221551.8B9271A2078@async.async.caltech.edu> Message-ID: <20110225013100.619CD1A2078@async.async.caltech.edu> Jay K writes: ... >m3cg is=2C technically=2C a gcc frontend=2C and therefore GPL. >However its input is not human authored text. >Its input is the output of a "real" front end -- m3front/cm3. >By making the "call mechanism" be write to a file and run a process=2C the = >GPL doesn't cross the barrier=2C >leaving cm3/m3front=2C and more importantly=2C m3core/libm3=2C not GPL. >=20 >=20 >I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. >=20 >=20 Good points all about C but I wonder about what is really lost. Also ...... Modula-3 is supposed to be portable to systems without C. You're supposed to be able to write an entire system in Modula-3. Ok generating C obviously doesn't break that feature but you might have to worry a bit about having too much C sneak into the runtime libraries. Well given that CM3 is released under what is more or less an MIT/BSD license, I think the problem is rather that Some Evil Software Company could write a completely non-free front-end and use m3cg to generate code... Mika From jay.krell at cornell.edu Fri Feb 25 02:55:52 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 25 Feb 2011 01:55:52 +0000 Subject: [M3devel] new target? :) In-Reply-To: <20110225013100.619CD1A2078@async.async.caltech.edu> References: , , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , <20110225013100.619CD1A2078@async.async.caltech.edu> Message-ID: When you write that system that doesn't have the support of any C compiler, sure, go ahead and write a backend just for it, and maybe a linker, and rewrite the small parts that depend on C and Posix in some sort of native way -- including writing all the underlying stuff. Getting Modula-3 working on that system will be a relatively small task, compared to whatever else is needed to build the system. Note for example, we can't e.g. open files in Modula-3, w/o underlying Win32 or Posix functionality. Everywhere that varies for Win32 vs. Posix would have to be revisited in this hypothetical system. This has little/no bearing on the vast vast vast vast majority of real world systems. The current Modula-3 system is NOT implemented entirely in Modula-3. It MUST depend on SOMETHING. So far, for the vast vast vast vast vast majority of systems, that something is written in C, and has a C runtime, compiler, and linker. Yes, I've heard of SPIN. You know..how are you actually going to network I/O or keyboard I/O or power management or video I/O or mouse I/O or sound I/O or preemptive multi-processor scheduling? You need device drivers. You need a processor-specific context switcher. i.e. you need a kernel and you need C AND/OR you need to do a lot of work. If you consider that our system is already forked along Win32 and Posix lines, you can consider these hypothetical systems a third (fourth, fifth, etc.) fork. A third (fourth) implementation of threading. A third implementation of file I/O. A third implementation of Trestle. etc. But this has little/no bearing how we should target existing real systems. You can say, not that "Modula-3 depends on C (runtime, compiler, linker)", but that "Modula-3 on Linux, NT, *BSD, Solaris, Darwin depends on C (runtime, compiler, linker)", and there's absolutely nothing wrong with that, as C and C++ are well supported on those systems and always will be. Increasing that dependency, on those systems, is fine. It would be nice to port the integrated backend to lots more systems, and to write our own linker. But not easy. And notice how the integrated backend also has the worst debugging story of the two/three options, currently. (I say three options because m3cg kind of has two modes, one is incomplete and currently disabled, the other requires m3gdb). One problem I see here, is that, you kind of sort of maybe forseeable want a backend that outputs Java or C#. C# particularly for Windows Phone 7/Nokia. Java kind of for Android, where Java is preferred over C/C++. i.e. there are systems that sort of lack C and C++. Not that they replace it with Modula-3 though. However this isn't super easy, like if you are to include the existing m3core/garbage collector. C# is optionally safe. I don't know if it would serve as a good target. (Optional safety imho makes C# among one of the closest languages to Modula-3. It is missing "compile to native", but that maybe is a good thing.) In addition, C# and Java do each compile down to one well documented "object" file format, so maybe skipping the text is probably sensible. Esp. since with C# there are in-proc libraries for codegen. More likely, if a Java/C# target were really needed, it'd include forking within m3core and replace the garbage collector. This is all highly specuative stuff. - Jay > To: jay.krell at cornell.edu > Date: Thu, 24 Feb 2011 17:31:00 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] new target? :) > > Jay K writes: > ... > >m3cg is=2C technically=2C a gcc frontend=2C and therefore GPL. > >However its input is not human authored text. > >Its input is the output of a "real" front end -- m3front/cm3. > >By making the "call mechanism" be write to a file and run a process=2C the = > >GPL doesn't cross the barrier=2C > >leaving cm3/m3front=2C and more importantly=2C m3core/libm3=2C not GPL. > >=20 > >=20 > >I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. > >=20 > >=20 > > Good points all about C but I wonder about what is really lost. Also ...... > Modula-3 is supposed to be portable to systems without C. You're supposed > to be able to write an entire system in Modula-3. Ok generating C obviously > doesn't break that feature but you might have to worry a bit about having > too much C sneak into the runtime libraries. > > Well given that CM3 is released under what is more or less an MIT/BSD license, > I think the problem is rather that Some Evil Software Company could write > a completely non-free front-end and use m3cg to generate code... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Fri Feb 25 03:35:28 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Thu, 24 Feb 2011 18:35:28 -0800 Subject: [M3devel] new target? :) In-Reply-To: References: , , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , <20110225013100.619CD1A2078@async.async.caltech.edu> Message-ID: <20110225023528.A615D1A2078@async.async.caltech.edu> In general I don't disagree with anything you say. On Linux, BSD, et al., we obviously use C for low-level system-specific stuff because the system call interfaces on those systems are documented for C and not for other languages. As for OSes in M3 I wasn't thinking as much about SPIN as about Topaz. Ok so it was Modula-2+, but it was the direct inspiration for the Modula-3 project: you're supposed to be able to write device drivers and such things in Modula-3. I agree though that targeting C doesn't directly weaken the case for M3 as a systems programming language. There are more similarities between C# and M3 than "unsafe". Look at the way methods are overridden in C#... (compared to Java). I also think almost everyone who had anything to do with M3 at DEC is now at Microsoft. Mika Jay K writes: >--_4e7f3be7-1b6e-44e9-92a2-cc8c4f6b0b8c_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >When you write that system that doesn't have the support of any C compiler= >=2C sure=2C go ahead and >write a backend just for it=2C and maybe a linker=2C and rewrite the small = >parts that depend on C and Posix in some >sort of native way -- including writing all the underlying stuff. >=20 >=20 >Getting Modula-3 working on that system will be a relatively small task=2C = >compared to whatever else > is needed to build the system. >=20 >Note for example=2C we can't e.g. open files in Modula-3=2C w/o underlying = >Win32 or Posix functionality. > Everywhere that varies for Win32 vs. Posix would have to be revisited in = >this hypothetical system. >=20 >=20 >This has little/no bearing on the vast vast vast vast majority of real worl= >d systems. >The current Modula-3 system is NOT implemented entirely in Modula-3. >It MUST depend on SOMETHING. >So far=2C for the vast vast vast vast vast majority of systems=2C that some= >thing is written in C=2C >and has a C runtime=2C compiler=2C and linker. >=20 >=20 >Yes=2C I've heard of SPIN. >=20 >=20 >You know..how are you actually going to network I/O or keyboard I/O or powe= >r management >or video I/O or mouse I/O or sound I/O or preemptive multi-processor schedu= >ling? >You need device drivers. You need a processor-specific context switcher. >i.e. you need a kernel and you need C AND/OR you need to do a lot of work. >=20 >=20 >If you consider that our system is already forked along Win32 and Posix lin= >es=2C you can consider >these hypothetical systems a third (fourth=2C fifth=2C etc.) fork. A third = >(fourth) implementation >of threading. A third implementation of file I/O. A third implementation of= > Trestle. etc. >=20 >=20 >But this has little/no bearing how we should target existing real systems. > You can say=2C not that "Modula-3 depends on C (runtime=2C compiler=2C lin= >ker)"=2C but that "Modula-3 on Linux=2C NT=2C *BSD=2C Solaris=2C Darwin >depends on C (runtime=2C compiler=2C linker)"=2C and there's absolutely not= >hing wrong with that=2C as C and C++ >are well supported on those systems and always will be. Increasing that dep= >endency=2C on those systems=2C is fine. >=20 >=20 >It would be nice to port the integrated backend to lots more systems=2C and= > to write our own linker. >But not easy. >And notice how the integrated backend also has the worst debugging story of= > the two/three options=2C currently. > (I say three options because m3cg kind of has two modes=2C one is incomple= >te and currently disabled=2C > the other requires m3gdb). >=20 >=20 >One problem I see here=2C is that=2C you kind of sort of maybe forseeable w= >ant a backend that outputs Java or C#. > C# particularly for Windows Phone 7/Nokia. > Java kind of for Android=2C where Java is preferred over C/C++. >=20 >=20 >i.e. there are systems that sort of lack C and C++. Not that they replace i= >t with Modula-3 though. >=20 >=20 >However this isn't super easy=2C like if you are to include the existing m3= >core/garbage collector. >C# is optionally safe. I don't know if it would serve as a good target. > (Optional safety imho makes C# among one of the closest languages to Modu= >la-3. It is > missing "compile to native"=2C but that maybe is a good thing.) >In addition=2C C# and Java do each compile down to one well documented "obj= >ect" file format=2C >so maybe skipping the text is probably sensible. Esp. since with C# there a= >re in-proc libraries for codegen. >=20 >=20 >More likely=2C if a Java/C# target were really needed=2C it'd include forki= >ng within m3core=20 >and replace the garbage collector. This is all highly specuative stuff. >=20 >=20 > - Jay > From jay.krell at cornell.edu Fri Feb 25 04:09:01 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 25 Feb 2011 03:09:01 +0000 Subject: [M3devel] new target? :) In-Reply-To: <20110225023528.A615D1A2078@async.async.caltech.edu> References: , , ,,<758561.47161.qm@web29710.mail.ird.yahoo.com>, ,,, , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , , , <20110225013100.619CD1A2078@async.async.caltech.edu>, , <20110225023528.A615D1A2078@async.async.caltech.edu> Message-ID: > I also think almost everyone who had anything to do with M3 at DEC is now at Microsoft Not just M3. M3 wasn't many people. Lots of people at Microsoft were at Digital. > Modula-3 project: you're supposed to be able to write device drivers > and such things in Modula-3. I agree though that targeting C doesn't Understood, agreed -- you can hypothetically write anything in Modula-3. The optional safety is fairly unique and important. Except for C# and Modula-3, all languages seem to be strictly safe or unsafe. I think, you can imagine, either an entire system in Modula-3, with m3core/libm3 coded "directly" in Modula-3 and simply statically or dynamically linked using "standard mechanism", or if there is still a user/kernel boundary, you either have to hack the compiler a bit, or resort to some assembly (possibly embedded in .c files). That is, calling the kernel is inevitably a special call sequence, and cm3 doesn't currently understand anything like that. About device drivers -- a critical piece is the runtime. Imagine writing an NT or Linux driver. Well, if you separate language from library...and limit language features to accomodate missing runtime...and retarget small parts of the runtime perhaps.. you know, there's a certain isomorphism? You can write almost anything in C or Modula-3 and most things will look very similar and map one-to-one. I think I read that Modula-3 can describe anything C can, except unions. You know, don't expect to have garbage collection in the kernel. Maybe not exception handling. Not necessarily the file I/O you are used to. But something. Not the threading you are used to. But something (at least on NT). But certainly you can interface Modula-3 to the NTkernel/Linux/*BSD runtimes, just like we interface to Win32/Posix. I'm not sure it'd buy much. I guess if you really like the syntax... It's another area where compiling to C would help. esp. for the exception handling on NT! __try/__finally are available in C on NT, usermode and kernel, and the implementation is much more efficient than what Modula-3 does currently. I really need to stop saying all this stuff though, either be quiet or do it... - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From dabenavidesd at yahoo.es Fri Feb 25 15:20:48 2011 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Fri, 25 Feb 2011 14:20:48 +0000 (GMT) Subject: [M3devel] new target? :) In-Reply-To: Message-ID: <669680.12232.qm@web29711.mail.ird.yahoo.com> Hi all: It was that there is some OS in Modula-2+ which run enterely in user space and you didn't want to get into the kernel but just for any option you would need to be unsafe (like machine starting, perhaps the OS managed software faults isolated from kernel space) that had the Runtime to time-out protect deadlocks, and it did it pretty much in a microkernel, just that company didn't feel sure o fsuch sophisticated system. Nowadays would be good if one could cooperate it in ARM VM for web platform developing, without the need for? GUI kernel, which indeed could crash the system if exploted, instead feels safer to run from "microkernel" traceable user space threads but making calls to a sort of video controller see below. Someone has the sources of it, just don't how complete, they did develop a UIMS and a window toolkit, plenty of tools to check: http://lists.cloud9.co.uk/pipermail/bbc-micro/2010-January/007795.html Apparently it had an Acorn C and Modula runtime: http://homepage.mac.com/dr2chase/resume.html Also about the machine and video chip: http://www.khantazi.org/Archives/MachineLst.html#A500 And its window toolkit: http://pages.sbcglobal.net/btlewis/resume-formatted.html Thanks in advance --- El jue, 24/2/11, Jay K escribi?: De: Jay K Asunto: Re: [M3devel] new target? :) Para: "Mika Nystrom" CC: "m3devel" Fecha: jueves, 24 de febrero, 2011 22:09 ?> ?I also think almost everyone who had anything to do with M3 at DEC is now at Microsoft ? ? Not just M3. M3 wasn't many people. Lots of?people at Microsoft were?at Digital. ? ? > Modula-3 project: you're supposed to be able to write device drivers > and such things in Modula-3. I agree though that targeting C doesn't ? Understood, agreed -- you can hypothetically write anything in Modula-3. The optional safety is fairly unique and important. Except for C# and Modula-3, all languages seem to be strictly safe or unsafe. ? ? I think, you can imagine, either an entire system in Modula-3, with m3core/libm3 coded "directly" in Modula-3 and simply statically or dynamically linked using "standard mechanism", or if there is still a user/kernel boundary, you either have to hack the compiler a bit, or resort to some assembly (possibly embedded in .c files). That is, calling the kernel is inevitably a special call sequence, and cm3 doesn't currently understand anything like that. ? ? About device drivers -- a critical piece is the runtime. Imagine writing an NT or Linux driver. Well, if you separate language from library...and limit language features to accomodate?missing runtime...and retarget small parts of the runtime perhaps.. you know, there's a certain isomorphism? You can write almost anything in C or Modula-3 and most things will look very?similar and map one-to-one. I think I read that Modula-3 can describe anything C can, except unions. ? ? You know, don't expect to have garbage collection in the kernel. Maybe not exception handling. Not necessarily the file I/O you are used to. But something. Not the threading you are used to. But something (at least on NT). But certainly you can interface Modula-3 to the NTkernel/Linux/*BSD runtimes, just like we? interface to Win32/Posix. ? ? I'm not sure it'd buy much. I guess if you really like the syntax... ? ? It's another area where compiling to C would help. esp. for the exception handling on NT! __try/__finally are available in C on NT, usermode and kernel, and the implementation is much more efficient than what Modula-3 does currently. I really need to stop saying all this stuff though, either be quiet or do it... ? ? ?- Jay ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hosking at cs.purdue.edu Fri Feb 25 16:08:15 2011 From: hosking at cs.purdue.edu (Tony Hosking) Date: Fri, 25 Feb 2011 10:08:15 -0500 Subject: [M3devel] new target? :) In-Reply-To: References: , , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , <20110225013100.619CD1A2078@async.async.caltech.edu> Message-ID: <04F51416-6EAC-4BBD-8C4B-623245437ECD@cs.purdue.edu> Just to follow up briefly as I have lots of other things on my plate. One of the reasons for targeting the gcc-based backend was to support easy generation of *source*-level debugging information. It is important that the debugger see types as Modula-3 sees them, rather than as C types generated form the Modula-3. Now, I agree that debugging the C-translated types is probably better than nothing. On Feb 24, 2011, at 8:55 PM, Jay K wrote: > When you write that system that doesn't have the support of any C compiler, sure, go ahead and > write a backend just for it, and maybe a linker, and rewrite the small parts that depend on C and Posix in some > sort of native way -- including writing all the underlying stuff. > > > Getting Modula-3 working on that system will be a relatively small task, compared to whatever else > is needed to build the system. > > Note for example, we can't e.g. open files in Modula-3, w/o underlying Win32 or Posix functionality. > Everywhere that varies for Win32 vs. Posix would have to be revisited in this hypothetical system. > > > This has little/no bearing on the vast vast vast vast majority of real world systems. > The current Modula-3 system is NOT implemented entirely in Modula-3. > It MUST depend on SOMETHING. > So far, for the vast vast vast vast vast majority of systems, that something is written in C, > and has a C runtime, compiler, and linker. > > > Yes, I've heard of SPIN. > > > You know..how are you actually going to network I/O or keyboard I/O or power management > or video I/O or mouse I/O or sound I/O or preemptive multi-processor scheduling? > You need device drivers. You need a processor-specific context switcher. > i.e. you need a kernel and you need C AND/OR you need to do a lot of work. > > > If you consider that our system is already forked along Win32 and Posix lines, you can consider > these hypothetical systems a third (fourth, fifth, etc.) fork. A third (fourth) implementation > of threading. A third implementation of file I/O. A third implementation of Trestle. etc. > > > But this has little/no bearing how we should target existing real systems. > You can say, not that "Modula-3 depends on C (runtime, compiler, linker)", but that "Modula-3 on Linux, NT, *BSD, Solaris, Darwin > depends on C (runtime, compiler, linker)", and there's absolutely nothing wrong with that, as C and C++ > are well supported on those systems and always will be. Increasing that dependency, on those systems, is fine. > > > It would be nice to port the integrated backend to lots more systems, and to write our own linker. > But not easy. > And notice how the integrated backend also has the worst debugging story of the two/three options, currently. > (I say three options because m3cg kind of has two modes, one is incomplete and currently disabled, > the other requires m3gdb). > > > One problem I see here, is that, you kind of sort of maybe forseeable want a backend that outputs Java or C#. > C# particularly for Windows Phone 7/Nokia. > Java kind of for Android, where Java is preferred over C/C++. > > > i.e. there are systems that sort of lack C and C++. Not that they replace it with Modula-3 though. > > > However this isn't super easy, like if you are to include the existing m3core/garbage collector. > C# is optionally safe. I don't know if it would serve as a good target. > (Optional safety imho makes C# among one of the closest languages to Modula-3. It is > missing "compile to native", but that maybe is a good thing.) > In addition, C# and Java do each compile down to one well documented "object" file format, > so maybe skipping the text is probably sensible. Esp. since with C# there are in-proc libraries for codegen. > > > More likely, if a Java/C# target were really needed, it'd include forking within m3core > and replace the garbage collector. This is all highly specuative stuff. > > > - Jay > > > > To: jay.krell at cornell.edu > > Date: Thu, 24 Feb 2011 17:31:00 -0800 > > From: mika at async.caltech.edu > > CC: m3devel at elegosoft.com > > Subject: Re: [M3devel] new target? :) > > > > Jay K writes: > > ... > > >m3cg is=2C technically=2C a gcc frontend=2C and therefore GPL. > > >However its input is not human authored text. > > >Its input is the output of a "real" front end -- m3front/cm3. > > >By making the "call mechanism" be write to a file and run a process=2C the = > > >GPL doesn't cross the barrier=2C > > >leaving cm3/m3front=2C and more importantly=2C m3core/libm3=2C not GPL. > > >=20 > > >=20 > > >I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. > > >=20 > > >=20 > > > > Good points all about C but I wonder about what is really lost. Also ...... > > Modula-3 is supposed to be portable to systems without C. You're supposed > > to be able to write an entire system in Modula-3. Ok generating C obviously > > doesn't break that feature but you might have to worry a bit about having > > too much C sneak into the runtime libraries. > > > > Well given that CM3 is released under what is more or less an MIT/BSD license, > > I think the problem is rather that Some Evil Software Company could write > > a completely non-free front-end and use m3cg to generate code... > > > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Fri Feb 25 19:50:12 2011 From: jay.krell at cornell.edu (Jay K) Date: Fri, 25 Feb 2011 18:50:12 +0000 Subject: [M3devel] new target? :) In-Reply-To: <04F51416-6EAC-4BBD-8C4B-623245437ECD@cs.purdue.edu> References: , , , <758561.47161.qm@web29710.mail.ird.yahoo.com>, , , , <20110224221551.8B9271A2078@async.async.caltech.edu>, , <20110225013100.619CD1A2078@async.async.caltech.edu> , <04F51416-6EAC-4BBD-8C4B-623245437ECD@cs.purdue.edu> Message-ID: #line directives should help source debugging. I agree the types would be a bit off. For example arrays, objects, texts, would have internals showing, and texts might take extra steps to see what you really want. - Jay Subject: Re: [M3devel] new target? :) From: hosking at cs.purdue.edu Date: Fri, 25 Feb 2011 10:08:15 -0500 CC: mika at async.caltech.edu; m3devel at elegosoft.com To: jay.krell at cornell.edu Just to follow up briefly as I have lots of other things on my plate.One of the reasons for targeting the gcc-based backend was to support easy generation of *source*-level debugging information. It is important that the debugger see types as Modula-3 sees them, rather than as C types generated form the Modula-3. Now, I agree that debugging the C-translated types is probably better than nothing. On Feb 24, 2011, at 8:55 PM, Jay K wrote:When you write that system that doesn't have the support of any C compiler, sure, go ahead and write a backend just for it, and maybe a linker, and rewrite the small parts that depend on C and Posix in some sort of native way -- including writing all the underlying stuff. Getting Modula-3 working on that system will be a relatively small task, compared to whatever else is needed to build the system. Note for example, we can't e.g. open files in Modula-3, w/o underlying Win32 or Posix functionality. Everywhere that varies for Win32 vs. Posix would have to be revisited in this hypothetical system. This has little/no bearing on the vast vast vast vast majority of real world systems. The current Modula-3 system is NOT implemented entirely in Modula-3. It MUST depend on SOMETHING. So far, for the vast vast vast vast vast majority of systems, that something is written in C, and has a C runtime, compiler, and linker. Yes, I've heard of SPIN. You know..how are you actually going to network I/O or keyboard I/O or power management or video I/O or mouse I/O or sound I/O or preemptive multi-processor scheduling? You need device drivers. You need a processor-specific context switcher. i.e. you need a kernel and you need C AND/OR you need to do a lot of work. If you consider that our system is already forked along Win32 and Posix lines, you can consider these hypothetical systems a third (fourth, fifth, etc.) fork. A third (fourth) implementation of threading. A third implementation of file I/O. A third implementation of Trestle. etc. But this has little/no bearing how we should target existing real systems. You can say, not that "Modula-3 depends on C (runtime, compiler, linker)", but that "Modula-3 on Linux, NT, *BSD, Solaris, Darwin depends on C (runtime, compiler, linker)", and there's absolutely nothing wrong with that, as C and C++ are well supported on those systems and always will be. Increasing that dependency, on those systems, is fine. It would be nice to port the integrated backend to lots more systems, and to write our own linker. But not easy. And notice how the integrated backend also has the worst debugging story of the two/three options, currently. (I say three options because m3cg kind of has two modes, one is incomplete and currently disabled, the other requires m3gdb). One problem I see here, is that, you kind of sort of maybe forseeable want a backend that outputs Java or C#. C# particularly for Windows Phone 7/Nokia. Java kind of for Android, where Java is preferred over C/C++. i.e. there are systems that sort of lack C and C++. Not that they replace it with Modula-3 though. However this isn't super easy, like if you are to include the existing m3core/garbage collector. C# is optionally safe. I don't know if it would serve as a good target. (Optional safety imho makes C# among one of the closest languages to Modula-3. It is missing "compile to native", but that maybe is a good thing.) In addition, C# and Java do each compile down to one well documented "object" file format, so maybe skipping the text is probably sensible. Esp. since with C# there are in-proc libraries for codegen. More likely, if a Java/C# target were really needed, it'd include forking within m3core and replace the garbage collector. This is all highly specuative stuff. - Jay > To: jay.krell at cornell.edu > Date: Thu, 24 Feb 2011 17:31:00 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] new target? :) > > Jay K writes: > ... > >m3cg is=2C technically=2C a gcc frontend=2C and therefore GPL. > >However its input is not human authored text. > >Its input is the output of a "real" front end -- m3front/cm3. > >By making the "call mechanism" be write to a file and run a process=2C the = > >GPL doesn't cross the barrier=2C > >leaving cm3/m3front=2C and more importantly=2C m3core/libm3=2C not GPL. > >=20 > >=20 > >I do wonder if m3cg/cm3/m3front/m3core/libm3 could be LGPL to please RMS. > >=20 > >=20 > > Good points all about C but I wonder about what is really lost. Also ...... > Modula-3 is supposed to be portable to systems without C. You're supposed > to be able to write an entire system in Modula-3. Ok generating C obviously > doesn't break that feature but you might have to worry a bit about having > too much C sneak into the runtime libraries. > > Well given that CM3 is released under what is more or less an MIT/BSD license, > I think the problem is rather that Some Evil Software Company could write > a completely non-free front-end and use m3cg to generate code... > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragisha at m3w.org Fri Feb 25 20:08:49 2011 From: dragisha at m3w.org (dragisha at m3w.org) Date: Fri, 25 Feb 2011 20:08:49 +0100 Subject: [M3devel] new target? :) In-Reply-To: References: Message-ID: <775012e5aaa51d171471059dac6ab8be@rstel.net> I am on the road so this is easiest way to amend this... As I see, some people steered this discussion to web dev... One of things I made some time ago is verry complete FastCGI support for Modula-3, and I will contribute it one of these days, I just need more time to go through everything and ensure it works on all platforms I have available now (OSX, Linux, Windows...)... FastCGI is server side, and there is no question about Modula-3 perfectness for anything server. This link I gave is a way for us to get our perfect platform client side. No oblets (IIRC), no nothing... Pure Modula-3. On Thu, 24 Feb 2011 20:16:58 +0100, Dragi?a Duri? wrote: > http://code.google.com/chrome/nativeclient/ [1] > > Links: > ------ > [1] http://code.google.com/chrome/nativeclient/ From rcolebur at SCIRES.COM Sat Feb 26 10:47:02 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sat, 26 Feb 2011 04:47:02 -0500 Subject: [M3devel] results of threadtest program on Windows7 Message-ID: Mika: I've finally managed to get cm3 rebuilt on Windows 7 again. So, I ran your threadtest program. Here are the results. Note the "..." is where I cut out a bunch of the repeating "ERROR FApply" messages. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe Writing file...done Creating read threads...done Creating fork threads...done Creating alloc threads...done Creating lock threads...done running...printing oldest/median age/newest ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. . . . *** *** runtime error: *** An enumeration or subrange value was out of range. *** file "..\src\Main.m3", line 340 *** laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. . . . laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. . . . ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified. Stack trace: FP PC Procedure --------- --------- ------------------------------- 0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 Regards, Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 26 11:25:51 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 26 Feb 2011 10:25:51 +0000 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: Message-ID: The FApply problems are because the wants to run "sleep". I've checked in a sleep subset including a Win32 port (untested, but it is very simple). We should get that available to the test and have it run m3sleep instead of sleep. Or you can manually copy this m3sleep.exe to %windir%\system32\sleep.exe or such. Or put Cygwin sleep.exe in %PATH%, but that is overkill. Probably the thread tester could implement a -sleep flag and just call itself with that.. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Sat, 26 Feb 2011 04:47:02 -0500 Subject: [M3devel] results of threadtest program on Windows7 Mika: I?ve finally managed to get cm3 rebuilt on Windows 7 again. So, I ran your threadtest program. Here are the results. Note the ??? is where I cut out a bunch of the repeating ?ERROR FApply? messages. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exeWriting file...doneCreating read threads...doneCreating fork threads...doneCreating alloc threads...doneCreating lock threads...donerunning...printing oldest/median age/newestERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....****** runtime error:*** An enumeration or subrange value was out of range.*** file "..\src\Main.m3", line 340*** laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.Stack trace: FP PC Procedure--------- --------- ------------------------------- 0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 Regards,Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.krell at cornell.edu Sat Feb 26 11:42:47 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 26 Feb 2011 10:42:47 +0000 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: , Message-ID: On Darwin, we often get back EBUSY from pthread_mutex_delete. And often: *** *** runtime error: *** An array subscript was out of range. *** file "../src/runtime/common/RTCollector.m3", line 418 *** Abort trap I'll try to look into this more/sooner... - Jay From: jay.krell at cornell.edu To: rcolebur at scires.com; m3devel at elegosoft.com Date: Sat, 26 Feb 2011 10:25:51 +0000 Subject: Re: [M3devel] results of threadtest program on Windows7 The FApply problems are because the wants to run "sleep". I've checked in a sleep subset including a Win32 port (untested, but it is very simple). We should get that available to the test and have it run m3sleep instead of sleep. Or you can manually copy this m3sleep.exe to %windir%\system32\sleep.exe or such. Or put Cygwin sleep.exe in %PATH%, but that is overkill. Probably the thread tester could implement a -sleep flag and just call itself with that.. - Jay From: rcolebur at SCIRES.COM To: m3devel at elegosoft.com Date: Sat, 26 Feb 2011 04:47:02 -0500 Subject: [M3devel] results of threadtest program on Windows7 Mika: I?ve finally managed to get cm3 rebuilt on Windows 7 again. So, I ran your threadtest program. Here are the results. Note the ??? is where I cut out a bunch of the repeating ?ERROR FApply? messages. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exeWriting file...doneCreating read threads...doneCreating fork threads...doneCreating alloc threads...doneCreating lock threads...donerunning...printing oldest/median age/newestERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....****** runtime error:*** An enumeration or subrange value was out of range.*** file "..\src\Main.m3", line 340*** laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified....ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.ERROR FApply: OSError.E: ErrorCode=2: The system cannot find the file specified.Stack trace: FP PC Procedure--------- --------- ------------------------------- 0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 Regards,Randy Coleburn -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sat Feb 26 18:55:11 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 26 Feb 2011 09:55:11 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: Message-ID: <20110226175511.3F7D21A2078@async.async.caltech.edu> Hi Randy, Hm yes it looks like my Windows programming skills leave something to be desired. You can run the thread tester while skipping a test as follows threadtest -tests ALL,-fork (for instance) if you just run threadtest -sadfassdaf it'll print the tests that are available. As it happens, I just had to upgrade my windows 2000 system to windows 7. Can you give me a very brief description of what you did to install Modula-3 on this system? Mika "Coleburn, Randy" writes: >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/plain; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > >Mika: > >I've finally managed to get cm3 rebuilt on Windows 7 again. > >So, I ran your threadtest program. > >Here are the results. Note the "..." is where I cut out a bunch of the rep= >eating "ERROR FApply" messages. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >*** >*** runtime error: >*** An enumeration or subrange value was out of range. >*** file "..\src\Main.m3", line 340 >*** > >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >Stack trace: > FP PC Procedure >--------- --------- ------------------------------- >0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 > >Regards, >Randy Coleburn > >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/html; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > From mika at async.caltech.edu Sat Feb 26 19:10:44 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 26 Feb 2011 10:10:44 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: Message-ID: <20110226181044.A44DD1A2078@async.async.caltech.edu> Hi Randy, I just implemented Jay's suggestion for removing the dependency on the external program "sleep". You can CVS update and try again now. Mika From jay.krell at cornell.edu Sat Feb 26 22:13:27 2011 From: jay.krell at cornell.edu (Jay K) Date: Sat, 26 Feb 2011 21:13:27 +0000 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: <20110226181044.A44DD1A2078@async.async.caltech.edu> References: , <20110226181044.A44DD1A2078@async.async.caltech.edu> Message-ID: Thanks. jbook2:thread jay$ ./I386_DARWIN/threadtest Writing file...done Creating read threads...done Creating fork threads...done Creating alloc threads...done Creating lock threads...done running...printing oldest/median age/newest ..........laziest thread is 1/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 1/0/0 lock 1/1/1) ..........laziest thread is 1/0/0 (tests: read 1/0/0 fork 1/1/0 alloc 0/0/0 lock 0/0/0) those aren't errors, right? I can reproduce problems on Darwin -- assertion failures, errors in ThreadPThreadC.c like EBUSY and I think EINVAL. EBUSY is from pthread_mutex_destroy, which means we are freeing a mutex while it is locked, which means there is a problem. Such as the garbage collector collecting non-garbage. It ccould be something else though. I'm trying with user threads. User threads do print the above. But so far has a longer runtime than with pthreads, hasn't hit any SIGSEGV or assertion failure or anything yet. I do wonder .. oh, and then w/ user threads: All tests complete. Congratulations. I do wonder about scheduling order and when the preemptions occur. For example user threads does offer Disable/EnableScheduling. Maybe the timeslices vary a lot and userthreads preempt a lot less? Maybe we can try w/ tuning the user thread timeslice to match whatever pthread uses on whatever platform ( there likely no guarantee or documentation, but for experimental purposes we can likely figure it out experimentally). It will be interesting to see the results on Windows too. I'll probably try that right away. - Jay > To: rcolebur at SCIRES.COM > Date: Sat, 26 Feb 2011 10:10:44 -0800 > From: mika at async.caltech.edu > CC: m3devel at elegosoft.com > Subject: Re: [M3devel] results of threadtest program on Windows7 > > > Hi Randy, > > I just implemented Jay's suggestion for removing the dependency on the > external program "sleep". You can CVS update and try again now. > > Mika -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika at async.caltech.edu Sun Feb 27 00:10:55 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sat, 26 Feb 2011 15:10:55 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: , <20110226181044.A44DD1A2078@async.async.caltech.edu> Message-ID: <20110226231055.932CA1A2078@async.async.caltech.edu> Jay K writes: >--_fe2b6a60-67dc-46f1-b844-2023fbc40258_ >Content-Type: text/plain; charset="iso-8859-1" >Content-Transfer-Encoding: quoted-printable > > >Thanks. > >jbook2:thread jay$ ./I386_DARWIN/threadtest=20 >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >..........laziest thread is 1/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 1/0/0= > lock 1/1/1) >..........laziest thread is 1/0/0 (tests: read 1/0/0 fork 1/1/0 alloc 0/0/0= > lock 0/0/0) > >those aren't errors=2C right? No those are just the noral diagnostics. The numbers are how many seconds the various threads are lagging. User threads do have a "more unfair" scheduler than you can expect with Darwin (see comments in the code)... You can change the number of threads etc but I find that the defaults seem to catch stuff. Lowering the number of threads per test from default 3 to "-n 2" simplifies debugging, perhaps. (More comments in the code.) Mika > >I can reproduce problems on Darwin -- assertion failures=2C errors in Threa= >dPThreadC.c like EBUSY and I think EINVAL. > EBUSY is from pthread_mutex_destroy=2C which means we are freeing a mutex= > while it is locked=2C which means > there is a problem. Such as the garbage collector collecting non-garbage= >. It ccould be something else though. > >I'm trying with user threads. User threads do print the above. But so far h= >as a longer runtime than with pthreads=2C > hasn't hit any SIGSEGV or assertion failure or anything yet. > >I do wonder .. oh=2C and then w/ user threads: > >All tests complete. Congratulations. > >I do wonder about scheduling order and when the preemptions occur. >For example user threads does offer Disable/EnableScheduling. >Maybe the timeslices vary a lot and userthreads preempt a lot less? >Maybe we can try w/ tuning the user thread timeslice to match whatever pthr= >ead uses on whatever platform ( >there likely no guarantee or documentation=2C but for experimental purposes= > we can likely figure it out >experimentally). > >It will be interesting to see the results on Windows too. I'll probably try= > that right away. > > - Jay From rcolebur at SCIRES.COM Sun Feb 27 20:09:08 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sun, 27 Feb 2011 14:09:08 -0500 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: <20110226175511.3F7D21A2078@async.async.caltech.edu> References: <20110226175511.3F7D21A2078@async.async.caltech.edu> Message-ID: Mika: Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. Here is what happens now when I run your threadtest program on Windows 7. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-fork Writing file...done Creating read threads...done Creating nxread threads...done Creating tryexcept threads...done Creating forktoomuch threads...done Creating alloc threads...done Creating creat threads...done Creating lock threads...done running...printing oldest/median age/newest *** *** runtime error: *** An array subscript was out of range. *** file "..\src\runtime\common\RTCollector.m3", line 418 *** *** *** runtime error: *** <*ASSERT*> failed. *** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 *** The last error repeats ad infinitum until I press CNTRL-C to abort. I'll send more info on the Windows install of Modula3 in a subsequent post. Regards, Randy Coleburn -----Original Message----- From: Mika Nystrom [mailto:mika at async.caltech.edu] Sent: Saturday, February 26, 2011 12:55 PM To: Coleburn, Randy Cc: m3devel at elegosoft.com Subject: Re: [M3devel] results of threadtest program on Windows7 Hi Randy, Hm yes it looks like my Windows programming skills leave something to be desired. You can run the thread tester while skipping a test as follows threadtest -tests ALL,-fork (for instance) if you just run threadtest -sadfassdaf it'll print the tests that are available. As it happens, I just had to upgrade my windows 2000 system to windows 7. Can you give me a very brief description of what you did to install Modula-3 on this system? Mika "Coleburn, Randy" writes: >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/plain; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > >Mika: > >I've finally managed to get cm3 rebuilt on Windows 7 again. > >So, I ran your threadtest program. > >Here are the results. Note the "..." is where I cut out a bunch of the rep= >eating "ERROR FApply" messages. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >*** >*** runtime error: >*** An enumeration or subrange value was out of range. >*** file "..\src\Main.m3", line 340 >*** > >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >Stack trace: > FP PC Procedure >--------- --------- ------------------------------- >0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 > >Regards, >Randy Coleburn > >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/html; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > From rcolebur at SCIRES.COM Sun Feb 27 20:45:29 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sun, 27 Feb 2011 14:45:29 -0500 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: <20110226175511.3F7D21A2078@async.async.caltech.edu> Message-ID: Mika et al: Thought I would try something else. I took the sources of your thread test program to an older XP machine that has CM3 circa August 2008. This is the machine and implementation I used when building a major project I did a couple years back. The thread test program does indeed build on this old system, but when I run it, I get different results than with the latest HEAD branch code. After it prints "running...printing oldest/median age/newest", on the next line I get two periods ".." and now the program seems hung. I'll let it "run" for a few more minutes to see if anything else happens before killing it. At least we don't get the subscript and assertion failures on this older CM3 platform. Regards, Randy Coleburn -----Original Message----- From: Coleburn, Randy Sent: Sunday, February 27, 2011 2:09 PM To: m3devel at elegosoft.com Subject: Re: [M3devel] results of threadtest program on Windows7 Mika: Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. Here is what happens now when I run your threadtest program on Windows 7. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-fork Writing file...done Creating read threads...done Creating nxread threads...done Creating tryexcept threads...done Creating forktoomuch threads...done Creating alloc threads...done Creating creat threads...done Creating lock threads...done running...printing oldest/median age/newest *** *** runtime error: *** An array subscript was out of range. *** file "..\src\runtime\common\RTCollector.m3", line 418 *** *** *** runtime error: *** <*ASSERT*> failed. *** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 *** The last error repeats ad infinitum until I press CNTRL-C to abort. I'll send more info on the Windows install of Modula3 in a subsequent post. Regards, Randy Coleburn -----Original Message----- From: Mika Nystrom [mailto:mika at async.caltech.edu] Sent: Saturday, February 26, 2011 12:55 PM To: Coleburn, Randy Cc: m3devel at elegosoft.com Subject: Re: [M3devel] results of threadtest program on Windows7 Hi Randy, Hm yes it looks like my Windows programming skills leave something to be desired. You can run the thread tester while skipping a test as follows threadtest -tests ALL,-fork (for instance) if you just run threadtest -sadfassdaf it'll print the tests that are available. As it happens, I just had to upgrade my windows 2000 system to windows 7. Can you give me a very brief description of what you did to install Modula-3 on this system? Mika "Coleburn, Randy" writes: >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/plain; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > >Mika: > >I've finally managed to get cm3 rebuilt on Windows 7 again. > >So, I ran your threadtest program. > >Here are the results. Note the "..." is where I cut out a bunch of the rep= >eating "ERROR FApply" messages. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >*** >*** runtime error: >*** An enumeration or subrange value was out of range. >*** file "..\src\Main.m3", line 340 >*** > >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D2: The system c= >annot find the file specified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >. >. >. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >ERROR FApply: OSError.E: ErrorCode=3D2: The system cannot find the file sp= >ecified. >Stack trace: > FP PC Procedure >--------- --------- ------------------------------- >0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 > >Regards, >Randy Coleburn > >--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >Content-Type: text/html; charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > From mika at async.caltech.edu Sun Feb 27 21:29:50 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 27 Feb 2011 12:29:50 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: <20110226175511.3F7D21A2078@async.async.caltech.edu> Message-ID: <20110227202950.46F3B1A2078@async.async.caltech.edu> Hi Randy, You can try it with -tests POSIX as well. I find on user threads it runs very slowly (but it does run) because of how unfair the thread scheduler is. Next step might be to whittle down the tests and see if you can get a failure with a single test running and -n 2. That would likely be the simplest scenario to start further debugging from. Mika "Coleburn, Randy" writes: >Mika et al: > >Thought I would try something else. > >I took the sources of your thread test program to an older XP machine that = >has CM3 circa August 2008. This is the machine and implementation I used w= >hen building a major project I did a couple years back. > >The thread test program does indeed build on this old system, but when I ru= >n it, I get different results than with the latest HEAD branch code. =20 > >After it prints "running...printing oldest/median age/newest", on the next = >line I get two periods ".." and now the program seems hung. I'll let it "r= >un" for a few more minutes to see if anything else happens before killing i= >t. > >At least we don't get the subscript and assertion failures on this older CM= >3 platform. > >Regards, >Randy Coleburn > > >-----Original Message----- >From: Coleburn, Randy=20 >Sent: Sunday, February 27, 2011 2:09 PM >To: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7 > >Mika: > >Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. > >Here is what happens now when I run your threadtest program on Windows 7. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-for= >k >Writing file...done >Creating read threads...done >Creating nxread threads...done >Creating tryexcept threads...done >Creating forktoomuch threads...done >Creating alloc threads...done >Creating creat threads...done >Creating lock threads...done >running...printing oldest/median age/newest > > >*** >*** runtime error: >*** An array subscript was out of range. >*** file "..\src\runtime\common\RTCollector.m3", line 418 >*** > > > >*** >*** runtime error: >*** <*ASSERT*> failed. >*** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 >*** > >The last error repeats ad infinitum until I press CNTRL-C to abort. > >I'll send more info on the Windows install of Modula3 in a subsequent post. > >Regards, >Randy Coleburn > >-----Original Message----- >From: Mika Nystrom [mailto:mika at async.caltech.edu]=20 >Sent: Saturday, February 26, 2011 12:55 PM >To: Coleburn, Randy >Cc: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7=20 > >Hi Randy, > >Hm yes it looks like my Windows programming skills leave something >to be desired. > >You can run the thread tester while skipping a test as follows > > threadtest -tests ALL,-fork > >(for instance) > >if you just run=20 > > threadtest -sadfassdaf > >it'll print the tests that are available. > >As it happens, I just had to upgrade my windows 2000 system to windows 7. >Can you give me a very brief description of what you did to install Modula-= >3 >on this system? > > Mika > >"Coleburn, Randy" writes: >>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>Content-Type: text/plain; charset=3D"us-ascii" >>Content-Transfer-Encoding: quoted-printable >> >>Mika: >> >>I've finally managed to get cm3 rebuilt on Windows 7 again. >> >>So, I ran your threadtest program. >> >>Here are the results. Note the "..." is where I cut out a bunch of the re= >p=3D >>eating "ERROR FApply" messages. >> >>C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >>Writing file...done >>Creating read threads...done >>Creating fork threads...done >>Creating alloc threads...done >>Creating lock threads...done >>running...printing oldest/median age/newest >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>*** >>*** runtime error: >>*** An enumeration or subrange value was out of range. >>*** file "..\src\Main.m3", line 340 >>*** >> >>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D2: The syste= >m c=3D >>annot find the file specified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D2: The syste= >m c=3D >>annot find the file specified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>Stack trace: >> FP PC Procedure >>--------- --------- ------------------------------- >>0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >>0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 >> >>Regards, >>Randy Coleburn >> >>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>Content-Type: text/html; charset=3D"us-ascii" >>Content-Transfer-Encoding: quoted-printable >> From rcolebur at SCIRES.COM Sun Feb 27 22:03:13 2011 From: rcolebur at SCIRES.COM (Coleburn, Randy) Date: Sun, 27 Feb 2011 16:03:13 -0500 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: <20110227202950.46F3B1A2078@async.async.caltech.edu> References: <20110226175511.3F7D21A2078@async.async.caltech.edu> <20110227202950.46F3B1A2078@async.async.caltech.edu> Message-ID: Mika: No change with "-tests POSIX". Interesting twist: On Windows 7, I thought I'd see what the command line options are, and I typed "threadtest -help" rather than reading the code. First time, it produced what appears to be a NIL deref crash. Then, I tried it again and it ran to completion. Something seems non-deterministic here. See below. C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe -help Writing file...done Creating read threads...done Creating fork threads...done Creating alloc threads...done Creating lock threads...done running...printing oldest/median age/newest . *** *** runtime error: *** Attempt to reference an illegal memory location. *** pc = 0x77762262 *** Stack trace: FP PC Procedure --------- --------- ------------------------------- 0xcdf998 0x130351b SystemError + 0x64 in ..\src\runtime\NT386\RTSignal.m3 0xcdf9c0 0x77762262 0xcdf9d8 0x12e83b7 LockMutex + 0x4f in ..\src\thread\WIN32\ThreadWin32.m3 0xcdfa00 0x12c7b08 GetChar + 0x28 in ..\src\rw\Rd.m3 0xcdfb38 0x12c12e3 RApply + 0xd3 in ..\src\Main.m3 0xcdfb74 0x12e971f ThreadBase + 0x254 in ..\src\thread\WIN32\ThreadWin32.m3 0xcdfb80 0x76543677 0xcdfbc0 0x77779f02 ......... ......... ... more frames ... C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe -help Writing file...done Creating read threads...done Creating fork threads...done Creating alloc threads...done Creating lock threads...done running...printing oldest/median age/newest ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0) All tests complete. Congratulations. Regards, Randy Coleburn -----Original Message----- From: Mika Nystrom [mailto:mika at async.caltech.edu] Sent: Sunday, February 27, 2011 3:30 PM To: Coleburn, Randy Cc: m3devel at elegosoft.com Subject: Re: [M3devel] results of threadtest program on Windows7 Hi Randy, You can try it with -tests POSIX as well. I find on user threads it runs very slowly (but it does run) because of how unfair the thread scheduler is. Next step might be to whittle down the tests and see if you can get a failure with a single test running and -n 2. That would likely be the simplest scenario to start further debugging from. Mika "Coleburn, Randy" writes: >Mika et al: > >Thought I would try something else. > >I took the sources of your thread test program to an older XP machine that = >has CM3 circa August 2008. This is the machine and implementation I used w= >hen building a major project I did a couple years back. > >The thread test program does indeed build on this old system, but when I ru= >n it, I get different results than with the latest HEAD branch code. =20 > >After it prints "running...printing oldest/median age/newest", on the next = >line I get two periods ".." and now the program seems hung. I'll let it "r= >un" for a few more minutes to see if anything else happens before killing i= >t. > >At least we don't get the subscript and assertion failures on this older CM= >3 platform. > >Regards, >Randy Coleburn > > >-----Original Message----- >From: Coleburn, Randy=20 >Sent: Sunday, February 27, 2011 2:09 PM >To: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7 > >Mika: > >Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. > >Here is what happens now when I run your threadtest program on Windows 7. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-for= >k >Writing file...done >Creating read threads...done >Creating nxread threads...done >Creating tryexcept threads...done >Creating forktoomuch threads...done >Creating alloc threads...done >Creating creat threads...done >Creating lock threads...done >running...printing oldest/median age/newest > > >*** >*** runtime error: >*** An array subscript was out of range. >*** file "..\src\runtime\common\RTCollector.m3", line 418 >*** > > > >*** >*** runtime error: >*** <*ASSERT*> failed. >*** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 >*** > >The last error repeats ad infinitum until I press CNTRL-C to abort. > >I'll send more info on the Windows install of Modula3 in a subsequent post. > >Regards, >Randy Coleburn > >-----Original Message----- >From: Mika Nystrom [mailto:mika at async.caltech.edu]=20 >Sent: Saturday, February 26, 2011 12:55 PM >To: Coleburn, Randy >Cc: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7=20 > >Hi Randy, > >Hm yes it looks like my Windows programming skills leave something >to be desired. > >You can run the thread tester while skipping a test as follows > > threadtest -tests ALL,-fork > >(for instance) > >if you just run=20 > > threadtest -sadfassdaf > >it'll print the tests that are available. > >As it happens, I just had to upgrade my windows 2000 system to windows 7. >Can you give me a very brief description of what you did to install Modula-= >3 >on this system? > > Mika > >"Coleburn, Randy" writes: >>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>Content-Type: text/plain; charset=3D"us-ascii" >>Content-Transfer-Encoding: quoted-printable >> >>Mika: >> >>I've finally managed to get cm3 rebuilt on Windows 7 again. >> >>So, I ran your threadtest program. >> >>Here are the results. Note the "..." is where I cut out a bunch of the re= >p=3D >>eating "ERROR FApply" messages. >> >>C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >>Writing file...done >>Creating read threads...done >>Creating fork threads...done >>Creating alloc threads...done >>Creating lock threads...done >>running...printing oldest/median age/newest >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>*** >>*** runtime error: >>*** An enumeration or subrange value was out of range. >>*** file "..\src\Main.m3", line 340 >>*** >> >>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D2: The syste= >m c=3D >>annot find the file specified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D2: The syste= >m c=3D >>annot find the file specified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>. >>. >>. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>ERROR FApply: OSError.E: ErrorCode=3D3D2: The system cannot find the file= > sp=3D >>ecified. >>Stack trace: >> FP PC Procedure >>--------- --------- ------------------------------- >>0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >>0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 >> >>Regards, >>Randy Coleburn >> >>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>Content-Type: text/html; charset=3D"us-ascii" >>Content-Transfer-Encoding: quoted-printable >> From mika at async.caltech.edu Mon Feb 28 00:11:25 2011 From: mika at async.caltech.edu (Mika Nystrom) Date: Sun, 27 Feb 2011 15:11:25 -0800 Subject: [M3devel] results of threadtest program on Windows7 In-Reply-To: References: <20110226175511.3F7D21A2078@async.async.caltech.edu> <20110227202950.46F3B1A2078@async.async.caltech.edu> Message-ID: <20110227231125.3DC611A2078@async.async.caltech.edu> Ah, it just doesn't check command-line arguments that carefully. I think what you did is equivalent to "-tests STD". Mika "Coleburn, Randy" writes: >Mika: > >No change with "-tests POSIX". > >Interesting twist: On Windows 7, I thought I'd see what the command line o= >ptions are, and I typed "threadtest -help" rather than reading the code. > >First time, it produced what appears to be a NIL deref crash. Then, I trie= >d it again and it ran to completion. Something seems non-deterministic her= >e. See below. > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe -help >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >. > >*** >*** runtime error: >*** Attempt to reference an illegal memory location. >*** pc =3D 0x77762262 >*** > >Stack trace: > FP PC Procedure >--------- --------- ------------------------------- > 0xcdf998 0x130351b SystemError + 0x64 in ..\src\runtime\NT386\RTSignal.m= >3 > 0xcdf9c0 0x77762262 > 0xcdf9d8 0x12e83b7 LockMutex + 0x4f in ..\src\thread\WIN32\ThreadWin32.m= >3 > 0xcdfa00 0x12c7b08 GetChar + 0x28 in ..\src\rw\Rd.m3 > 0xcdfb38 0x12c12e3 RApply + 0xd3 in ..\src\Main.m3 > 0xcdfb74 0x12e971f ThreadBase + 0x254 in ..\src\thread\WIN32\ThreadWin32= >.m3 > 0xcdfb80 0x76543677 > 0xcdfbc0 0x77779f02 >......... ......... ... more frames ... > >C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe -help >Writing file...done >Creating read threads...done >Creating fork threads...done >Creating alloc threads...done >Creating lock threads...done >running...printing oldest/median age/newest >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0= > lock 0/0/0) >All tests complete. Congratulations. > >Regards, >Randy Coleburn > >-----Original Message----- >From: Mika Nystrom [mailto:mika at async.caltech.edu]=20 >Sent: Sunday, February 27, 2011 3:30 PM >To: Coleburn, Randy >Cc: m3devel at elegosoft.com >Subject: Re: [M3devel] results of threadtest program on Windows7=20 > >Hi Randy, > >You can try it with -tests POSIX as well. > >I find on user threads it runs very slowly (but it does run) because of how= > unfair >the thread scheduler is. > >Next step might be to whittle down the tests and see if you can get a failu= >re with >a single test running and -n 2. That would likely be the simplest scenario= > to >start further debugging from. > > Mika > >"Coleburn, Randy" writes: >>Mika et al: >> >>Thought I would try something else. >> >>I took the sources of your thread test program to an older XP machine that= > =3D >>has CM3 circa August 2008. This is the machine and implementation I used = >w=3D >>hen building a major project I did a couple years back. >> >>The thread test program does indeed build on this old system, but when I r= >u=3D >>n it, I get different results than with the latest HEAD branch code. =3D20 >> >>After it prints "running...printing oldest/median age/newest", on the next= > =3D >>line I get two periods ".." and now the program seems hung. I'll let it "= >r=3D >>un" for a few more minutes to see if anything else happens before killing = >i=3D >>t. >> >>At least we don't get the subscript and assertion failures on this older C= >M=3D >>3 platform. >> >>Regards, >>Randy Coleburn >> >> >>-----Original Message----- >>From: Coleburn, Randy=3D20 >>Sent: Sunday, February 27, 2011 2:09 PM >>To: m3devel at elegosoft.com >>Subject: Re: [M3devel] results of threadtest program on Windows7 >> >>Mika: >> >>Ok, I've updated to latest HEAD and I've also built Jay's m3sleep program. >> >>Here is what happens now when I run your threadtest program on Windows 7. >> >>C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest -tests ALL,-fo= >r=3D >>k >>Writing file...done >>Creating read threads...done >>Creating nxread threads...done >>Creating tryexcept threads...done >>Creating forktoomuch threads...done >>Creating alloc threads...done >>Creating creat threads...done >>Creating lock threads...done >>running...printing oldest/median age/newest >> >> >>*** >>*** runtime error: >>*** An array subscript was out of range. >>*** file "..\src\runtime\common\RTCollector.m3", line 418 >>*** >> >> >> >>*** >>*** runtime error: >>*** <*ASSERT*> failed. >>*** file "..\src\thread\WIN32\ThreadWin32.m3", line 841 >>*** >> >>The last error repeats ad infinitum until I press CNTRL-C to abort. >> >>I'll send more info on the Windows install of Modula3 in a subsequent post= >. >> >>Regards, >>Randy Coleburn >> >>-----Original Message----- >>From: Mika Nystrom [mailto:mika at async.caltech.edu]=3D20 >>Sent: Saturday, February 26, 2011 12:55 PM >>To: Coleburn, Randy >>Cc: m3devel at elegosoft.com >>Subject: Re: [M3devel] results of threadtest program on Windows7=3D20 >> >>Hi Randy, >> >>Hm yes it looks like my Windows programming skills leave something >>to be desired. >> >>You can run the thread tester while skipping a test as follows >> >> threadtest -tests ALL,-fork >> >>(for instance) >> >>if you just run=3D20 >> >> threadtest -sadfassdaf >> >>it'll print the tests that are available. >> >>As it happens, I just had to upgrade my windows 2000 system to windows 7. >>Can you give me a very brief description of what you did to install Modula= >-=3D >>3 >>on this system? >> >> Mika >> >>"Coleburn, Randy" writes: >>>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>>Content-Type: text/plain; charset=3D3D"us-ascii" >>>Content-Transfer-Encoding: quoted-printable >>> >>>Mika: >>> >>>I've finally managed to get cm3 rebuilt on Windows 7 again. >>> >>>So, I ran your threadtest program. >>> >>>Here are the results. Note the "..." is where I cut out a bunch of the r= >e=3D >>p=3D3D >>>eating "ERROR FApply" messages. >>> >>>C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe >>>Writing file...done >>>Creating read threads...done >>>Creating fork threads...done >>>Creating alloc threads...done >>>Creating lock threads...done >>>running...printing oldest/median age/newest >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>. >>>. >>>. >>>*** >>>*** runtime error: >>>*** An enumeration or subrange value was out of range. >>>*** file "..\src\Main.m3", line 340 >>>*** >>> >>>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The sy= >ste=3D >>m c=3D3D >>>annot find the file specified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>. >>>. >>>. >>>laziest thread is 0/0/ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The sy= >ste=3D >>m c=3D3D >>>annot find the file specified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>. >>>. >>>. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>ERROR FApply: OSError.E: ErrorCode=3D3D3D2: The system cannot find the f= >ile=3D >> sp=3D3D >>>ecified. >>>Stack trace: >>> FP PC Procedure >>>--------- --------- ------------------------------- >>>0x30fbd0 0x127218a PutStats + 0x1a3 in ..\src\Main.m3 >>>0x30fcc0 0x1273825 Main_M3 + 0x11db(!) in ..\src\Main.m3 >>> >>>Regards, >>>Randy Coleburn >>> >>>--_000_D67F02DDC62F7545A6B84C285F88F3E6EE25C849atlex02srv_ >>>Content-Type: text/html; charset=3D3D"us-ascii" >>>Content-Transfer-Encoding: quoted-printable >>>