[M3devel] deadlock... in ThreadPThread.m3?

Sat Jan 8 03:08:26 CET 2011

The basic requirement, I know it sounds wierd at first, but is to hold *all* mutexes across the fork.
Consider a multi-threaded program, that calls fork, possibly on multiple threads.

If some threads are doing "random" work and some are calling fork, then the state of the mutexes will be arbitrary in the child process and the child process will inevitably hand/deadlock upon attempted to use them.

The model in mind is that there is not any central "control". Arbitrary code, in aribtrary threads, can be calling fork at arbitrary times.
And it makes a fair amount of sense, I think.
Any "component" (library, module, etc.) that uses any mutexes should register a fork prepare and takes all of its mutexes.

What problems would it cause to take the heap lock around the mutex init?
upgrade and build-all works, which does excerise this path some.
  Granted, the compiler isn't aggressively multithreaded, but does have the GC thread.

I guess we can release initMu before registering the cleanup??

 - Jay

----------------------------------------
> From: hosking at cs.purdue.edu
> Date: Fri, 7 Jan 2011 20:31:36 -0500
> To: jay.krell at cornell.edu
> CC: m3devel at elegosoft.com
> Subject: Re: [M3devel] deadlock... in ThreadPThread.m3?
>
> No, we don't want that since it can cause other problems. Better is not to be holding LockHeap when you acquire initMu in AtForkPrepare.
> Also, do we really need to hold LockHeap across the fork? I don't see the need for it.
>
>
> Antony Hosking | Associate Professor | Computer Science | Purdue University
> 305 N. University Street | West Lafayette | IN 47907 | USA
> Office +1 765 494 6001 | Mobile +1 765 427 5484
>
>
>
>
> On Jan 7, 2011, at 7:24 PM, Jay K wrote:
>
> >
> > ?
> >
> > Index: PTHREAD/ThreadPThread.m3
> > ===================================================================
> > RCS file: /usr/cvs/cm3/m3-libs/m3core/src/thread/PTHREAD/ThreadPThread.m3,v
> > retrieving revision 1.244
> > diff -u -w -r1.244 ThreadPThread.m3
> > --- PTHREAD/ThreadPThread.m3 7 Jan 2011 20:28:59 -0000 1.244
> > +++ PTHREAD/ThreadPThread.m3 8 Jan 2011 00:24:26 -0000
> > @@ -97,8 +97,10 @@
> > Clean: PROCEDURE(root: REFANY)) =
> > VAR mutex := pthread_mutex_new();
> > BEGIN
> > + LockHeap();
> > TRY
> > WITH r = pthread_mutex_lock(initMu) DO <*ASSERT r=0*> END;
> > + TRY
> > (* Did someone else win? *)
> > IF m # NIL THEN RETURN END;
> > (* We won, but we might have failed to allocate! *)
> > @@ -109,6 +111,9 @@
> > FINALLY
> > WITH r = pthread_mutex_unlock(initMu) DO <*ASSERT r=0*> END;
> > pthread_mutex_delete(mutex);
> > + END
> > + FINALLY
> > + UnlockHeap();
> > END;
> > END InitMutex;
> >
> >
> >
> > I'll try this.
> >
> >
> > - Jay
> >
> >
> > ----------------------------------------
> >> From: jay.krell at cornell.edu
> >> To: hosking at cs.purdue.edu; mika at async.caltech.edu
> >> Date: Fri, 7 Jan 2011 23:51:16 +0000
> >> CC: m3devel at elegosoft.com
> >> Subject: Re: [M3devel] deadlock... in ThreadPThread.m3?
> >>
> >>
> >> Btw, I didn't entirely guess, but it looks like my analysis was incomplete.
> >> Mostly the locks aren't taken together. But not entirely..
> >>
> >>
> >> I think I see the problem or almost.
> >>
> >>
> >>>> Thread 64 (process 32362):
> >>>> #0 0x9710b142 in semaphore_wait_signal_trap ()
> >>>> #1 0x97110c46 in pthread_mutex_lock ()
> >>>> #2 0x003c4a0a in ThreadPThread__pthread_mutex_lock (i=0x613ca0) at ../src/thread/PTHREAD/ThreadPThreadC.c:452
> >>>> #3 0x003c3bb3 in RTOS__LockHeap () at ../src/thread/PTHREAD/ThreadPThread.m3:1336
> >>>> #4 0x003b023b in RTHeapRep__RegisterFinalCleanup (r=0x2430f38, p=0x3bed6f) at ../src/runtime/common/RTCollector.m3:2148
> >>>> #5 0x003bef65 in ThreadPThread__InitMutex (m=0x2430f3c, root=0x2430f38, Clean=0x3bed6f) at ../src/thread/PTHREAD/ThreadPThread.m3:106
> >>>> #6 0x003befce in ThreadPThread__LockMutex (m=0x2430f38) at ../src/thread/PTHREAD/ThreadPThread.m3:117
> >>
> >>
> >> has initMu and is trying to LockHeap
> >>
> >>
> >> We can release initMu before calling RegisterFinalCleanup here, right?
> >>
> >>
> >> And then AtForkPrepare locks the heap before initMu.
> >>
> >>
> >> locks := ARRAY [0..3] OF pthread_mutex_t{activeMu, slotsMu, initMu, perfMu};
> >>
> >>
> >> PROCEDURE AtForkPrepare() =
> >> VAR me := GetActivation();
> >> act: Activation;
> >> cond: Condition;
> >> BEGIN
> >> Acquire(joinMu);
> >> LockHeap();
> >> FOR i := FIRST(locks) TO LAST(locks) DO
> >> PThreadLockMutex(locks[i], ThisLine(), i);
> >> END;
> >>
> >>
> >> - Jay
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> ----------------------------------------
> >>> From: jay.krell at cornell.edu
> >>> To: hosking at cs.purdue.edu; mika at async.caltech.edu
> >>> CC: m3devel at elegosoft.com
> >>> Subject: RE: [M3devel] deadlock... in ThreadPThread.m3?
> >>> Date: Fri, 7 Jan 2011 20:31:00 +0000
> >>>
> >>>
> >>> We need a lock order. I guessed. Maybe poorly. I/we should study the code and these stacks more and determine its validity.
> >>>
> >>>
> >>> Here is the code, *slightly* altered just now for debuggability.
> >>>
> >>>
> >>> VAR
> >>> locks := ARRAY [0..3] OF pthread_mutex_t{activeMu, slotsMu, initMu, perfMu};
> >>>
> >>>
> >>>
> >>> PROCEDURE AtForkPrepare() =
> >>> VAR me := GetActivation();
> >>> act: Activation;
> >>> cond: Condition;
> >>> BEGIN
> >>> Acquire(joinMu);
> >>> LockHeap();
> >>> FOR i := FIRST(locks) TO LAST(locks) DO
> >>> PThreadLockMutex(locks[i], ThisLine(), i);
> >>> END;
> >>> (* Walk activations and lock all threads, conditions.
> >>> * NOTE: We have initMu, activeMu, so slots
> >>> * won't change, conditions and mutexes
> >>> * won't be initialized on-demand.
> >>> *)
> >>> act := me;
> >>> REPEAT
> >>> PThreadLockMutex(act.mutex, ThisLine(), -1);
> >>> (*PThreadLockMutex(act.waitingOn, ThisLine());*)
> >>> cond := slots[act.slot].join;
> >>> IF cond # NIL THEN PThreadLockMutex(cond.mutex, ThisLine(), -2) END;
> >>> act := act.next;
> >>> UNTIL act = me;
> >>> END AtForkPrepare;
> >>>
> >>>
> >>>
> >>> Mika, if this easy to reproduce, please try again from head.
> >>> The problem is not fixed, but there's a little more information that will appear.
> >>>
> >>>
> >>> - Jay
> >>>
> >>>
> >>>
> >>> ----------------------------------------
> >>>> From: hosking at cs.purdue.edu
> >>>> Date: Fri, 7 Jan 2011 10:42:38 -0500
> >>>> To: mika at async.caltech.edu
> >>>> CC: m3devel at elegosoft.com
> >>>> Subject: Re: [M3devel] deadlock... in ThreadPThread.m3?
> >>>>
> >>>> This has something to do with the fork handlers Jay introduced, which run when a process is forked. They invoke LockHeap in the parent before the fork, and UnlockHeap in the parent after the fork. It appears that your threads are stuck waiting on LockHeap. Also, we have two threads in your trace trying to acquire initMu. Jay, I think we have a deadlock caused by AtForkPrepare. Ideas?
> >>>>
> >>>> On Jan 7, 2011, at 10:14 AM, Mika Nystrom wrote:
> >>>>
> >>>>> Hi m3devel (especially Tony),
> >>>>>
> >>>>> I'm having a problem with a Modula-3 program that seems to deadlock
> >>>>> with one of my 60+ threads stuck in Process.Create. Is it possible
> >>>>> there is a bug in ThreadPThread.m3? (Would be really nice to have a
> >>>>> deadlock detector here, you know... rather than have the program go
> >>>>> catatonic when there's a deadlock. Deadlock is stable so the detector
> >>>>> could run very infrequently---i.e., be very inefficient---and still be
> >>>>> extremely useful....)
> >>>>>
> >>>>> I don't think there is anything I ought to be able to do from "user
> >>>>> code" that would cause Process.Create to be part of a deadlock cycle.
> >>>>> Or could a "user code" deadlock cycle (cycle of Modula-3 MUTEXes)
> >>>>> possibly exhibit itself as a failure of Process.Create to complete??
> >>>>>
> >>>>> Below are backtraces of the threads stuck in some sort of locking
> >>>>> activity, platform is I386_DARWIN at head as of a couple of weeks ago.
> >>>>> (I deleted the threads that are up to other things than locking.)
> >>>>>
> >>>>> Mika
> >>>>>
> >>>>> Thread 68 (process 32362):
> >>>>> #0 0x9710b142 in semaphore_wait_signal_trap ()
> >>>>> #1 0x97110c46 in pthread_mutex_lock ()
> >>>>> #2 0x003c4a0a in ThreadPThread__pthread_mutex_lock (i=0x613c20) at ../src/thread/PTHREAD/ThreadPThreadC.c:452
> >>>>> #3 0x003c36e1 in ThreadPThread__PThreadLockMutex (mutex=0x613c20, line=1279) at ../src/thread/PTHREAD/ThreadPThread.m3:1256
> >>>>> #4 0x003c37e3 in ThreadPThread__AtForkPrepare () at ../src/thread/PTHREAD/ThreadPThread.m3:1279
> >>>>> #5 0x97137217 in _cthread_fork_prepare ()
> >>>>> #6 0x97137155 in fork ()
> >>>>> #7 0x00352910 in ProcessPosixCommon__Create_ForkExec (cmd=0x242c2ac, params=0x242c3ac, env=0x0, wd=0x57bf44, stdin=0x0, stdout=0x239f9a4, stderr=0x2433508) at ../src/os/POSIX/ProcessPosixC
> >>>>> ommon.m3:75
> >>>>> #8 0x00353de5 in Process__Create (cmd=0x242c2ac, params=0x242c3ac, env=0x0, wd=0x57bf44, stdin=0x0, stdout=0x239f9a4, stderr=0x2433508) at ../src/os/POSIX/ProcessPosix.m3:21
> >>>>> #9 0x00316384 in ProcUtils__Apply__Exec.1072 () at ../src/ProcUtils.m3:180
> >>>>> #10 0x00316d6e in ProcUtils__Apply (self=0x242f160) at ../src/ProcUtils.m3:274
> >>>>> #11 0x003c0ea5 in ThreadPThread__RunThread (me=0x2009600) at ../src/thread/PTHREAD/ThreadPThread.m3:450
> >>>>> #12 0x003c0b9a in ThreadPThread__ThreadBase (param=0x2009600) at ../src/thread/PTHREAD/ThreadPThread.m3:422
> >>>>> #13 0x9713885d in _pthread_start ()
> >>>>> #14 0x971386e2 in thread_start ()
> >>>>>
> >>>>> Thread 65 (process 32362):
> >>>>> #0 0x9710b142 in semaphore_wait_signal_trap ()
> >>>>> #1 0x97110c46 in pthread_mutex_lock ()
> >>>>> #2 0x003c4a0a in ThreadPThread__pthread_mutex_lock (i=0x613c20) at ../src/thread/PTHREAD/ThreadPThreadC.c:452
> >>>>> #3 0x003beefd in ThreadPThread__InitMutex (m=0x242a6f8, root=0x242a6f4, Clean=0x3bed6f) at ../src/thread/PTHREAD/ThreadPThread.m3:101
> >>>>> #4 0x003befce in ThreadPThread__LockMutex (m=0x242a6f4) at ../src/thread/PTHREAD/ThreadPThread.m3:117
> >>>>> #5 0x00355ff6 in Rd__GetChar (rd=0x242a6f4) at ../src/rw/Rd.m3:33
> >>>>> #6 0x00364b5d in Lex__Skip (rd=0x242a6f4, cs=0x587ff8) at ../src/fmtlex/Lex.m3:46
> >>>>> #7 0x00366e53 in Lex__ReadReal (rd=0x242a6f4, sign=0xb1cf0603, digits=0xb1cf061c, digCnt=0xb1cf0608, exp=0xb1cf060c) at ../src/fmtlex/Lex.m3:569
> >>>>> #8 0x003678f0 in Lex__LongReal (rd=0x242a6f4) at ../src/fmtlex/Lex.m3:662
> >>>>> #9 0x0036834e in Scan__LongReal (txt=0x1753494) at ../src/fmtlex/Scan.m3:77
> >>>>> #10 0x002603f5 in SchemeInputPort__NextToken (t=0x2380de4, wx=0x242a4a8) at ../src/SchemeInputPort.m3:439
> >>>>> #11 0x00260756 in SchemeInputPort__ReadTail2 (t=0x2380de4, wx=0x242a4a8) at ../src/SchemeInputPort.m3:240
> >>>>> #12 0x0025ebdd in SchemeInputPort__Read (t=0x2380de4) at ../src/SchemeInputPort.m3:179
> >>>>> #13 0x0025edad in SchemeInputPort__Read (t=0x2380de4) at ../src/SchemeInputPort.m3:191
> >>>>> #14 0x00260738 in SchemeInputPort__ReadTail2 (t=0x2380de4, wx=0x242a40c) at ../src/SchemeInputPort.m3:240
> >>>>> #15 0x0025ebdd in SchemeInputPort__Read (t=0x2380de4) at ../src/SchemeInputPort.m3:179
> >>>>> #16 0x00260738 in SchemeInputPort__ReadTail2 (t=0x2380de4, wx=0x239ddc4) at ../src/SchemeInputPort.m3:240
> >>>>> #17 0x0025ebdd in SchemeInputPort__Read (t=0x2380de4) at ../src/SchemeInputPort.m3:179
> >>>>> #18 0x00260738 in SchemeInputPort__ReadTail2 (t=0x2380de4, wx=0x239dc4c) at ../src/SchemeInputPort.m3:240
> >>>>> #19 0x0025ebdd in SchemeInputPort__Read (t=0x2380de4) at ../src/SchemeInputPort.m3:179
> >>>>> #20 0x00260738 in SchemeInputPort__ReadTail2 (t=0x2380de4, wx=0x239da8c) at ../src/SchemeInputPort.m3:240
> >>>>> #21 0x0025ebdd in SchemeInputPort__Read (t=0x2380de4) at ../src/SchemeInputPort.m3:179
> >>>>> #22 0x00260738 in SchemeInputPort__ReadTail2 (t=0x2380de4, wx=0x239d9b4) at ../src/SchemeInputPort.m3:240
> >>>>> #23 0x0025ebdd in SchemeInputPort__Read (t=0x2380de4) at ../src/SchemeInputPort.m3:179
> >>>>> #24 0x0024db82 in Scheme__LoadPort (t=0x237018c, in=0x2380de4) at ../src/Scheme.m3:317
> >>>>> #25 0x0024ae56 in Scheme__LoadRd (t=0x237018c, rd=0x2379cfc, fn=0x0) at ../src/Scheme.m3:174
> >>>>> #26 0x0024e659 in Scheme__ReadInitialFiles (t=0x237018c, files=0x4afc8c) at ../src/Scheme.m3:134
> >>>>> #27 0x0024ea46 in Scheme__Init2 (t=0x237018c, input=0x15ac4ec, output=0x15ac514, files=0x4afc8c, env=0x0) at ../src/Scheme.m3:127
> >>>>> #28 0x0024a89a in Scheme__Init (t=0x237018c, files=0x4afc8c, env=0x0) at ../src/Scheme.m3:102
> >>>>> #29 0x0002abbc in SchemeCalculator__Init (t=0x237017c, code=0x225541c) at ../src/SchemeCalculator.m3:28
> >>>>> #30 0x0000e373 in EuroDriver__OApply (cl=0x2367b70) at ../src/EuroDriver.m3:865
> >>>>> #31 0x003c0ea5 in ThreadPThread__RunThread (me=0x2008b80) at ../src/thread/PTHREAD/ThreadPThread.m3:450
> >>>>> #32 0x003c0b9a in ThreadPThread__ThreadBase (param=0x2008b80) at ../src/thread/PTHREAD/ThreadPThread.m3:422
> >>>>> #33 0x9713885d in _pthread_start ()
> >>>>> #34 0x971386e2 in thread_start ()
> >>>>>
> >>>>> Thread 64 (process 32362):
> >>>>> #0 0x9710b142 in semaphore_wait_signal_trap ()
> >>>>> #1 0x97110c46 in pthread_mutex_lock ()
> >>>>> #2 0x003c4a0a in ThreadPThread__pthread_mutex_lock (i=0x613ca0) at ../src/thread/PTHREAD/ThreadPThreadC.c:452
> >>>>> #3 0x003c3bb3 in RTOS__LockHeap () at ../src/thread/PTHREAD/ThreadPThread.m3:1336
> >>>>> #4 0x003b023b in RTHeapRep__RegisterFinalCleanup (r=0x2430f38, p=0x3bed6f) at ../src/runtime/common/RTCollector.m3:2148
> >>>>> #5 0x003bef65 in ThreadPThread__InitMutex (m=0x2430f3c, root=0x2430f38, Clean=0x3bed6f) at ../src/thread/PTHREAD/ThreadPThread.m3:106
> >>>>> #6 0x003befce in ThreadPThread__LockMutex (m=0x2430f38) at ../src/thread/PTHREAD/ThreadPThread.m3:117
> >>>>> #7 0x00355ff6 in Rd__GetChar (rd=0x2430f38) at ../src/rw/Rd.m3:33
> >>>>> #8 0x00364b5d in Lex__Skip (rd=0x2430f38, cs=0x587ff8) at ../src/fmtlex/Lex.m3:46
> >>>>> #9 0x00366e53 in Lex__ReadReal (rd=0x2430f38, sign=0xb1c6e7c3, digits=0xb1c6e7dc, digCnt=0xb1c6e7c8, exp=0xb1c6e7cc) at ../src/fmtlex/Lex.m3:569
> >>>>> #10 0x003678f0 in Lex__LongReal (rd=0x2430f38) at ../src/fmtlex/Lex.m3:662
> >>>>> #11 0x0036834e in Scan__LongReal (txt=0x1753494) at ../src/fmtlex/Scan.m3:77
> >>>>> #12 0x002603f5 in SchemeInputPort__NextToken (t=0x237ede4, wx=0x2430e44) at ../src/SchemeInputPort.m3:439
> >>>>> #13 0x00260756 in SchemeInputPort__ReadTail2 (t=0x237ede4, wx=0x2430e44) at ../src/SchemeInputPort.m3:240
> >>>>> #14 0x0025ebdd in SchemeInputPort__Read (t=0x237ede4) at ../src/SchemeInputPort.m3:179
> >>>>> #15 0x00260738 in SchemeInputPort__ReadTail2 (t=0x237ede4, wx=0x2430db0) at ../src/SchemeInputPort.m3:240
> >>>>> #16 0x0025ebdd in SchemeInputPort__Read (t=0x237ede4) at ../src/SchemeInputPort.m3:179
> >>>>> #17 0x00260738 in SchemeInputPort__ReadTail2 (t=0x237ede4, wx=0x2430cd8) at ../src/SchemeInputPort.m3:240
> >>>>> #18 0x0025ebdd in SchemeInputPort__Read (t=0x237ede4) at ../src/SchemeInputPort.m3:179
> >>>>> #19 0x0024db82 in Scheme__LoadPort (t=0x236e18c, in=0x237ede4) at ../src/Scheme.m3:317
> >>>>> #20 0x0024ae56 in Scheme__LoadRd (t=0x236e18c, rd=0x2377cfc, fn=0x0) at ../src/Scheme.m3:174
> >>>>> #21 0x0024e659 in Scheme__ReadInitialFiles (t=0x236e18c, files=0x4afc8c) at ../src/Scheme.m3:134
> >>>>> #22 0x0024ea46 in Scheme__Init2 (t=0x236e18c, input=0x15ac4ec, output=0x15ac514, files=0x4afc8c, env=0x0) at ../src/Scheme.m3:127
> >>>>> #23 0x0024a89a in Scheme__Init (t=0x236e18c, files=0x4afc8c, env=0x0) at ../src/Scheme.m3:102
> >>>>> #24 0x0002abbc in SchemeCalculator__Init (t=0x236e17c, code=0x225541c) at ../src/SchemeCalculator.m3:28
> >>>>> #25 0x0000e373 in EuroDriver__OApply (cl=0x2367ad8) at ../src/EuroDriver.m3:865
> >>>>> #26 0x003c0ea5 in ThreadPThread__RunThread (me=0x2008af0) at ../src/thread/PTHREAD/ThreadPThread.m3:450
> >>>>> #27 0x003c0b9a in ThreadPThread__ThreadBase (param=0x2008af0) at ../src/thread/PTHREAD/ThreadPThread.m3:422
> >>>>> #28 0x9713885d in _pthread_start ()
> >>>>> #29 0x971386e2 in thread_start ()
> >>>>>
> >>>>> Thread 60 (process 32362):
> >>>>> #0 0x9710b142 in semaphore_wait_signal_trap ()
> >>>>> #1 0x97110c46 in pthread_mutex_lock ()
> >>>>> #2 0x003c4a0a in ThreadPThread__pthread_mutex_lock (i=0x2006bc0) at ../src/thread/PTHREAD/ThreadPThreadC.c:452
> >>>>> #3 0x003befff in ThreadPThread__LockMutex (m=0x22cc724) at ../src/thread/PTHREAD/ThreadPThread.m3:119
> >>>>> #4 0x003bed47 in Thread__Acquire (m=0x22cc724) at ../src/thread/PTHREAD/ThreadPThread.m3:81
> >>>>> #5 0x002d9fb9 in SX__Lock (arr=0x235a9a4) at ../src/SX.m3:318
> >>>>> #6 0x002d983f in SXSelect__WaitE (on=0xb1beccd8, except=0x0, touched=0x0) at ../src/SX.m3:241
> >>>>> #7 0x002d8e0d in SXSelect__Wait (on=0xb1beccd8, touched=0x0) at ../src/SX.m3:149
> >>>>> #8 0x00010c7d in EuroDriver__OApply__MainLoop.1174 () at ../src/EuroDriver.m3:832
> >>>>> #9 0x0000e830 in EuroDriver__OApply (cl=0x22aad10) at ../src/EuroDriver.m3:892
> >>>>> #10 0x003c0ea5 in ThreadPThread__RunThread (me=0x2007080) at ../src/thread/PTHREAD/ThreadPThread.m3:450
> >>>>> #11 0x003c0b9a in ThreadPThread__ThreadBase (param=0x2007080) at ../src/thread/PTHREAD/ThreadPThread.m3:422
> >>>>> #12 0x9713885d in _pthread_start ()
> >>>>> #13 0x971386e2 in thread_start ()
> >>>>>
> >>>>> Thread 59 (process 32362):
> >>>>> #0 0x9710b142 in semaphore_wait_signal_trap ()
> >>>>> #1 0x97110c46 in pthread_mutex_lock ()
> >>>>> #2 0x003c4a0a in ThreadPThread__pthread_mutex_lock (i=0x613c20) at ../src/thread/PTHREAD/ThreadPThreadC.c:452
> >>>>> #3 0x003beefd in ThreadPThread__InitMutex (m=0x2354a24, root=0x2354a20, Clean=0x3bed6f) at ../src/thread/PTHREAD/ThreadPThread.m3:101
> >>>>> #4 0x003befce in ThreadPThread__LockMutex (m=0x2354a20) at ../src/thread/PTHREAD/ThreadPThread.m3:117
> >>>>> #5 0x003bed47 in Thread__Acquire (m=0x2354a20) at ../src/thread/PTHREAD/ThreadPThread.m3:81
> >>>>> #6 0x00359e60 in RdClass__Lock (rd=0x2354a20) at ../src/rw/Rd.m3:521
> >>>>> #7 0x0025e4c4 in SchemeInputPort__Lock (t=0x2354b08) at ../src/SchemeInputPort.m3:72
> >>>>> #8 0x0025f3dc in SchemeInputPort__NextToken (t=0x2354b08, wx=0x2354b28) at ../src/SchemeInputPort.m3:299
> >>>>> #9 0x0025eb94 in SchemeInputPort__Read (t=0x2354b08) at ../src/SchemeInputPort.m3:177
> >>>>> #10 0x0024aea9 in Scheme__LoadEval (t=0x224418c, rd=0x2354a20) at ../src/Scheme.m3:291
> >>>>> #11 0x0024af22 in Scheme__LoadEvalText (t=0x224418c, txt=0x225541c) at ../src/Scheme.m3:306
> >>>>> #12 0x0002ad28 in SchemeCalculator__Calc (t=0x224417c, arr=0xb1b6ab04) at ../src/SchemeCalculator.m3:47
> >>>>> #13 0x0000f37c in EuroDriver__OApply__CalcOrderSize.1137 () at ../src/EuroDriver.m3:564
> >>>>> #14 0x0000ece1 in EuroDriver__OApply__SafetyViolation.1126 () at ../src/EuroDriver.m3:507
> >>>>> #15 0x00010a04 in EuroDriver__OApply__MainLoop.1174 () at ../src/EuroDriver.m3:795
> >>>>> #16 0x0000e830 in EuroDriver__OApply (cl=0x22aac78) at ../src/EuroDriver.m3:892
> >>>>> #17 0x003c0ea5 in ThreadPThread__RunThread (me=0x2007020) at ../src/thread/PTHREAD/ThreadPThread.m3:450
> >>>>> #18 0x003c0b9a in ThreadPThread__ThreadBase (param=0x2007020) at ../src/thread/PTHREAD/ThreadPThread.m3:422
> >>>>> #19 0x9713885d in _pthread_start ()
> >>>>> #20 0x971386e2 in thread_start ()
> >>>>>
> >>>>> Thread 52 (process 32362):
> >>>>> #0 0x9710b142 in semaphore_wait_signal_trap ()
> >>>>> #1 0x97110c46 in pthread_mutex_lock ()
> >>>>> #2 0x003c4a0a in ThreadPThread__pthread_mutex_lock (i=0x613ca0) at ../src/thread/PTHREAD/ThreadPThreadC.c:452
> >>>>> #3 0x003c3bb3 in RTOS__LockHeap () at ../src/thread/PTHREAD/ThreadPThread.m3:1336
> >>>>> #4 0x003a66fc in RTAllocator__AllocTraced (dataSize=24, dataAlignment=4, thread=0x161e144) at ../src/runtime/common/RTAllocator.m3:363
> >>>>> #5 0x003a5a99 in RTAllocator__GetTracedObj (def=0x616d54) at ../src/runtime/common/RTAllocator.m3:224
> >>>>> #6 0x003a550f in RTHooks__AllocateTracedObj (defn=0x616d54) at ../src/runtime/common/RTAllocator.m3:122
> >>>>> #7 0x003d3c0e in RTHooks__Concat (t=0x23669ec, u=0x56c460) at ../src/text/TextCat.m3:28
> >>>>> #8 0x002b0dc2 in TWSTester__MakeString (contract=0xb19e4dc0) at ../src/TWSTester.m3:92
> >>>>> #9 0x002b11ac in TWSTester__ReqMktData (t=0x2220800, tickerId=3, contract=0xb19e4dc0) at ../src/TWSTester.m3:136
> >>>>> #10 0x002bccaf in TWSTestBridge__MreqMktData (t=0x2220800, tickerId=3, contract=0xb19e4dc0) at ../src/TWSTestBridge.m3 => ../src/TWSBridgeG.mg:580
> >>>>> #11 0x002978ec in PMTWS__TSCApply (cl=0x2221044) at ../src/PMTWS.m3:210
> >>>>> #12 0x003c0ea5 in ThreadPThread__RunThread (me=0x161e100) at ../src/thread/PTHREAD/ThreadPThread.m3:450
> >>>>> #13 0x003c0b9a in ThreadPThread__ThreadBase (param=0x161e100) at ../src/thread/PTHREAD/ThreadPThread.m3:422
> >>>>> #14 0x9713885d in _pthread_start ()
> >>>>> #15 0x971386e2 in thread_start ()
> >>>>>
> >>>>> Thread 51 (process 32362):
> >>>>> #0 0x9710b142 in semaphore_wait_signal_trap ()
> >>>>> #1 0x97110c46 in pthread_mutex_lock ()
> >>>>> #2 0x003c4a0a in ThreadPThread__pthread_mutex_lock (i=0x613ca0) at ../src/thread/PTHREAD/ThreadPThreadC.c:452
> >>>>> #3 0x003c3bb3 in RTOS__LockHeap () at ../src/thread/PTHREAD/ThreadPThread.m3:1336
> >>>>> #4 0x003a66fc in RTAllocator__AllocTraced (dataSize=136, dataAlignment=8, thread=0x16265a4) at ../src/runtime/common/RTAllocator.m3:363
> >>>>> #5 0x003a581e in RTAllocator__GetTracedRef (def=0x5b1280) at ../src/runtime/common/RTAllocator.m3:202
> >>>>> #6 0x003a54da in RTHooks__AllocateTracedRef (defn=0x5b1280) at ../src/runtime/common/RTAllocator.m3:115
> >>>>> #7 0x000b03d0 in IntDBTable_gcoms_instrumentTbl__Put (tbl=0x235f464, key=0xb1962d7c, val=0xb1962d00) at ../I386_DARWIN/IntDBTable_gcoms_instrumentTbl.m3 => /usr/local/cm3/pkg/libm3/src/table/Table.mg:126
> >>>>> #8 0x000ad244 in DBTable_gcoms_instrumentMonitor__CopyC (cl=0x228c080, res=0x235f464) at ../I386_DARWIN/DBTable_gcoms_instrumentMonitor.m3 => /Users/mika/t/calarm/ratsql/src/TableMonitor.mg:362
> >>>>> #9 0x0008c6de in GCOMSPortfolio__Lock (t=0x2255cd8) at ../src/GCOMSPortfolio.m3:94
> >>>>> #10 0x00029faa in SXPortfolio__Scan (t=0x229a888) at ../src/SXPortfolio.m3:47
> >>>>> #11 0x0002a998 in SXPortfolio__Apply (cl=0x222057c) at ../src/SXPortfolio.m3:113
> >>>>> #12 0x003c0ea5 in ThreadPThread__RunThread (me=0x1626560) at ../src/thread/PTHREAD/ThreadPThread.m3:450
> >>>>> #13 0x003c0b9a in ThreadPThread__ThreadBase (param=0x1626560) at ../src/thread/PTHREAD/ThreadPThread.m3:422
> >>>>> #14 0x9713885d in _pthread_start ()
> >>>>> #15 0x971386e2 in thread_start ()
> >>>>>
> >>>>> Thread 27 (process 32362):
> >>>>> #0 0x9710b142 in semaphore_wait_signal_trap ()
> >>>>> #1 0x97110c46 in pthread_mutex_lock ()
> >>>>> #2 0x003c4a0a in ThreadPThread__pthread_mutex_lock (i=0x161e470) at ../src/thread/PTHREAD/ThreadPThreadC.c:452
> >>>>> #3 0x003befff in ThreadPThread__LockMutex (m=0x22440a4) at ../src/thread/PTHREAD/ThreadPThread.m3:119
> >>>>> #4 0x002dbd63 in SXLongReal__Update (v=0x2244010, newValue=1293721230.087589, when=1293721230.087605) at ../I386_DARWIN/SXLongReal.m3 => ../src/SXType.mg:79
> >>>>> #5 0x002dc086 in SXLongReal__SetVar (v=0x2244010, newValue=1293721230.087589, when=1293721230.087605) at ../I386_DARWIN/SXLongReal.m3 => ../src/SXType.mg:98
> >>>>> #6 0x00300c16 in SXTimer__Loop (cl=0x173934c) at ../src/SXTimer.m3:64
> >>>>> #7 0x003c0ea5 in ThreadPThread__RunThread (me=0x2001ec0) at ../src/thread/PTHREAD/ThreadPThread.m3:450
> >>>>> #8 0x003c0b9a in ThreadPThread__ThreadBase (param=0x2001ec0) at ../src/thread/PTHREAD/ThreadPThread.m3:422
> >>>>> #9 0x9713885d in _pthread_start ()
> >>>>> #10 0x971386e2 in thread_start ()
> >>>>>
> >>>>> Thread 2 (process 32362):
> >>>>> #0 0x9710b142 in semaphore_wait_signal_trap ()
> >>>>> #1 0x97110c46 in pthread_mutex_lock ()
> >>>>> #2 0x003c4a0a in ThreadPThread__pthread_mutex_lock (i=0x613c20) at ../src/thread/PTHREAD/ThreadPThreadC.c:452
> >>>>> #3 0x003beefd in ThreadPThread__InitMutex (m=0x23480ac, root=0x23480a8, Clean=0x3bed6f) at ../src/thread/PTHREAD/ThreadPThread.m3:101
> >>>>> #4 0x003befce in ThreadPThread__LockMutex (m=0x23480a8) at ../src/thread/PTHREAD/ThreadPThread.m3:117
> >>>>> #5 0x00355ff6 in Rd__GetChar (rd=0x23480a8) at ../src/rw/Rd.m3:33
> >>>>> #6 0x00364b5d in Lex__Skip (rd=0x23480a8, cs=0x587ff8) at ../src/fmtlex/Lex.m3:46
> >>>>> #7 0x00366e53 in Lex__ReadReal (rd=0x23480a8, sign=0xb0080ca3, digits=0xb0080cbc, digCnt=0xb0080ca8, exp=0xb0080cac) at ../src/fmtlex/Lex.m3:569
> >>>>> #8 0x003678f0 in Lex__LongReal (rd=0x23480a8) at ../src/fmtlex/Lex.m3:662
> >>>>> #9 0x0036834e in Scan__LongReal (txt=0x234807c) at ../src/fmtlex/Scan.m3:77
> >>>>> #10 0x003260a0 in XTime__ClApply (cl=0x15971b8) at ../src/XTime.m3:185
> >>>>> #11 0x003c0ea5 in ThreadPThread__RunThread (me=0x161a0b0) at ../src/thread/PTHREAD/ThreadPThread.m3:450
> >>>>> #12 0x003c0b9a in ThreadPThread__ThreadBase (param=0x161a0b0) at ../src/thread/PTHREAD/ThreadPThread.m3:422
> >>>>> #13 0x9713885d in _pthread_start ()
> >>>>> #14 0x971386e2 in thread_start ()
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>