[M3devel] CM3 crashing

Mon Jan 13 22:02:52 CET 2014

Tony:

Here are a couple more runs on 64-bit Windows 7 that yield different results, albeit using the "-verbose" option.

These may give some more clues.

I note that in both cases it appears a read thread and a fork thread get the same ID (0) which looks very suspicious.

By my read of the code (WITH id = i * nPer + j), I think the 2nd fork thread should be ID=10, not ID=0.

Oops, now I see the problem:  there is a bug in the PutInt procedure.  The test (IF c > 10) should read instead (IF c >= 10).  I'll commit a fix for that momentarily.

But this fix just solves the output problem; it doesn't affect the fact that the program is misbehaving and crashing.

RUN #1:

C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe @M3paranoidgc -verbose
Writing file...done
Creating read threads...
   read=0
   read=1
   read=2
done
Creating fork threads...
   fork=9
   fork=0
   fork=11
done
Creating alloc threads...
   alloc=15
   alloc=16
   alloc=17
done
Creating lock threads...
   lock=21
   lock=22
   lock=23
done
running...printing oldest/median age/newest

***
*** runtime error:
***    Attempt to reference an illegal memory location.
***    pc = 0xc286e0 = Init + 0x79 in ..\src\rw\FileRd.m3
***

Stack trace:
   FP         PC      Procedure
---------  ---------  -------------------------------
0x7cfd60   0xc6afeb  SystemError + 0x64 in ..\src\runtime\NT386\RTSignal.m3
0x7cfda8   0xc286e0  Init + 0x79 in ..\src\rw\FileRd.m3
0x7cfdd4   0xc2865d  Open + 0x4d in ..\src\rw\FileRd.m3
0x7cff0c   0xc21350  RApply + 0x160 in ..\src\Main.m3
0x7cff48   0xc4b71b  ThreadBase + 0x255 in ..\src\thread\WIN32\ThreadWin32.m3
0x7cff54  0x76db336a  <???>
0x7cff94  0x7770bf32  <???>
.........  .........  ... more frames ...

RUN #2:

C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe -verbose
Writing file...done
Creating read threads...
   read=0
   read=1
   read=2
done
Creating fork threads...
   fork=9
   fork=0
   fork=11
done
Creating alloc threads...
   alloc=15
   alloc=16
   alloc=17
done
Creating lock threads...
   lock=21
   lock=22
   lock=23
done
running...printing oldest/median age/newest
..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0)
   read Thread 0 completed 534 loops.
   read Thread 1 completed 606 loops.
   read Thread 2 completed 398 loops.
   fork Thread 9 completed 8 loops.
   fork Thread 10 completed 8 loops.
   fork Thread 11 completed 8 loops.
   alloc Thread 15 completed 18296 loops.
   alloc Thread 16 completed 44871 loops.
   alloc Thread 17 completed 79125 loops.
   lock Thread 21 completed 3845080 loops.
   lock Thread 22 completed 2537613 loops.
   lock Thread 23 completed 4506151 loops.
..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0)
   read Thread 0 completed 790 loops.
   read Thread 1 completed 786 loops.
   read Thread 2 completed 675 loops.
   fork Thread 9 completed 9 loops.
   fork Thread 10 completed 9 loops.
   fork Thread 11 completed 9 loops.
   alloc Thread 15 completed 22091 loops.
   alloc Thread 16 completed 47532 loops.
   alloc Thread 17 completed 68901 loops.
  lock Thread 21 completed 4560731 loops.
   lock Thread 22 completed 3440795 loops.
   lock Thread 23 completed 6433538 loops.
..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0)
   read Thread 0 completed 725 loops.
   read Thread 1 completed 705 loops.
   read Thread 2 completed 617 loops.
   fork Thread 9 completed 10 loops.
   fork Thread 10 completed 9 loops.
   fork Thread 11 completed 9 loops.
   alloc Thread 15 completed 18560 loops.
   alloc Thread 16 completed 44487 loops.
   alloc Thread 17 completed 87219 loops.
   lock Thread 21 completed 3769840 loops.
   lock Thread 22 completed 3037581 loops.
   lock Thread 23 completed 6097922 loops.
..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0)
   read Thread 0 completed 825 loops.
   read Thread 1 completed 802 loops.
   read Thread 2 completed 669 loops.
   fork Thread 9 completed 9 loops.
   fork Thread 10 completed 10 loops.
   fork Thread 11 completed 9 loops.
   alloc Thread 15 completed 16877 loops.
   alloc Thread 16 completed 53277 loops.
   alloc Thread 17 completed 77986 loops.
   lock Thread 21 completed 4087218 loops.
   lock Thread 22 completed 3116357 loops.
   lock Thread 23 completed 6466214 loops.
..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0)
   read Thread 0 completed 684 loops.
   read Thread 1 completed 686 loops.
   read Thread 2 completed 648 loops.
   fork Thread 9 completed 10 loops.
   fork Thread 10 completed 9 loops.
   fork Thread 11 completed 10 loops.
   alloc Thread 15 completed 19717 loops.
   alloc Thread 16 completed 47797 loops.
   alloc Thread 17 completed 78110 loops.
   lock Thread 21 completed 4580435 loops.
   lock Thread 22 completed 2974247 loops.
   lock Thread 23 completed 5843850 loops.
..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0)
   read Thread 0 completed 748 loops.
   read Thread 1 completed 755 loops.
   read Thread 2 completed 639 loops.
   fork Thread 9 completed 9 loops.
   fork Thread 10 completed 10 loops.
   fork Thread 11 completed 9 loops.
   alloc Thread 15 completed 18037 loops.
   alloc Thread 16 completed 46293 loops.
   alloc Thread 17 completed 82560 loops.
   lock Thread 21 completed 4210008 loops.
   lock Thread 22 completed 2708008 loops.
   lock Thread 23 completed 6139723 loops.
..

***
*** runtime error:
***    Attempt to reference an illegal memory location.
***    pc = 0xc4e39a = Move + 0x50 in ..\src\runtime\common\RTCollector.m3
***

***
*** runtime error:
***    <*ASSERT*> failed.
***    file "..\src\thread\WIN32\ThreadWin32.m3", line 823
***

***
*** runtime error:
***    <*ASSERT*> failed.
***    file "..\src\thread\WIN32\ThreadWin32.m3", line 823
***
This last assert repeats indefinitely until you press CTRL-C.

--Randy

From: Coleburn, Randy
Sent: Monday, January 13, 2014 3:14 PM
To: Tony Hosking
Cc: m3devel
Subject: Re: [M3devel] CM3 crashing

Tony:

Yes, it is from the HEAD as of 17 December 2013.

Using the @M3paranoidgc flag,

On 32-bit Windows XP I get:

C:\cm3\Sandbox\cm3\m3-libs\m3core\tests\thread\NT386>threadtest.exe @M3paranoidgc
Writing file...done
Creating read threads...done
Creating fork threads...done
Creating alloc threads...done
Creating lock threads...done
running...printing oldest/median age/newest
.

***
*** runtime error:
***    Attempt to reference an illegal memory location.
***    pc = 0x431755 = RefSanityCheck + 0x2c in ..\src\runtime\common\RTCollector.m3
***

***
*** runtime error:
***    <*ASSERT*> failed.
***    file "..\src\thread\WIN32\ThreadWin32.m3", line 823
***

***
*** runtime error:
***    <*ASSERT*> failed.
***    file "..\src\thread\WIN32\ThreadWin32.m3", line 823
***
This last assert repeats indefinitely until you press CTRL-C.

On 64-bit Windows 7 I get:

C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe @M3paranoidgc
Writing file...done
Creating read threads...done
Creating fork threads...done
Creating alloc threads...done
Creating lock threads...done
running...printing oldest/median age/newest
.

***
*** runtime error:
***    An array subscript was out of range.
***    file "..\src\rw\FileRd.m3", line 83
***

***
*** runtime error:
***    An enumeration or subrange value was out of range.
***    file "..\src\runtime\common\RTType.m3", line 71
***

***
*** runtime error:
***    <*ASSERT*> failed.
***    file "..\src\thread\WIN32\ThreadWin32.m3", line 823
***

***
*** runtime error:
***    <*ASSERT*> failed.
***    file "..\src\thread\WIN32\ThreadWin32.m3", line 823
***
This last assert repeats indefinitely until you press CTRL-C.

--Randy

From: Tony Hosking [mailto:hosking at cs.purdue.edu]
Sent: Monday, January 13, 2014 3:03 PM
To: Coleburn, Randy
Cc: Peter McKinna; m3devel
Subject: EXT:Re: [M3devel] CM3 crashing

Is this from the head of source?
Also, please run with @M3paranoidgc flag.

On Jan 13, 2014, at 2:55 PM, Coleburn, Randy <rcolebur at SCIRES.COM<mailto:rcolebur at SCIRES.COM>> wrote:

I've also tested on 32-bit Windows XP and the thread test program crashes there also.

C:\cm3\Sandbox\cm3\m3-libs>set CM3_TARGET=NT386

C:\cm3\Sandbox\cm3\m3-libs>cm3 --version
Critical Mass Modula-3 version d5.9.0
  last updated: 2010-07-21
  compiled: 2013-12-17 17:28:52
  configuration: C:\cm3\bin\cm3.cfg
  host: NT386
  target: NT386

Here is the output:

C:\cm3\Sandbox\cm3\m3-libs\m3core\tests\thread\NT386>threadtest.exe --help
Writing file...done
Creating read threads...done
Creating fork threads...done
Creating alloc threads...done
Creating lock threads...done
running...printing oldest/median age/newest

***
*** runtime error:
***    Attempt to reference an illegal memory location.
***    pc = 0x42e34a = Move + 0x50 in ..\src\runtime\common\RTCollector.m3
***

***
*** runtime error:
***    <*ASSERT*> failed.
***    file "..\src\thread\WIN32\ThreadWin32.m3", line 823
***

***
*** runtime error:
***    <*ASSERT*> failed.
***    file "..\src\thread\WIN32\ThreadWin32.m3", line 823
***

This last assert repeats indefinitely until you press CTRL-C.

--Randy

From: Coleburn, Randy
Sent: Monday, January 13, 2014 2:17 PM
To: Tony Hosking; Peter McKinna
Cc: m3devel
Subject: Re: [M3devel] CM3 crashing

Tony et al:

The thread test program fails for me on 64-bit Windows 7.

I've listed output from a couple of runs below:

C:\cm3\Sandbox>cm3 --version
Critical Mass Modula-3 version d5.9.0
  last updated: 2010-07-21
  compiled: 2014-01-07 06:55:14
  configuration: C:\cm3\bin\cm3.cfg
  host: NT386
  target: NT386

FIRST RUN:

C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe
Writing file...done
Creating read threads...done
Creating fork threads...done
Creating alloc threads...done
Creating lock threads...done
running...printing oldest/median age/newest

***
*** runtime error:
***    Attempt to reference an illegal memory location.

***
*** runtime error:
***    <*ASSERT*> failed.
***    file "..\src\runtime\common\RTCollector.m3", line 418
***

***
*** runtime error:
***    <*ASSERT*> failed.
***    file "..\src\thread\WIN32\ThreadWin32.m3", line 823
***

***
*** runtime error:
***    <*ASSERT*> failed.
***    file "..\src\thread\WIN32\ThreadWin32.m3", line 823
***

               ...this last assertion repeats until you press CTRL-C to abort the program...

SECOND RUN:

C:\cm3\Sandbox\m3-libs\m3core\tests\thread\NT386>threadtest.exe /help
Writing file...done
Creating read threads...done
Creating fork threads...done
Creating alloc threads...done
Creating lock threads...done
running...printing oldest/median age/newest
..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock 0/0/0)
....

***
*** runtime error:
***    Attempt to reference an illegal memory location.
***    pc = 0x776f22d2
***

Stack trace:
   FP         PC      Procedure
---------  ---------  -------------------------------
0xa6f9a8   0xc6afeb  SystemError + 0x64 in ..\src\runtime\NT386\RTSignal.m3
0xa6f9d0  0x776f22d2  <???>
0xa6f9e8   0xc4a39b  LockMutex + 0x4f in ..\src\thread\WIN32\ThreadWin32.m3
0xa6fa10   0xc299f8  GetChar + 0x28 in ..\src\rw\Rd.m3
0xa6fb48   0xc213a8  RApply + 0x1b8 in ..\src\Main.m3
0xa6fb84   0xc4b71b  ThreadBase + 0x255 in ..\src\thread\WIN32\ThreadWin32.m3
0xa6fb90  0x76db336a  <???>
0xa6fbd0  0x7770bf32  <???>
.........  .........  ... more frames ...

Maybe this info will be useful in tracking down the problem.

--Randy

From: Tony Hosking [mailto:hosking at cs.purdue.edu]
Sent: Monday, January 13, 2014 11:04 AM
To: Peter McKinna
Cc: m3devel
Subject: EXT:Re: [M3devel] CM3 crashing

Let us assume that the user-level threads are functioning properly w.r.to GC (can someone confirm?).
In which case, it would be good to have as many eyes as possible take a look at the differences between ThreadPosixC.c (ProcessContext) and ThreadPThreadC.c (ProcessLive) to see if we can spot a problem.

As I understand it, the crash occurs even for non-concurrent (@M3noincremental) and non-generational (@M3nogenerational) GC.  Combined with my assumption that user threads work fine, it would seem to point the finger at stack scanning.  Can someone confirm?

If the failure is *only* with concurrent or generational collection then we might suspect unsafe code (perhaps newly introduced?) messing with heap references without keeping the collector informed.

On Jan 12, 2014, at 10:54 PM, Peter McKinna <peter.mckinna at gmail.com<mailto:peter.mckinna at gmail.com>> wrote:

Hey,
  I was trying to get a handle on that problem last year. The threadtest program is really a stress tester of the collector/allocator with pthreads. If you run it with just the tests read and alloc you pretty much always get a crash. If you run them with paranoidgc it will crash in the heap checker. Tony thought it was a clear cut  problem of the roots of some ref not being found on a thread stack. I mucked around with code to get some output and the stacks looked ok to me but I could be wrong. All stacks are checked whilst the threads are blocked in a signal handler, the design of which looks fine as far as I can tell. This test is characterised by some slow threads (the read threads) and a bunch of fast threads (the alloc threads). Even if you modify the test to have only one read thread the problem occurs. I have had misgivings about mixing signals and threads having been bitten many years ago, but really this is the only way the collector can get its raw refs to check.
My guess is its a subtle timing or lock problem maybe a lurking bug in the collector itself. One thing I noticed is that in ThreadPThread__ProcessStopped  the second call to   p(context, ((char *)context) + sizeof(ucontext_t)); according to the comment is to process the registers. But the registers should already be on the stack and anyway this call is a partial duplicate of the previous one.
It would be good to raise the priority on this problem. Trust in the collector has always been at the heart of m3 programs.
Regards Peter

On Mon, Jan 13, 2014 at 11:25 AM, <mika at async.caltech.edu<mailto:mika at async.caltech.edu>> wrote:
Yes it works in PM3 (still, since I use PM3 on FreeBSD4, never saw a reason to switch to CM3).

I figured there aren't actually implementations of Word.Xor, Word.Or,
and Word.And as procedures but they get inlined somehow?

BTW, any idea about what's wrong with pthreads?  Do you think the issue
is with the garbage collector or with the threads, off the top of your
head?

Tony Hosking writes:
>Are you saying passing these used to work in PM3?
>Sounds like a front-end bug.  I=92m curious what changed to break it.
>
>On Jan 12, 2014, at 12:58 PM, mika at async.caltech.edu<mailto:mika at async.caltech.edu> wrote:
>
>>=20
>> The code is:
>>=20
>>              HIntExpr.Xor =3D> RETURN NewConst(CBitwise(av, bv, =
>Word.Xor), ab)
>>            |
>>              HIntExpr.Bor =3D> RETURN NewConst(CBitwise(av, bv, =
>Word.Or), ab)
>>            |
>>              HIntExpr.Band =3D> RETURN NewConst(CBitwise(av, bv, =
>Word.And), ab)
>>=20
>> I guess it doesn't like passing Word.Xor, Word.Or, and Word.And ...=20
>
>
>
>Antony Hosking | Associate Professor | Computer Science | Purdue =
>University
>305 N. University Street | West Lafayette | IN 47907 | USA
>Mobile +1 765 427 5484<tel:%2B1%20765%20427%205484>
>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20140113/eea82060/attachment-0002.html>