[M3devel] results of threadtest program on Windows7

[Tony Hosking]

I'd like to debug the current problem because it does not seem to imply any problem with suspend/resume.  More likely is losing a reference in the stacks somehow.

Antony Hosking | Associate Professor | Computer Science | Purdue University
305 N. University Street | West Lafayette | IN 47907 | USA
Office +1 765 494 6001 | Mobile +1 765 427 5484




On Mar 12, 2011, at 7:40 PM, Jay K wrote:

> I thought maybe they'd show a way for us to get away with "preemptive suspend",
> a way to use SuspendThread/GetThreadContext with confidence.
> 
> So, you agree we want cooperative suspend then?
> And we should maybe get going on implementing it?
> Maybe do that before debugging Mika's problem?
>   Well, to repeat myself, I'm ok with looking into 32bit-only Windows for Mika's problem.
>   Though, clearly, as i've said, I have A LOT less time for Modula-3 now and probably ever.
> 
>  - Jay
> 
> Subject: Re: [M3devel] results of threadtest program on Windows7
> From: hosking at cs.purdue.edu
> Date: Sat, 12 Mar 2011 11:23:04 -0500
> CC: m3devel at elegosoft.com
> To: jay.krell at cornell.edu
> 
> I know they use cooperative suspend.
> We do in Jikes RVM too.
> That's another code base to look at.  Probably easier as it is all in Java.
> 
> On Mar 12, 2011, at 4:57 AM, Jay K wrote:
> 
> There is a lack of evidence that Java does things in a way at all resembling what Modula-3 does, on Win32.
>  There is precious little in the way of calling the Win32 functions SuspendThread and GetThreadContext.
> There is evidence that they use cooperative suspend.
> It isn't conclusive, to me..just because there is so much there, it is not immediately clear.
> That is, pretty clear there is code implementing cooperative suspend, but is that their entire story? I don't know.
> 
>  - Jay
> 
> From: jay.krell at cornell.edu
> To: hosking at cs.purdue.edu
> CC: m3devel at elegosoft.com
> Subject: RE: [M3devel] results of threadtest program on Windows7
> Date: Thu, 10 Mar 2011 21:03:58 +0000
> 
> I often forget this. Thank you.
> 
>  - Jay
> 
> From: hosking at cs.purdue.edu
> Date: Thu, 10 Mar 2011 11:50:40 -0500
> To: jay.krell at cornell.edu
> CC: m3devel at elegosoft.com
> Subject: Re: [M3devel] results of threadtest program on Windows7
> 
> Even better perhaps is to look at OpenJDK!
> 
> On Mar 10, 2011, at 6:05 AM, Jay K wrote:
> 
> Tony, more and more I think we want "cooperative suspend".
> That would address this, avoid scanning the entire stack only the live stack on FreeBSD and OpenBSD, and
> be much more portable, and bigger and slower.
> I'll see what Mono and Rotor ("sscli") do though.
> http://en.wikipedia.org/wiki/WoW64
> Apparently the Boehm GC has problems too.
>  
> We could probably do something like, note the ranges of all Modula-3 .dlls, and check that EIP is in one of them,
> else let the thread run longer and try suspend again. Maybe..
>  
>  - Jay
>  
> From: jay.krell at cornell.edu
> To: rcolebur at scires.com; m3devel at elegosoft.com
> Date: Thu, 10 Mar 2011 10:40:58 +0000
> Subject: Re: [M3devel] results of threadtest program on Windows7
> 
> oh, but to repeat, use of SuspendThread / GetThreadContext makes me nervous
>  
> http://zachsaw.blogspot.com/2010/11/wow64-bug-getthreadcontext-may-return.html
>  
> some of the talk is only about the wrong esp, but I wonder if it is all wrong sometimes.
> I think we can range check esp and reject if it is out of range, similar to the hack
> that was there for Win9x.
>  
> Maybe stick to 32bit OS while investigating threadtest?
>  
> Or maybe ignore Win32 for now and find when pthreads worked?
>  
>  - Jay
>  
> From: jay.krell at cornell.edu
> To: rcolebur at scires.com; m3devel at elegosoft.com
> Date: Thu, 10 Mar 2011 10:09:02 +0000
> Subject: Re: [M3devel] results of threadtest program on Windows7
> 
> I spoke too soon, alas.
> 5.2.6 Win32:
> 
> ..........laziest thread is 0/0/0 (tests: read 0/0/0 fork 0/0/0 alloc 0/0/0 lock
>  0/0/0)
> .
> 
> ***
> *** runtime error:
> ***    <*ASSERT*> failed.
> ***    file "..\src\runtime\common\RTCollector.m3", line 1073
> ***
> 
> Stack trace:
>    FP         PC      Procedure
> ---------  ---------  -------------------------------
> 0xbf8f5f0  0x7531f4b3  CleanBetween + 0x48 in ..\src\runtime\common\RTCollector.m3
> 0xbf8f628  0x7531f326  CleanPage + 0x83 in ..\src\runtime\common\RTCollector.m3
> 0xbf8f658  0x7531ebca  FinishThreadPages + 0xa2 in ..\src\runtime\common\RTCollector.m3
> 0xbf8f6a0  0x7531eae6  CollectSomeInStateZero + 0x5a3 in ..\src\runtime\common\RTCollector.m3
> 0xbf8f6b4  0x7531e507  CollectSome + 0x6e in ..\src\runtime\common\RTCollector.m3
> 0xbf8f6cc  0x7531e084  CollectEnough + 0x9c in ..\src\runtime\common\RTCollector.m3
> 0xbf8f714  0x7532015c  LongAlloc + 0x74 in ..\src\runtime\common\RTCollector.m3
> 0xbf8f74c  0x75320086  AllocTraced + 0x8a in ..\src\runtime\common\RTCollector.m3
> 0xbf8f790  0x75316fa4  GetOpenArray + 0x8e in ..\src\runtime\common\RTAllocator.m3
> 0xbf8f7c0  0x7531688a  AllocateOpenArray + 0x2d in ..\src\runtime\common\RTAllocator.m3
> .........  .........  ... more frames ...
> 
> 
> I should double check these results.
> And then I'm afraid I'm tempted to try 3.6 (DEC release) and 4.1 (commercial release)...
> 
>  - Jay
> 
> From: jay.krell at cornell.edu
> To: rcolebur at scires.com; m3devel at elegosoft.com
> Subject: RE: [M3devel] results of threadtest program on Windows7
> Date: Wed, 9 Mar 2011 20:10:37 +0000
> 
> 5.2.6 seems to have no problem with the thread test, on Win32.
> Now to either:
>   search for when the problems started.
>   and/or try replacing current ThreadWin32.m3 with 5.2.6 or vice versa
> 
>  - Jay
> 
> From: jay.krell at cornell.edu
> To: rcolebur at scires.com; m3devel at elegosoft.com
> Subject: RE: [M3devel] results of threadtest program on Windows7
> Date: Mon, 7 Mar 2011 06:58:45 +0000
> 
> Hm.. I think the first change may not be advisable. There are multiple such address range walkers and the interface probably doesn't constrain the order they go on.
> So, instead, another change would be to touch every stack page at thread start, so that later walks going "out of order" won't trigger stack overflow exceptions.
> This has the side benefit of not needing the backend to learn about chkstk.
>  
>  
> That is, on Win32, stack pages must be touched in order from highest to lowest.
> That is, the first time a stack page is touched, it should be only one lower than the lowest stack page previously touched.
> Once some stack pages have been touched, they can later be touched in any order.
> Functions with more than a page of locals are supposed to touch all of their frame's pages.
> Otherwise function call's pushing a return address suffices.
> The Modula-3 backend doesn't do this.
> I've never seen this clearly documented, but it is clearly the intent of the C compiler's generated code.
>   The C compiler generates a call to chkstk for functions with more than a page of locals, and chkstk touches a byte on each page.
>   And chkstk and alloca are the same function.
>   If this wasn't the case, the compiler could just inline subtraction.
>  
>  
> And any GC code that walks the stack could possible violate this, depending on what it uses for stack bounds.
> Currently it uses "accurate" bounds on Win32, and so is ok.
>  
>  
>  - Jay
>  
> From: jay.krell at cornell.edu
> To: rcolebur at scires.com; m3devel at elegosoft.com
> Date: Mon, 7 Mar 2011 06:53:06 +0000
> Subject: Re: [M3devel] results of threadtest program on Windows7
> 
> I'm going to make one or two changes because of this.
>  
>  - Where we process memory in a range, I'm going to go in stack growth order instead from lowest address to highest. If we don't already.
>     Or possibly recieve a direction parameter and pass it as 1 everywhere except for -1 when processing Win32 stacks.
>  
>  
>  - On Win32 I'm going to process the "entire" stack, instead of just from the current stack pointer to the "initial" stack pointer (highest address).
>    This is wasteful, will avoid reclaiming some garbage, will scan generally far more than needed.
>    However it is also what we do on FreeBSD and OpenBSD (see ThreadOpenBSD.c's use of pthread_stackseg_np and ThreadFreeBSD.c's use of pthread_stackseg_np)
>  
>  
> I think the main other alternative is the more portable "cooperative" scheme Tony has proposed where each thread has a boolean
> it checks occasionally as to if suspension is requested. It is a nice scheme, since it would remove a bunch of platform-specific code
> and posix-specific code (i.e. all signal stuff that isn't for user threads).
>  
>  
> On the matter of garbage lingering in stack frames that have been popped...should we consider NIL'ing local pointers before turn?
> Or it is too expensive and doesn't gain much?
> And they'll tend to get overwritten fairly soon anyway??
> Seems very gray.
>  
>  
>  - Jay
>  
>  
> From: jay.krell at cornell.edu
> To: rcolebur at scires.com; m3devel at elegosoft.com
> Date: Mon, 7 Mar 2011 05:51:32 +0000
> Subject: Re: [M3devel] results of threadtest program on Windows7
> 
> I find this worrisome, and want to understand it, and workaround it, somehow.
>  
> http://zachsaw.blogspot.com/2010/11/wow64-bug-getthreadcontext-may-return.html
>  
>  - Jay
>  
> From: jay.krell at cornell.edu
> To: rcolebur at scires.com; m3devel at elegosoft.com
> Date: Mon, 7 Mar 2011 05:39:53 +0000
> Subject: Re: [M3devel] results of threadtest program on Windows7
> 
> Ok. I see it fail often on Windows now too. I didn't change anything.
> I enabled pageheap to try to narrow it down, only to find an operating system bug, which I have worked around.
> "pageheap" causes heap allocations to be followed immediately by an unused page, to catch heap overruns right away.
>  Modulo alignment restrictions -- i.e. malloc(1) will still have a few bytes you can silently corrupt before the unused page.
>  
> Here is one instance.
>  
> 0:000> g
> Writing file...done
> Creating read threads...done
> Creating fork threads...done
> Creating alloc threads...done
> Creating lock threads...done
> running...printing oldest/median age/newest
> .ModLoad: 73b20000 73b6b000   C:\Windows\SysWOW64\apphelp.dll
> (1028.ca0): Access violation - code c0000005 (first chance)
> First chance exceptions are reported before any exception handling.
> This exception may be expected and handled.
> eax=01155e18 ebx=00200000 ecx=00000005 edx=0111caee esi=001ffffc edi=00a80018
> eip=0111cb3e esp=06d9f5c4 ebp=06d9f5fc iopl=0         nv up ei pl nz ac pe nc
> cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010216
> *** WARNING: Unable to verify checksum for threadtest.exe
> threadtest!RTCollector__Move+0x50:
> 0111cb3e 8b1e            mov     ebx,dword ptr [esi]  ds:002b:001ffffc=????????
> 0:009> r esi
> esi=001ffffc
> 0:009> db @esi
> 001ffffc  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
> 0020000c  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
> 0020001c  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
> 0020002c  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
> 0020003c  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
> 0020004c  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
> 0020005c  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
> 0020006c  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
> 0:009> ~*k
>    0  Id: 1028.d60 Suspend: 2 Teb: 7efdd000 Unfrozen
> ChildEBP RetAddr
> 
> ...
> 0045f674 0111a448 KERNELBASE!Sleep+0xf
> 0045f6b4 0111a218 threadtest!ThreadWin32__XPause+0x1cd
> 0045f6d8 010f356e threadtest!Thread__Pause+0x37
> 0045f7cc 01115268 threadtest!Main_M3+0xe67
> 0045f810 01114818 threadtest!RTLinker__RunMainBody+0x257
> 0045f828 011148bf threadtest!RTLinker__AddUnitI+0xf7
> 0045f848 010f1038 threadtest!RTLinker__AddUnit+0x9f
> 0045f864 011515d3 threadtest!main+0x38
> 0045f8a8 75e63677 threadtest!__tmainCRTStartup+0x122
> 0045f8b4 771c9f02 kernel32!BaseThreadInitThunk+0x12
> ...
>  
>    1  Id: 1028.ab8 Suspend: 2 Teb: 7efda000 Unfrozen
> ChildEBP RetAddr
> 00f6f680 771c8cc8 ntdll!NtWaitForSingleObject+0x15
> 00f6f6a8 0111b262 ntdll!RtlIntegerToUnicodeString+0x20b
> 00f6f6bc 01117b86 threadtest!RTOS__LockHeap+0x2c
> 00f6f6fc 011171cc threadtest!RTAllocator__AllocTraced+0xcd
> 00f6f730 01116ce6 threadtest!RTAllocator__GetTracedObj+0x89
> 00f6f74c 010f6829 threadtest!RTHooks__AllocateTracedObj+0x15
> 00f6f774 010f1288 threadtest!FileRd__Open+0x19
> 00f6f8ac 01119ca4 threadtest!Main__RApply+0x78
> 00f6f8e8 75e63677 threadtest!ThreadWin32__ThreadBase+0x25a
> ...
>  
>    2  Id: 1028.d50 Suspend: 2 Teb: 7efd7000 Unfrozen
> ChildEBP RetAddr
> ...
> 0106f9b0 011188ef threadtest!ThreadWin32__InitMutex+0x56
> 0106f9c8 010f7c08 threadtest!ThreadWin32__LockMutex+0x42
> 0106f9f0 010f12e3 threadtest!Rd__GetChar+0x28
> 0106fb28 01119ca4 threadtest!Main__RApply+0xd3
> 0106fb64 75e63677 threadtest!ThreadWin32__ThreadBase+0x25a
> ...
>  
>    3  Id: 1028.ef0 Suspend: 2 Teb: 7efaf000 Unfrozen
> ChildEBP RetAddr
> ...
> 062ff510 01117b86 threadtest!RTOS__LockHeap+0x2c
> 062ff550 011171cc threadtest!RTAllocator__AllocTraced+0xcd
> 062ff584 01116ce6 threadtest!RTAllocator__GetTracedObj+0x89
> 062ff5a0 011076b2 threadtest!RTHooks__AllocateTracedObj+0x15
> 062ff5d8 01106302 threadtest!FileWin32__New+0x52
> 062ff600 010f6839 threadtest!FS__OpenFileReadonly+0x8a
> 062ff628 010f1288 threadtest!FileRd__Open+0x29
> 062ff760 01119ca4 threadtest!Main__RApply+0x78
> 062ff79c 75e63677 threadtest!ThreadWin32__ThreadBase+0x25a
> ...
>  
>    4  Id: 1028.10c8 Suspend: 2 Teb: 7efa9000 Unfrozen
> ChildEBP RetAddr
> ...
> 0674fc44 010f17b9 threadtest!Process__Wait+0x8c
> 0674fd00 01119ca4 threadtest!Main__FApply+0xa2
> 0674fd3c 75e63677 threadtest!ThreadWin32__ThreadBase+0x25a
> ...
>  
>    5  Id: 1028.c54 Suspend: 2 Teb: 7efac000 Unfrozen
> ChildEBP RetAddr
> ...
> 0646fcc8 010f17b9 threadtest!Process__Wait+0x8c
> 0646fd84 01119ca4 threadtest!Main__FApply+0xa2
> 0646fdc0 75e63677 threadtest!ThreadWin32__ThreadBase+0x25a
> ...
>    6  Id: 1028.101c Suspend: 2 Teb: 7efa3000 Unfrozen
> ChildEBP RetAddr
> ...
> 06a1fdac 01117b86 threadtest!RTOS__LockHeap+0x2c
> 06a1fdec 0111772a threadtest!RTAllocator__AllocTraced+0xcd
> 06a1fe28 01116d9b threadtest!RTAllocator__GetOpenArray+0x94
> 06a1fe4c 010f1ada threadtest!RTHooks__AllocateOpenArray+0x19
> 06a1fea8 01119ca4 threadtest!Main__AApply+0x46
> 06a1fee4 75e63677 threadtest!ThreadWin32__ThreadBase+0x25a
> 06a1fef0 771c9f02 kernel32!BaseThreadInitThunk+0x12
> ...
>    7  Id: 1028.d5c Suspend: 2 Teb: 7efa6000 Unfrozen
> ChildEBP RetAddr
> ...
> 0684f6d0 01118eb1 threadtest!ThreadWin32__InitCondition+0xb0
> 0684f6f0 0111b328 threadtest!Thread__Broadcast+0x3f
> 0684f70c 01117c31 threadtest!RTOS__UnlockHeap+0xb3
> 0684f728 01117bbe threadtest!RTAllocator_M3_LINE_368+0x15
> 0684f768 011171cc threadtest!RTAllocator__AllocTraced+0x105
> 0684f79c 01116ce6 threadtest!RTAllocator__GetTracedObj+0x89
> 0684f7b8 011410c9 threadtest!RTHooks__AllocateTracedObj+0x15
> 0684f7d4 0112ce59 threadtest!Text8CString__New+0x19
> 0684f7ec 0112b63a threadtest!M3toC__StoT+0x16
> 0684f818 010fb22d threadtest!RTArgs__GetArg+0x70
> 0684f840 010f1776 threadtest!Params__Get+0x5d
> 0684f8fc 01119ca4 threadtest!Main__FApply+0x5f
> 0684f938 75e63677 threadtest!ThreadWin32__ThreadBase+0x25a
> ...
>    8  Id: 1028.d14 Suspend: 2 Teb: 7efa0000 Unfrozen
> ChildEBP RetAddr
> WARNING: Stack unwind information not available. Following frames may be wrong.
> ...
> 06c6f6c8 01117b86 threadtest!RTOS__LockHeap+0x2c
> 06c6f708 0111772a threadtest!RTAllocator__AllocTraced+0xcd
> 06c6f744 01116d9b threadtest!RTAllocator__GetOpenArray+0x94
> 06c6f768 010f1ada threadtest!RTHooks__AllocateOpenArray+0x19
> 06c6f7c4 01119ca4 threadtest!Main__AApply+0x46
> 06c6f800 75e63677 threadtest!ThreadWin32__ThreadBase+0x25a
> ...
>  
> #  9  Id: 1028.ca0 Suspend: 1 Teb: 7ef9d000 Unfrozen
> ChildEBP RetAddr
> 06d9f5fc 0113e6bd threadtest!RTCollector__Move+0x50
> 06d9f640 0113dfa8 threadtest!RTHeapMap__Walk+0x45a
> 06d9f664 0113df83 threadtest!RTHeapMap__DoWalkRef+0x62
> 06d9f688 0113df83 threadtest!RTHeapMap__DoWalkRef+0x3d
> 06d9f6ac 0113df3e threadtest!RTHeapMap__DoWalkRef+0x3d
> 06d9f6d8 0111e65f threadtest!RTHeapMap__WalkRef+0xfe
> 06d9f6fc 0111e4ad threadtest!RTCollector__CleanBetween+0xeb
> 06d9f724 0111de48 threadtest!RTCollector__CleanPage+0x54
> 06d9f778 0111d8ab threadtest!RTCollector__CollectSomeInStateZero+0x562
> 06d9f788 0111d549 threadtest!RTCollector__CollectSome+0x6e
> 06d9f7cc 01117b8c threadtest!RTHeapRep__CollectEnough+0x9b
> 06d9f80c 0111772a threadtest!RTAllocator__AllocTraced+0xd3
> 06d9f848 01116d9b threadtest!RTAllocator__GetOpenArray+0x94
> 06d9f86c 010f1ada threadtest!RTHooks__AllocateOpenArray+0x19
> 06d9f8c8 01119ca4 threadtest!Main__AApply+0x46
> 06d9f904 75e63677 threadtest!ThreadWin32__ThreadBase+0x25a
> ...
>   10  Id: 1028.4cc Suspend: 2 Teb: 7ef97000 Unfrozen
> ChildEBP RetAddr
> WARNING: Stack unwind information not available. Following frames may be wrong.
> 0664fdec 771c8cc8 ntdll!NtWaitForSingleObject+0x15
> 0664fe14 011188ff ntdll!RtlIntegerToUnicodeString+0x20b
> 0664fe2c 010f1d9b threadtest!ThreadWin32__LockMutex+0x52
> 0664fe74 01119ca4 threadtest!Main__LApply+0x7d
> 0664feb0 75e63677 threadtest!ThreadWin32__ThreadBase+0x25a
> ...
>  
>   11  Id: 1028.b48 Suspend: 2 Teb: 7ef9a000 Unfrozen
> ChildEBP RetAddr
> ...
> 06f1fe38 010f1d9b threadtest!ThreadWin32__LockMutex+0x52
> 06f1fe80 01119ca4 threadtest!Main__LApply+0x7d
> 06f1febc 75e63677 threadtest!ThreadWin32__ThreadBase+0x25a
> ...
>  
>   12  Id: 1028.a0c Suspend: 2 Teb: 7ef94000 Unfrozen
> ChildEBP RetAddr
> ...
> 06b1fa98 6c2a016a kernel32!HeapFree+0x14
> 06b1faac 0113328d MSVCR100!free+0x1c
> 06b1fab8 01116e2e threadtest!Cstdlib__free+0xd
> 06b1facc 011184fa threadtest!RTHooks__DisposeUntracedRef+0x2c
> 06b1fae0 011185c0 threadtest!ThreadWin32__DelCriticalSection+0x2d
> 06b1fb14 011188ef threadtest!ThreadWin32__InitMutex+0x6e
> 06b1fb2c 010f1d9b threadtest!ThreadWin32__LockMutex+0x42
> 06b1fb74 01119ca4 threadtest!Main__LApply+0x7d
> 06b1fbb0 75e63677 threadtest!ThreadWin32__ThreadBase+0x25a
> ...
> 
>  
> Suggestions?
>  
> 
> That last thread is a bit different, but I guess merely suggesting heavy stress.
>   Init goes to Del only when another thread calls Init at about the same time on the same mutex.
>  
>  
> Maybe try user threads but with a smaller quanta?
> Maybe try a much older Win32 thread implementation?
> Maybe go back to uniproc? Just for anecdotal evidence?
> I'll try with noinc/nogen/paranoid on Win32 also.
>  
>  
>  - Jay
> 
>  
> From: jay.krell at cornell.edu
> To: rcolebur at scires.com; m3devel at elegosoft.com
> Date: Mon, 7 Mar 2011 01:25:59 +0000
> Subject: Re: [M3devel] results of threadtest program on Windows7
> 
> Visual Studio should work.
> You just open the .exe you want to debug, and it should decide you have opened it because you want to debug it.
> Does it let you set breakpoints by typing in function names?
> e.g. main or cm3!main or threadtest!main.
> 
> windbg is also very good and a free download.
> e.g.
> \bin\x86\windbg threadtest
> bp threadtest!main
> bp threadtest!RTError__MsgS
> g
> 
>  - Jay
> 
> 
> From: rcolebur at SCIRES.COM
> To: m3devel at elegosoft.com
> Date: Sat, 5 Mar 2011 20:35:41 -0500
> Subject: Re: [M3devel] results of threadtest program on Windows7
> 
> Jay:
>  
> I have Visual Studio Express installed.  Can I use its debugger?  If so, how do I make it work with cm3?  Or, do you suggest something else?
>  
> Regards,
> Randy
>  
> From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K
> Sent: Saturday, March 05, 2011 12:38 AM
> To: Coleburn, Randy; m3devel at elegosoft.com
> Subject: RE: [M3devel] results of threadtest program on Windows7
>  
> Microsoft debuggers are freely downloadable. 
> 
> Jay/phone
> 
> 
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20110313/f58adf71/attachment-0002.html>