[M3devel] thread tester on 32bit Windows
Jay K
jay.krell at cornell.edu
Sun May 22 10:58:50 CEST 2011
I'm not sure if I posted this.
This is on Windows XP.
C:\dev2\cm3.2\m3-libs\m3core\tests\thread>\bin\x86\cdb NT386\threadtest.exe @M3paranoidgc
Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
Copyright (c) Microsoft Corporation. All rights reserved.
CommandLine: NT386\threadtest.exe @M3paranoidgc
Symbol search path is: SRV*C:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00400000 00c7b000 threadtest.exe
ModLoad: 7c900000 7c9b2000 ntdll.dll
ModLoad: 7c800000 7c8f6000 C:\WINDOWS2\system32\kernel32.dll
ModLoad: 78520000 785c3000 C:\WINDOWS2\WinSxS\x86_Microsoft.VC90.CRT_1fc8b3b9a1e18e3b_9.0.30729.4974_x-ww_d889290f\MSVCR90.dll
(6ec.1010): Break instruction exception - code 80000003 (first chance)
eax=00251eb4 ebx=7ffdf000 ecx=00000005 edx=00000020 esi=00251f48 edi=00251eb4
eip=7c90120e esp=0012fb20 ebp=0012fc94 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
ntdll!DbgBreakPoint:
7c90120e cc int 3
0:000> .lines
Line number information will be loaded
0:000> g
Writing file...done
Creating read threads...done
Creating fork threads...done
Creating alloc threads...done
Creating lock threads...done
running...printing oldest/median age/newest
.(6ec.116c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=001ffffc ebx=00200000 ecx=00000002 edx=004317f0 esi=00380018 edi=00380018
eip=0043181c esp=0117fb98 ebp=0117fbc0 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010202
threadtest!RTCollector__RefSanityCheck+0x2c:
0043181c 8b18 mov ebx,dword ptr [eax] ds:0023:001ffffc=????????
0:005> db @eax
001ffffc ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
0020000c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
0020001c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
0020002c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
0020003c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
0020004c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
0020005c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
0020006c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
0:005> ~*k
0 Id: 6ec.1010 Suspend: 2 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr
0012fcd8 7c90d21a ntdll!KiFastSystemCallRet
0012fcdc 7c8023f1 ntdll!NtDelayExecution+0xc
0012fd34 7c802455 kernel32!SleepEx+0x61
0012fd44 0042bd6e kernel32!Sleep+0xf
0012fd84 0042bb47 threadtest!ThreadWin32__XPause+0x1c7 [..\src\thread\WIN32\ThreadWin32.m3 @ 765]
0012fda8 00404870 threadtest!Thread__Pause+0x31 [..\src\thread\WIN32\ThreadWin32.m3 @ 727]
0012fee4 00426c95 threadtest!Main_M3+0x13e2 [..\src\Main.m3 @ 558]
0012ff28 00426262 threadtest!RTLinker__RunMainBody+0x250 [..\src\runtime\common\RTLinker.m3 @ 406]
0012ff40 00426306 threadtest!RTLinker__AddUnitI+0xf7 [..\src\runtime\common\RTLinker.m3 @ 113]
0012ff60 00401038 threadtest!RTLinker__AddUnit+0x9c [..\src\runtime\common\RTLinker.m3 @ 122]
0012ff7c 00462900 threadtest!main+0x38 [_m3main.mc @ 4]
0012ffc0 7c817077 threadtest!__tmainCRTStartup+0x10f [f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c @ 582]
0012fff0 00000000 kernel32!BaseProcessStart+0x23
1 Id: 6ec.1168 Suspend: 2 Teb: 7ffdd000 Unfrozen
ChildEBP RetAddr
00d7fc64 7c90df5a ntdll!KiFastSystemCallRet
00d7fc68 7c919b23 ntdll!NtWaitForSingleObject+0xc
00d7fcf0 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
00d7fcf8 0042a153 ntdll!RtlEnterCriticalSection+0x46
00d7fd30 0042a830 threadtest!ThreadWin32__InitCondition+0xa1 [..\src\thread\WIN32\ThreadWin32.m3 @ 186]
00d7fd50 0042cc08 threadtest!Thread__Broadcast+0x3f [..\src\thread\WIN32\ThreadWin32.m3 @ 409]
00d7fd6c 004295f9 threadtest!RTOS__UnlockHeap+0xb3 [..\src\thread\WIN32\ThreadWin32.m3 @ 1181]
00d7fd88 00429586 threadtest!RTAllocator_M3_LINE_370+0x15 [..\src\runtime\common\RTAllocator.m3 @ 371]
00d7fdc8 00428ba8 threadtest!RTAllocator__AllocTraced+0x105 [..\src\runtime\common\RTAllocator.m3 @ 368]
00d7fdfc 004286dc threadtest!RTAllocator__GetTracedObj+0x86 [..\src\runtime\common\RTAllocator.m3 @ 224]
00d7fe18 00408679 threadtest!RTHooks__AllocateTracedObj+0x15 [..\src\runtime\common\RTAllocator.m3 @ 122]
00d7fe40 00401360 threadtest!FileRd__Open+0x19 [..\src\rw\FileRd.m3 @ 16]
00d7ff78 0042b5ff threadtest!Main__RApply+0x160 [..\src\Main.m3 @ 182]
00d7ffb4 7c80b729 threadtest!ThreadWin32__ThreadBase+0x254 [..\src\thread\WIN32\ThreadWin32.m3 @ 615]
00d7ffec 00000000 kernel32!BaseThreadStart+0x37
2 Id: 6ec.1174 Suspend: 2 Teb: 7ffdc000 Unfrozen
ChildEBP RetAddr
00e7fc64 7c90df5a ntdll!KiFastSystemCallRet
00e7fc68 7c919b23 ntdll!NtWaitForSingleObject+0xc
00e7fcf0 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
00e7fcf8 0042a153 ntdll!RtlEnterCriticalSection+0x46
00e7fd30 0042a830 threadtest!ThreadWin32__InitCondition+0xa1 [..\src\thread\WIN32\ThreadWin32.m3 @ 186]
00e7fd50 0042cc08 threadtest!Thread__Broadcast+0x3f [..\src\thread\WIN32\ThreadWin32.m3 @ 409]
00e7fd6c 004295f9 threadtest!RTOS__UnlockHeap+0xb3 [..\src\thread\WIN32\ThreadWin32.m3 @ 1181]
00e7fd88 00429586 threadtest!RTAllocator_M3_LINE_370+0x15 [..\src\runtime\common\RTAllocator.m3 @ 371]
00e7fdc8 00428ba8 threadtest!RTAllocator__AllocTraced+0x105 [..\src\runtime\common\RTAllocator.m3 @ 368]
00e7fdfc 004286dc threadtest!RTAllocator__GetTracedObj+0x86 [..\src\runtime\common\RTAllocator.m3 @ 224]
00e7fe18 00408679 threadtest!RTHooks__AllocateTracedObj+0x15 [..\src\runtime\common\RTAllocator.m3 @ 122]
00e7fe40 00401360 threadtest!FileRd__Open+0x19 [..\src\rw\FileRd.m3 @ 16]
00e7ff78 0042b5ff threadtest!Main__RApply+0x160 [..\src\Main.m3 @ 182]
00e7ffb4 7c80b729 threadtest!ThreadWin32__ThreadBase+0x254 [..\src\thread\WIN32\ThreadWin32.m3 @ 615]
00e7ffec 00000000 kernel32!BaseThreadStart+0x37
3 Id: 6ec.14cc Suspend: 2 Teb: 7ffdb000 Unfrozen
ChildEBP RetAddr
00f7fd1c 7c90d0ba ntdll!KiFastSystemCallRet
00f7fd20 7c8109b6 ntdll!NtCreateFile+0xc
00f7fdb8 7c801a53 kernel32!CreateFileW+0x35f
00f7fddc 00417f10 kernel32!CreateFileA+0x30
00f7fe18 00408689 threadtest!FS__OpenFileReadonly+0x3c [..\src\os\WIN32\FSWin32.m3 @ 226]
00f7fe40 00401360 threadtest!FileRd__Open+0x29 [..\src\rw\FileRd.m3 @ 16]
00f7ff78 0042b5ff threadtest!Main__RApply+0x160 [..\src\Main.m3 @ 182]
00f7ffb4 7c80b729 threadtest!ThreadWin32__ThreadBase+0x254 [..\src\thread\WIN32\
ThreadWin32.m3 @ 615]
00f7ffec 00000000 kernel32!BaseThreadStart+0x37
4 Id: 6ec.d20 Suspend: 2 Teb: 7ffda000 Unfrozen
ChildEBP RetAddr
0107fcf8 7c90df5a ntdll!KiFastSystemCallRet
0107fcfc 7c919b23 ntdll!NtWaitForSingleObject+0xc
0107fd84 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
0107fd8c 0042cb42 ntdll!RtlEnterCriticalSection+0x46
0107fda0 0042954e threadtest!RTOS__LockHeap+0x2c [..\src\thread\WIN32\ThreadWin32.m3 @ 1168]
0107fde0 00428ba8 threadtest!RTAllocator__AllocTraced+0xcd [..\src\runtime\common\RTAllocator.m3 @ 367]
0107fe14 004286dc threadtest!RTAllocator__GetTracedObj+0x86 [..\src\runtime\common\RTAllocator.m3 @ 224]
0107fe30 00452f79 threadtest!RTHooks__AllocateTracedObj+0x15 [..\src\runtime\common\RTAllocator.m3 @ 122]
0107fe4c 0043e62a threadtest!Text8CString__New+0x19 [..\src\text\Text8CString.m3 @ 17]
0107fe64 0043cdf3 threadtest!M3toC__StoT+0x16 [..\src\C\Common\M3toC.m3 @ 96]
0107fe90 0040d037 threadtest!RTArgs__GetArg+0x6e [..\src\runtime\WIN32\RTArgs.m3 @ 139]
0107feb8 00401d6a threadtest!Params__Get+0x57 [..\src\params\Params.m3 @ 18]
0107ff78 0042b5ff threadtest!Main__FApply+0x141 [..\src\Main.m3 @ 245]
0107ffb4 7c80b729 threadtest!ThreadWin32__ThreadBase+0x254 [..\src\thread\WIN32\ThreadWin32.m3 @ 615]
0107ffec 00000000 kernel32!BaseThreadStart+0x37
# 5 Id: 6ec.116c Suspend: 1 Teb: 7ffd9000 Unfrozen
ChildEBP RetAddr
0117fbc0 0044fbdc threadtest!RTCollector__RefSanityCheck+0x2c [..\src\runtime\common\RTCollector.m3 @ 1702]
0117fc04 0044f4df threadtest!RTHeapMap__Walk+0x448 [..\src\runtime\common\RTHeapMap.m3 @ 202]
0117fc28 0044f4ba threadtest!RTHeapMap__DoWalkRef+0x62 [..\src\runtime\common\RTHeapMap.m3 @ 62]
0117fc4c 0044f4ba threadtest!RTHeapMap__DoWalkRef+0x3d [..\src\runtime\common\RTHeapMap.m3 @ 57]
0117fc70 0044f475 threadtest!RTHeapMap__DoWalkRef+0x3d [..\src\runtime\common\RTHeapMap.m3 @ 57]
0117fc9c 00431607 threadtest!RTHeapMap__WalkRef+0xf5 [..\src\runtime\common\RTHeapMap.m3 @ 47]
0117fcd8 0044eab4 threadtest!RTCollector__SanityCheck+0x23c [..\src\runtime\common\RTCollector.m3 @ 1660]
0117fcf8 0042f379 threadtest!RTHeapRep__InvokeMonitors+0x84 [..\src\runtime\common\RTHeapRep.m3 @ 56]
0117fd4c 0042f15e threadtest!RTCollector__CollectSomeInStateZero+0x1e0 [..\src\runtime\common\RTCollector.m3 @ 786]
0117fd5c 0042edfc threadtest!RTCollector__CollectSome+0x6e [..\src\runtime\common\RTCollector.m3 @ 720]
0117fda0 00429554 threadtest!RTHeapRep__CollectEnough+0x9b [..\src\runtime\common\RTCollector.m3 @ 655]
0117fde0 00428ba8 threadtest!RTAllocator__AllocTraced+0xd3 [..\src\runtime\common\RTAllocator.m3 @ 368]
0117fe14 004286dc threadtest!RTAllocator__GetTracedObj+0x86 [..\src\runtime\common\RTAllocator.m3 @ 224]
0117fe30 00452f79 threadtest!RTHooks__AllocateTracedObj+0x15 [..\src\runtime\common\RTAllocator.m3 @ 122]
0117fe4c 0043e62a threadtest!Text8CString__New+0x19 [..\src\text\Text8CString.m3 @ 17]
0117fe64 0043cdf3 threadtest!M3toC__StoT+0x16 [..\src\C\Common\M3toC.m3 @ 96]
0117fe90 0040d037 threadtest!RTArgs__GetArg+0x6e [..\src\runtime\WIN32\RTArgs.m3 @ 139]
0117feb8 00401d6a threadtest!Params__Get+0x57 [..\src\params\Params.m3 @ 18]
0117ff78 0042b5ff threadtest!Main__FApply+0x141 [..\src\Main.m3 @ 245]
0117ffb4 7c80b729 threadtest!ThreadWin32__ThreadBase+0x254 [..\src\thread\WIN32\ThreadWin32.m3 @ 615]
6 Id: 6ec.11b4 Suspend: 2 Teb: 7ffd8000 Unfrozen
ChildEBP RetAddr
0127fcf8 7c90df5a ntdll!KiFastSystemCallRet
0127fcfc 7c919b23 ntdll!NtWaitForSingleObject+0xc
0127fd84 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
0127fd8c 0042cb42 ntdll!RtlEnterCriticalSection+0x46
0127fda0 0042954e threadtest!RTOS__LockHeap+0x2c [..\src\thread\WIN32\ThreadWin32.m3 @ 1168]
0127fde0 00428ba8 threadtest!RTAllocator__AllocTraced+0xcd [..\src\runtime\common\RTAllocator.m3 @ 367]
0127fe14 004286dc threadtest!RTAllocator__GetTracedObj+0x86 [..\src\runtime\common\RTAllocator.m3 @ 224]
0127fe30 00452f79 threadtest!RTHooks__AllocateTracedObj+0x15 [..\src\runtime\common\RTAllocator.m3 @ 122]
0127fe4c 0043e62a threadtest!Text8CString__New+0x19 [..\src\text\Text8CString.m3 @ 17]
0127fe64 0043cdf3 threadtest!M3toC__StoT+0x16 [..\src\C\Common\M3toC.m3 @ 96]
0127fe90 0040d037 threadtest!RTArgs__GetArg+0x6e [..\src\runtime\WIN32\RTArgs.m3 @ 139]
0127feb8 00401d6a threadtest!Params__Get+0x57 [..\src\params\Params.m3 @ 18]
0127ff78 0042b5ff threadtest!Main__FApply+0x141 [..\src\Main.m3 @ 245]
0127ffb4 7c80b729 threadtest!ThreadWin32__ThreadBase+0x254 [..\src\thread\WIN32\ThreadWin32.m3 @ 615]
0127ffec 00000000 kernel32!BaseThreadStart+0x37
7 Id: 6ec.1038 Suspend: 2 Teb: 7ffd7000 Unfrozen
ChildEBP RetAddr
0137fdd4 7c90df5a ntdll!KiFastSystemCallRet
0137fdd8 7c919b23 ntdll!NtWaitForSingleObject+0xc
0137fe60 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
0137fe68 0042cb42 ntdll!RtlEnterCriticalSection+0x46
0137fe7c 0042954e threadtest!RTOS__LockHeap+0x2c [..\src\thread\WIN32\ThreadWin32.m3 @ 1168]
0137febc 004290fd threadtest!RTAllocator__AllocTraced+0xcd [..\src\runtime\common\RTAllocator.m3 @ 367]
0137fef8 00428788 threadtest!RTAllocator__GetOpenArray+0x91 [..\src\runtime\common\RTAllocator.m3 @ 296]
0137ff1c 00402430 threadtest!RTHooks__AllocateOpenArray+0x19 [..\src\runtime\common\RTAllocator.m3 @ 143]
0137ff78 0042b5ff threadtest!Main__AApply+0x128 [..\src\Main.m3 @ 283]
0137ffb4 7c80b729 threadtest!ThreadWin32__ThreadBase+0x254 [..\src\thread\WIN32\ThreadWin32.m3 @ 615]
0137ffec 00000000 kernel32!BaseThreadStart+0x37
8 Id: 6ec.1034 Suspend: 2 Teb: 7ffd6000 Unfrozen
ChildEBP RetAddr
0147fdd4 7c90df5a ntdll!KiFastSystemCallRet
0147fdd8 7c919b23 ntdll!NtWaitForSingleObject+0xc
0147fe60 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
0147fe68 0042cb42 ntdll!RtlEnterCriticalSection+0x46
0147fe7c 0042954e threadtest!RTOS__LockHeap+0x2c [..\src\thread\WIN32\ThreadWin32.m3 @ 1168]
0147febc 004290fd threadtest!RTAllocator__AllocTraced+0xcd [..\src\runtime\common\RTAllocator.m3 @ 367]
0147fef8 00428788 threadtest!RTAllocator__GetOpenArray+0x91 [..\src\runtime\common\RTAllocator.m3 @ 296]
0147ff1c 00402430 threadtest!RTHooks__AllocateOpenArray+0x19 [..\src\runtime\common\RTAllocator.m3 @ 143]
0147ff78 0042b5ff threadtest!Main__AApply+0x128 [..\src\Main.m3 @ 283]
0147ffb4 7c80b729 threadtest!ThreadWin32__ThreadBase+0x254 [..\src\thread\WIN32\ThreadWin32.m3 @ 615]
0147ffec 00000000 kernel32!BaseThreadStart+0x37
9 Id: 6ec.10d0 Suspend: 2 Teb: 7ffd5000 Unfrozen
ChildEBP RetAddr
0157fdd4 7c90df5a ntdll!KiFastSystemCallRet
0157fdd8 7c919b23 ntdll!NtWaitForSingleObject+0xc
0157fe60 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
0157fe68 0042cb42 ntdll!RtlEnterCriticalSection+0x46
0157fe7c 0042954e threadtest!RTOS__LockHeap+0x2c [..\src\thread\WIN32\ThreadWin32.m3 @ 1168]
0157febc 004290fd threadtest!RTAllocator__AllocTraced+0xcd [..\src\runtime\common\RTAllocator.m3 @ 367]
0157fef8 00428788 threadtest!RTAllocator__GetOpenArray+0x91 [..\src\runtime\common\RTAllocator.m3 @ 296]
0157ff1c 00402430 threadtest!RTHooks__AllocateOpenArray+0x19 [..\src\runtime\common\RTAllocator.m3 @ 143]
0157ff78 0042b5ff threadtest!Main__AApply+0x128 [..\src\Main.m3 @ 283]
0157ffb4 7c80b729 threadtest!ThreadWin32__ThreadBase+0x254 [..\src\thread\WIN32\ThreadWin32.m3 @ 615]
0157ffec 00000000 kernel32!BaseThreadStart+0x37
10 Id: 6ec.ff8 Suspend: 2 Teb: 7ffd4000 Unfrozen
ChildEBP RetAddr
0167fe48 7c90df5a ntdll!KiFastSystemCallRet
0167fe4c 7c919b23 ntdll!NtWaitForSingleObject+0xc
0167fed4 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
0167fedc 00429f52 ntdll!RtlEnterCriticalSection+0x46
0167ff10 0042a287 threadtest!ThreadWin32__InitMutex+0x53 [..\src\thread\WIN32\ThreadWin32.m3 @ 148]
0167ff28 00402a5f threadtest!ThreadWin32__LockMutex+0x3f [..\src\thread\WIN32\ThreadWin32.m3 @ 210]
0167ff78 0042b5ff threadtest!Main__LApply+0x15f [..\src\Main.m3 @ 319]
0167ffb4 7c80b729 threadtest!ThreadWin32__ThreadBase+0x254 [..\src\thread\WIN32\ThreadWin32.m3 @ 615]
0167ffec 00000000 kernel32!BaseThreadStart+0x37
11 Id: 6ec.10e8 Suspend: 2 Teb: 7ffaf000 Unfrozen
ChildEBP RetAddr
0177fdfc 7c90df5a ntdll!KiFastSystemCallRet
0177fe00 7c919b23 ntdll!NtWaitForSingleObject+0xc
0177fe88 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
0177fe90 0042cb42 ntdll!RtlEnterCriticalSection+0x46
0177fea4 004332ce threadtest!RTOS__LockHeap+0x2c [..\src\thread\WIN32\ThreadWin2.m3 @ 1168]
0177fed8 00429faf threadtest!RTHeapRep__RegisterFinalCleanup+0x73 [..\src\runtime\common\RTCollector.m3 @ 2150]
0177ff10 0042a287 threadtest!ThreadWin32__InitMutex+0xb0 [..\src\thread\WIN32\ThreadWin32.m3 @ 156]
0177ff28 00402a5f threadtest!ThreadWin32__LockMutex+0x3f [..\src\thread\WIN32\ThreadWin32.m3 @ 210]
0177ff78 0042b5ff threadtest!Main__LApply+0x15f [..\src\Main.m3 @ 319]
0177ffb4 7c80b729 threadtest!ThreadWin32__ThreadBase+0x254 [..\src\thread\WIN32\ThreadWin32.m3 @ 615]
0177ffec 00000000 kernel32!BaseThreadStart+0x37
12 Id: 6ec.10d8 Suspend: 2 Teb: 7ffae000 Unfrozen
ChildEBP RetAddr
0187fe48 7c90df5a ntdll!KiFastSystemCallRet
0187fe4c 7c919b23 ntdll!NtWaitForSingleObject+0xc
0187fed4 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
0187fedc 00429f52 ntdll!RtlEnterCriticalSection+0x46
0187ff10 0042a287 threadtest!ThreadWin32__InitMutex+0x53 [..\src\thread\WIN32\ThreadWin32.m3 @ 148]
0187ff28 00402a5f threadtest!ThreadWin32__LockMutex+0x3f [..\src\thread\WIN32\ThreadWin32.m3 @ 210]
0187ff78 0042b5ff threadtest!Main__LApply+0x15f [..\src\Main.m3 @ 319]
0187ffb4 7c80b729 threadtest!ThreadWin32__ThreadBase+0x254 [..\src\thread\WIN32\ThreadWin32.m3 @ 615]
0187ffec 00000000 kernel32!BaseThreadStart+0x37
0:005>
I suspect more logging/tracing is needed, to see what was moved where.
- Jay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20110522/e21c84b6/attachment-0001.html>
More information about the M3devel
mailing list