[M3devel] OpenBSD/x86 pthreads
Jay K
jay.krell at cornell.edu
Fri Nov 20 21:22:57 CET 2009
I tried:
void
ThreadPThread__ProcessStopped (m3_pthread_t mt, char *start, char *end,
void (*p)(void *start, void *end))
{
stack_t sinfo;
char* ss_sp;
if (pthread_stackseg_np(PTHREAD_FROM_M3(mt), &sinfo) != 0) abort();
ss_sp = (char*)sinfo.ss_sp;
assert(start == 0);
assert((ss_sp - sinfo.ss_size) < end); /* man page says ss_sp is "top" */
assert(end < ss_sp);
/* we don't have a reliable sp, so... */
p((ss_sp - sinfo.ss_size), end);
}
the C assertions don't trigger, but always:
Starting program: /home/jay/cm3/bin/Juno
***
*** runtime error:
*** <*ASSERT*> failed.
*** file "../src/runtime/common/RTCollector.m3", line 589
***
Program received signal SIGABRT, Aborted.
[Switching to process 6216, thread 0x7d6f5800]
0x0a58df81 in kill () from /usr/lib/libc.so.50.1
(gdb) bt
#0 0x0a58df81 in kill () from /usr/lib/libc.so.50.1
#1 0x0a5da907 in abort () at /usr/src/lib/libc/stdlib/abort.c:68
#2 0x01095cb2 in RTOS__Crash () at ../src/runtime/POSIX/RTOS.m3:20
#3 0x0108b77f in RTProcess__Crash (M3_Bd56fi_msg=0x0)
at ../src/runtime/common/RTProcess.m3:65
#4 0x010892da in RTError__EndError (M3_AicXUJ_crash=1 '\001')
at ../src/runtime/common/RTError.m3:118
#5 0x01088f9e in RTError__MsgS (M3_AJWxb1_file=0x210568c5,
M3_AcxOUs_line=589, M3_Bd56fi_msgA=0x21058368, M3_Bd56fi_msgB=0x210526d0,
M3_Bd56fi_msgC=0x21058368) at ../src/runtime/common/RTError.m3:40
#6 0x01089742 in RTException__Crash (M3_Cblw37_a=0xcfbd9658,
M3_AicXUJ_raises=0 '\0', M3_AJWxb1_rte=0x21052580)
at ../src/runtime/common/RTException.m3:79
#7 0x01089469 in RTException__DefaultBackstop (M3_Cblw37_a=0xcfbd9658,
M3_AicXUJ_raises=0 '\0') at ../src/runtime/common/RTException.m3:39
#8 0x010893a5 in RTException__InvokeBackstop (M3_Cblw37_a=0xcfbd9658,
M3_AicXUJ_raises=0 '\0') at ../src/runtime/common/RTException.m3:25
#9 0x010966bc in RTException__Raise (M3_Cblw37_act=0xcfbd9658)
at ../src/runtime/ex_frame/RTExFrame.m3:29
#10 0x01089502 in RTException__DefaultBackstop (M3_Cblw37_a=0xcfbd9658,
M3_AicXUJ_raises=0 '\0') at ../src/runtime/common/RTException.m3:47
#11 0x010893a5 in RTException__InvokeBackstop (M3_Cblw37_a=0xcfbd9658,
M3_AicXUJ_raises=0 '\0') at ../src/runtime/common/RTException.m3:25
#12 0x010966bc in RTException__Raise (M3_Cblw37_act=0xcfbd9658)
---Type <return> to continue, or q <return> to quit---
at ../src/runtime/ex_frame/RTExFrame.m3:29
#13 0x010742f6 in RTHooks__ReportFault (M3_AJWxb1_module=0x21056a40,
M3_AcxOUs_info=18848) at ../src/runtime/common/RTHooks.m3:110
#14 0x01086bb9 in _m3_fault (M3_AcxOUs_arg=18848) from /cm3/lib/libm3core.so.5
#15 0x0107dc9b in RTCollector__GrayBetween (M3_Edk2y1_h=0x7c7e5978,
M3_Edk2y1_he=0x7c7e6000, M3_AZrJv4_r=6 '\006')
at ../src/runtime/common/RTCollector.m3:589
#16 0x0107dbfc in RTCollector__PromotePage (M3_Cq39JE_page=0x7c7e4000,
M3_AZrJv4_r=6 '\006') at ../src/runtime/common/RTCollector.m3:578
#17 0x0107d7bb in RTCollector__NoteStackLocations (
M3_AJWxb1_start=0x89922000, M3_AJWxb1_stop=0x89935fac)
at ../src/runtime/common/RTCollector.m3:528
#18 0x0109cfc0 in ThreadPThread__ProcessStopped (mt=0x7d6f5000, start=0x0,
end=0x89935fb0 "", p=0x107d72c <RTCollector__NoteStackLocations>)
at ../src/thread/PTHREAD/ThreadPThreadC.c:185
#19 0x0109b8a3 in ThreadPThread__ProcessOther (M3_DKqeuR_act=0x7d9cc380,
M3_Ad3xEV_p=0x107d72c) at ../src/thread/PTHREAD/ThreadPThread.m3:1018
#20 0x0109b196 in ThreadInternal__ProcessStacks (M3_Ad3xEV_p=0x107d72c)
at ../src/thread/PTHREAD/ThreadPThread.m3:861
#21 0x0107e9a4 in RTCollector__CollectSomeInStateZero ()
at ../src/runtime/common/RTCollector.m3:819
#22 0x0107e412 in RTCollector__CollectSome ()
at ../src/runtime/common/RTCollector.m3:719
#23 0x0107dec5 in RTHeapRep__CollectEnough ()
---Type <return> to continue, or q <return> to quit---
at ../src/runtime/common/RTCollector.m3:653
#24 0x01075fc1 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=12,
M3_Cwb5VA_dataAlignment=4, M3_AmBNir_thread=0x7d9cc544)
at ../src/runtime/common/RTAllocator.m3:365
#25 0x01074f70 in RTAllocator__GetTracedRef (M3_Eic7CK_def=0x23453ca0)
at ../src/runtime/common/RTAllocator.m3:202
#26 0x01074b8a in RTHooks__AllocateTracedRef (M3_AJWxb1_defn=0x23453ca0)
at ../src/runtime/common/RTAllocator.m3:113
#27 0x034c8196 in IntIntTbl__Put (M3_DTcOE5_tbl=0x7c7ea3f8,
M3_EN2A1V_key=0xcfbd9a3c, M3_EN2A1V_val=0xcfbd9a88)
at ../I386_OPENBSD/IntIntTbl.m3 => ../src/table/Table.mg:127
#28 0x08761656 in KeyFilter__Set (M3_Bkn9rd_a=45 '-', M3_Bkn9rd_b=111 'o',
M3_AcxOUs_c=245, M3_AicXUJ_bothCases=0 '\0', M3_AicXUJ_reversed=1 '\001')
at ../src/etext/KeyFilter.m3:288
#29 0x08761a0a in KeyFilter__Set (M3_Bkn9rd_a=111 'o', M3_Bkn9rd_b=45 '-',
M3_AcxOUs_c=245, M3_AicXUJ_bothCases=0 '\0', M3_AicXUJ_reversed=0 '\0')
at ../src/etext/KeyFilter.m3:304
#30 0x087620a2 in KeyFilter_M3 (M3_AcxOUs_mode=1)
at ../src/etext/KeyFilter.m3:437
#31 0x01088444 in RTLinker__RunMainBody (M3_DjPxE3_m=0x287046a0)
at ../src/runtime/common/RTLinker.m3:399
#32 0x010882e6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x287070c0)
at ../src/runtime/common/RTLinker.m3:379
#33 0x010882e6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x287094e0)
---Type <return> to continue, or q <return> to quit---
at ../src/runtime/common/RTLinker.m3:379
#34 0x010882e6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x28706b40)
at ../src/runtime/common/RTLinker.m3:379
#35 0x010882e6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x287060e0)
at ../src/runtime/common/RTLinker.m3:379
#36 0x010882e6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x3c010780)
at ../src/runtime/common/RTLinker.m3:379
#37 0x010877fe in RTLinker__AddUnitI (M3_DjPxE3_m=0x3c010780)
at ../src/runtime/common/RTLinker.m3:113
#38 0x0108788c in RTLinker__AddUnit (M3_DjPxE5_b=0x1c05c83a)
at ../src/runtime/common/RTLinker.m3:122
#39 0x1c008706 in main (argc=1, argv=0xcfbd9ebc, envp=0xcfbd9ec4)
at _m3main.mc:4
(gdb)
Presumably should look at the Java and Boehm stuff..later..
- Jay
From: jay.krell at cornell.edu
To: hosking at cs.purdue.edu
Date: Fri, 20 Nov 2009 19:16:53 +0000
CC: m3devel at elegosoft.com
Subject: Re: [M3devel] OpenBSD/x86 pthreads
$ /cm3/bin/Juno
assertion "sinfo.ss_sp <= end" failed: file "../src/thread/PTHREAD/ThreadPThread
C.c", line 180, function "ThreadPThread__ProcessStopped"
Abort trap (core dumped)
$ gdb /cm3/bin/Juno
GNU gdb 6.3
...
(gdb) r
Starting program: /home/jay/cm3/bin/Juno
assertion "sinfo.ss_sp <= end" failed: file "../src/thread/PTHREAD/ThreadPThread
C.c", line 180, function "ThreadPThread__ProcessStopped"
Program received signal SIGABRT, Aborted.
[Switching to process 32202, thread 0x8ad0c000]
0x03931f81 in kill () from /usr/lib/libc.so.50.1
(gdb) up
#1 0x0397e907 in abort () at /usr/src/lib/libc/stdlib/abort.c:68
...
(gdb) up
#2 0x0390d85b in __assert2 (
...
(gdb) up
#3 0x04bfff34 in ThreadPThread__ProcessStopped (mt=0x8ad0c800, start=0x0,
end=0x864fefb0, p=0x4be06ec <RTCollector__NoteStackLocations>)
at ../src/thread/PTHREAD/ThreadPThreadC.c:180
180 assert(sinfo.ss_sp <= end); /* man page says ss_sp is "top" */
(gdb) info locals
sinfo = {ss_sp = 0x864ff000, ss_size = 81920, ss_flags = 0}
(gdb) thread apply all bt
Appears to be this thread.
Thread 4 (process 32202, thread 0x8ad0c800):
#0 _thread_kern_sched (scp=0x0)
at /usr/src/lib/libpthread/uthread/uthread_kern.c:482
#1 0x05e01200 in _thread_kern_sched_state_unlock (state=PS_SIGTHREAD,
lock=0x8ad0c8b0, fname=0x1 <Address 0x1 out of bounds>, lineno=1)
at /usr/src/lib/libpthread/uthread/uthread_kern.c:581
#2 0x05dfebc9 in pthread_cond_wait (cond=0x882737b0, mutex=0x88273870)
at /usr/src/lib/libpthread/uthread/uthread_cond.c:261
#3 0x04c003b4 in ThreadPThread__pthread_cond_wait (cond=0x882737b0,
mutex=0x88273870) at ../src/thread/PTHREAD/ThreadPThreadC.c:506
#4 0x04bfad66 in ThreadPThread__XWait (M3_DKqeuR_self=0x7f820500,
M3_AYIbX3_m=0x83281998, M3_Bl0jv4_c=0x832819a4,
M3_AicXUJ_alertable=0 '\0') at ../src/thread/PTHREAD/ThreadPThread.m3:209
#5 0x04bfae91 in Thread__Wait (M3_AYIbX3_m=0x83281998,
M3_Bl0jv4_c=0x832819a4) at ../src/thread/PTHREAD/ThreadPThread.m3:231
#6 0x0a7ad5bf in VBTRep__MeterMaid (M3_EMTrVz_self=0x832819cc)
at ../src/vbt/VBTRep.m3:439
#7 0x04bfc850 in ThreadPThread__RunThread (M3_DKqeuR_me=0x7f820500)
at ../src/thread/PTHREAD/ThreadPThread.m3:502
#8 0x04bfc55b in ThreadPThread__ThreadBase (M3_AJWxb1_param=0x7f820500)
at ../src/thread/PTHREAD/ThreadPThread.m3:476
#9 0x05df837f in _thread_start ()
at /usr/src/lib/libpthread/uthread/uthread_create.c:240
---Type <return> to continue, or q <return> to quit---
#10 0x0000002b in ?? ()
#11 0x00000000 in ?? ()
Thread 3 (process 32202, thread 0x8aae0c00):
#0 _thread_kern_sched (scp=0x0)
at /usr/src/lib/libpthread/uthread/uthread_kern.c:482
#1 0x05e01200 in _thread_kern_sched_state_unlock (state=PS_SIGTHREAD,
lock=0x8aae0cb0, fname=0x1 <Address 0x1 out of bounds>, lineno=1)
at /usr/src/lib/libpthread/uthread/uthread_kern.c:581
#2 0x05dfee2d in pthread_cond_timedwait (cond=0x25df30e0, mutex=0x25df30dc,
abstime=0x7effafa8) at /usr/src/lib/libpthread/uthread/uthread_cond.c:431
#3 0x05df85a7 in _thread_gc (arg=0x0)
at /usr/src/lib/libpthread/uthread/uthread_gc.c:181
#4 0x05df837f in _thread_start ()
at /usr/src/lib/libpthread/uthread/uthread_create.c:240
#5 0x0000002b in ?? ()
#6 0x00000000 in ?? ()
Thread 2 (process 32202, thread 0x8aae0800):
#0 _thread_start () at /usr/src/lib/libpthread/uthread/uthread_create.c:233
#1 0x0000002b in ?? ()
#2 0x00000000 in ?? ()
Thread 1 (process 32202, thread 0x8ad0c000):
---Type <return> to continue, or q <return> to quit---
#0 0x03931f81 in kill () from /usr/lib/libc.so.50.1
#1 0x0397e907 in abort () at /usr/src/lib/libc/stdlib/abort.c:68
#2 0x0390d85b in __assert2 (
file=0x24bb2700 "../src/thread/PTHREAD/ThreadPThreadC.c", line=180,
func=0x24bb27cf "ThreadPThread__ProcessStopped",
failedexpr=0x24bb27f8 "sinfo.ss_sp <= end")
at /usr/src/lib/libc/gen/assert.c:52
#3 0x04bfff34 in ThreadPThread__ProcessStopped (mt=0x8ad0c800, start=0x0,
end=0x864fefb0, p=0x4be06ec <RTCollector__NoteStackLocations>)
at ../src/thread/PTHREAD/ThreadPThreadC.c:180
#4 0x04bfe85e in ThreadPThread__ProcessOther (M3_DKqeuR_act=0x7f820500,
M3_Ad3xEV_p=0x4be06ec) at ../src/thread/PTHREAD/ThreadPThread.m3:1017
#5 0x04bfe151 in ThreadInternal__ProcessStacks (M3_Ad3xEV_p=0x4be06ec)
at ../src/thread/PTHREAD/ThreadPThread.m3:860
#6 0x04be1964 in RTCollector__CollectSomeInStateZero ()
at ../src/runtime/common/RTCollector.m3:819
#7 0x04be13d2 in RTCollector__CollectSome ()
at ../src/runtime/common/RTCollector.m3:719
#8 0x04be0e85 in RTHeapRep__CollectEnough ()
at ../src/runtime/common/RTCollector.m3:653
#9 0x04bd8f81 in RTAllocator__AllocTraced (M3_Cwb5VA_dataSize=12,
M3_Cwb5VA_dataAlignment=4, M3_AmBNir_thread=0x7f820344)
at ../src/runtime/common/RTAllocator.m3:365
#10 0x04bd7f30 in RTAllocator__GetTracedRef (M3_Eic7CK_def=0x283aaca0)
---Type <return> to continue, or q <return> to quit---
at ../src/runtime/common/RTAllocator.m3:202
#11 0x04bd7b4a in RTHooks__AllocateTracedRef (M3_AJWxb1_defn=0x283aaca0)
at ../src/runtime/common/RTAllocator.m3:113
#12 0x0841f196 in IntIntTbl__Put (M3_DTcOE5_tbl=0x832863f8,
M3_EN2A1V_key=0xcfbfc1ec, M3_EN2A1V_val=0xcfbfc238)
at ../I386_OPENBSD/IntIntTbl.m3 => ../src/table/Table.mg:127
#13 0x0ad56656 in KeyFilter__Set (M3_Bkn9rd_a=45 '-', M3_Bkn9rd_b=111 'o',
M3_AcxOUs_c=245, M3_AicXUJ_bothCases=0 '\0', M3_AicXUJ_reversed=1 '\001')
at ../src/etext/KeyFilter.m3:288
#14 0x0ad56a0a in KeyFilter__Set (M3_Bkn9rd_a=111 'o', M3_Bkn9rd_b=45 '-',
M3_AcxOUs_c=245, M3_AicXUJ_bothCases=0 '\0', M3_AicXUJ_reversed=0 '\0')
at ../src/etext/KeyFilter.m3:304
#15 0x0ad570a2 in KeyFilter_M3 (M3_AcxOUs_mode=1)
at ../src/etext/KeyFilter.m3:437
#16 0x04beb404 in RTLinker__RunMainBody (M3_DjPxE3_m=0x2acf96a0)
at ../src/runtime/common/RTLinker.m3:399
#17 0x04beb2a6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x2acfc0c0)
at ../src/runtime/common/RTLinker.m3:379
#18 0x04beb2a6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x2acfe4e0)
at ../src/runtime/common/RTLinker.m3:379
#19 0x04beb2a6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x2acfbb40)
at ../src/runtime/common/RTLinker.m3:379
#20 0x04beb2a6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x2acfb0e0)
at ../src/runtime/common/RTLinker.m3:379
---Type <return> to continue, or q <return> to quit---
#21 0x04beb2a6 in RTLinker__RunMainBody (M3_DjPxE3_m=0x3c010780)
at ../src/runtime/common/RTLinker.m3:379
#22 0x04bea7be in RTLinker__AddUnitI (M3_DjPxE3_m=0x3c010780)
at ../src/runtime/common/RTLinker.m3:113
#23 0x04bea84c in RTLinker__AddUnit (M3_DjPxE5_b=0x1c05c83a)
at ../src/runtime/common/RTLinker.m3:122
#24 0x1c008706 in main (argc=1, argv=0xcfbfc66c, envp=0xcfbfc674)
at _m3main.mc:4
(gdb)
(gdb) up
#4 0x04bfe85e in ThreadPThread__ProcessOther (M3_DKqeuR_act=0x7f820500,
M3_Ad3xEV_p=0x4be06ec) at ../src/thread/PTHREAD/ThreadPThread.m3:1017
1017 THEN ProcessStopped(act.handle, act.sp, act.stackbase, p);
(gdb) p M3_DKqeuR_act
$1 = (void *) 0x7f820500
(gdb) x 0x7f820500
0x7f820500: 0x88273870
(gdb) x
0x7f820504: 0x882737b0
(gdb)
0x7f820508: 0x00000000
(gdb)
0x7f82050c: 0x882738a0
(gdb)
0x7f820510: 0x00000000
(gdb)
0x7f820514: 0x7f820300
(gdb)
0x7f820518: 0x7f820300
(gdb)
0x7f82051c: 0x8ad0c800
(gdb) thread 4
[Switching to thread 4 (process 32202, thread 0x8ad0c800)]#0 _thread_kern_sched
(scp=0x0) at /usr/src/lib/libpthread/uthread/uthread_kern.c:482
482 /usr/src/lib/libpthread/uthread/uthread_kern.c: No such file or director
y.
in /usr/src/lib/libpthread/uthread/uthread_kern.c
(gdb) up
#1 0x05e01200 in _thread_kern_sched_state_unlock (state=PS_SIGTHREAD,
lock=0x8ad0c8b0, fname=0x1 <Address 0x1 out of bounds>, lineno=1)
at /usr/src/lib/libpthread/uthread/uthread_kern.c:581
581 in /usr/src/lib/libpthread/uthread/uthread_kern.c
(gdb) up
#2 0x05dfebc9 in pthread_cond_wait (cond=0x882737b0, mutex=0x88273870)
at /usr/src/lib/libpthread/uthread/uthread_cond.c:261
261 /usr/src/lib/libpthread/uthread/uthread_cond.c: No such file or director
y.
in /usr/src/lib/libpthread/uthread/uthread_cond.c
(gdb) up
#3 0x04c003b4 in ThreadPThread__pthread_cond_wait (cond=0x882737b0,
mutex=0x88273870) at ../src/thread/PTHREAD/ThreadPThreadC.c:506
506 return pthread_cond_wait(cond, mutex);
(gdb) info frame
Stack level 3, frame at 0x864fee50:
eip = 0x4c003b4 in ThreadPThread__pthread_cond_wait
(../src/thread/PTHREAD/ThreadPThreadC.c:506); saved eip 0x4bfad66
called by frame at 0x864feea0, caller of frame at 0x864fee30
source language c.
Arglist at 0x864fee48, args: cond=0x882737b0, mutex=0x88273870
Locals at 0x864fee48, Previous frame's sp is 0x864fee50
Saved registers:
ebx at 0x864fee44, ebp at 0x864fee48, esi at 0x864fee20, edi at 0x864fee24,
eip at 0x864fee4c
(gdb)
(gdb) down
#1 0x05e01200 in _thread_kern_sched_state_unlock (state=PS_SIGTHREAD,
lock=0x8ad0c8b0, fname=0x1 <Address 0x1 out of bounds>, lineno=1)
at /usr/src/lib/libpthread/uthread/uthread_kern.c:581
581 /usr/src/lib/libpthread/uthread/uthread_kern.c: No such file or director
y.
in /usr/src/lib/libpthread/uthread/uthread_kern.c
(gdb) down
#0 _thread_kern_sched (scp=0x0)
at /usr/src/lib/libpthread/uthread/uthread_kern.c:482
482 in /usr/src/lib/libpthread/uthread/uthread_kern.c
(gdb) down
Bottom (i.e., innermost) frame selected; you cannot go down.
(gdb) info frame
Stack level 0, frame at 0x864fedd0:
eip = 0x5e00f70 in _thread_kern_sched
(/usr/src/lib/libpthread/uthread/uthread_kern.c:482); saved eip 0x5e01200
called by frame at 0x864fedf0
source language c.
Arglist at 0x864fedc8, args: scp=0x0
Locals at 0x864fedc8, Previous frame's sp is 0x864fedd0
Saved registers:
ebx at 0x864fedbc, ebp at 0x864fedc8, esi at 0x864fedc0, edi at 0x864fedc4,
eip at 0x864fedcc
I'll try reversing the assert?
- Jay
From: hosking at cs.purdue.edu
To: jay.krell at cornell.edu
Date: Fri, 20 Nov 2009 09:58:38 -0500
CC: m3devel at elegosoft.com
Subject: Re: [M3devel] OpenBSD/x86 pthreads
This looks odd -- the assertion should not fail. It is reporting that recorded stack base is not is not in range of the reported stack info. What exactly does OpenBSD report in the sinfo? Which thread is this? I need a full stack trace of all the threads to understand what is going on here.
On 20 Nov 2009, at 04:01, Jay K wrote:
Tony, I'm not sure we really care, but:
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/jay/cm3/bin/Juno
assertion "end >= sinfo.ss_sp" failed: file "../src/thread/PTHREAD/ThreadPThread
C.c", line 180, function "ThreadPThread__ProcessStopped"
Program received signal SIGABRT, Aborted.
[Switching to process 5091, thread 0x85a73400]
0x042d5f81 in kill () from /usr/lib/libc.so.50.1
(gdb) up
#1 0x04322907 in abort () at /usr/src/lib/libc/stdlib/abort.c:68
...
(gdb) up
#2 0x042b185b in __assert2 (
...
(gdb) up
#3 0x00b92f24 in ThreadPThread__ProcessStopped (mt=0x85a73800, start=0x0,
end=0x7fe80fb0, p=0xb736dc <RTCollector__NoteStackLocations>)
at ../src/thread/PTHREAD/ThreadPThreadC.c:180
180 assert(end >= sinfo.ss_sp);
(gdb) info locals
sinfo = {ss_sp = 0x7fe81000, ss_size = 81920, ss_flags = 0}
(gdb)
- Jay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20091120/40984fde/attachment-0002.html>
More information about the M3devel
mailing list