[M3devel] more crashers

Tony Hosking hosking at cs.purdue.edu
Mon Aug 17 16:47:46 CEST 2009 

Indeed, we might be better off investing effort in getting rid of try- 
blocks implemented using setjmp/longjmp so as to avoid the stack- 
allocated jmpbuf state.  Or perhaps allocate that state in the heap.

On 17 Aug 2009, at 09:18, Jay K wrote:

>
> Not as much luck as I'd like.
>
>
> PTHREAD_STACK_MIN
> Solaris: 4K
> Linux/sparc: 24K
> Linux/x86: 16K
> Linux/powerpc: 128K
>
>
> The overflowing stack, with 16K:
>
>
> current thread: t at 12
> =>[1] Rd__CharsReady(0xfd61a00c, 0xfd617410, 0xfd59fb6c, 0x0,  
> 0xfd59fb84, 0xfd59
> fb8c), at 0xfec58578
>  ---- 1 following frame from gwindows -- possible stack overflow
>  [2] SynScan__LookChar(0xfd883670, 0xfd8aa81c, 0xfd59fb6c, 0x0,  
> 0xfd59fb84, 0xf
> d59fb8c), at 0xfead71fc
>  [3] SynScan__NextToken(0xfd883670, 0xfd8ab980, 0x9, 0x0,  
> 0xfd59fb84, 0xfd59fb8
> c), at 0xfead91c4
>  [4] SynScan__LookToken(0xfd883670, 0x4ae, 0x9, 0x0, 0xfd59fb84,  
> 0xfd59fb8c), a
> t 0xfead9628
>  [5] SynScan__HaveTokenName(0xfd883670, 0xfd5f7768, 0x8, 0x0,  
> 0xfd59fb84, 0xfd5
> 9fb8c), at 0xfeadd084
>  [6] SynParse__Read0(0xfd883654, 0xfd6174ac, 0xfd59fb6c, 0x0,  
> 0xfd59fb84, 0xfd5
> 9fb8c), at 0xfeae4760
>  [7] SynParse__Read0(0xfd883654, 0xfd617460, 0xfd59fb6c, 0x0,  
> 0xfd59fb84, 0xfd5
> 9fb8c), at 0xfeae5e8c
>
>
> hundreds of frames of Lots of SynParse__Read0, using about 1K per  
> frame; there is a try in there.
>
> d59fb8c), at 0xfeae348c
>  [666] SynParse__Read0(0xfd883654, 0xfd6172d8, 0xfd59fb6c, 0x0,  
> 0xfd59fb84, 0xf
> d59fb8c), at 0xfeae5e8c
>  [667] SynParse__Read0(0xfd883654, 0xfd8aaa6c, 0xfd59fb6c, 0x0,  
> 0xfd59fb84, 0xf
> d59fb8c), at 0xfeae3dd4
>  [668] SynParse__Read0(0xfd883654, 0xfd617288, 0xfd59fb6c, 0x0,  
> 0xfd59fb84, 0xf
> d59fb8c), at 0xfeae3168
>  [669] SynParse__Read0(0xfd883654, 0xfd8aacec, 0xfd59fb6c, 0x0,  
> 0xfd59fb84, 0xf
> d59fb8c), at 0xfeae6230
>  [670] SynParse__Read1(0xfd883654, 0xfd8aacec, 0x0, 0xfd59fb84,  
> 0xfd59fb8c, 0x0
> ), at 0xfeae2674
>  [671] SynParse__Read(0xfd883654, 0xfd8aacec, 0x0, 0xfd803a00, 0x0,  
> 0xfebee308)
> , at 0xfeadfb60
>  [672] SynParse__ReadNonTerminal(0xfd883654, 0xfefed45c, 0xfd61b250,  
> 0xfd61b8cc
> , 0xfd61b280, 0xfd89a27c), at 0xfeadfae8
>  [673] ObliqParser__ParsePhrase(0xfd883654, 0x0, 0xfd61b250,  
> 0xfd61b8cc, 0xfd61
> b280, 0x0), at 0xfefd5ff0
>  [674] ObliqOnline__Interact(0xfd89a848, 0xc32a0, 0xfd61a00c, 0x1,  
> 0x1, 0xfebee
> 308), at 0xff36b80c
>  [675] ObliqRuntime__ForkedDo(0xfd3ddfb8, 0x9658aac0, 0xff000000,  
> 0x0, 0xfd9100
> 90, 0x0), at 0x80db0
>  [676] ThreadPThread__RunThread(0xef2b0, 0x0, 0x0, 0x0, 0xfd803a00,  
> 0x1), at 0x
> fe06bc88
>  [677] ThreadPThread__ThreadBase(0xef2b0, 0xfd5a0000, 0x0, 0x0,  
> 0xfe06b7c8, 0x1
> ), at 0xfe06b818
>
>
> One of the problems with defining a unit of stack size is the large  
> variety of jmpbuf size.
> But heck, why 1K per frame here when SOLsun has a stack walker? Hm.
>
> - Jay
>
>
> ----------------------------------------
>> From: jay.krell at cornell.edu
>> To: wagner at elegosoft.com; m3devel at elegosoft.com
>> Subject: RE: [M3devel] more crashers
>> Date: Mon, 17 Aug 2009 11:35:40 +0000
>>
>>
>> - I think the whole machine stack mechanism is pretty hopelessly  
>> broken.
>> Not just in Modula-3. Therefore my enthusiasm to attempt to "fix"
>> it -- which is impossible -- isn't very high.
>>
>> - I'm hoping to find that no platform other than Solaris lets you
>> even use such a small stack. I have to experiment some in a bit.
>> Depending on that experiment, we'll be in a better position, without
>> any further changes, but again, depending on what I find.
>> That is, if every other platform already had a larger stack, then  
>> this
>> just brings Solaris inline with the rest. If the other platforms
>> were actually successfully using a 4K stack, then indeed a problem.
>>
>>
>> - It could also be a factor of optimization.
>> Maybe ensuring -O2 or -O3 is used more would cover it up??
>>
>>
>> - Or some other changes.
>> That is, if we have drastically increased stack usage, we should try
>> to reduce it back down. But given that 4K was the previous value
>> and even 16K doesn't work...I'm not hopeful we could fit it back  
>> down in 4K.
>>
>> It is easy enough to see the bug and then look at the various frames
>> and see if they use a lot of stack and can be reduced.
>>
>> - Jay
>>
>>
>>
>> ----------------------------------------
>>> Date: Mon, 17 Aug 2009 13:30:26 +0200
>>> From: wagner at elegosoft.com
>>> To: m3devel at elegosoft.com
>>> Subject: Re: [M3devel] more crashers
>>>
>>> Quoting Jay K :
>>>
>>>> visobliq was stack overflow.
>>>> Raising the default from 4K to 16K didn't fix, but 32K did.
>>>
>>> We need to address this somehow. It cannot be correct if the user
>>> needs to increase the default stack size on one platform by 8
>>> and not at all on another. Do we have any concept of a 'standard
>>> stack size unit' ;-)? Something that can be easily understood and
>>> implemented and be used instead of the machine words in the current
>>> interfaces? Any ideas?
>>>
>>> Olaf
>>>
>>>
>>>> - Jay
>>>>
>>>> ----------------------------------------
>>>>> From: jay.krell at cornell.edu
>>>>> To: hosking at cs.purdue.edu
>>>>> CC: m3devel at elegosoft.com
>>>>> Subject: RE: [M3devel] more crashers
>>>>> Date: Sun, 16 Aug 2009 20:32:31 +0000
>>>>>
>>>>>
>>>>> m3browser:
>>>>>
>>>>> It is specific to my config file
>>>>> m3-sys/cminstall/src/config-no-install/cm3.cfg that reachs back
>>>>> into the source tree. It won't happen with what most people do. I
>>>>> have to do that M3Predefines or such work so that all the uses of
>>>>> m3quake besides cm3 are much more like cm3.
>>>>>
>>>>>
>>>>> visobliq:
>>>>> not the same thing
>>>>> Does not occur on Linux/x86 or Linux/sparc.
>>>>> They do print
>>>>> Syntax error just before: -helvetica-bold-\*R-\*240-\*\")  
>>>>> (LabelFo ...
>>>>> ill-formed character escape sequence
>>>>> Error detected (file file generated by Visual Obliq) (line 18,  
>>>>> char 10)
>>>>> VideoVBT.Start Jv ServerFailure:errno=0
>>>>>
>>>>> but then gui comes up.
>>>>> I'll have to finish my Solaris reinstall and get dbx.
>>>>>
>>>>>
>>>>> - Jay
>>>>>
>>>>>
>>>>> ----------------------------------------
>>>>>> From: jay.krell at cornell.edu
>>>>>> To: hosking at cs.purdue.edu
>>>>>> Date: Sun, 16 Aug 2009 20:22:49 +0000
>>>>>> CC: m3devel at elegosoft.com
>>>>>> Subject: Re: [M3devel] more crashers
>>>>>>
>>>>>>
>>>>>> sorry, this is simple.
>>>>>>
>>>>>> derived_dirs := IntList.List1 (ID.Add  
>>>>>> (MxConfig.Get("BUILD_DIR")));
>>>>>>
>>>>>>
>>>>>> - Jay
>>>>>>
>>>>>>
>>>>>>
>>>>>> ----------------------------------------
>>>>>>> From: jay.krell at cornell.edu
>>>>>>> To: hosking at cs.purdue.edu
>>>>>>> Date: Sun, 16 Aug 2009 19:52:03 +0000
>>>>>>> CC: m3devel at elegosoft.com
>>>>>>> Subject: Re: [M3devel] more crashers
>>>>>>>
>>>>>>>
>>>>>>> Here is the relevant code in m3browser ID.m3:
>>>>>>>
>>>>>>>
>>>>>>> C:\dev2\cm3.2\m3-tools\m3markup\src\ID.m3(38): empty_id := Add  
>>>>>>> ("");
>>>>>>>
>>>>>>> empty_id := Add ("");
>>>>>>>
>>>>>>> (*--------------------------------------------------------------
>>>>>>> exported ---*)
>>>>>>> PROCEDURE Add (x: TEXT): T =
>>>>>>> VAR
>>>>>>> t: T;
>>>>>>> a: REF ARRAY OF CHAR;
>>>>>>> l: INTEGER;
>>>>>>> BEGIN
>>>>>>> l := Text.Length(x);
>>>>>>> ...
>>>>>>> END Add;
>>>>>>>
>>>>>>>
>>>>>>> The empty string constant ends up as null.
>>>>>>>
>>>>>>> .stabn 68,0,38,.LM115-.LFBB10
>>>>>>> .LM115:
>>>>>>> movl L_1 at GOT(%ebx), %eax
>>>>>>> leal 24(%eax), %eax
>>>>>>> movl %eax, (%esp)
>>>>>>> call ID__Add at PLT
>>>>>>> movl %eax, -8(%ebp)
>>>>>>> movl -8(%ebp), %ecx
>>>>>>> movl MM_ID at GOT(%ebx), %edx
>>>>>>> movl 248(%edx), %eax
>>>>>>> andl $0, %eax
>>>>>>> orl %ecx, %eax
>>>>>>> movl %eax, 248(%edx)
>>>>>>> .L136:
>>>>>>> movl MM_ID at GOT(%ebx), %eax
>>>>>>> .LBE16:
>>>>>>> addl $52, %esp
>>>>>>> popl %ebx
>>>>>>> popl %ebp
>>>>>>> ret
>>>>>>>
>>>>>>> L_1:
>>>>>>> 0 .long RTHooks__TextLitInfo
>>>>>>> 4 .long RTHooks__TextLitGetChar
>>>>>>> 8 .long RTHooks__TextLitGetWideChar
>>>>>>> 12 .long RTHooks__TextLitGetChars
>>>>>>> 16 .long RTHooks__TextLitGetWideChars
>>>>>>> 20 .long 2
>>>>>>> 24 .long L_1
>>>>>>> .long 0
>>>>>>> .zero 4
>>>>>>> .ascii "ID_M3"
>>>>>>> .zero 1
>>>>>>> .ascii "ExpandHashTable"
>>>>>>> .zero 1
>>>>>>> .ascii "ExpandIDs"
>>>>>>> .zero 1
>>>>>>> .ascii "ExpandChars"
>>>>>>> .zero 1
>>>>>>> .ascii "Compare"
>>>>>>> .zero 1
>>>>>>>
>>>>>>> If I change "" to "foo", I get:
>>>>>>>
>>>>>>> L_1:
>>>>>>> 0 .long RTHooks__TextLitInfo
>>>>>>> 4 .long RTHooks__TextLitGetChar
>>>>>>> 8 .long RTHooks__TextLitGetWideChar
>>>>>>> 12 .long RTHooks__TextLitGetChars
>>>>>>> 16 .long RTHooks__TextLitGetWideChars
>>>>>>> 20 .long 2
>>>>>>> 24 .long L_1
>>>>>>> .long 3
>>>>>>> .ascii "foo"
>>>>>>> .zero 1
>>>>>>> .ascii "ID_M3"
>>>>>>>
>>>>>>> and it still crashes.
>>>>>>>
>>>>>>>
>>>>>>> It seems to hit ID__Add twice when only once would be expected.
>>>>>>>
>>>>>>>
>>>>>>> - Jay
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------
>>>>>>>> From: jay.krell at cornell.edu
>>>>>>>> To: hosking at cs.purdue.edu
>>>>>>>> Date: Sun, 16 Aug 2009 19:33:34 +0000
>>>>>>>> CC: m3devel at elegosoft.com
>>>>>>>> Subject: Re: [M3devel] more crashers
>>>>>>>>
>>>>>>>>
>>>>>>>> Could be. I've also seen it on PPC_DARWIN a while ago. I only
>>>>>>>> recall seeing it on sparc and ppc.
>>>>>>>> It doesn't always occur.
>>>>>>>>
>>>>>>>>
>>>>>>>> Here is the m3browser crash on a system that can be debugged.
>>>>>>>> See why I want the full paths in the debug info?
>>>>>>>>
>>>>>>>>
>>>>>>>> C:\Users\jay\Desktop>\cygwin\bin\ssh xobsd.
>>>>>>>> $ cd /cm3
>>>>>>>> $ cd bin
>>>>>>>> $ export DISPLAY=192.168.1.120:0.0
>>>>>>>> $ gdb ./m3browser
>>>>>>>> GNU gdb 6.3
>>>>>>>> This GDB was configured as "i386-unknown-openbsd4.5"...
>>>>>>>> (gdb) r
>>>>>>>> Starting program: /home/jay/cm3/bin/m3browser
>>>>>>>> Program received signal SIGSEGV, Segmentation fault.
>>>>>>>> [Switching to process 31063, thread 0x7eafe800]
>>>>>>>> 0x01592fd4 in Text__Length (M3_Bd56fi_t=0x0) at ../src/text/ 
>>>>>>>> Text.m3:16
>>>>>>>> 16 t.get_info (i);
>>>>>>>> (gdb) bt
>>>>>>>> #0 0x01592fd4 in Text__Length (M3_Bd56fi_t=0x0) at ../src/ 
>>>>>>>> text/Text.m3:16
>>>>>>>> #1 0x09fd9464 in ID__Add (M3_Bd56fi_x=0x0) at ../src/ID.m3:48
>>>>>>>> #2 0x1c01e644 in Main_M3 (M3_AcxOUs_mode=1) at ../src/ 
>>>>>>>> Main.m3:37
>>>>>>>> #3 0x0156f1bc in RTLinker__RunMainBody (M3_DjPxE3_m=0x3c003ec0)
>>>>>>>> at ../src/runtime/common/RTLinker.m3:399
>>>>>>>> #4 0x0156e576 in RTLinker__AddUnitI (M3_DjPxE3_m=0x3c003ec0)
>>>>>>>> at ../src/runtime/common/RTLinker.m3:113
>>>>>>>> #5 0x0156e604 in RTLinker__AddUnit (M3_DjPxE5_b=0x1c01e54d)
>>>>>>>> at ../src/runtime/common/RTLinker.m3:122
>>>>>>>> #6 0x1c004676 in main (argc=1, argv=0xcfbf6a3c,  
>>>>>>>> envp=0xcfbf6a44)
>>>>>>>> at _m3main.mc:4
>>>>>>>>
>>>>>>>>
>>>>>>>> - Jay
>>>>>>>>
>>>>>>>>
>>>>>>>> ________________________________
>>>>>>>>> From: hosking at cs.purdue.edu
>>>>>>>>> To: jay.krell at cornell.edu
>>>>>>>>> Date: Sun, 16 Aug 2009 15:26:49 -0400
>>>>>>>>> CC: m3devel at elegosoft.com
>>>>>>>>> Subject: Re: [M3devel] more crashers
>>>>>>>>>
>>>>>>>>> Could this be endianness-related?
>>>>>>>>>
>>>>>>>>> On 16 Aug 2009, at 15:21, Jay K wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I see the assertion failure on PPC_LINUX too.
>>>>>>>>> I tried VAR defaultStackSize := 1024 * 1024; (not sure why
>>>>>>>>> there was any suspicion of stack size).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Debugging doesn't work PPC_LINUX.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> jay at plin:/cm3/bin$ gdb ./cm3
>>>>>>>>> GNU gdb 6.8-debian
>>>>>>>>> Copyright (C) 2008 Free Software Foundation, Inc.
>>>>>>>>> License GPLv3+: GNU GPL version 3 or later
>>>>>>>>> This is free software: you are free to change and  
>>>>>>>>> redistribute it.
>>>>>>>>> There is NO WARRANTY, to the extent permitted by law. Type
>>>>>>>>> "show copying"
>>>>>>>>> and "show warranty" for details.
>>>>>>>>> This GDB was configured as "powerpc-linux-gnu"...
>>>>>>>>> break ma(gdb) break main
>>>>>>>>> Breakpoint 1 at 0x86fc4: file _m3main.mc, line 2.
>>>>>>>>> (gdb) r
>>>>>>>>> Starting program: /home/jay/cm3/bin/cm3
>>>>>>>>> Warning:
>>>>>>>>> Cannot insert breakpoint 1.
>>>>>>>>> Error accessing memory address 0x86fc4: Input/output error.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> No problem with C:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> jay at plin:/cm3/bin$ echo "main(){}"> 1.c
>>>>>>>>> jay at plin:/cm3/bin$ gcc -g 1.c
>>>>>>>>> jay at plin:/cm3/bin$ gdb ./a.out
>>>>>>>>> GNU gdb 6.8-debian
>>>>>>>>> Copyright (C) 2008 Free Software Foundation, Inc.
>>>>>>>>> License GPLv3+: GNU GPL version 3 or later
>>>>>>>>> This is free software: you are free to change and  
>>>>>>>>> redistribute it.
>>>>>>>>> There is NO WARRANTY, to the extent permitted by law. Type
>>>>>>>>> "show copying"
>>>>>>>>> and "show warranty" for details.
>>>>>>>>> This GDB was configured as "powerpc-linux-gnu"...
>>>>>>>>> (gdb) break main
>>>>>>>>> Breakpoint 1 at 0x10000448: file 1.c, line 1.
>>>>>>>>> (gdb) r
>>>>>>>>> Starting program: /home/jay/cm3/bin/a.out
>>>>>>>>> Breakpoint 1, main () at 1.c:1
>>>>>>>>> 1 main(){}
>>>>>>>>> (gdb)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> jay at plin:/cm3/bin$ gcc -gstabs+ 1.c
>>>>>>>>> jay at plin:/cm3/bin$ gdb ./a.out
>>>>>>>>> GNU gdb 6.8-debian
>>>>>>>>> Copyright (C) 2008 Free Software Foundation, Inc.
>>>>>>>>> License GPLv3+: GNU GPL version 3 or later
>>>>>>>>> This is free software: you are free to change and  
>>>>>>>>> redistribute it.
>>>>>>>>> There is NO WARRANTY, to the extent permitted by law. Type
>>>>>>>>> "show copying"
>>>>>>>>> and "show warranty" for details.
>>>>>>>>> This GDB was configured as "powerpc-linux-gnu"...
>>>>>>>>> (gdb) break main
>>>>>>>>> Breakpoint 1 at 0x10000448: file 1.c, line 1.
>>>>>>>>> (gdb) r
>>>>>>>>> Starting program: /home/jay/cm3/bin/a.out
>>>>>>>>> Breakpoint 1, main () at 1.c:1
>>>>>>>>> 1 main(){}
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> jay at plin:/cm3/bin$ gcc -gstabs 1.c
>>>>>>>>> jay at plin:/cm3/bin$ gdb ./a.out
>>>>>>>>> GNU gdb 6.8-debian
>>>>>>>>> Copyright (C) 2008 Free Software Foundation, Inc.
>>>>>>>>> License GPLv3+: GNU GPL version 3 or later
>>>>>>>>> This is free software: you are free to change and  
>>>>>>>>> redistribute it.
>>>>>>>>> There is NO WARRANTY, to the extent permitted by law. Type
>>>>>>>>> "show copying"
>>>>>>>>> and "show warranty" for details.
>>>>>>>>> This GDB was configured as "powerpc-linux-gnu"...
>>>>>>>>> (gdb) break main
>>>>>>>>> Breakpoint 1 at 0x10000448: file 1.c, line 1.
>>>>>>>>> (gdb) r
>>>>>>>>> Starting program: /home/jay/cm3/bin/a.out
>>>>>>>>> Breakpoint 1, main () at 1.c:1
>>>>>>>>> 1 main(){}
>>>>>>>>> (gdb)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - Jay
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----------------------------------------
>>>>>>>>> From: jay.krell at cornell.edu
>>>>>>>>> To: m3devel at elegosoft.com
>>>>>>>>> Subject: more crashers
>>>>>>>>> Date: Sun, 16 Aug 2009 10:06:15 +0000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is head on SOLgnu:
>>>>>>>>> I'll try another and debug and see if fix is easy.
>>>>>>>>>
>>>>>>>>> -bash-3.00$ ./m3browser
>>>>>>>>>
>>>>>>>>> ***
>>>>>>>>> *** runtime error:
>>>>>>>>> *** Segmentation violation - possible attempt to dereference  
>>>>>>>>> NIL
>>>>>>>>> *** pc = 0xfe484f04 = String16_M3 + 0xe0 in ../src/text/ 
>>>>>>>>> String16.m3
>>>>>>>>> ***
>>>>>>>>>
>>>>>>>>> -bash-3.00$ ./visobliq
>>>>>>>>> Internal Obliq interpreter installed...
>>>>>>>>> Establishing 'templates'
>>>>>>>>> (Created frame 'templates')
>>>>>>>>> let
>>>>>>>>> Templates =
>>>>>>>>> {boolean=> ... ,
>>>>>>>>> browser=> ... ,
>>>>>>>>> button=> ... ,
>>>>>>>>> choice=> ... ,
>>>>>>>>> filebrowser=> ... ,
>>>>>>>>> form=> ... ,
>>>>>>>>> frame=> ... ,
>>>>>>>>> hscroll=> ... ,
>>>>>>>>> menu=> ... ,
>>>>>>>>> numeric=> ... ,
>>>>>>>>> text=> ... ,
>>>>>>>>> textedit=> ... ,
>>>>>>>>> typein=> ... ,
>>>>>>>>> video=> ... ,
>>>>>>>>> vscroll=> ... }
>>>>>>>>> Establishing 'vowidgets'
>>>>>>>>> (Created frame 'vowidgets')
>>>>>>>>> ok
>>>>>>>>> Establishing 'vocheckpt'
>>>>>>>>> (Created frame 'vocheckpt')
>>>>>>>>> ok
>>>>>>>>> Establishing 'volib'
>>>>>>>>> (Created frame 'volib')
>>>>>>>>> Segmentation Fault (core dumped)
>>>>>>>>> -bash-3.00$
>>>>>>>>>
>>>>>>>>>
>>>
>>>
>>>
>>> --
>>> Olaf Wagner -- elego Software Solutions GmbH
>>> Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
>>> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23  
>>> 45 86 95
>>> http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz:  
>>> Berlin
>>> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr:  
>>> DE163214194
>>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20090817/e5f60f64/attachment-0002.html>

More information about the M3devel mailing list