[M3devel] threading on Windows?

Tue Feb 15 02:34:59 CET 2011

I'll have to look into it. It is possible I dropped support for "NT386" in favor of "I386_NT".
I've definitely used the Python a fair amount on Windows, but less lately.
You can probably force it by first setting the environment variable CM3_TARGET to NT386 or I386_NT.
Or by giving either of those parameters on the command line.
But you shouldn't have to.

 - Jay

From: rcolebur at SCIRES.COM
To: jay.krell at cornell.edu; m3devel at elegosoft.com
Date: Mon, 14 Feb 2011 18:54:50 -0500
Subject: Re: [M3devel] threading on Windows?

Jay:

Up to now, I’ve been using my Windows CMD scripts to “upgrade”, namely:  scripts\dev\windows\ RCC_upgradeCM3.CMD

But, based on your message, I am trying to do as you suggest, i.e., use your “upgrade.py”.

I am running Windows 7.  I have just installed Python 2.7.1 via http://www.python.org/ftp/python/2.7.1/python-2.7.1.msi 

Here is error message I get:

C:\cm3\Sandbox\scripts\python>upgrade.py
Traceback (most recent call last):
  File "C:\cm3\Sandbox\scripts\python\upgrade.py", line 4, in <module>
    import pylib
  File "C:\cm3\Sandbox\scripts\python\pylib.py", line 612, in <module>
    if Target.startswith("NT386"):
AttributeError: 'NoneType' object has no attribute 'startswith'

Please advise.

Regards,
Randy Coleburn

From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K
Sent: Saturday, February 12, 2011 5:39 AM
To: Coleburn, Randy; m3devel
Subject: RE: [M3devel] threading on Windows?

TextUtils.mo : error LNK2001: unresolved external symbol __alloca
TextUtils.mo : error LNK2001: unresolved external symbol _Csetjmp__Jumpbuf_size
sysutils.dll : fatal error LNK1120: 2 unresolved externals

Most likely you didn't build it correctly.
Go back to a working cm3/m3core/libm3 and run upgrade.py.
There should be no longer any references to alloca of Csetjmp__Jumpbuf_size.
Really, Csetjmp__Jumpbuf_size should never have been a problem. That likely also indicates
some sort of incorrect bootstrap.
 > Are you saying you do, or do not, want me to work further on the solving the “problem,” regardless of how we describe it ?

I still want alloca to work, either via a function call or a specific m3cg method.
(I still don't see that RTHooks.Alloca makes better sense; I still don't see a way
to avoid hacking the compiler, given the custom calling convention.) 

It will be difficult to test the fix until/unless m3front is changed back.

 - Jay

From: rcolebur at SCIRES.COM
To: m3devel at elegosoft.com
Date: Fri, 11 Feb 2011 18:17:15 -0500
Subject: Re: [M3devel] threading on Windows?

Jay:

Ok, I don’t mean to start a new issue or upset anyone.

I see you checked in some new changes.  I updated to latest HEAD and tried to build, but again I get unresolved symbols beginning with m3-libs\sysutils  (see attached sysutils.lst)

So, not sure what you mean by “it should be ok now”.

Are you saying you do, or do not, want me to work further on the solving the “problem,” regardless of how we describe it ?

Regards,
Randy

From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K
Sent: Friday, February 11, 2011 4:52 PM
To: Tony; Coleburn, Randy
Cc: m3devel
Subject: RE: [M3devel] threading on Windows?

It isn't heap-based!
  And saying it "leaks" is misleading. It doesn't leak how most people would consider "leak".
  It calls _alloca. If you call _alloca in a loop, you get more and more storage, until you return from the function.
  I understand this. Tony understand it. But calling it "leak" will mislead almost everyone else.
It should be ok now. But still only a temporary solution. We have to either move all the back
to the old way, or make the new way work.

As we have two implementations of many things, it is common for one or the other to break.
The fix is really not difficult. We can't be stuck like this on everything.
But granted, m3front remains confusing to me, so I am stuck for a bit
doing this much better.

Anyway, my time is now significantly reallocated, so it all matters less.

 - Jay

From: hosking at cs.purdue.edu
Date: Fri, 11 Feb 2011 16:29:43 -0500
To: rcolebur at SCIRES.COM
CC: m3devel at elegosoft.com
Subject: Re: [M3devel] threading on Windows?

Randy,

I am in complete agreement with you.

Antony Hosking | Associate Professor | Computer Science | Purdue University

305 N. University Street | West Lafayette | IN 47907 | USA

Office +1 765 494 6001 | Mobile +1 765 427 5484

On Feb 11, 2011, at 4:26 PM, Coleburn, Randy wrote:

Tony:

Thank you very much for this explanation.

So, it appears that the changes introduced to support the new “heap-based jmpbuf allocation” have introduced at least 2 problems that must be solved if we are to move forward on the Windows platform, namely:

1.       Space leak problem.

2.       Binding to appropriate “alloca” routine on Windows platform.

Assuming we want to retain this new heap-based jmpbuf allocation capability, someone needs to solve these 2 problems for the Windows platform.

I will begin looking at the code and see if I can make some progress on these 2 problems.  Before I “do” anything, I’ll run my proposed solution thru the m3devel group first.  Any further insights you or Jay can give in helping me work on the solution are welcome and appreciated.

Finally, let me also suggest that we all may need to revisit the conversation we had on this forum many months back about contributors making sure their changes are tested not to cause major breaks before committing them to HEAD branch. 

Seems to me that if the “heap based jumpbuf allocation scheme” was known not to be compatible with Windows, that it should have been checked in on a different experimental branch until all the kinks could be worked out for all platforms.

Otherwise, without some mechanism to warn everyone of known problems on the HEAD branch, folks can easily download stuff that renders their implementation broken.

If my understanding of HEAD is not the prevailing one, I for one would like clarification as to what is STABLE for checkout and use and what is UNSTABLE.

Regards,

Randy Coleburn

From: Tony Hosking [mailto:hosking at cs.purdue.edu] 
Sent: Thursday, February 10, 2011 4:26 PM
To: Coleburn, Randy
Cc: m3devel
Subject: Re: [M3devel] threading on Windows?

Hi Randy,

A far as I know, the changes are as follows.

TRY blocks using setjmp/longjmp to implement the exception handling each require statically one instance of a jmpbuf.  Historically, the storage for this was reserved as stack local storage (using CG.Declare_param) of the target-dependent jmpbuf size.  This meant having to declare the jmp_buf accurately for the target.  What Jay did was to replace the stack local with a scalar pointer local variable (also using Declare_param), and to call alloca to obtain a pointer to stack allocated jmpbuf storage.  The idea was to add a level of indirection, and have the size of the jmpbuf obtained from a C variable initialized appropriately,  so that targets need no longer declare their jmpbuf_size.   This was intended to make porting easier.  There are several problems here.  First, the local pointer variable needs to be initialized *once* on entry to the procedure to the value NIL, so that we check for NIL on the first invocation of the TRY and alloca only in that case.  The problem is that currently that initialization is not being performed *once* but instead on each invocation of the TRY so loops with TRY blocks in them currently have a space leak.

The other problem was how to bind to the appropriate alloca routine.  Ideally it should be inlined, and on Windows this is achieved by calling a *different* function, not called alloca.  My suggestion is that making the alloca call a runtime hook will allow targets to easily redirect to the correct function.  On non-Windows targets using the gcc-based backend all works out because alloca is a gcc intrinsic function that gets inlined.

To fix the space leak problem I proposed that Marker in the M3 compiler front end m3front should track, per-procedure, the jmpbuf pointer variables as it parses or checks the program, and have a callback to initialize them at the top of the procedure during code generation.  I don't think this will involve much work, but I don't currently have any spare cycles to spend on it.

-- Tony

Antony Hosking | Associate Professor | Computer Science | Purdue University

305 N. University Street | West Lafayette | IN 47907 | USA

Office +1 765 494 6001 | Mobile +1 765 427 5484

On Feb 10, 2011, at 3:59 PM, Coleburn, Randy wrote:

Jay / Tony:

I am seeing in this thread that the approach to fixing the problem is not yet agreed upon.

I will be glad to help in any way I can, but right now, I am “in the dark”, both as to what change was introduced that caused the problem, and the various potential solution paths.  All I know is that it has been broken for a number of weeks now.

I know Jay has asked for help, and I am willing to help, but it may cost him or you some time/effort in getting me up to speed on what went wrong and the pros/cons of potential solutions.  If you choose to help enlighten me, I pledge to try and put that knowledge to good use both now and in the future.  Otherwise, I will have to wait for one of you to solve the problem.

BTW, when I look at the linker error reports for the various packages, I’m seeing several different symbols that don’t resolve, not just the one dealing with memory allocation.

Regards,

Randy

From: Tony Hosking [mailto:hosking at cs.purdue.edu] 
Sent: Thursday, February 10, 2011 3:33 PM
To: jay.krell at cornell.edu
Cc: Coleburn, Randy; m3devel
Subject: Re: [M3devel] threading on Windows?

Why is it better?  Because rebinding to a different function is just a matter of changing an INTERFACE in the libraries, rather than hacking the compiler.

On Feb 10, 2011, at 6:01 AM, jay.krell at cornell.edu wrote:

1. I never understood why this any better.
2. It how things are currently.

3. Why not m3cg.alloca? (but see #2)

 - Jay/iPad

On Feb 9, 2011, at 9:48 PM, Tony Hosking <hosking at cs.purdue.edu> wrote:

What happened to fixing this with runtime hooks?

On Feb 9, 2011, at 11:40 PM, Jay K wrote:

Of course not.
It is a made-up function that the frontend generates a call to.
That it never previously did.
Using the general function call interface.
You need to add a special case, in the general function call code, to do something different and specific
for this specific function.

 - Jay 

From: rcolebur at SCIRES.COM
To: jay.krell at cornell.edu; m3devel at elegosoft.com
Date: Wed, 9 Feb 2011 22:28:45 -050
Subject: Re: [M3devel] threading on Windows?

Jay:

I looked thru the 4,618 lines of M3x86.m3, but I don’t see any reference to “m3_alloca” or even “alloc” in this file.

Regards,

Randy

From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K
Sent: Wednesday, February 09, 2011 9:01 PM
To: Coleburn, Randy; m3devel
Subject: RE: [M3devel] threading on Windows?

In m3back/src/M3x86.m3.

 - Jay

From: rcolebur at SCIRES.COM
To: m3devel at elegosoft.com
Date: Wed, 9 Feb 2011 18:22:56 -0500
Subject: Re: [M3devel] threading on Windows?

I am certainly willing to work on the problem, but need more context info about what caused it in order to know how to resolve.

All I know is that everything was working fine until I checked out the HEAD repository.

When you say “special case” calls “m3_alloca”, where do I go about finding this “special case”?

Are we talking Modula-3 code, C code, Assembler, what?  What package/module?

Regards,

Randy

From: jayk123 at hotmail.com [mailto:jayk123 at hotmail.com] On Behalf Of Jay K
Sent: Wednesday, February 09, 2011 5:59 PM
To: Coleburn, Randy; Mika Nystrom; m3devel
Subject: RE: [M3devel] threading on Windows?

Maybe someone else can do it?
The fix is: special case calls to the function "m3_alloca".
Change it to call "_chkstk" (or maybe "chkstk", whatever works).
The one parameter is an unsigned 32bit quantity, passed in eax, and the return value is a pointer, returned in esp.
To some extent I drag my feet hoping anyone else might become motivated enough to do it and start learning how.

 - Jay

> From: rcolebur at SCIRES.COM
> To: mika at async.caltech.edu; m3devel at elegosoft.com
> Date: Wed, 9 Feb 2011 17:25:20 -0500
> Subject: Re: [M3devel] threading on Windows?
> 
> Mika:
> 
> Sorry, but my Windows build is broken. Jay seems to indicate he is working on a "fix" and that this fix is relatively simple, but so far no solution has been checked in.
> 
> I've been wanting to run your program ever since you first checked it in, but that was about same time the HEAD branch update introduced a build problem. Problem is an unresolved symbol during link.
> 
> As soon as I can get the build problem resolved, I'll try out your test program.
> 
> Regards,
> Randy Coleburn
> 
> -----Original Message-----
> From: Mika Nystrom [mailto:mika at async.caltech.edu] 
> Sent: Wednesday, February 09, 2011 5:01 PM
> To: m3devel at elegosoft.com
> Subject: [M3devel] threading on Windows?
> 
> Hi m3devel,
> 
> I'm just curious if anyone out there who's running CM3 on Windows has had
> a chance to try my thread testing program yet. Current status on Unix
> (Linux, FreeBSD) appears to be that there are problems in pthreads (but
> I think only under heavy load) and that user threading works perfectly.
> So I now wonder how things are on the third threading platform (Windows).
> The thread testing program is at m3-libs/m3core/tests/thread .
> 
> (Note however that there is a memory leak in TRY-EXCEPT in the current
> head so don't try to update everything to the CVS head. The previous
> release should be OK.)
> 
> Also has anyone else noticed that debugging information has recently
> broken? m3gdb is very unhappy on all platforms for me...
> 
> Mika

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20110215/3f5bf114/attachment-0002.html>