<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style>
</head>
<body class='hmmessage'>
"fork() and do more work" is portable, to Posix, which is many systems.<BR>
It is easily done in Modula-3 by calling fork().<BR><BR>
<BR>
Given a choice between fixing fork() and do more work, vs. changing<BR>
cvsupd to not depend on it, and given what I know about<BR>
fixing fork() and do more work, I chose fork() and do more work.<BR>
Or just abandon cvsupd.<BR><BR><BR>
cvsupd is far from the only user of this pattern, in the wider non-Modula-3 world.<BR>
<BR>
<BR>
> But why would user threads work? We're skipping pthread_atfork for them too. <BR>
<BR>
<BR>
It is quite possible that fork and do more work + user threads + FreeBSD < 6, doesn't work.<BR>
It is quite possible that it works intermittently.<BR>
<BR>
<BR>
Let me try to explain what pthread_atfork is for.<BR>
Though the opengroup spec does discuss it in detail and is clear (Bing for "opengroup pthread_atfork"; the first hit is it).<BR>
<BR>
<BR>
In a multithreaded system, arbitrary mutexes are held at arbitrary times.<BR>
Aribrary libraries spawn arbitrary worker threads, and use mutexes therein.<BR>
As well, such programs might fork and do more work.<BR>
There is not necessarily any central arbitrator among worker threads and the thread<BR>
that calls fork(). That is, when you fork, you might be holding multiple mutexes.<BR>
There isn't a way for the forking thread to request all other threads to enter some quiescent<BR>
state in which no mutexes are held.<BR>
<BR>
<BR>
If you fork and do more work, the new process has only one thread,<BR>
the thread that called fork(). But any mutex ownership at the time of fork() is inherited.<BR>
As a result, attemps to use mutexes in the new child process will have a high propensity<BR>
to hang/deadlock.<BR>
<BR>
<BR>
Therefore...go back..and do introduce an arbitrator.<BR>
The implementation of fork() is therefore integrated with the implementation of pthreads.<BR>
pthread_atfork establishes a set of callbacks. 3 sets actually.<BR>
fork() calls them at the appropriate times -- before fork in the parent, and after fork<BR>
in both the parent and the child.<BR>
What the callbacks are meant to do is acquire all locks in the parent before the fork,<BR>
and release them all in the parent and child after the work. And possibly reinitialize<BR>
globals in the child after the fork.<BR>
<BR>
<BR>
In the Modula-3 user thread system, threads exist by virtue of having<BR>
memory allocated for their stack, small memory allocated for their processor context/registers/etc.,<BR>
and by being on a global linked list. Scheduling is triggered by an occasional timer.<BR>
<BR>
<BR>
<BR>
Historically, upon fork(), all that memory and the global linked list were unchanged.<BR>
Therefore the child process would continue to have all the same threads as the parent.<BR>
But compare this to pthreads. Solaris has an option to be either way, but in everything<BR>
else I've seen -- OpenBSD, FreeBSD, NetBSD, Darwin, Linux -- and I think Cygwin, HP-UX,<BR>
Irix, AIX, and probably the Posix spec -- fork() is specified as creating a child process<BR>
with only one thread. Solaris has fork1() and forkall() and the behavior of plain fork()<BR>
has varied from release to release.<BR>
<BR>
<BR>
So, again, upon fork(), historical Modula-3 user threads retain all threads.<BR>
If a thread held a mutex at fork() time, that's ok, it will continue running in the child<BR>
and eventually release the mutex. No hang.<BR>
<BR>
<BR>
If fork() is followed shortly by exec, well, then indeed, even with user threads,<BR>
alll threads are gone, globals get reinitialized (to zero), there is a new thread.<BR>
I guess you just have to be a bit careful what you do between fork() and exec().<BR>
Notice that things like Process.Create() suspend scheduling and probably after<BR>
that avoid allocating any memory and possibly avoid touching garbage collected memory<BR>
and probably avoid taking any locks. That is, upon suspending scheduling, they<BR>
don't depend on any mutex to need to be released by any other thread.<BR>
The various threads/mutexes can be in an arbitrary state. In the child they will<BR>
go away shortly (upon exec). In the parent, scheduling will shortly resume.<BR>
<BR>
<BR>
Does some of this make sense?<BR>
<BR>
<BR>
> It is quite possible that fork and do more work + user threads + FreeBSD < 6, doesn't work.<BR>
<BR>
<BR>
Historically it did work -- child process retained all the threads.<BR>
These days I expect..I expect you still get that same behavior.<BR>
<BR>
<BR>
We could fix it, I believe, by never calling fork() directly.<BR>
Have a wrapper Process.Fork().<BR>
On most systems, it would just call fork.<BR>
But on OpenVMS and FreeBSD < 6...there is also the Register function.<BR>
On most systems, it'd just call pthread_atfork.<BR>
On OpenVMS and FreeBSD < 6, it should maintain the lists itself.<BR>
And then Process.Fork on those systems, make the call outs.<BR>
Indeed, the OpenVMS/FreeBSD<6 implementation could be the one and only implementation.<BR>
If fork() was disallowed and Process.Fork() the only allowed replacement.<BR>
Or, it could be tested that way, and then actually commited the other way.<BR>
I very well might do this.<BR>
<BR>
<BR>
<BR>
- Jay<BR><BR> <BR>
> To: wagner@elegosoft.com<BR>> Date: Sat, 12 Feb 2011 12:25:50 -0800<BR>> From: mika@async.caltech.edu<BR>> CC: m3devel@elegosoft.com<BR>> Subject: Re: [M3devel] pthread_atfork<BR>> <BR>> How do you do "fork-and-do-work" from Modula-3?<BR>> <BR>> All my programs that create new Unix processes do it with Process.Create,<BR>> which as far as I know does a fork-and-exec...<BR>> <BR>> I'm not sure what if anything is broken here, either. But then again I<BR>> didn't fully understand Jay's last mail on the subject.<BR>> <BR>> Mika<BR>> <BR>> Olaf Wagner writes:<BR>> >Well, I think actually the fork-and-do-work pattern has been<BR>> >the default for a long time with the M3 user threads implementation.<BR>> >It wasn't specified though, and I think no application except<BR>> >CVSup did use it.<BR>> ><BR>> >Jay is not responsible for any breakage here, he just tried to fix<BR>> >CVSup and establish a consistent behaviour after our change to<BR>> >system pthreads on most platforms.<BR>> ><BR>> >Olaf<BR>> ><BR>> >Quoting Mika Nystrom <mika@async.caltech.edu>:<BR>> ><BR>> >> Hi Jay,<BR>> >><BR>> >> I think "fork and do work" qualifies as a bug in the application, if<BR>> >> the application wants to be portable. Why can't it fork and exec instead?<BR>> >> If it wants to do more work it should just fork and exec another copy<BR>> >> of itself.<BR>> >><BR>> >> fork() is also not part of Modula-3... Process.Create() is, however.<BR>> >> Don't tell me you broke that as part of "fixing" things for cvsupd...<BR>> >><BR>> >> There is still one aspect of your comment I don't understand. What does<BR>> >> the call to pthread_atfork actually accomplish when running with user<BR>> >> threads? You say below that "don't expect pthreads to work on FreeBSD <<BR>> >> 6 [using the fork-and-do-work pattern]". Fair enough. But why would<BR>> >> user threads work? We're skipping pthread_atfork for them too.<BR>> >><BR>> >> Do we not need pthread_atfork with user threads? What you're saying<BR>> >> suggests that FreeBSD >=3D 6 and all other platforms need pthread_atfork<BR>> >> with user threads. Is that really what you mean?<BR>> >><BR>> >> At the moment I'm just concerned with getting user threads to work as<BR>> >> I know there are bugs with pthreads.<BR>> >><BR>> >> I also think we should not waste time and effort making the "fork and do<BR>> >> work" pattern work unless it's really, really necessary (which I really<BR>> >> don't see why it would be). Fix the application instead of introducing<BR>> >> bugs in the libraries and language implementation!<BR>> >><BR>> >> I've been trying for a long, long time to switch from PM3 to CM3 (almost<BR>> >> ten years, I think). These bugs from "enhancements" have stopped me<BR>> >> every time. Maybe things are finally working well enough...<BR>> >><BR>> >> (some comments inline in your quoted text below)<BR>> >><BR>> >> Mika<BR>> >><BR>> >><BR>> >> Jay K writes:<BR>> >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_<BR>> >>> Content-Type: text/plain; charset=3D"iso-8859-1"<BR>> >>> Content-Transfer-Encoding: quoted-printable<BR>> >>><BR>> >>><BR>> >>> I wrote that.<BR>> >>><BR>> >>> As I recall=3D2C FreeBSD < 6 doesn't have pthread_atfork.<BR>> >>><BR>> >>> I think that was based on reading the man pages.<BR>> >>><BR>> >>> So=3D2C yes. What would you suggest?<BR>> >>><BR>> >>><BR>> >>><BR>> >>><BR>> >>><BR>> >>> All this atfork stuff came about as part of getting cvsupd to work.<BR>> >>><BR>> >>> With pthreads/kernel threads. It historically worked=3D2C with Modula-3 =<BR>> >use=3D<BR>> >>> r threads.<BR>> >>><BR>> >>> Modula-3 user threads have this funny property that all threads survive =<BR>> >w=3D<BR>> >>> ork.<BR>> >><BR>> >> you mean survive fork don't you?<BR>> >><BR>> >>><BR>> >>> With pthreads/kernel threads=3D2C fork gives you a new process with just=<BR>> > on=3D<BR>> >>> e thread.<BR>> >>><BR>> >>> (Solaris has fork1 and forkall=3D2C and fork maybe used to be forkall=<BR>> >=3D2C b=3D<BR>> >>> ut now it is fork1).<BR>> >>><BR>> >>><BR>> >>><BR>> >>><BR>> >>><BR>> >>> cvsupd is one of those slightly unusual programs that does "fork and then=<BR>> > d=3D<BR>> >>> o more work"<BR>> >>><BR>> >>> as supposed to the more typical "fork and exec".<BR>> >>><BR>> >>><BR>> >>><BR>> >>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o=<BR>> >n Fr=3D<BR>> >>> eeBSD < 6?<BR>> >>><BR>> >>> At least not in a process that ever calls fork? Or does "fork and then mo=<BR>> >re=3D<BR>> >>> work"?<BR>> >>><BR>> >>><BR>> >>><BR>> >>><BR>> >>><BR>> >>> In reality=3D2C i think a lot of libraries have trouble with "fork and th=<BR>> >en d=3D<BR>> >>> o more work"=3D2C but<BR>> >>><BR>> >>> I don't know. I recall vague notices in the Apple/Darwin manpages not to =<BR>> >as=3D<BR>> >>> sume<BR>> >>><BR>> >>> this works -- the obvious implication that there is plenty of use of pthr=<BR>> >ea=3D<BR>> >>> ds e.g. mutexes=3D2C<BR>> >>><BR>> >>> but relatively little use of pthread_atfork.<BR>> >>><BR>> >>><BR>> >>><BR>> >>><BR>> >>><BR>> >>> You could perhaps replace all uses of fork with a function that called th=<BR>> >e =3D<BR>> >>> fork handlers=3D2C<BR>> >>><BR>> >>> then fork=3D2C then the other handlers. That is probably what I meant in =<BR>> >the =3D<BR>> >>> comment<BR>> >>><BR>> >>> about this being easy to fix.<BR>> >>><BR>> >>> You might though then run into the problem that callers of fork can't do =<BR>> >mu=3D<BR>> >>> ch<BR>> >>><BR>> >>> before exec...oh=3D2C that's vfork I bet actually.<BR>> >><BR>> >> I don't understand this paragraph either :(<BR>> >><BR>> >>><BR>> >>><BR>> >>><BR>> >>><BR>> >>><BR>> >>> - Jay<BR>> >>><BR>> >>><BR>> >>><BR>> >>><BR>> >>><BR>> >>><BR>> >>>> To: hosking@cs.purdue.edu<BR>> >>>> Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800<BR>> >>>> From: mika@async.caltech.edu<BR>> >>>> CC: m3devel@elegosoft.com=3D3B jay.krell@cornell.edu<BR>> >>>> Subject: [M3devel] pthread_atfork<BR>> >>>> =3D20<BR>> >>>> Tony=3D2C<BR>> >>>> =3D20<BR>> >>>> What's RTProcessC.c doing?<BR>> >>>> =3D20<BR>> >>>> I note the following cryptic comment:<BR>> >>>> =3D20<BR>> >>>> /* NOTE: Even userthreads now depends<BR>> >>>> * on availability of pthreads.<BR>> >>>> * This can be fixed if need be.<BR>> >>>> */<BR>> >>>> =3D20<BR>> >>>> now further down we can read:<BR>> >>>> =3D20<BR>> >>>> #if defined(_WIN32) \<BR>> >>>> || defined(__vms) \<BR>> >>>> || (defined(__FreeBSD__) && (__FreeBSD__ < 6))<BR>> >>>> return 0=3D3B<BR>> >>>> #else<BR>> >>>> while (1)<BR>> >>>> {<BR>> >>>> int i =3D3D pthread_atfork(prepare=3D2C parent=3D2C child)=3D3B<BR>> >>>> if (i !=3D3D EAGAIN)<BR>> >>>> return i=3D3B<BR>> >>>> sleep(0)=3D3B<BR>> >>>> }<BR>> >>>> #endif<BR>> >>>> =3D20<BR>> >>>> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=3D2C<BR>> >>>> RTProcess.RegisterForkHandlers does nothing?<BR>> >>>> =3D20<BR>> >>>> Hmmm.....<BR>> >>>> =3D20<BR>> >>>> Mika<BR>> >>> =09=09 =09 =09=09 =3D<BR>> >>><BR>> >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_<BR>> >>> Content-Type: text/html; charset=3D"iso-8859-1"<BR>> >>> Content-Transfer-Encoding: quoted-printable<BR>> >>><BR>> >>> <html><BR>> >>> <head><BR>> >>> <style><!--<BR>> >>> .hmmessage P<BR>> >>> {<BR>> >>> margin:0px=3D3B<BR>> >>> padding:0px<BR>> >>> }<BR>> >>> body.hmmessage<BR>> >>> {<BR>> >>> font-size: 10pt=3D3B<BR>> >>> font-family:Tahoma<BR>> >>> }<BR>> >>> --></style><BR>> >>> </head><BR>> >>> <body class=3D3D'hmmessage'><BR>> >>> I wrote that.<br><BR>> >>> As I recall=3D2C FreeBSD <=3D3B 6 doesn't have pthread_atfork.<br><BR>> >>> I think that was based on reading the man pages.<br><BR>> >>> So=3D2C yes. What would you suggest?<br><BR>> >>> <br><BR>> >>> <br><BR>> >>> All this atfork stuff came about as part of getting cvsupd to work.<br><BR>> >>>  =3D3B With pthreads/kernel threads. It historically worked=3D2C with=<BR>> > Modul=3D<BR>> >>> a-3 user threads.<br><BR>> >>>  =3D3B Modula-3 user threads have this funny property that all thread=<BR>> >s su=3D<BR>> >>> rvive work.<br><BR>> >>>  =3D3B With pthreads/kernel threads=3D2C fork gives you a new process=<BR>> > with =3D<BR>> >>> just one thread.<br><BR>> >>>  =3D3B =3D3B (Solaris has fork1 and forkall=3D2C and fork maybe u=<BR>> >sed to b=3D<BR>> >>> e forkall=3D2C but now it is fork1).<br><BR>> >>> <br><BR>> >>> <br><BR>> >>> cvsupd is one of those slightly unusual programs that does "fork and then=<BR>> > d=3D<BR>> >>> o more work"<br><BR>> >>> as supposed to the more typical "fork and exec".<br><BR>> >>> <br><BR>> >>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o=<BR>> >n Fr=3D<BR>> >>> eeBSD <=3D3B 6?<br><BR>> >>> At least not in a process that ever calls fork? Or does "fork and then mo=<BR>> >re=3D<BR>> >>> work"?<br><BR>> >>> <br><BR>> >>> <br><BR>> >>> In reality=3D2C i think a lot of libraries have trouble with "fork and th=<BR>> >en d=3D<BR>> >>> o more work"=3D2C but<br><BR>> >>> I don't know. I recall vague notices in the Apple/Darwin manpages not to =<BR>> >as=3D<BR>> >>> sume<br><BR>> >>> this works -- the obvious implication that there is plenty of use of pthr=<BR>> >ea=3D<BR>> >>> ds e.g. mutexes=3D2C<br><BR>> >>> but relatively little use of pthread_atfork.<br><BR>> >>> <br><BR>> >>> <br><BR>> >>> You could perhaps replace all uses of fork with a function that called th=<BR>> >e =3D<BR>> >>> fork handlers=3D2C<br><BR>> >>> then fork=3D2C then the other handlers. That is probably what I meant in =<BR>> >the =3D<BR>> >>> comment<br><BR>> >>> about this being easy to fix.<br><BR>> >>> You might though then run into the problem that callers of fork can't do =<BR>> >mu=3D<BR>> >>> ch<br><BR>> >>> before exec...oh=3D2C that's vfork I bet actually.<br><BR>> >>> <br><BR>> >>> <br><BR>> >>>  =3D3B- Jay<br><br><br><br><br><br><br>>=3D3B To: hosking@cs.purdue=<BR>> >.edu<b=3D<BR>> >>> r>>=3D3B Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800<br>>=3D3B From: mi=<BR>> >ka@async=3D<BR>> >>> .caltech.edu<br>>=3D3B CC: m3devel@elegosoft.com=3D3B jay.krell@cornell=<BR>> >.edu<b=3D<BR>> >>> r>>=3D3B Subject: [M3devel] pthread_atfork<br>>=3D3B <br>>=3D3B Ton=<BR>> >y=3D2C<br>=3D<BR>> >>> >=3D3B <br>>=3D3B What's RTProcessC.c doing?<br>>=3D3B <br>>=3D3B=<BR>> > I note th=3D<BR>> >>> e following cryptic comment:<br>>=3D3B <br>>=3D3B /* NOTE: Even usert=<BR>> >hreads=3D<BR>> >>> now depends<br>>=3D3B * on availability of pthreads.<br>>=3D3B * Th=<BR>> >is ca=3D<BR>> >>> n be fixed if need be.<br>>=3D3B */<br>>=3D3B <br>>=3D3B now furth=<BR>> >er down =3D<BR>> >>> we can read:<br>>=3D3B <br>>=3D3B #if defined(_WIN32) \<br>>=3D3B =<BR>> > |=3D<BR>> >>> | defined(__vms) \<br>>=3D3B || (defined(__FreeBSD__) &=3D3B=<BR>> >&=3D<BR>> >>> =3D3B (__FreeBSD__ <=3D3B 6))<br>>=3D3B return 0=3D3B<br>>=3D3B=<BR>> > #else<br>&g=3D<BR>> >>> t=3D3B while (1)<br>>=3D3B {<br>>=3D3B int i =3D3D pthr=<BR>> >ead_atfork=3D<BR>> >>> (prepare=3D2C parent=3D2C child)=3D3B<br>>=3D3B if (i !=3D3D EAGA=<BR>> >IN)<br>>=3D<BR>> >>> =3D3B return i=3D3B<br>>=3D3B sleep(0)=3D3B<br>>=3D3B =<BR>> > }<br>>=3D<BR>> >>> =3D3B #endif<br>>=3D3B <br>>=3D3B so on FreeBSD 5 (what I'm running o=<BR>> >n my "Fr=3D<BR>> >>> eeBSD4" system)=3D2C<br>>=3D3B RTProcess.RegisterForkHandlers does noth=<BR>> >ing?<b=3D<BR>> >>> r>>=3D3B <br>>=3D3B Hmmm.....<br>>=3D3B <br>>=3D3B Mika<br><BR>> >>> =09 =09 =09=09 =3D<BR>> >>> </body><BR>> >>> </html>=3D<BR>> >>><BR>> >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_--<BR>> >><BR>> ><BR>> ><BR>> ><BR>> >--=20<BR>> >Olaf Wagner -- elego Software Solutions GmbH<BR>> > Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa=<BR>> >ny<BR>> >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95<BR>> > http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz=<BR>> >: Berlin<BR>> >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194<BR> </body>
</html>