[M3devel] pthread_atfork

[Jay K]

"fork() and do more work" is portable, to Posix, which is many systems.
It is easily done in Modula-3 by calling fork().

 
Given a choice between fixing fork() and do more work, vs. changing
cvsupd to not depend on it, and given what I know about
fixing fork() and do more work, I chose fork() and do more work.
Or just abandon cvsupd.


cvsupd is far from the only user of this pattern, in the wider non-Modula-3 world.
 
 
 > But why would user threads work? We're skipping pthread_atfork for them too. 
 
 
It is quite possible that fork and do more work + user threads + FreeBSD < 6, doesn't work.
It is quite possible that it works intermittently.
 
 
Let me try to explain what pthread_atfork is for.
 Though the opengroup spec does discuss it in detail and is clear (Bing for "opengroup pthread_atfork"; the first hit is it).
 
 
In a multithreaded system, arbitrary mutexes are held at arbitrary times.
Aribrary libraries spawn arbitrary worker threads, and use mutexes therein.
As well, such programs might fork and do more work.
There is not necessarily any central arbitrator among worker threads and the thread
that calls fork(). That is, when you fork, you might be holding multiple mutexes.
There isn't a way for the forking thread to request all other threads to enter some quiescent
state in which no mutexes are held.
 
 
If you fork and do more work, the new process has only one thread,
the thread that called fork(). But any mutex ownership at the time of fork() is inherited.
As a result, attemps to use mutexes in the new child process will have a high propensity
to hang/deadlock.
 
 
Therefore...go back..and do introduce an arbitrator.
The implementation of fork() is therefore integrated with the implementation of pthreads.
pthread_atfork establishes a set of callbacks. 3 sets actually.
fork() calls them at the appropriate times -- before fork in the parent, and after fork
in both the parent and the child.
What the callbacks are meant to do is acquire all locks in the parent before the fork,
and release them all in the parent and child after the work. And possibly reinitialize
globals in the child after the fork.
 
 
In the Modula-3 user thread system, threads exist by virtue of having
memory allocated for their stack, small memory allocated for their processor context/registers/etc.,
and by being on a global linked list. Scheduling is triggered by an occasional timer.
 
 
 
Historically, upon fork(), all that memory and the global linked list were unchanged.
Therefore the child process would continue to have all the same threads as the parent.
But compare this to pthreads. Solaris has an option to be either way, but in everything
else I've seen -- OpenBSD, FreeBSD, NetBSD, Darwin, Linux -- and I think Cygwin, HP-UX,
Irix, AIX, and probably the Posix spec -- fork() is specified as creating a child process
with only one thread. Solaris has fork1() and forkall() and the behavior of plain fork()
has varied from release to release.
 
 
So, again, upon fork(), historical Modula-3 user threads retain all threads.
If a thread held a mutex at fork() time, that's ok, it will continue running in the child
and eventually release the mutex. No hang.
 
 
If fork() is followed shortly by exec, well, then indeed, even with user threads,
alll threads are gone, globals get reinitialized (to zero), there is a new thread.
I guess you just have to be a bit careful what you do between fork() and exec().
Notice that things like Process.Create() suspend scheduling and probably after
that avoid allocating any memory and possibly avoid touching garbage collected memory
and probably avoid taking any locks. That is, upon suspending scheduling, they
don't depend on any mutex to need to be released by any other thread.
The various threads/mutexes can be in an arbitrary state. In the child they will
go away shortly (upon exec). In the parent, scheduling will shortly resume.
 
 
Does some of this make sense?
 
 
 > It is quite possible that fork and do more work + user threads + FreeBSD < 6, doesn't work.
 
 
Historically it did work -- child process retained all the threads.
These days I expect..I expect you still get that same behavior.
 
 
We could fix it, I believe, by never calling fork() directly.
Have a wrapper Process.Fork().
On most systems, it would just call fork.
But on OpenVMS and FreeBSD < 6...there is also the Register function.
On most systems, it'd just call pthread_atfork.
On OpenVMS and FreeBSD < 6, it should maintain the lists itself.
And then Process.Fork on those systems, make the call outs.
Indeed, the OpenVMS/FreeBSD<6 implementation could be the one and only implementation.
If fork() was disallowed and Process.Fork() the only allowed replacement.
Or, it could be tested that way, and then actually commited the other way.
I very well might do this.
 
 
 
 - Jay

 
> To: wagner at elegosoft.com
> Date: Sat, 12 Feb 2011 12:25:50 -0800
> From: mika at async.caltech.edu
> CC: m3devel at elegosoft.com
> Subject: Re: [M3devel] pthread_atfork
> 
> How do you do "fork-and-do-work" from Modula-3?
> 
> All my programs that create new Unix processes do it with Process.Create,
> which as far as I know does a fork-and-exec...
> 
> I'm not sure what if anything is broken here, either. But then again I
> didn't fully understand Jay's last mail on the subject.
> 
> Mika
> 
> Olaf Wagner writes:
> >Well, I think actually the fork-and-do-work pattern has been
> >the default for a long time with the M3 user threads implementation.
> >It wasn't specified though, and I think no application except
> >CVSup did use it.
> >
> >Jay is not responsible for any breakage here, he just tried to fix
> >CVSup and establish a consistent behaviour after our change to
> >system pthreads on most platforms.
> >
> >Olaf
> >
> >Quoting Mika Nystrom <mika at async.caltech.edu>:
> >
> >> Hi Jay,
> >>
> >> I think "fork and do work" qualifies as a bug in the application, if
> >> the application wants to be portable. Why can't it fork and exec instead?
> >> If it wants to do more work it should just fork and exec another copy
> >> of itself.
> >>
> >> fork() is also not part of Modula-3... Process.Create() is, however.
> >> Don't tell me you broke that as part of "fixing" things for cvsupd...
> >>
> >> There is still one aspect of your comment I don't understand. What does
> >> the call to pthread_atfork actually accomplish when running with user
> >> threads? You say below that "don't expect pthreads to work on FreeBSD <
> >> 6 [using the fork-and-do-work pattern]". Fair enough. But why would
> >> user threads work? We're skipping pthread_atfork for them too.
> >>
> >> Do we not need pthread_atfork with user threads? What you're saying
> >> suggests that FreeBSD >=3D 6 and all other platforms need pthread_atfork
> >> with user threads. Is that really what you mean?
> >>
> >> At the moment I'm just concerned with getting user threads to work as
> >> I know there are bugs with pthreads.
> >>
> >> I also think we should not waste time and effort making the "fork and do
> >> work" pattern work unless it's really, really necessary (which I really
> >> don't see why it would be). Fix the application instead of introducing
> >> bugs in the libraries and language implementation!
> >>
> >> I've been trying for a long, long time to switch from PM3 to CM3 (almost
> >> ten years, I think). These bugs from "enhancements" have stopped me
> >> every time. Maybe things are finally working well enough...
> >>
> >> (some comments inline in your quoted text below)
> >>
> >> Mika
> >>
> >>
> >> Jay K writes:
> >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_
> >>> Content-Type: text/plain; charset=3D"iso-8859-1"
> >>> Content-Transfer-Encoding: quoted-printable
> >>>
> >>>
> >>> I wrote that.
> >>>
> >>> As I recall=3D2C FreeBSD < 6 doesn't have pthread_atfork.
> >>>
> >>> I think that was based on reading the man pages.
> >>>
> >>> So=3D2C yes. What would you suggest?
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> All this atfork stuff came about as part of getting cvsupd to work.
> >>>
> >>> With pthreads/kernel threads. It historically worked=3D2C with Modula-3 =
> >use=3D
> >>> r threads.
> >>>
> >>> Modula-3 user threads have this funny property that all threads survive =
> >w=3D
> >>> ork.
> >>
> >> you mean survive fork don't you?
> >>
> >>>
> >>> With pthreads/kernel threads=3D2C fork gives you a new process with just=
> > on=3D
> >>> e thread.
> >>>
> >>> (Solaris has fork1 and forkall=3D2C and fork maybe used to be forkall=
> >=3D2C b=3D
> >>> ut now it is fork1).
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> cvsupd is one of those slightly unusual programs that does "fork and then=
> > d=3D
> >>> o more work"
> >>>
> >>> as supposed to the more typical "fork and exec".
> >>>
> >>>
> >>>
> >>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o=
> >n Fr=3D
> >>> eeBSD < 6?
> >>>
> >>> At least not in a process that ever calls fork? Or does "fork and then mo=
> >re=3D
> >>> work"?
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> In reality=3D2C i think a lot of libraries have trouble with "fork and th=
> >en d=3D
> >>> o more work"=3D2C but
> >>>
> >>> I don't know. I recall vague notices in the Apple/Darwin manpages not to =
> >as=3D
> >>> sume
> >>>
> >>> this works -- the obvious implication that there is plenty of use of pthr=
> >ea=3D
> >>> ds e.g. mutexes=3D2C
> >>>
> >>> but relatively little use of pthread_atfork.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> You could perhaps replace all uses of fork with a function that called th=
> >e =3D
> >>> fork handlers=3D2C
> >>>
> >>> then fork=3D2C then the other handlers. That is probably what I meant in =
> >the =3D
> >>> comment
> >>>
> >>> about this being easy to fix.
> >>>
> >>> You might though then run into the problem that callers of fork can't do =
> >mu=3D
> >>> ch
> >>>
> >>> before exec...oh=3D2C that's vfork I bet actually.
> >>
> >> I don't understand this paragraph either :(
> >>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> - Jay
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> To: hosking at cs.purdue.edu
> >>>> Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800
> >>>> From: mika at async.caltech.edu
> >>>> CC: m3devel at elegosoft.com=3D3B jay.krell at cornell.edu
> >>>> Subject: [M3devel] pthread_atfork
> >>>> =3D20
> >>>> Tony=3D2C
> >>>> =3D20
> >>>> What's RTProcessC.c doing?
> >>>> =3D20
> >>>> I note the following cryptic comment:
> >>>> =3D20
> >>>> /* NOTE: Even userthreads now depends
> >>>> * on availability of pthreads.
> >>>> * This can be fixed if need be.
> >>>> */
> >>>> =3D20
> >>>> now further down we can read:
> >>>> =3D20
> >>>> #if defined(_WIN32) \
> >>>> || defined(__vms) \
> >>>> || (defined(__FreeBSD__) && (__FreeBSD__ < 6))
> >>>> return 0=3D3B
> >>>> #else
> >>>> while (1)
> >>>> {
> >>>> int i =3D3D pthread_atfork(prepare=3D2C parent=3D2C child)=3D3B
> >>>> if (i !=3D3D EAGAIN)
> >>>> return i=3D3B
> >>>> sleep(0)=3D3B
> >>>> }
> >>>> #endif
> >>>> =3D20
> >>>> so on FreeBSD 5 (what I'm running on my "FreeBSD4" system)=3D2C
> >>>> RTProcess.RegisterForkHandlers does nothing?
> >>>> =3D20
> >>>> Hmmm.....
> >>>> =3D20
> >>>> Mika
> >>> =09=09 =09 =09=09 =3D
> >>>
> >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_
> >>> Content-Type: text/html; charset=3D"iso-8859-1"
> >>> Content-Transfer-Encoding: quoted-printable
> >>>
> >>> <html>
> >>> <head>
> >>> <style><!--
> >>> .hmmessage P
> >>> {
> >>> margin:0px=3D3B
> >>> padding:0px
> >>> }
> >>> body.hmmessage
> >>> {
> >>> font-size: 10pt=3D3B
> >>> font-family:Tahoma
> >>> }
> >>> --></style>
> >>> </head>
> >>> <body class=3D3D'hmmessage'>
> >>> I wrote that.<br>
> >>> As I recall=3D2C FreeBSD &lt=3D3B 6 doesn't have pthread_atfork.<br>
> >>> I think that was based on reading the man pages.<br>
> >>> So=3D2C yes. What would you suggest?<br>
> >>> <br>
> >>> <br>
> >>> All this atfork stuff came about as part of getting cvsupd to work.<br>
> >>> &nbsp=3D3B With pthreads/kernel threads. It historically worked=3D2C with=
> > Modul=3D
> >>> a-3 user threads.<br>
> >>> &nbsp=3D3B Modula-3 user threads have this funny property that all thread=
> >s su=3D
> >>> rvive work.<br>
> >>> &nbsp=3D3B With pthreads/kernel threads=3D2C fork gives you a new process=
> > with =3D
> >>> just one thread.<br>
> >>> &nbsp=3D3B&nbsp=3D3B (Solaris has fork1 and forkall=3D2C and fork maybe u=
> >sed to b=3D
> >>> e forkall=3D2C but now it is fork1).<br>
> >>> <br>
> >>> <br>
> >>> cvsupd is one of those slightly unusual programs that does "fork and then=
> > d=3D
> >>> o more work"<br>
> >>> as supposed to the more typical "fork and exec".<br>
> >>> <br>
> >>> Which is to say..I don't know. Maybe=3D2C don't expect pthreads to work o=
> >n Fr=3D
> >>> eeBSD &lt=3D3B 6?<br>
> >>> At least not in a process that ever calls fork? Or does "fork and then mo=
> >re=3D
> >>> work"?<br>
> >>> <br>
> >>> <br>
> >>> In reality=3D2C i think a lot of libraries have trouble with "fork and th=
> >en d=3D
> >>> o more work"=3D2C but<br>
> >>> I don't know. I recall vague notices in the Apple/Darwin manpages not to =
> >as=3D
> >>> sume<br>
> >>> this works -- the obvious implication that there is plenty of use of pthr=
> >ea=3D
> >>> ds e.g. mutexes=3D2C<br>
> >>> but relatively little use of pthread_atfork.<br>
> >>> <br>
> >>> <br>
> >>> You could perhaps replace all uses of fork with a function that called th=
> >e =3D
> >>> fork handlers=3D2C<br>
> >>> then fork=3D2C then the other handlers. That is probably what I meant in =
> >the =3D
> >>> comment<br>
> >>> about this being easy to fix.<br>
> >>> You might though then run into the problem that callers of fork can't do =
> >mu=3D
> >>> ch<br>
> >>> before exec...oh=3D2C that's vfork I bet actually.<br>
> >>> <br>
> >>> <br>
> >>> &nbsp=3D3B- Jay<br><br><br><br><br><br><br>&gt=3D3B To: hosking at cs.purdue=
> >.edu<b=3D
> >>> r>&gt=3D3B Date: Fri=3D2C 11 Feb 2011 19:05:25 -0800<br>&gt=3D3B From: mi=
> >ka at async=3D
> >>> .caltech.edu<br>&gt=3D3B CC: m3devel at elegosoft.com=3D3B jay.krell at cornell=
> >.edu<b=3D
> >>> r>&gt=3D3B Subject: [M3devel] pthread_atfork<br>&gt=3D3B <br>&gt=3D3B Ton=
> >y=3D2C<br>=3D
> >>> &gt=3D3B <br>&gt=3D3B What's RTProcessC.c doing?<br>&gt=3D3B <br>&gt=3D3B=
> > I note th=3D
> >>> e following cryptic comment:<br>&gt=3D3B <br>&gt=3D3B /* NOTE: Even usert=
> >hreads=3D
> >>> now depends<br>&gt=3D3B * on availability of pthreads.<br>&gt=3D3B * Th=
> >is ca=3D
> >>> n be fixed if need be.<br>&gt=3D3B */<br>&gt=3D3B <br>&gt=3D3B now furth=
> >er down =3D
> >>> we can read:<br>&gt=3D3B <br>&gt=3D3B #if defined(_WIN32) \<br>&gt=3D3B =
> > |=3D
> >>> | defined(__vms) \<br>&gt=3D3B || (defined(__FreeBSD__) &amp=3D3B=
> >&amp=3D
> >>> =3D3B (__FreeBSD__ &lt=3D3B 6))<br>&gt=3D3B return 0=3D3B<br>&gt=3D3B=
> > #else<br>&g=3D
> >>> t=3D3B while (1)<br>&gt=3D3B {<br>&gt=3D3B int i =3D3D pthr=
> >ead_atfork=3D
> >>> (prepare=3D2C parent=3D2C child)=3D3B<br>&gt=3D3B if (i !=3D3D EAGA=
> >IN)<br>&gt=3D
> >>> =3D3B return i=3D3B<br>&gt=3D3B sleep(0)=3D3B<br>&gt=3D3B =
> > }<br>&gt=3D
> >>> =3D3B #endif<br>&gt=3D3B <br>&gt=3D3B so on FreeBSD 5 (what I'm running o=
> >n my "Fr=3D
> >>> eeBSD4" system)=3D2C<br>&gt=3D3B RTProcess.RegisterForkHandlers does noth=
> >ing?<b=3D
> >>> r>&gt=3D3B <br>&gt=3D3B Hmmm.....<br>&gt=3D3B <br>&gt=3D3B Mika<br>
> >>> =09 =09 =09=09 =3D
> >>> </body>
> >>> </html>=3D
> >>>
> >>> --_19a7daf7-3f48-43cb-bea8-6f0fa13b0fef_--
> >>
> >
> >
> >
> >--=20
> >Olaf Wagner -- elego Software Solutions GmbH
> > Gustav-Meyer-Allee 25 / Geb=C3=A4ude 12, 13355 Berlin, Germa=
> >ny
> >phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95
> > http://www.elegosoft.com | Gesch=C3=A4ftsf=C3=BChrer: Olaf Wagner | Sitz=
> >: Berlin
> >Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20110212/e74eb6fc/attachment-0002.html>