[M3devel] 5.8.6 LINUXLIBC6 breakage, kernel 2.6.23, glibc-2.6-4
Jay K
jay.krell at cornell.edu
Tue Apr 19 22:32:54 CEST 2011
> This change depends on pthread_atfork, for pthreads and user threads.
Clarification: There's no change described there for pthreads, and indeed, getting
this behavior with pthreads doesn't depend on using pthread_atfork. That is just
how pthreads are -- except optionally on Solaris (fork1() vs. forkall() vs. fork()).
The Solaris default used to be fork => forkall().
The current Solaris default is fork => fork1().
There really do exist on Solaris to functions you can call, fork1() and forkall().
All other systems and the Posix specification have the "fork1" semantic for fork.
Therefore "fork1" is what we should provide unconditionally.
If you really want "forkall", then you write:
#ifdef __sun
forkall()
#else
#error
#endif
and/or you go and hack up m3core/src/thread/POSIX, branch off m3core/src/thread/POSIX_FORKALL or such.
I don't expect anyone to ever do that.
The need for pthread_atfork is sort of different for user threads vs. pthreads, but it is needed either way.
For pthreads, there is this problem -- at the time a thread calls fork(), what are other threads doing?
Answer: Anything.
Follow up question: What locks might they hold?
Answer: Any.
Follow up: So, if arbitrary locks are held, and you fork, and don't exec, and the child process
attempts to use some of the locks that were held in the parent, and only the forking thread survived,
then the other threads will never leave their locks, and later attempts to acquire them in the children
will deadlock. Therefore, what pthread_atfork provides for, what you are supposed to use it for,
is roughly speaking, right before fork(), in the parent, acquire all locks, and then fork(), and then in the
parent and children, release all locks. Therefore, you have to come up with a locking order.
Things are a little different for user threads.
In user threads, the existance of threads is established by virtue of a global array or list of data describing the threads.
And a timer to occasionally preempt. When the timer interrupts, the scheduler picks another thread from the global
array/list. Previously, the data went unchanged when fork() was called. Therefore all threads survived.
What we do now is that in the child "handler" for pthread_atfork, is reinitialize the various data, so as to forget about
all but the current thread. Something like that.
I suspect if we had RTProcess.ForkBeforeExec and RTProcess.ForkBeforeMoreWork, then ForkBeforeExec
might get away with just fork() and ignore all this.
So far we have not drawn this line -- there is just the one fork and it is used prior to exec or do-more-work
and does the same thing w/o knowing what is to follow.
The Posix documentation I think describes this all fairly well.
Bing for "posix opengroup pthread_atfork".
Really. I think it is saying much of the same thing I am saying, but in language that has received
much more thought. Be sure to read the "rationale" part.
- Jay
From: jay.krell at cornell.edu
To: mika at async.caltech.edu
CC: m3devel at elegosoft.com
Subject: RE: [M3devel] 5.8.6 LINUXLIBC6 breakage, kernel 2.6.23, glibc-2.6-4
Date: Tue, 19 Apr 2011 15:06:38 +0000
Previously: user threads: all threads survived fork()
Previously: pthreads: only the thread calling fork() survived fork()
Now: user threads and pthreads: only the thread calling fork() survives fork()
This must be how pthreads behave, and this makes user threads and pthreads consistent.
This change depends on pthread_atfork, for pthreads and user threads.
It only really matters to the rare "fork + do more work" program, such as cvsupd.
Most programs either never fork, or exec almost immediately after fork.
pthread_atfork offers a good model.
A sort of "distributed" model.
You don't have to go and change all the calls to fork().
Each module with a need to do something before/after fork, calls the central pthread_atfork,
and fork and pthread_atfork cooperate to do what is needed.
No function pointer is needed.
Instead move the code to m3core/thread.
If you really must not use -pthread, then you must implement pthread_atfork functionality
yourself and have all fork() calls go through your own fork() wrapper that cooperates
with your pthread_atfork replacement.
There is no free lunch -- there is a downside to this approach, as plain fork() calls
are ok and correct if pthread_atfork is used, but now become incorrect.
Pick your poison:
user thread/pthread inconsistency
cvsupd incompatibility with pthreads
user threads using pthread_atfork/-pthread
fork() calls having to go through a wrapper (ok -- you could miss this and
not likely notice -- only fork() calls in fork+do-more-work programs need the wrapper).
(Most of this has been explained multiple times, but people only pay attention
when they think it affects them. I'm guilty of the same thing.)
- Jay
> To: jay.krell at cornell.edu
> Date: Mon, 18 Apr 2011 19:30:58 -0700
> From: mika at async.caltech.edu
> CC: m3devel at elegosoft.com
> Subject: Re: [M3devel] 5.8.6 LINUXLIBC6 breakage, kernel 2.6.23, glibc-2.6-4
>
>
> If you or Tony could describe roughly what you think needs to be done
> I'd be happy to look into it.
>
> The basic problem is that a decision that's made/described in
> m3core/thread/m3makefile needs to someone find its way to controlling
> what C code gets compiled elsewhere in the system. Or maybe there should
> be an indirection into the thread library to pick up pthread_atfork (or not).
>
> But then again you still haven't explained why you made the user threads use
> pthread_atfork. I just remove it from my installations, but then again I'm
> not trying to run CVSup so I don't know if it breaks that program to do so.
>
> Mika
>
> Jay K writes:
> >--_3dd397d4-ac1d-4148-a9ff-059d27dd794a_
> >Content-Type: text/plain; charset="iso-8859-1"
> >Content-Transfer-Encoding: quoted-printable
> >
> >
> >> The following code from RTProcessC.c ensures that it is neeeded on every =
> >Unix except
> >> > /* NOTE: Even userthreads now depends
> >> > * on availability of pthreads.
> >> > * This can be fixed if need be.
> >> > */
> >
> >=20
> >Ok=2C "we" should probably go ahead and fix that.
> >I'll try to=2C but no promises=2C sorry.
> >=20
> > - Jay
> >=20
> >> From: hosking at cs.purdue.edu
> >> Date: Mon=2C 18 Apr 2011 18:11:26 -0400
> >> To: mika at async.caltech.edu
> >> CC: m3devel at elegosoft.com
> >> Subject: Re: [M3devel] 5.8.6 LINUXLIBC6 breakage=2C kernel 2.6.23=2C glib=
> >c-2.6-4
> >>=20
> >> Probably unnecessary=2C given that I think there is another entry point t=
> >o forking a process (I forget where) in the thread-specific portion of m3co=
> >re. In which case the necessary work might be done there?
> >>=20
> >> On Apr 18=2C 2011=2C at 2:45 PM=2C Mika Nystrom wrote:
> >>=20
> >> > Tony Hosking writes:
> >> > ...
> >> >> pthread_atfork should not be needed under user threads.
> >> > ...
> >> >=20
> >> > The following code from RTProcessC.c ensures that it is neeeded on ever=
> >y Unix except
> >> > FreeBSD before 6. The comment is from the checked-in source file.
> >> >=20
> >> > /* NOTE: Even userthreads now depends
> >> > * on availability of pthreads.
> >> > * This can be fixed if need be.
> >> > */
> >> >=20
> >> > INTEGER
> >> > __cdecl
> >> > RTProcess__RegisterForkHandlers(
> >> > ForkHandler prepare=2C
> >> > ForkHandler parent=2C
> >> > ForkHandler child)
> >> > {
> >> > /* FreeBSD < 6 lacks pthread_atfork. Would be good to use autoconf.
> >> > * VMS lacks pthread_atfork? Would be good to use autoconf.
> >> > * Win32 lacks pthread_atfork and fork. OK.
> >> > *
> >> > * As well=2C for all Posix systems=2C we could implement
> >> > * atfork ourselves=2C as long as we provide a fork()
> >> > * wrapper that code uses.
> >> > */
> >> > #if defined(_WIN32) \
> >> > || defined(__vms) \
> >> > || (defined(__FreeBSD__) /* && (__FreeBSD__ < 6)*/ )
> >> > return 0=3B
> >> > #else
> >> > while (1)
> >> > {
> >> > int i =3D pthread_atfork(prepare=2C parent=2C child)=3B
> >> > if (i !=3D EAGAIN)
> >> > return i=3B
> >> > sleep(0)=3B
> >> > }
> >> > #endif
> >> > }
> >>=20
> > =
> >
> >--_3dd397d4-ac1d-4148-a9ff-059d27dd794a_
> >Content-Type: text/html; charset="iso-8859-1"
> >Content-Transfer-Encoding: quoted-printable
> >
> ><html>
> ><head>
> ><style><!--
> >.hmmessage P
> >{
> >margin:0px=3B
> >padding:0px
> >}
> >body.hmmessage
> >{
> >font-size: 10pt=3B
> >font-family:Tahoma
> >}
> >--></style>
> ></head>
> ><body class=3D'hmmessage'>
> >>=3B The following code from RTProcessC.c ensures that it is neeeded on e=
> >very Unix except<BR>>=3B >=3B /* NOTE: Even userthreads now depends<BR>=
> >>=3B >=3B * on availability of pthreads.<BR>>=3B >=3B * This can be=
> > fixed if need be.<BR>>=3B >=3B */<BR><BR>
> > =3B<BR>
> >Ok=2C "we" should probably go ahead and fix that.<BR>
> >I'll try to=2C but no =3Bpromises=2C sorry.<BR>
> > =3B<BR>
> > =3B- Jay<BR> =3B<BR>
> >>=3B From: hosking at cs.purdue.edu<BR>>=3B Date: Mon=2C 18 Apr 2011 18:11=
> >:26 -0400<BR>>=3B To: mika at async.caltech.edu<BR>>=3B CC: m3devel at elegos=
> >oft.com<BR>>=3B Subject: Re: [M3devel] 5.8.6 LINUXLIBC6 breakage=2C kerne=
> >l 2.6.23=2C glibc-2.6-4<BR>>=3B <BR>>=3B Probably unnecessary=2C given =
> >that I think there is another entry point to forking a process (I forget wh=
> >ere) in the thread-specific portion of m3core. In which case the necessary =
> >work might be done there?<BR>>=3B <BR>>=3B On Apr 18=2C 2011=2C at 2:45=
> > PM=2C Mika Nystrom wrote:<BR>>=3B <BR>>=3B >=3B Tony Hosking writes:=
> ><BR>>=3B >=3B ...<BR>>=3B >=3B>=3B pthread_atfork should not be n=
> >eeded under user threads.<BR>>=3B >=3B ...<BR>>=3B >=3B <BR>>=3B =
> >>=3B The following code from RTProcessC.c ensures that it is neeeded on e=
> >very Unix except<BR>>=3B >=3B FreeBSD before 6. The comment is from the=
> > checked-in source file.<BR>>=3B >=3B <BR>>=3B >=3B /* NOTE: Even u=
> >serthreads now depends<BR>>=3B >=3B * on availability of pthreads.<BR>&=
> >gt=3B >=3B * This can be fixed if need be.<BR>>=3B >=3B */<BR>>=3B =
> >>=3B <BR>>=3B >=3B INTEGER<BR>>=3B >=3B __cdecl<BR>>=3B >=3B =
> >RTProcess__RegisterForkHandlers(<BR>>=3B >=3B ForkHandler prepare=2C<BR=
> >>>=3B >=3B ForkHandler parent=2C<BR>>=3B >=3B ForkHandler child)<BR=
> >>>=3B >=3B {<BR>>=3B >=3B /* FreeBSD <=3B 6 lacks pthread_atfork.=
> > Would be good to use autoconf.<BR>>=3B >=3B * VMS lacks pthread_atfork=
> >? Would be good to use autoconf.<BR>>=3B >=3B * Win32 lacks pthread_atf=
> >ork and fork. OK.<BR>>=3B >=3B *<BR>>=3B >=3B * As well=2C for all =
> >Posix systems=2C we could implement<BR>>=3B >=3B * atfork ourselves=2C =
> >as long as we provide a fork()<BR>>=3B >=3B * wrapper that code uses.<B=
> >R>>=3B >=3B */<BR>>=3B >=3B #if defined(_WIN32) \<BR>>=3B >=3B =
> >|| defined(__vms) \<BR>>=3B >=3B || (defined(__FreeBSD__) /* &=3B&am=
> >p=3B (__FreeBSD__ <=3B 6)*/ )<BR>>=3B >=3B return 0=3B<BR>>=3B >=
> >=3B #else<BR>>=3B >=3B while (1)<BR>>=3B >=3B {<BR>>=3B >=3B in=
> >t i =3D pthread_atfork(prepare=2C parent=2C child)=3B<BR>>=3B >=3B if (=
> >i !=3D EAGAIN)<BR>>=3B >=3B return i=3B<BR>>=3B >=3B sleep(0)=3B<BR=
> >>>=3B >=3B }<BR>>=3B >=3B #endif<BR>>=3B >=3B }<BR>>=3B <BR> =
> > </body>
> ></html>=
> >
> >--_3dd397d4-ac1d-4148-a9ff-059d27dd794a_--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20110419/c06bc7eb/attachment-0002.html>
More information about the M3devel
mailing list