<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
--></style>
</head>
<body class='hmmessage'>
To repeat myself:<BR>
<BR>
> and basically *any* library that uses pthreads is obligated to use<BR> > pthread_atfork, be it a C library or the Modula-3 library, etc..<BR><BR>
<BR>
This is the crux of the matter.<BR>
We are being "bad pthread citizens" by not using pthread_atfork.<BR>Granted, we are probably in large company.<BR>
<BR>
<BR>There is a smaller fix and a larger fix, but it seems very clear we<BR>
should take one of them.<BR>
<BR>
<BR>
The small fix is:<BR>
common/Thread.i3: add PThreadAtFork<BR>
It does nothing on posix/nt.<BR>
It calls pthread_atfork on pthread.<BR>
<BR>
<BR>
In RTCollector.m3 give a child handler that stores FALSE in the<BR>
three booleans that indicate the three threads have started.<BR>
No prepare or parent handler I believe.<BR>
Though I should check for global locks there.<BR>
Probably the locks in ThreadPThread suffice?<BR>
<BR>
<BR>
In ThreadPThread.m3 provide three handlers:<BR>
prepare: lock "everything", at least the 5 static locks.<BR>
I might try walking all the threads and locking them too.<BR>
parent: unlock everything<BR>
child: unlock everything and reinitialize the globals<BR>
<BR>
<BR>
Note that the atfork handlers run in many more programs than cvsup.<BR>
They would be used also with fork + exec.<BR>
Perhaps a way to disable that -- no way in Posix, we could just<BR>
provide a boolean to ourselves.<BR>
<BR>
<BR>
The larger fix would be to provide a common/Thread.i3 function<BR>
to call fork() *and* recreate all Modula-3 threads in the child process.<BR>
That is what I sent out.<BR>
<BR>
<BR>
In the second case, you also then need to allow that the caller<BR>
may or may not use that function, so you still need atfork handlers,<BR>
but they have to be slightly different, as the diffs I sent show.<BR>
<BR>
<BR>
I believe strongly this is the way to go.<BR>
It is how one is a "good pthread citizen".<BR>
Now, granted, I don't think most libraries are.<BR>
Witness the caveat about fork() in the Darwin manpage and<BR>
I think even the Posix spec.<BR>
But pthread_atfork is meant to solve this.<BR>
<BR>
<BR>
I don't suggest a full audit and add atfork handlers everywhere.<BR>
But m3core we can at least do, and that should satisfy cvsup.<BR>
Should check libm3 too.<BR>
<BR>
<BR>
Plus, as long as each module takes reponsibility for itself,<BR>
it shouldn't be so hard.<BR>
That is, I don't think the case is all that strong for providing the "forkall"<BR>
function that is in the diffs I sent.<BR>
<BR>
<BR>
I'll do more testing and send out diffs "later".<BR>
It could be as long as a week, I have a bunch of things to do.<BR>
Or maybe just a day. :)<BR>
<BR>
<BR>
I tested a form of this fix + cvsup + user threads and it appears<BR>
that cvsup can do one small thing and thereby work either way,<BR>
without knowledge as to the way.<BR>
<BR>
<BR>
That is, it can shutdown the dispatcher "directly", instead of queuing to it.<BR>
I'm not even sure the dispatcher thread is really needed.<BR>
<BR>
<BR>
As I said, I don't believe the application can handle this all itself.<BR>
Any library that uses pthreads is obligated to play into the general<BR>
mechanism. Generally the internals are not exposed for the application<BR>
to do it.<BR>
<BR>
Treating Modula-3 as a closed system in which nobody uses<BR>
anything outside of or "below" m3core/libm3 I don't think is practical.<BR>
<BR>
<BR>
"applications", arguably defines as "processes", are often composed<BR>
of fairly disparate bodies of code that don't necessarily expose much<BR>
to each other. For example, they might each create their own worker<BR>
threads and not expose them.<BR>
This is what pthread_atfork is for.<BR>
Actually Tony, you gave me the lead here. :)<BR>
<BR>
<BR>
There is a problem that m3posixthreads (user) and pthreads semantics<BR>
diverage, and..I haven't been able to think of an way to fix that.<BR>
<BR>
<BR>
Well, it is actually easy to make "forkall" the default and only behavior actually,<BR>
on user and pthreads (but not NT).<BR>
<BR>
<BR>
But I don't see a way to make "fork1" the behavior for user threads.<BR>
You can associate a pid with each thread, and notice when the pid changes,<BR>
but I don't know which one thread you would keep, and you wouldn't<BR>
necessarily be able to register pthread_atfork handlers, unless maybe<BR>
user threads were used only on platforms that actually have pthreads..<BR>
<BR>
<BR>
And still there's no good way to it on NT I expect.<BR>
<BR>
<BR>
- Jay<BR> <BR>
<HR id=stopSpelling>
From: jay.krell@cornell.edu<BR>To: dragisha@m3w.org<BR>Date: Thu, 18 Mar 2010 10:09:38 +0000<BR>CC: m3devel@elegosoft.com<BR>Subject: Re: [M3devel] cvsup<BR><BR>
<STYLE>
.ExternalClass .ecxhmmessage P
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Verdana;}
</STYLE>
Dragisha, Really? The server? With the -C flag and/or -f?<BR> <BR>Evidence is very very very very very good as to what is going on here.<BR> It is all related to "fork1" vs. "forkall".<BR> fork1 is the overwhelming usual behavior (Solaris has an option),<BR> and basically *any* library that uses pthreads is obligated to use<BR> pthread_atfork, be it a C library or the Modula-3 library, etc..<BR> <BR> The application cannot do things, unless<BR> the library exposes its internal locks. The Posix spec for<BR> pthread_atfork explains the problem.<BR> <BR> Consider C code that links in Modula-3 code.<BR> C code is fairly free to use the fork + do work model. It isn't very common,<BR> but it does exist, and people have gone to extra length to keep it viable.<BR> bash and ccache I believe both use this model.<BR> <BR> <BR>Granted, if you had your own kernel thread implementation, it might<BR>have addressed this.<BR> <BR> <BR>I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions.<BR>The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork.<BR>I want to still try out the "bigger" change, but with the simpler lock/unlock.<BR>And test on Solaris and Darwin.<BR> <BR> <BR>I think for SuspendOthers to not work is not surprising.<BR>The threads can still hold a few locks even if suspended, I'm pretty sure.<BR>The point is to fork in a "controlled" fashion -- all locks held, and in<BR>the right order, so that both parent and child can release them.<BR> <BR> <BR>Taking "all" the locks in Prepare is more reliable.<BR> <BR> <BR>I do wonder if really "all" locks need to be taken.<BR>I only take the static ones declared in PThreadThread.c.<BR> <BR> - Jay<BR><BR> <BR>> Subject: Re: [M3devel] cvsup<BR>> From: dragisha@m3w.org<BR>> To: jay.krell@cornell.edu<BR>> CC: hosking@cs.purdue.edu; m3devel@elegosoft.com<BR>> Date: Wed, 17 Mar 2010 13:29:43 +0100<BR>> <BR>> Yes, I can. I am building it regularly, from version to version, at<BR>> least few times a year. And it also worked with my ancient kernel thread<BR>> implementation. Was one of stress tests.<BR>> <BR>> On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote:<BR>> > <BR>> > (Can anyone claim to have seen cvsup server work with kernel threads?)<BR>> -- <BR>> Dragiša Durić <dragisha@m3w.org><BR>> <BR> </body>
</html>