<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
--></style>
</head>
<body class='hmmessage'>
> Name should probably be something more like Thread.AtFork. <BR>
<BR>
<BR>I disagree.<BR>
It should be PThreadAtFork, because it only does anything when<BR>using pthreads. If you are on a non-pthread system (user threads or Win32),<BR>I expect, I think, the function won't have any affect.<BR>Granted, we could be using user threads on a system that has pthreads,<BR>so there is ambiguity, there is ambiguity either way.<BR>
<BR>
<BR>Does it mean, hey, thin wrapper over pthread_atfork, and I'll call it<BR>even if using user threads? Or does it mean, only call it if<BR>using pthreads?<BR>
<BR>
<BR>
> I wonder if we should only fork when the collector is turned off?<BR>
<BR>
<BR>I don't quite follow.<BR>You mean, anyone calling fork() must GC.Disable()?<BR>Or LockHeap()<BR>Or, take all the thread locks?<BR>That last point is part of what I'm saying, since the "prepare" callback<BR>would do that.<BR>
<BR> <BR>
pthread_atfork lets you establish preconditions for fork() "distributed"<BR>out around the code, including with access to internals.<BR>
Without having to "coordinate" with the caller of fork().<BR>
<BR>
<BR> > fork + exec is probably relatively safe already.<BR>
<BR>
<BR>Right, but this change "unnecessarily" alters it.<BR>The "prepare" callback would still run.<BR>You can't discern fork + noexec vs. fork + exec<BR>They start the same.<BR>
<BR>
<BR>
I think we are converging on agreement.<BR>Let me do some more tuning (stylistic, diff reduction, not perf) and testing.<BR>Diffs later, maybe tonight, maybe in a week, depending on life.<BR>
<BR>
<BR>
> Perhaps a way to disable that -- no way in Posix, we could just<BR> > provide a boolean to ourselves.<BR>
> Not sure I follow.<BR>
<BR>
<BR>I meant, something like, in m3core/libm3 where we have a fork+exec<BR>sequence, set a boolean somewhere to tell all of our "prepare" handlers<BR>not to waste their time doing anything. fork + exec need not<BR>run the prepare handlers.<BR>And I worry that all their lock taking might deadlock..but I suspect<BR>that would indicate a bug. ?<BR>
<BR>
<BR> > recreate the threads<BR> > That's tough to do portably!<BR>
<BR> <BR>
Easy with user threads -- automatically happens.<BR>Easy with pthreads -- systems with fork().<BR>Hard/impossible on Win32.<BR>
<BR>
<BR> > Really we only need to be concerned with internal threads<BR> > to the run-time, right? It's up to apps to restart threads<BR> > in the children as necessary.<BR>
<BR>
<BR>I can agree with that.<BR>It is kind of just a lazy app that says "hey, please recreate<BR>all my threads, I didn't keep track of them, so I don't<BR>know what to restart, I know you kept track of all of them,<BR>so please do it for me".<BR>
<BR>
<BR>
You should be able to "imagine" what my diff looks like.<BR>
And maybe ok it without seeing it?<BR>
<BR>
<BR>
Anyway, I definitely want to see Darwin not hang first.<BR>
<BR>
<BR>
- Jay<BR><BR><BR><BR><BR><BR><BR> <BR>
<HR id=stopSpelling>
Subject: Re: [M3devel] cvsup<BR>From: hosking@cs.purdue.edu<BR>Date: Thu, 18 Mar 2010 13:16:03 -0400<BR>CC: dragisha@m3w.org; m3devel@elegosoft.com<BR>To: jay.krell@cornell.edu<BR><BR><BASE>
<DIV>On 18 Mar 2010, at 11:45, Jay K wrote:</DIV>
<DIV><BR class=ecxApple-interchange-newline>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV style="FONT-FAMILY: Verdana; FONT-SIZE: 10pt" class=ecxhmmessage>To repeat myself:<BR> <BR> > and basically *any* library that uses pthreads is obligated to use<BR> > pthread_atfork, be it a C library or the Modula-3 library, etc..<BR><BR> <BR>This is the crux of the matter.<BR>We are being "bad pthread citizens" by not using pthread_atfork.<BR>Granted, we are probably in large company.<BR> <BR><BR>There is a smaller fix and a larger fix, but it seems very clear we<BR>should take one of them.<BR> <BR> <BR>The small fix is:<BR> common/Thread.i3: add PThreadAtFork<BR></DIV></SPAN></BLOCKQUOTE>
<DIV><BR></DIV>
<DIV>Name should probably be something more like Thread.AtFork.</DIV><BR>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV style="FONT-FAMILY: Verdana; FONT-SIZE: 10pt" class=ecxhmmessage> It does nothing on posix/nt.<BR> It calls pthread_atfork on pthread.<BR> <BR> <BR>In RTCollector.m3 give a child handler that stores FALSE in the<BR>three booleans that indicate the three threads have started.<BR>No prepare or parent handler I believe.<BR> Though I should check for global locks there.<BR> Probably the locks in ThreadPThread suffice?<BR></DIV></SPAN></BLOCKQUOTE>
<DIV><BR></DIV>
<DIV>I wonder if we should only fork when the collector is turned off?</DIV><BR>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV style="FONT-FAMILY: Verdana; FONT-SIZE: 10pt" class=ecxhmmessage> In ThreadPThread.m3 provide three handlers:<BR> prepare: lock "everything", at least the 5 static locks.<BR></DIV></SPAN></BLOCKQUOTE>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV style="FONT-FAMILY: Verdana; FONT-SIZE: 10pt" class=ecxhmmessage> I might try walking all the threads and locking them too.<BR> parent: unlock everything<BR> child: unlock everything and reinitialize the globals<BR> <BR> <BR>Note that the atfork handlers run in many more programs than cvsup.<BR>They would be used also with fork + exec.<BR></DIV></SPAN></BLOCKQUOTE>
<DIV><BR></DIV>
<DIV>fork + exec is probably relatively safe already.</DIV><BR>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV style="FONT-FAMILY: Verdana; FONT-SIZE: 10pt" class=ecxhmmessage>Perhaps a way to disable that -- no way in Posix, we could just<BR>provide a boolean to ourselves.<BR></DIV></SPAN></BLOCKQUOTE>
<DIV><BR></DIV>
<DIV>Not sure I follow.</DIV><BR>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV style="FONT-FAMILY: Verdana; FONT-SIZE: 10pt" class=ecxhmmessage> The larger fix would be to provide a common/Thread.i3 function<BR>to call fork() *and* recreate all Modula-3 threads in the child process.<BR>That is what I sent out.<BR></DIV></SPAN></BLOCKQUOTE>
<DIV><BR></DIV>
<DIV>That's tough to do portably!</DIV><BR>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV style="FONT-FAMILY: Verdana; FONT-SIZE: 10pt" class=ecxhmmessage> In the second case, you also then need to allow that the caller<BR>may or may not use that function, so you still need atfork handlers,<BR>but they have to be slightly different, as the diffs I sent show.<BR> <BR> <BR>I believe strongly this is the way to go.<BR>It is how one is a "good pthread citizen".<BR>Now, granted, I don't think most libraries are.<BR>Witness the caveat about fork() in the Darwin manpage and<BR>I think even the Posix spec.<BR>But pthread_atfork is meant to solve this.<BR> <BR> <BR>I don't suggest a full audit and add atfork handlers everywhere.<BR>But m3core we can at least do, and that should satisfy cvsup.<BR>Should check libm3 too.<BR></DIV></SPAN></BLOCKQUOTE>
<DIV><BR></DIV>
<DIV>Really we only need to be concerned with internal threads to the run-time, right? It's up to apps to restart threads in the children as necessary.</DIV><BR>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV style="FONT-FAMILY: Verdana; FONT-SIZE: 10pt" class=ecxhmmessage> Plus, as long as each module takes reponsibility for itself,<BR>it shouldn't be so hard.<BR>That is, I don't think the case is all that strong for providing the "forkall"<BR>function that is in the diffs I sent.<BR></DIV></SPAN></BLOCKQUOTE>
<DIV><BR></DIV>
<DIV>Yes, I think that is overkill.</DIV><BR>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV style="FONT-FAMILY: Verdana; FONT-SIZE: 10pt" class=ecxhmmessage>I'll do more testing and send out diffs "later".<BR>It could be as long as a week, I have a bunch of things to do.<BR>Or maybe just a day. :)<BR> <BR> <BR>I tested a form of this fix + cvsup + user threads and it appears<BR>that cvsup can do one small thing and thereby work either way,<BR>without knowledge as to the way.<BR> <BR> <BR>That is, it can shutdown the dispatcher "directly", instead of queuing to it.<BR>I'm not even sure the dispatcher thread is really needed.<BR> <BR> <BR>As I said, I don't believe the application can handle this all itself.<BR>Any library that uses pthreads is obligated to play into the general<BR>mechanism. Generally the internals are not exposed for the application<BR>to do it.<BR> <BR>Treating Modula-3 as a closed system in which nobody uses<BR>anything outside of or "below" m3core/libm3 I don't think is practical.<BR> <BR> <BR>"applications", arguably defines as "processes", are often composed<BR>of fairly disparate bodies of code that don't necessarily expose much<BR>to each other. For example, they might each create their own worker<BR>threads and not expose them.<BR>This is what pthread_atfork is for.<BR>Actually Tony, you gave me the lead here. :)<BR></DIV></SPAN></BLOCKQUOTE>
<DIV><BR></DIV>
<DIV>;-)</DIV><BR>
<BLOCKQUOTE><SPAN style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: medium Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; WORD-SPACING: 0px" class=ecxApple-style-span>
<DIV style="FONT-FAMILY: Verdana; FONT-SIZE: 10pt" class=ecxhmmessage> <BR> <BR>There is a problem that m3posixthreads (user) and pthreads semantics<BR>diverage, and..I haven't been able to think of an way to fix that.<BR> <BR> <BR>Well, it is actually easy to make "forkall" the default and only behavior actually,<BR>on user and pthreads (but not NT).<BR> <BR> <BR>But I don't see a way to make "fork1" the behavior for user threads.<BR>You can associate a pid with each thread, and notice when the pid changes,<BR>but I don't know which one thread you would keep, and you wouldn't<BR>necessarily be able to register pthread_atfork handlers, unless maybe<BR>user threads were used only on platforms that actually have pthreads..<BR> <BR> <BR>And still there's no good way to it on NT I expect.<BR> <BR> <BR> - Jay<BR> <BR>
<HR id=ecxstopSpelling>
From:<SPAN class=ecxApple-converted-space> </SPAN><A href="mailto:jay.krell@cornell.edu">jay.krell@cornell.edu</A><BR>To:<SPAN class=ecxApple-converted-space> </SPAN><A href="mailto:dragisha@m3w.org">dragisha@m3w.org</A><BR>Date: Thu, 18 Mar 2010 10:09:38 +0000<BR>CC:<SPAN class=ecxApple-converted-space> </SPAN><A href="mailto:m3devel@elegosoft.com">m3devel@elegosoft.com</A><BR>Subject: Re: [M3devel] cvsup<BR><BR>Dragisha, Really? The server? With the -C flag and/or -f?<BR> <BR>Evidence is very very very very very good as to what is going on here.<BR> It is all related to "fork1" vs. "forkall".<BR> fork1 is the overwhelming usual behavior (Solaris has an option),<BR> and basically *any* library that uses pthreads is obligated to use<BR> pthread_atfork, be it a C library or the Modula-3 library, etc..<BR> <BR> The application cannot do things, unless<BR> the library exposes its internal locks. The Posix spec for<BR> pthread_atfork explains the problem.<BR> <BR> Consider C code that links in Modula-3 code.<BR> C code is fairly free to use the fork + do work model. It isn't very common,<BR> but it does exist, and people have gone to extra length to keep it viable.<BR> bash and ccache I believe both use this model.<BR> <BR> <BR>Granted, if you had your own kernel thread implementation, it might<BR>have addressed this.<BR> <BR> <BR>I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions.<BR>The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork.<BR>I want to still try out the "bigger" change, but with the simpler lock/unlock.<BR>And test on Solaris and Darwin.<BR> <BR> <BR>I think for SuspendOthers to not work is not surprising.<BR>The threads can still hold a few locks even if suspended, I'm pretty sure.<BR>The point is to fork in a "controlled" fashion -- all locks held, and in<BR>the right order, so that both parent and child can release them.<BR> <BR> <BR>Taking "all" the locks in Prepare is more reliable.<BR> <BR> <BR>I do wonder if really "all" locks need to be taken.<BR>I only take the static ones declared in PThreadThread.c.<BR> <BR> - Jay<BR><BR> <BR>> Subject: Re: [M3devel] cvsup<BR>> From:<SPAN class=ecxApple-converted-space> </SPAN><A href="mailto:dragisha@m3w.org">dragisha@m3w.org</A><BR>> To:<SPAN class=ecxApple-converted-space> </SPAN><A href="mailto:jay.krell@cornell.edu">jay.krell@cornell.edu</A><BR>> CC:<SPAN class=ecxApple-converted-space> </SPAN><A href="mailto:hosking@cs.purdue.edu">hosking@cs.purdue.edu</A>;<SPAN class=ecxApple-converted-space> </SPAN><A href="mailto:m3devel@elegosoft.com">m3devel@elegosoft.com</A><BR>> Date: Wed, 17 Mar 2010 13:29:43 +0100<BR>><SPAN class=ecxApple-converted-space> </SPAN><BR>> Yes, I can. I am building it regularly, from version to version, at<BR>> least few times a year. And it also worked with my ancient kernel thread<BR>> implementation. Was one of stress tests.<BR>><SPAN class=ecxApple-converted-space> </SPAN><BR>> On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote:<BR>> ><SPAN class=ecxApple-converted-space> </SPAN><BR>> > (Can anyone claim to have seen cvsup server work with kernel threads?)<BR>> --<SPAN class=ecxApple-converted-space> </SPAN><BR>> Dragiša Durić <<A href="mailto:dragisha@m3w.org">dragisha@m3w.org</A>><BR>><SPAN class=ecxApple-converted-space> </SPAN><BR></DIV></SPAN></BLOCKQUOTE></DIV><BR> </body>
</html>