<html><head><base href="x-msg://208/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>On 18 Mar 2010, at 11:45, Jay K wrote:</div><div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="hmmessage" style="font-size: 10pt; font-family: Verdana; ">To repeat myself:<br> <br> > and basically *any* library that uses pthreads is obligated to use<br> > pthread_atfork, be it a C library or the Modula-3 library, etc..<br><br> <br>This is the crux of the matter.<br>We are being "bad pthread citizens" by not using pthread_atfork.<br>Granted, we are probably in large company.<br> <br><br>There is a smaller fix and a larger fix, but it seems very clear we<br>should take one of them.<br> <br> <br>The small fix is:<br> common/Thread.i3: add PThreadAtFork<br></div></span></blockquote><div><br></div><div>Name should probably be something more like Thread.AtFork.</div><br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="hmmessage" style="font-size: 10pt; font-family: Verdana; "> It does nothing on posix/nt.<br> It calls pthread_atfork on pthread.<br> <br> <br>In RTCollector.m3 give a child handler that stores FALSE in the<br>three booleans that indicate the three threads have started.<br>No prepare or parent handler I believe.<br> Though I should check for global locks there.<br> Probably the locks in ThreadPThread suffice?<br></div></span></blockquote><div><br></div><div>I wonder if we should only fork when the collector is turned off?</div><br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="hmmessage" style="font-size: 10pt; font-family: Verdana; "> In ThreadPThread.m3 provide three handlers:<br> prepare: lock "everything", at least the 5 static locks.<br></div></span></blockquote><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="hmmessage" style="font-size: 10pt; font-family: Verdana; "> I might try walking all the threads and locking them too.<br> parent: unlock everything<br> child: unlock everything and reinitialize the globals<br> <br> <br>Note that the atfork handlers run in many more programs than cvsup.<br>They would be used also with fork + exec.<br></div></span></blockquote><div><br></div><div>fork + exec is probably relatively safe already.</div><br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="hmmessage" style="font-size: 10pt; font-family: Verdana; ">Perhaps a way to disable that -- no way in Posix, we could just<br>provide a boolean to ourselves.<br></div></span></blockquote><div><br></div><div>Not sure I follow.</div><br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="hmmessage" style="font-size: 10pt; font-family: Verdana; "> The larger fix would be to provide a common/Thread.i3 function<br>to call fork() *and* recreate all Modula-3 threads in the child process.<br>That is what I sent out.<br></div></span></blockquote><div><br></div><div>That's tough to do portably!</div><br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="hmmessage" style="font-size: 10pt; font-family: Verdana; "> In the second case, you also then need to allow that the caller<br>may or may not use that function, so you still need atfork handlers,<br>but they have to be slightly different, as the diffs I sent show.<br> <br> <br>I believe strongly this is the way to go.<br>It is how one is a "good pthread citizen".<br>Now, granted, I don't think most libraries are.<br>Witness the caveat about fork() in the Darwin manpage and<br>I think even the Posix spec.<br>But pthread_atfork is meant to solve this.<br> <br> <br>I don't suggest a full audit and add atfork handlers everywhere.<br>But m3core we can at least do, and that should satisfy cvsup.<br>Should check libm3 too.<br></div></span></blockquote><div><br></div><div>Really we only need to be concerned with internal threads to the run-time, right? It's up to apps to restart threads in the children as necessary.</div><br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="hmmessage" style="font-size: 10pt; font-family: Verdana; "> Plus, as long as each module takes reponsibility for itself,<br>it shouldn't be so hard.<br>That is, I don't think the case is all that strong for providing the "forkall"<br>function that is in the diffs I sent.<br></div></span></blockquote><div><br></div><div>Yes, I think that is overkill.</div><br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="hmmessage" style="font-size: 10pt; font-family: Verdana; ">I'll do more testing and send out diffs "later".<br>It could be as long as a week, I have a bunch of things to do.<br>Or maybe just a day. :)<br> <br> <br>I tested a form of this fix + cvsup + user threads and it appears<br>that cvsup can do one small thing and thereby work either way,<br>without knowledge as to the way.<br> <br> <br>That is, it can shutdown the dispatcher "directly", instead of queuing to it.<br>I'm not even sure the dispatcher thread is really needed.<br> <br> <br>As I said, I don't believe the application can handle this all itself.<br>Any library that uses pthreads is obligated to play into the general<br>mechanism. Generally the internals are not exposed for the application<br>to do it.<br> <br>Treating Modula-3 as a closed system in which nobody uses<br>anything outside of or "below" m3core/libm3 I don't think is practical.<br> <br> <br>"applications", arguably defines as "processes", are often composed<br>of fairly disparate bodies of code that don't necessarily expose much<br>to each other. For example, they might each create their own worker<br>threads and not expose them.<br>This is what pthread_atfork is for.<br>Actually Tony, you gave me the lead here. :)<br></div></span></blockquote><div><br></div><div>;-)</div><br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="hmmessage" style="font-size: 10pt; font-family: Verdana; "> <br> <br>There is a problem that m3posixthreads (user) and pthreads semantics<br>diverage, and..I haven't been able to think of an way to fix that.<br> <br> <br>Well, it is actually easy to make "forkall" the default and only behavior actually,<br>on user and pthreads (but not NT).<br> <br> <br>But I don't see a way to make "fork1" the behavior for user threads.<br>You can associate a pid with each thread, and notice when the pid changes,<br>but I don't know which one thread you would keep, and you wouldn't<br>necessarily be able to register pthread_atfork handlers, unless maybe<br>user threads were used only on platforms that actually have pthreads..<br> <br> <br>And still there's no good way to it on NT I expect.<br> <br> <br> - Jay<br> <br><hr id="stopSpelling">From:<span class="Apple-converted-space"> </span><a href="mailto:jay.krell@cornell.edu">jay.krell@cornell.edu</a><br>To:<span class="Apple-converted-space"> </span><a href="mailto:dragisha@m3w.org">dragisha@m3w.org</a><br>Date: Thu, 18 Mar 2010 10:09:38 +0000<br>CC:<span class="Apple-converted-space"> </span><a href="mailto:m3devel@elegosoft.com">m3devel@elegosoft.com</a><br>Subject: Re: [M3devel] cvsup<br><br>Dragisha, Really? The server? With the -C flag and/or -f?<br> <br>Evidence is very very very very very good as to what is going on here.<br> It is all related to "fork1" vs. "forkall".<br> fork1 is the overwhelming usual behavior (Solaris has an option),<br> and basically *any* library that uses pthreads is obligated to use<br> pthread_atfork, be it a C library or the Modula-3 library, etc..<br> <br> The application cannot do things, unless<br> the library exposes its internal locks. The Posix spec for<br> pthread_atfork explains the problem.<br> <br> Consider C code that links in Modula-3 code.<br> C code is fairly free to use the fork + do work model. It isn't very common,<br> but it does exist, and people have gone to extra length to keep it viable.<br> bash and ccache I believe both use this model.<br> <br> <br>Granted, if you had your own kernel thread implementation, it might<br>have addressed this.<br> <br> <br>I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions.<br>The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork.<br>I want to still try out the "bigger" change, but with the simpler lock/unlock.<br>And test on Solaris and Darwin.<br> <br> <br>I think for SuspendOthers to not work is not surprising.<br>The threads can still hold a few locks even if suspended, I'm pretty sure.<br>The point is to fork in a "controlled" fashion -- all locks held, and in<br>the right order, so that both parent and child can release them.<br> <br> <br>Taking "all" the locks in Prepare is more reliable.<br> <br> <br>I do wonder if really "all" locks need to be taken.<br>I only take the static ones declared in PThreadThread.c.<br> <br> - Jay<br><br> <br>> Subject: Re: [M3devel] cvsup<br>> From:<span class="Apple-converted-space"> </span><a href="mailto:dragisha@m3w.org">dragisha@m3w.org</a><br>> To:<span class="Apple-converted-space"> </span><a href="mailto:jay.krell@cornell.edu">jay.krell@cornell.edu</a><br>> CC:<span class="Apple-converted-space"> </span><a href="mailto:hosking@cs.purdue.edu">hosking@cs.purdue.edu</a>;<span class="Apple-converted-space"> </span><a href="mailto:m3devel@elegosoft.com">m3devel@elegosoft.com</a><br>> Date: Wed, 17 Mar 2010 13:29:43 +0100<br>><span class="Apple-converted-space"> </span><br>> Yes, I can. I am building it regularly, from version to version, at<br>> least few times a year. And it also worked with my ancient kernel thread<br>> implementation. Was one of stress tests.<br>><span class="Apple-converted-space"> </span><br>> On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote:<br>> ><span class="Apple-converted-space"> </span><br>> > (Can anyone claim to have seen cvsup server work with kernel threads?)<br>> --<span class="Apple-converted-space"> </span><br>> Dragiša Durić <<a href="mailto:dragisha@m3w.org">dragisha@m3w.org</a>><br>><span class="Apple-converted-space"> </span><br></div></span></blockquote></div><br></body></html>