[M3devel] cvsup

[Tony Hosking]

On 18 Mar 2010, at 11:45, Jay K wrote:

> To repeat myself:
>  
>  >  and basically *any* library that uses pthreads is obligated to use
>  >  pthread_atfork, be it a C library or the Modula-3 library, etc..
> 
>  
> This is the crux of the matter.
> We are being "bad pthread citizens" by not using pthread_atfork.
> Granted, we are probably in large company.
>  
> 
> There is a smaller fix and a larger fix, but it seems very clear we
> should take one of them.
>  
>  
> The small fix is:
>  common/Thread.i3: add PThreadAtFork

Name should probably be something more like Thread.AtFork.

>   It does nothing on posix/nt.
>   It calls pthread_atfork on pthread.
>  
>  
> In RTCollector.m3 give a child handler that stores FALSE in the
> three booleans that indicate the three threads have started.
> No prepare or parent handler I believe.
>  Though I should check for global locks there.
>   Probably the locks in ThreadPThread suffice?

I wonder if we should only fork when the collector is turned off?

>  In ThreadPThread.m3 provide three handlers:
>   prepare: lock "everything", at least the 5 static locks.
>     I might try walking all the threads and locking them too.
>   parent: unlock everything
>   child: unlock everything and reinitialize the globals
>  
>  
> Note that the atfork handlers run in many more programs than cvsup.
> They would be used also with fork + exec.

fork + exec is probably relatively safe already.

> Perhaps a way to disable that -- no way in Posix, we could just
> provide a boolean to ourselves.

Not sure I follow.

>  The larger fix would be to provide a common/Thread.i3 function
> to call fork() *and* recreate all Modula-3 threads in the child process.
> That is what I sent out.

That's tough to do portably!

>  In the second case, you also then need to allow that the caller
> may or may not use that function, so you still need atfork handlers,
> but they have to be slightly different, as the diffs I sent show.
>  
>  
> I believe strongly this is the way to go.
> It is how one is a "good pthread citizen".
> Now, granted, I don't think most libraries are.
> Witness the caveat about fork() in the Darwin manpage and
> I think even the Posix spec.
> But pthread_atfork is meant to solve this.
>  
>  
> I don't suggest a full audit and add atfork handlers everywhere.
> But m3core we can at least do, and that should satisfy cvsup.
> Should check libm3 too.

Really we only need to be concerned with internal threads to the run-time, right?  It's up to apps to restart threads in the children as necessary.

>  Plus, as long as each module takes reponsibility for itself,
> it shouldn't be so hard.
> That is, I don't think the case is all that strong for providing the "forkall"
> function that is in the diffs I sent.

Yes, I think that is overkill.

> I'll do more testing and send out diffs "later".
> It could be as long as a week, I have a bunch of things to do.
> Or maybe just a day. :)
>  
>  
> I tested a form of this fix + cvsup + user threads and it appears
> that cvsup can do one small thing and thereby work either way,
> without knowledge as to the way.
>  
>  
> That is, it can shutdown the dispatcher "directly", instead of queuing to it.
> I'm not even sure the dispatcher thread is really needed.
>  
>  
> As I said, I don't believe the application can handle this all itself.
> Any library that uses pthreads is obligated to play into the general
> mechanism. Generally the internals are not exposed for the application
> to do it.
>  
> Treating Modula-3 as a closed system in which nobody uses
> anything outside of or "below" m3core/libm3 I don't think is practical.
>  
>  
> "applications", arguably defines as "processes", are often composed
> of fairly disparate bodies of code that don't necessarily expose much
> to each other. For example, they might each create their own worker
> threads and not expose them.
> This is what pthread_atfork is for.
> Actually Tony, you gave me the lead here. :)

;-)

>  
>  
> There is a problem that m3posixthreads (user) and pthreads semantics
> diverage, and..I haven't been able to think of an way to fix that.
>  
>  
> Well, it is actually easy to make "forkall" the default and only behavior actually,
> on user and pthreads (but not NT).
>  
>  
> But I don't see a way to make "fork1" the behavior for user threads.
> You can associate a pid with each thread, and notice when the pid changes,
> but I don't know which one thread you would keep, and you wouldn't
> necessarily be able to register pthread_atfork handlers, unless maybe
> user threads were used only on platforms that actually have pthreads..
>  
>  
> And still there's no good way to it on NT I expect.
>  
>  
>  - Jay
>  
> From: jay.krell at cornell.edu
> To: dragisha at m3w.org
> Date: Thu, 18 Mar 2010 10:09:38 +0000
> CC: m3devel at elegosoft.com
> Subject: Re: [M3devel] cvsup
> 
> Dragisha, Really? The server? With the -C flag and/or -f?
>  
> Evidence is very very very very very good as to what is going on here.
>   It is all related to "fork1" vs. "forkall".
>   fork1 is the overwhelming usual behavior (Solaris has an option),
>   and basically *any* library that uses pthreads is obligated to use
>    pthread_atfork, be it a C library or the Modula-3 library, etc..
>  
>    The application cannot do things, unless
>    the library exposes its internal locks. The Posix spec for
>    pthread_atfork explains the problem.
>  
>   Consider C code that links in Modula-3 code.
>   C code is fairly free to use the fork + do work model. It isn't very common,
>    but it does exist, and people have gone to extra length to keep it viable.
>    bash and ccache I believe both use this model.
>  
>  
> Granted, if you had your own kernel thread implementation, it might
> have addressed this.
>  
>  
> I have a version now that has served over 1000 requests, similar to what I sent, but a bit reduced. The main change was I just called pthread_mutex_lock/unlock over the locks in ThreadPThread.m3, instead of using SuspendOthers. Haven't tested on Solaris/Darwin yet, just Linux. This version also doesn't expose the ForkProcessAndAllThreads functions.
> The client has to restart its threads, or in the cvsupd case, don't depend on the dispatcher thread to survive fork.
> I want to still try out the "bigger" change, but with the simpler lock/unlock.
> And test on Solaris and Darwin.
>  
>  
> I think for SuspendOthers to not work is not surprising.
> The threads can still hold a few locks even if suspended, I'm pretty sure.
> The point is to fork in a "controlled" fashion -- all locks held, and in
> the right order, so that both parent and child can release them.
>  
>  
> Taking "all" the locks in Prepare is more reliable.
>  
>  
> I do wonder if really "all" locks need to be taken.
> I only take the static ones declared in PThreadThread.c.
>  
>  - Jay
> 
>  
> > Subject: Re: [M3devel] cvsup
> > From: dragisha at m3w.org
> > To: jay.krell at cornell.edu
> > CC: hosking at cs.purdue.edu; m3devel at elegosoft.com
> > Date: Wed, 17 Mar 2010 13:29:43 +0100
> > 
> > Yes, I can. I am building it regularly, from version to version, at
> > least few times a year. And it also worked with my ancient kernel thread
> > implementation. Was one of stress tests.
> > 
> > On Tue, 2010-03-16 at 16:10 +0000, Jay K wrote:
> > > 
> > > (Can anyone claim to have seen cvsup server work with kernel threads?)
> > -- 
> > Dragiša Durić <dragisha at m3w.org>
> > 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20100318/91de1c54/attachment-0002.html>