<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style>
</head>
<body class='hmmessage'>
> > soon afterwards in the child process, thus resetting all states. In the<br>> > meantime, only a short list of async-signal-safe library routines are<br>> > promised to be available."<br><br><br>Any idea why? I don't know.<br><br><br>Is this such a burden?<br>We need a global lock order for it. Unreasonable?<br>I understand pthread_atfork is causing a problem with user threads, but<br> - It really ought not. The libc vs. libpthread design is broken. I believe many systems<br> don't have this broken design, but I guess some do. The broken design is that<br> libc and libpthread implement some functions with the same name, and you get<br> to chose which one. And some of them might be thread-safe.<br><br><br> > In Modula-3, every application has *always*<br> > had to assume that there are other threads floating around. There is<br><br><br>This is the case in C too.<br>I can't assume any library I link to doesn't create some helper threads with pthreads.<br>pthreads should be assumed always in use, and there should be no ramifications of this.<br>I shouldn't have a choice e.g. of a malloc/free/strtok that aren't thread safe vs. ones that are.<br><br><br>We really need pthreads to work and to worry less about user threads imho.<br><br><br>I believe we have seen the same problems with Win32 threads as with pthreads.<br> (Yes, I know, some people might protest that Win32 threads work fine, but pthreads<br> work very well too. It is only the new stress testing that is turning up problems as I understand,<br> and it turns up problems in pthreads and Win32 threads. Ok, there is now some new mention<br> of problems with other programs?) I suspect the problem is in the allocator/garbage collector,<br> but that user threads don't stress it enough. I suspect I suspect I suspect...whatever.. real<br> debugging is needed, (my own) suspicion and speculation isn't going to help much.<br> It would of course be good if someone anyone can find a point in the history where<br> pthreads and Win32 threads are really solid, with these new one or two test programs,<br> if such a point exists -- I'm not sure one does, or that recent changes are the problem.<br> But, again, I don't know. Time, debugging, investigation is needed.<br><br><br><br><br> - Jay<br><br><br>> From: hosking@cs.purdue.edu<br>> Date: Wed, 20 Apr 2011 10:58:06 -0400<br>> To: mika@async.caltech.edu<br>> CC: m3devel@elegosoft.com; jay.krell@cornell.edu<br>> Subject: Re: [M3devel] 5.8.6 LINUXLIBC6 breakage, kernel 2.6.23, glibc-2.6-4<br>> <br>> Agreed. But CVSup used it and we were trying to be supportive.<br>> <br>> Antony Hosking | Associate Professor | Computer Science | Purdue University<br>> 305 N. University Street | West Lafayette | IN 47907 | USA<br>> Office +1 765 494 6001 | Mobile +1 765 427 5484<br>> <br>> <br>> <br>> <br>> On Apr 20, 2011, at 1:55 AM, Mika Nystrom wrote:<br>> <br>> > <br>> > The rationale you directed us to says:<br>> > <br>> > "It is suggested that programs that use fork() call an exec function very<br>> > soon afterwards in the child process, thus resetting all states. In the<br>> > meantime, only a short list of async-signal-safe library routines are<br>> > promised to be available."<br>> > <br>> > and later<br>> > <br>> > "Application programs may not be aware that a multi-threaded library<br>> > is in use, and they feel free to call any number of library routines<br>> > between the fork() and exec calls, just as they always have. Indeed,<br>> > they may be extant single-threaded programs and cannot, therefore,<br>> > be expected to obey new restrictions imposed by the threads library."<br>> > <br>> > In C maybe that is true. In Modula-3, every application has *always*<br>> > had to assume that there are other threads floating around. There is<br>> > no such thing as a "single-threaded" Modula-3 program---or at least,<br>> > the distinction doesn't matter.<br>> > <br>> > I really think fork-and-do-more-work is just a bug. Maybe it is a special<br>> > pattern used by some special program---maybe. In any case support for it<br>> > I do not think belongs in m3core.<br>> > <br>> > Mika<br>> > <br>> > <br>> > <br>> > Jay K writes:<br>> >> --_0203aac2-5c8e-49c7-9074-6bc448ecf342_<br>> >> Content-Type: text/plain; charset="iso-8859-1"<br>> >> Content-Transfer-Encoding: quoted-printable<br>> >> <br>> >> <br>> >>> This change depends on pthread_atfork=2C for pthreads and user threads.<br>> >> <br>> >> <br>> >> Clarification: There's no change described there for pthreads=2C and indeed=<br>> >> =2C getting<br>> >> this behavior with pthreads doesn't depend on using pthread_atfork. That is=<br>> >> just<br>> >> how pthreads are -- except optionally on Solaris (fork1() vs. forkall() vs.=<br>> >> fork()).<br>> >> The Solaris default used to be fork =3D> forkall().<br>> >> The current Solaris default is fork =3D> fork1().<br>> >> There really do exist on Solaris to functions you can call=2C fork1() and f=<br>> >> orkall().<br>> >> All other systems and the Posix specification have the "fork1" semantic for=<br>> >> fork.<br>> >> Therefore "fork1" is what we should provide unconditionally.<br>> >> If you really want "forkall"=2C then you write:<br>> >> #ifdef __sun<br>> >> forkall()<br>> >> #else<br>> >> #error<br>> >> #endif<br>> >> <br>> >> <br>> >> <br>> >> and/or you go and hack up m3core/src/thread/POSIX=2C branch off m3core/src/=<br>> >> thread/POSIX_FORKALL or such.<br>> >> I don't expect anyone to ever do that.<br>> >> <br>> >> <br>> >> <br>> >> The need for pthread_atfork is sort of different for user threads vs. pthre=<br>> >> ads=2C but it is needed either way.<br>> >> For pthreads=2C there is this problem -- at the time a thread calls fork()=<br>> >> =2C what are other threads doing?<br>> >> Answer: Anything.<br>> >> Follow up question: What locks might they hold?<br>> >> Answer: Any.<br>> >> Follow up: So=2C if arbitrary locks are held=2C and you fork=2C and don't e=<br>> >> xec=2C and the child process<br>> >> attempts to use some of the locks that were held in the parent=2C and only =<br>> >> the forking thread survived=2C<br>> >> then the other threads will never leave their locks=2C and later attempts t=<br>> >> o acquire them in the children<br>> >> will deadlock. Therefore=2C what pthread_atfork provides for=2C what you ar=<br>> >> e supposed to use it for=2C<br>> >> is roughly speaking=2C right before fork()=2C in the parent=2C acquire all =<br>> >> locks=2C and then fork()=2C and then in the<br>> >> parent and children=2C release all locks. Therefore=2C you have to come up =<br>> >> with a locking order.<br>> >> <br>> >> <br>> >> <br>> >> Things are a little different for user threads.<br>> >> In user threads=2C the existance of threads is established by virtue of a g=<br>> >> lobal array or list of data describing the threads.<br>> >> And a timer to occasionally preempt. When the timer interrupts=2C the sched=<br>> >> uler picks another thread from the global<br>> >> array/list. Previously=2C the data went unchanged when fork() was called. T=<br>> >> herefore all threads survived.<br>> >> What we do now is that in the child "handler" for pthread_atfork=2C is rein=<br>> >> itialize the various data=2C so as to forget about<br>> >> all but the current thread. Something like that.<br>> >> <br>> >> <br>> >> <br>> >> I suspect if we had RTProcess.ForkBeforeExec and RTProcess.ForkBeforeMoreWo=<br>> >> rk=2C then ForkBeforeExec<br>> >> might get away with just fork() and ignore all this.<br>> >> So far we have not drawn this line -- there is just the one fork and it is =<br>> >> used prior to exec or do-more-work<br>> >> and does the same thing w/o knowing what is to follow.<br>> >> <br>> >> <br>> >> The Posix documentation I think describes this all fairly well.<br>> >> Bing for "posix opengroup pthread_atfork".<br>> >> Really. I think it is saying much of the same thing I am saying=2C but in l=<br>> >> anguage that has received<br>> >> much more thought. Be sure to read the "rationale" part.<br>> >> <br>> >> <br>> >> - Jay<br>> >> <br>> >> <br>> >> From: jay.krell@cornell.edu<br>> >> To: mika@async.caltech.edu<br>> >> CC: m3devel@elegosoft.com<br>> >> Subject: RE: [M3devel] 5.8.6 LINUXLIBC6 breakage=2C kernel 2.6.23=2C glibc-=<br>> >> 2.6-4<br>> >> Date: Tue=2C 19 Apr 2011 15:06:38 +0000<br>> >> <br>> >> <br>> >> <br>> >> <br>> >> <br>> >> <br>> >> <br>> >> <br>> >> Previously: user threads: all threads survived fork()<br>> >> Previously: pthreads: only the thread calling fork() survived fork()<br>> >> Now: user threads and pthreads: only the thread calling fork() survives for=<br>> >> k()<br>> >> This must be how pthreads behave=2C and this makes user threads and pthread=<br>> >> s consistent.<br>> >> <br>> >> This change depends on pthread_atfork=2C for pthreads and user threads.<br>> >> It only really matters to the rare "fork + do more work" program=2C such as=<br>> >> cvsupd.<br>> >> Most programs either never fork=2C or exec almost immediately after fork.<br>> >> <br>> >> <br>> >> pthread_atfork offers a good model.<br>> >> A sort of "distributed" model.<br>> >> You don't have to go and change all the calls to fork().<br>> >> Each module with a need to do something before/after fork=2C calls the cent=<br>> >> ral pthread_atfork=2C<br>> >> and fork and pthread_atfork cooperate to do what is needed.<br>> >> <br>> >> <br>> >> No function pointer is needed.<br>> >> Instead move the code to m3core/thread.<br>> >> <br>> >> <br>> >> If you really must not use -pthread=2C then you must implement pthread_atfo=<br>> >> rk functionality<br>> >> yourself and have all fork() calls go through your own fork() wrapper that =<br>> >> cooperates<br>> >> with your pthread_atfork replacement.<br>> >> There is no free lunch -- there is a downside to this approach=2C as plain =<br>> >> fork() calls<br>> >> are ok and correct if pthread_atfork is used=2C but now become incorrect.<br>> >> Pick your poison:<br>> >> user thread/pthread inconsistency<br>> >> cvsupd incompatibility with pthreads=20<br>> >> user threads using pthread_atfork/-pthread=20<br>> >> fork() calls having to go through a wrapper (ok -- you could miss this an=<br>> >> d=20<br>> >> not likely notice -- only fork() calls in fork+do-more-work programs nee=<br>> >> d the wrapper).=20<br>> >> <br>> >> <br>> >> (Most of this has been explained multiple times=2C but people only pay atte=<br>> >> ntion<br>> >> when they think it affects them. I'm guilty of the same thing.)<br>> >> <br>> >> <br>> >> - Jay<br>> >> <br>> >> <br>> >>> To: jay.krell@cornell.edu<br>> >>> Date: Mon=2C 18 Apr 2011 19:30:58 -0700<br>> >>> From: mika@async.caltech.edu<br>> >>> CC: m3devel@elegosoft.com<br>> >>> Subject: Re: [M3devel] 5.8.6 LINUXLIBC6 breakage=2C kernel 2.6.23=2C glib=<br>> >> c-2.6-4<br>> >>> =20<br>> >>> =20<br>> >>> If you or Tony could describe roughly what you think needs to be done<br>> >>> I'd be happy to look into it.<br>> >>> =20<br>> >>> The basic problem is that a decision that's made/described in<br>> >>> m3core/thread/m3makefile needs to someone find its way to controlling<br>> >>> what C code gets compiled elsewhere in the system. Or maybe there should<br>> >>> be an indirection into the thread library to pick up pthread_atfork (or n=<br>> >> ot).<br>> >>> =20<br>> >>> But then again you still haven't explained why you made the user threads =<br>> >> use<br>> >>> pthread_atfork. I just remove it from my installations=2C but then again=<br>> >> I'm<br>> >>> not trying to run CVSup so I don't know if it breaks that program to do s=<br>> >> o.<br>> >>> =20<br>> >>> Mika<br>> >>> =20<br>> >>> Jay K writes:<br>> >>>> --_3dd397d4-ac1d-4148-a9ff-059d27dd794a_<br>> >>>> Content-Type: text/plain=3B charset=3D"iso-8859-1"<br>> >>>> Content-Transfer-Encoding: quoted-printable<br>> >>>> <br>> >>>> <br>> >>>>> The following code from RTProcessC.c ensures that it is neeeded on eve=<br>> >> ry =3D<br>> >>>> Unix except<br>> >>>>>> /* NOTE: Even userthreads now depends<br>> >>>>>> * on availability of pthreads.<br>> >>>>>> * This can be fixed if need be.<br>> >>>>>> */<br>> >>>> <br>> >>>> =3D20<br>> >>>> Ok=3D2C "we" should probably go ahead and fix that.<br>> >>>> I'll try to=3D2C but no promises=3D2C sorry.<br>> >>>> =3D20<br>> >>>> - Jay<br>> >>>> =3D20<br>> >>>>> From: hosking@cs.purdue.edu<br>> >>>>> Date: Mon=3D2C 18 Apr 2011 18:11:26 -0400<br>> >>>>> To: mika@async.caltech.edu<br>> >>>>> CC: m3devel@elegosoft.com<br>> >>>>> Subject: Re: [M3devel] 5.8.6 LINUXLIBC6 breakage=3D2C kernel 2.6.23=3D=<br>> >> 2C glib=3D<br>> >>>> c-2.6-4<br>> >>>>> =3D20<br>> >>>>> Probably unnecessary=3D2C given that I think there is another entry po=<br>> >> int t=3D<br>> >>>> o forking a process (I forget where) in the thread-specific portion of m=<br>> >> 3co=3D<br>> >>>> re. In which case the necessary work might be done there?<br>> >>>>> =3D20<br>> >>>>> On Apr 18=3D2C 2011=3D2C at 2:45 PM=3D2C Mika Nystrom wrote:<br>> >>>>> =3D20<br>> >>>>>> Tony Hosking writes:<br>> >>>>>> ...<br>> >>>>>>> pthread_atfork should not be needed under user threads.<br>> >>>>>> ...<br>> >>>>>> =3D20<br>> >>>>>> The following code from RTProcessC.c ensures that it is neeeded on e=<br>> >> ver=3D<br>> >>>> y Unix except<br>> >>>>>> FreeBSD before 6. The comment is from the checked-in source file.<br>> >>>>>> =3D20<br>> >>>>>> /* NOTE: Even userthreads now depends<br>> >>>>>> * on availability of pthreads.<br>> >>>>>> * This can be fixed if need be.<br>> >>>>>> */<br>> >>>>>> =3D20<br>> >>>>>> INTEGER<br>> >>>>>> __cdecl<br>> >>>>>> RTProcess__RegisterForkHandlers(<br>> >>>>>> ForkHandler prepare=3D2C<br>> >>>>>> ForkHandler parent=3D2C<br>> >>>>>> ForkHandler child)<br>> >>>>>> {<br>> >>>>>> /* FreeBSD < 6 lacks pthread_atfork. Would be good to use autoconf.<br>> >>>>>> * VMS lacks pthread_atfork? Would be good to use autoconf.<br>> >>>>>> * Win32 lacks pthread_atfork and fork. OK.<br>> >>>>>> *<br>> >>>>>> * As well=3D2C for all Posix systems=3D2C we could implement<br>> >>>>>> * atfork ourselves=3D2C as long as we provide a fork()<br>> >>>>>> * wrapper that code uses.<br>> >>>>>> */<br>> >>>>>> #if defined(_WIN32) \<br>> >>>>>> || defined(__vms) \<br>> >>>>>> || (defined(__FreeBSD__) /* && (__FreeBSD__ < 6)*/ )<br>> >>>>>> return 0=3D3B<br>> >>>>>> #else<br>> >>>>>> while (1)<br>> >>>>>> {<br>> >>>>>> int i =3D3D pthread_atfork(prepare=3D2C parent=3D2C child)=3D3B<br>> >>>>>> if (i !=3D3D EAGAIN)<br>> >>>>>> return i=3D3B<br>> >>>>>> sleep(0)=3D3B<br>> >>>>>> }<br>> >>>>>> #endif<br>> >>>>>> }<br>> >>>>> =3D20<br>> >>>> =3D<br>> >>>> <br>> >>>> --_3dd397d4-ac1d-4148-a9ff-059d27dd794a_<br>> >>>> Content-Type: text/html=3B charset=3D"iso-8859-1"<br>> >>>> Content-Transfer-Encoding: quoted-printable<br>> >>>> <br>> >>>> <html><br>> >>>> <head><br>> >>>> <style><!--<br>> >>>> .hmmessage P<br>> >>>> {<br>> >>>> margin:0px=3D3B<br>> >>>> padding:0px<br>> >>>> }<br>> >>>> body.hmmessage<br>> >>>> {<br>> >>>> font-size: 10pt=3D3B<br>> >>>> font-family:Tahoma<br>> >>>> }<br>> >>>> --></style><br>> >>>> </head><br>> >>>> <body class=3D3D'hmmessage'><br>> >>>> >=3D3B The following code from RTProcessC.c ensures that it is neeeded=<br>> >> on e=3D<br>> >>>> very Unix except<BR>>=3D3B >=3D3B /* NOTE: Even userthreads now depe=<br>> >> nds<BR>=3D<br>> >>>> >=3D3B >=3D3B * on availability of pthreads.<BR>>=3D3B >=3D3B * =<br>> >> This can be=3D<br>> >>>> fixed if need be.<BR>>=3D3B >=3D3B */<BR><BR><br>> >>>>  =3D3B<BR><br>> >>>> Ok=3D2C "we" should probably go ahead and fix that.<BR><br>> >>>> I'll try to=3D2C but no =3D3Bpromises=3D2C sorry.<BR><br>> >>>>  =3D3B<BR><br>> >>>>  =3D3B- Jay<BR> =3D3B<BR><br>> >>>> >=3D3B From: hosking@cs.purdue.edu<BR>>=3D3B Date: Mon=3D2C 18 Apr 2=<br>> >> 011 18:11=3D<br>> >>>> :26 -0400<BR>>=3D3B To: mika@async.caltech.edu<BR>>=3D3B CC: m3devel=<br>> >> @elegos=3D<br>> >>>> oft.com<BR>>=3D3B Subject: Re: [M3devel] 5.8.6 LINUXLIBC6 breakage=3D2=<br>> >> C kerne=3D<br>> >>>> l 2.6.23=3D2C glibc-2.6-4<BR>>=3D3B <BR>>=3D3B Probably unnecessary=<br>> >> =3D2C given =3D<br>> >>>> that I think there is another entry point to forking a process (I forget=<br>> >> wh=3D<br>> >>>> ere) in the thread-specific portion of m3core. In which case the necessa=<br>> >> ry =3D<br>> >>>> work might be done there?<BR>>=3D3B <BR>>=3D3B On Apr 18=3D2C 2011=<br>> >> =3D2C at 2:45=3D<br>> >>>> PM=3D2C Mika Nystrom wrote:<BR>>=3D3B <BR>>=3D3B >=3D3B Tony Hosk=<br>> >> ing writes:=3D<br>> >>>> <BR>>=3D3B >=3D3B ...<BR>>=3D3B >=3D3B>=3D3B pthread_atfork sh=<br>> >> ould not be n=3D<br>> >>>> eeded under user threads.<BR>>=3D3B >=3D3B ...<BR>>=3D3B >=3D3B =<br>> >> <BR>>=3D3B =3D<br>> >>>> >=3D3B The following code from RTProcessC.c ensures that it is neeeded=<br>> >> on e=3D<br>> >>>> very Unix except<BR>>=3D3B >=3D3B FreeBSD before 6. The comment is f=<br>> >> rom the=3D<br>> >>>> checked-in source file.<BR>>=3D3B >=3D3B <BR>>=3D3B >=3D3B /* N=<br>> >> OTE: Even u=3D<br>> >>>> serthreads now depends<BR>>=3D3B >=3D3B * on availability of pthread=<br>> >> s.<BR>&=3D<br>> >>>> gt=3D3B >=3D3B * This can be fixed if need be.<BR>>=3D3B >=3D3B */=<br>> >> <BR>>=3D3B =3D<br>> >>>> >=3D3B <BR>>=3D3B >=3D3B INTEGER<BR>>=3D3B >=3D3B __cdecl<BR>&=<br>> >> gt=3D3B >=3D3B =3D<br>> >>>> RTProcess__RegisterForkHandlers(<BR>>=3D3B >=3D3B ForkHandler prepar=<br>> >> e=3D2C<BR=3D<br>> >>>>> >=3D3B >=3D3B ForkHandler parent=3D2C<BR>>=3D3B >=3D3B ForkHand=<br>> >> ler child)<BR=3D<br>> >>>>> >=3D3B >=3D3B {<BR>>=3D3B >=3D3B /* FreeBSD <=3D3B 6 lacks pt=<br>> >> hread_atfork.=3D<br>> >>>> Would be good to use autoconf.<BR>>=3D3B >=3D3B * VMS lacks pthread=<br>> >> _atfork=3D<br>> >>>> ? Would be good to use autoconf.<BR>>=3D3B >=3D3B * Win32 lacks pthr=<br>> >> ead_atf=3D<br>> >>>> ork and fork. OK.<BR>>=3D3B >=3D3B *<BR>>=3D3B >=3D3B * As well=<br>> >> =3D2C for all =3D<br>> >>>> Posix systems=3D2C we could implement<BR>>=3D3B >=3D3B * atfork ours=<br>> >> elves=3D2C =3D<br>> >>>> as long as we provide a fork()<BR>>=3D3B >=3D3B * wrapper that code =<br>> >> uses.<B=3D<br>> >>>> R>>=3D3B >=3D3B */<BR>>=3D3B >=3D3B #if defined(_WIN32) \<BR>>=<br>> >> =3D3B >=3D3B =3D<br>> >>>> || defined(__vms) \<BR>>=3D3B >=3D3B || (defined(__FreeBSD__) /* &am=<br>> >> p=3D3B&am=3D<br>> >>>> p=3D3B (__FreeBSD__ <=3D3B 6)*/ )<BR>>=3D3B >=3D3B return 0=3D3B<B=<br>> >> R>>=3D3B >=3D<br>> >>>> =3D3B #else<BR>>=3D3B >=3D3B while (1)<BR>>=3D3B >=3D3B {<BR>>=<br>> >> =3D3B >=3D3B in=3D<br>> >>>> t i =3D3D pthread_atfork(prepare=3D2C parent=3D2C child)=3D3B<BR>>=3D3=<br>> >> B >=3D3B if (=3D<br>> >>>> i !=3D3D EAGAIN)<BR>>=3D3B >=3D3B return i=3D3B<BR>>=3D3B >=3D3B=<br>> >> sleep(0)=3D3B<BR=3D<br>> >>>>> >=3D3B >=3D3B }<BR>>=3D3B >=3D3B #endif<BR>>=3D3B >=3D3B }<=<br>> >> BR>>=3D3B <BR> =3D<br>> >>>> </body><br>> >>>> </html>=3D<br>> >>>> <br>> >>>> --_3dd397d4-ac1d-4148-a9ff-059d27dd794a_--<br>> >> =<br>> >> <br>> >> --_0203aac2-5c8e-49c7-9074-6bc448ecf342_<br>> >> Content-Type: text/html; charset="iso-8859-1"<br>> >> Content-Transfer-Encoding: quoted-printable<br>> >> <br>> >> <html><br>> >> <head><br>> >> <style><!--<br>> >> .hmmessage P<br>> >> {<br>> >> margin:0px=3B<br>> >> padding:0px<br>> >> }<br>> >> body.hmmessage<br>> >> {<br>> >> font-size: 10pt=3B<br>> >> font-family:Tahoma<br>> >> }<br>> >> --></style><br>> >> </head><br>> >> <body class=3D'hmmessage'><br>> >>  =3B>=3B This change depends on pthread_atfork=2C for pthreads and us=<br>> >> er threads.<br><br><br>Clarification: There's no change described there for=<br>> >> pthreads=2C and indeed=2C getting<br>this behavior with pthreads doesn't d=<br>> >> epend on using pthread_atfork. That is just<br>how pthreads are -- except o=<br>> >> ptionally on Solaris (fork1() vs. forkall() vs. fork()).<br>The Solaris def=<br>> >> ault used to be fork =3D>=3B forkall().<br>The current Solaris default is=<br>> >> fork =3D>=3B fork1().<br>There really do exist on Solaris to functions y=<br>> >> ou can call=2C fork1() and forkall().<br>All other systems and the Posix sp=<br>> >> ecification have the "fork1" semantic for fork.<br>Therefore "fork1" is wha=<br>> >> t we should provide unconditionally.<br>If you really want "forkall"=2C the=<br>> >> n you write:<br>#ifdef __sun<br> =3B forkall()<br>#else<br> =3B #er=<br>> >> ror<br>#endif<br><br><br><br>and/or you go and hack up m3core/src/thread/PO=<br>> >> SIX=2C branch off m3core/src/thread/POSIX_FORKALL or such.<br>I don't expec=<br>> >> t anyone to ever do that.<br><br><br><br>The need for pthread_atfork is sor=<br>> >> t of different for user threads vs. pthreads=2C but it is needed either way=<br>> >> .<br>For pthreads=2C there is this problem -- at the time a thread calls fo=<br>> >> rk()=2C what are other threads doing?<br>Answer: Anything.<br>Follow up que=<br>> >> stion: What locks might they hold?<br>Answer: Any.<br>Follow up: So=2C if a=<br>> >> rbitrary locks are held=2C and you fork=2C and don't exec=2C and the child =<br>> >> process<br>attempts to use some of the locks that were held in the parent=<br>> >> =2C and only the forking thread survived=2C<br>then the other threads will =<br>> >> never leave their locks=2C and later attempts to acquire them in the childr=<br>> >> en<br>will deadlock. Therefore=2C what pthread_atfork provides for=2C what =<br>> >> you are supposed to use it for=2C<br>is roughly speaking=2C right before fo=<br>> >> rk()=2C in the parent=2C acquire all locks=2C and then fork()=2C and then i=<br>> >> n the<br>parent and children=2C release all locks. Therefore=2C you have to=<br>> >> come up with a locking order.<br><br><br><br>Things are a little different=<br>> >> for user threads.<br>In user threads=2C the existance of threads is establ=<br>> >> ished by virtue of a global array or list of data describing the threads.<b=<br>> >> r>And a timer to occasionally preempt. When the timer interrupts=2C the sch=<br>> >> eduler picks another thread from the global<br>array/list. Previously=2C th=<br>> >> e data went unchanged when fork() was called. Therefore all threads survive=<br>> >> d.<br>What we do now is that in the child "handler" for pthread_atfork=2C i=<br>> >> s reinitialize the various data=2C so as to forget about<br>all but the cur=<br>> >> rent thread. Something like that.<br><br><br><br>I suspect if we had RTProc=<br>> >> ess.ForkBeforeExec and RTProcess.ForkBeforeMoreWork=2C then ForkBeforeExec<=<br>> >> br>might get away with just fork() and ignore all this.<br>So far we have n=<br>> >> ot drawn this line -- there is just the one fork and it is used prior to ex=<br>> >> ec or do-more-work<br>and does the same thing w/o knowing what is to follow=<br>> >> .<br><br><br>The Posix documentation I think describes this all fairly well=<br>> >> .<br>Bing for "posix opengroup pthread_atfork".<br>Really. I think it is sa=<br>> >> ying much of the same thing I am saying=2C but in language that has receive=<br>> >> d<br>much more thought. Be sure to read the "rationale" part.<br><br><br>&n=<br>> >> bsp=3B- Jay<br><br><br><hr id=3D"stopSpelling">From: jay.krell@cornell.edu<=<br>> >> br>To: mika@async.caltech.edu<br>CC: m3devel@elegosoft.com<br>Subject: RE: =<br>> >> [M3devel] 5.8.6 LINUXLIBC6 breakage=2C kernel 2.6.23=2C glibc-2.6-4<br>Date=<br>> >> : Tue=2C 19 Apr 2011 15:06:38 +0000<br><br><br>> >> <br>> >> <meta http-equiv=3D"Content-Type" content=3D"text/html=3B charset=3Dunicode=<br>> >> "><br>> >> <meta name=3D"Generator" content=3D"Microsoft SafeHTML"><br>> >> <style><br>> >> .ExternalClass .ecxhmmessage P<br>> >> {padding:0px=3B}<br>> >> .ExternalClass body.ecxhmmessage<br>> >> {font-size:10pt=3Bfont-family:Tahoma=3B}<br>> >> <br>> >> </style><br>> >> <br>> >> <br>> >> Previously: user threads: all threads survived fork()<br>Previously: pthrea=<br>> >> ds: only the thread calling fork() survived fork()<br>Now: user threads and=<br>> >> pthreads: only the thread calling fork() survives fork()<br>This must be h=<br>> >> ow pthreads behave=2C and this makes user threads and pthreads consistent.<=<br>> >> br><br>> >> This change depends on pthread_atfork=2C for pthreads and user threads.<br>=<br>> >> It only really matters to the rare "fork + do more work" program=2C such as=<br>> >> cvsupd.<br>Most programs either never fork=2C or exec almost immediately a=<br>> >> fter fork.<br><br><br>pthread_atfork offers a good model.<br>A sort of "dis=<br>> >> tributed" model.<br>You don't have to go and change all the calls to fork()=<br>> >> .<br>Each module with a need to do something before/after fork=2C calls the=<br>> >> central pthread_atfork=2C<br>and fork and pthread_atfork cooperate to do w=<br>> >> hat is needed.<br><br><br>No function pointer is needed.<br>Instead move th=<br>> >> e code to m3core/thread.<br><br><br>If you really must not use -pthread=2C =<br>> >> then you must implement pthread_atfork functionality<br>yourself and have a=<br>> >> ll fork() calls go through your own fork() wrapper that cooperates<br>with =<br>> >> your pthread_atfork replacement.<br>There is no free lunch -- there is a do=<br>> >> wnside to this approach=2C as plain fork() calls<br>are ok and correct if p=<br>> >> thread_atfork is used=2C but now become incorrect.<br>Pick your poison:<br>=<br>> >>  =3B user thread/pthread inconsistency<br> =3B cvsupd incompatibili=<br>> >> ty with pthreads <br> =3B user threads using pthread_atfork/-pthread <b=<br>> >> r> =3B fork() calls having to go through a wrapper (ok -- you could mis=<br>> >> s this and <br> =3B =3B not likely notice -- only fork() calls in f=<br>> >> ork+do-more-work programs need the wrapper). <br><br><br>(Most of this has =<br>> >> been explained multiple times=2C but people only pay attention<br>when they=<br>> >> think it affects them. I'm guilty of the same thing.)<br><br><br> =3B-=<br>> >> Jay<br><br><br>>=3B To: jay.krell@cornell.edu<br>>=3B Date: Mon=2C 18 =<br>> >> Apr 2011 19:30:58 -0700<br>>=3B From: mika@async.caltech.edu<br>>=3B CC=<br>> >> : m3devel@elegosoft.com<br>>=3B Subject: Re: [M3devel] 5.8.6 LINUXLIBC6 b=<br>> >> reakage=2C kernel 2.6.23=2C glibc-2.6-4<br>>=3B <br>>=3B <br>>=3B If =<br>> >> you or Tony could describe roughly what you think needs to be done<br>>=<br>> >> =3B I'd be happy to look into it.<br>>=3B <br>>=3B The basic problem is=<br>> >> that a decision that's made/described in<br>>=3B m3core/thread/m3makefil=<br>> >> e needs to someone find its way to controlling<br>>=3B what C code gets c=<br>> >> ompiled elsewhere in the system. Or maybe there should<br>>=3B be an ind=<br>> >> irection into the thread library to pick up pthread_atfork (or not).<br>>=<br>> >> =3B <br>>=3B But then again you still haven't explained why you made the =<br>> >> user threads use<br>>=3B pthread_atfork. I just remove it from my instal=<br>> >> lations=2C but then again I'm<br>>=3B not trying to run CVSup so I don't =<br>> >> know if it breaks that program to do so.<br>>=3B <br>>=3B Mika<br>=<br>> >> >=3B <br>>=3B Jay K writes:<br>>=3B >=3B--_3dd397d4-ac1d-4148-a9ff-=<br>> >> 059d27dd794a_<br>>=3B >=3BContent-Type: text/plain=3B charset=3D"iso-88=<br>> >> 59-1"<br>>=3B >=3BContent-Transfer-Encoding: quoted-printable<br>>=3B=<br>> >> >=3B<br>>=3B >=3B<br>>=3B >=3B>=3B The following code from RTP=<br>> >> rocessC.c ensures that it is neeeded on every =3D<br>>=3B >=3BUnix exce=<br>> >> pt<br>>=3B >=3B>=3B >=3B /* NOTE: Even userthreads now depends<br>&=<br>> >> gt=3B >=3B>=3B >=3B * on availability of pthreads.<br>>=3B >=3B&g=<br>> >> t=3B >=3B * This can be fixed if need be.<br>>=3B >=3B>=3B >=3B *=<br>> >> /<br>>=3B >=3B<br>>=3B >=3B=3D20<br>>=3B >=3BOk=3D2C "we" shoul=<br>> >> d probably go ahead and fix that.<br>>=3B >=3BI'll try to=3D2C but no p=<br>> >> romises=3D2C sorry.<br>>=3B >=3B=3D20<br>>=3B >=3B - Jay<br>>=3B =<br>> >> >=3B=3D20<br>>=3B >=3B>=3B From: hosking@cs.purdue.edu<br>>=3B &g=<br>> >> t=3B>=3B Date: Mon=3D2C 18 Apr 2011 18:11:26 -0400<br>>=3B >=3B>=3B=<br>> >> To: mika@async.caltech.edu<br>>=3B >=3B>=3B CC: m3devel@elegosoft.co=<br>> >> m<br>>=3B >=3B>=3B Subject: Re: [M3devel] 5.8.6 LINUXLIBC6 breakage=<br>> >> =3D2C kernel 2.6.23=3D2C glib=3D<br>>=3B >=3Bc-2.6-4<br>>=3B >=3B&g=<br>> >> t=3B=3D20<br>>=3B >=3B>=3B Probably unnecessary=3D2C given that I thi=<br>> >> nk there is another entry point t=3D<br>>=3B >=3Bo forking a process (I=<br>> >> forget where) in the thread-specific portion of m3co=3D<br>>=3B >=3Bre=<br>> >> . In which case the necessary work might be done there?<br>>=3B >=3B>=<br>> >> =3B=3D20<br>>=3B >=3B>=3B On Apr 18=3D2C 2011=3D2C at 2:45 PM=3D2C Mi=<br>> >> ka Nystrom wrote:<br>>=3B >=3B>=3B=3D20<br>>=3B >=3B>=3B >=3B=<br>> >> Tony Hosking writes:<br>>=3B >=3B>=3B >=3B ...<br>>=3B >=3B>=<br>> >> =3B >=3B>=3B pthread_atfork should not be needed under user threads.<br=<br>> >>> >=3B >=3B>=3B >=3B ...<br>>=3B >=3B>=3B >=3B=3D20<br>>=<br>> >> =3B >=3B>=3B >=3B The following code from RTProcessC.c ensures that i=<br>> >> t is neeeded on ever=3D<br>>=3B >=3By Unix except<br>>=3B >=3B>=<br>> >> =3B >=3B FreeBSD before 6. The comment is from the checked-in source file=<br>> >> .<br>>=3B >=3B>=3B >=3B=3D20<br>>=3B >=3B>=3B >=3B /* NOTE:=<br>> >> Even userthreads now depends<br>>=3B >=3B>=3B >=3B * on availabili=<br>> >> ty of pthreads.<br>>=3B >=3B>=3B >=3B * This can be fixed if need b=<br>> >> e.<br>>=3B >=3B>=3B >=3B */<br>>=3B >=3B>=3B >=3B=3D20<br>&=<br>> >> gt=3B >=3B>=3B >=3B INTEGER<br>>=3B >=3B>=3B >=3B __cdecl<br>=<br>> >> >=3B >=3B>=3B >=3B RTProcess__RegisterForkHandlers(<br>>=3B >=<br>> >> =3B>=3B >=3B ForkHandler prepare=3D2C<br>>=3B >=3B>=3B >=3B For=<br>> >> kHandler parent=3D2C<br>>=3B >=3B>=3B >=3B ForkHandler child)<br>&g=<br>> >> t=3B >=3B>=3B >=3B {<br>>=3B >=3B>=3B >=3B /* FreeBSD <=3B =<br>> >> 6 lacks pthread_atfork. Would be good to use autoconf.<br>>=3B >=3B>=<br>> >> =3B >=3B * VMS lacks pthread_atfork? Would be good to use autoconf.<br>&g=<br>> >> t=3B >=3B>=3B >=3B * Win32 lacks pthread_atfork and fork. OK.<br>>=<br>> >> =3B >=3B>=3B >=3B *<br>>=3B >=3B>=3B >=3B * As well=3D2C for =<br>> >> all Posix systems=3D2C we could implement<br>>=3B >=3B>=3B >=3B * a=<br>> >> tfork ourselves=3D2C as long as we provide a fork()<br>>=3B >=3B>=3B =<br>> >> >=3B * wrapper that code uses.<br>>=3B >=3B>=3B >=3B */<br>>=3B=<br>> >> >=3B>=3B >=3B #if defined(_WIN32) \<br>>=3B >=3B>=3B >=3B ||=<br>> >> defined(__vms) \<br>>=3B >=3B>=3B >=3B || (defined(__FreeBSD__) /*=<br>> >> &=3B&=3B (__FreeBSD__ <=3B 6)*/ )<br>>=3B >=3B>=3B >=3B re=<br>> >> turn 0=3D3B<br>>=3B >=3B>=3B >=3B #else<br>>=3B >=3B>=3B >=<br>> >> =3B while (1)<br>>=3B >=3B>=3B >=3B {<br>>=3B >=3B>=3B >=3B=<br>> >> int i =3D3D pthread_atfork(prepare=3D2C parent=3D2C child)=3D3B<br>>=3B =<br>> >> >=3B>=3B >=3B if (i !=3D3D EAGAIN)<br>>=3B >=3B>=3B >=3B retu=<br>> >> rn i=3D3B<br>>=3B >=3B>=3B >=3B sleep(0)=3D3B<br>>=3B >=3B>=<br>> >> =3B >=3B }<br>>=3B >=3B>=3B >=3B #endif<br>>=3B >=3B>=3B &g=<br>> >> t=3B }<br>>=3B >=3B>=3B=3D20<br>>=3B >=3B =3D<br>>=<br>> >> =3B >=3B<br>>=3B >=3B--_3dd397d4-ac1d-4148-a9ff-059d27dd794a_<br>>=<br>> >> =3B >=3BContent-Type: text/html=3B charset=3D"iso-8859-1"<br>>=3B >=<br>> >> =3BContent-Transfer-Encoding: quoted-printable<br>>=3B >=3B<br>>=3B &=<br>> >> gt=3B<=3Bhtml>=3B<br>>=3B >=3B<=3Bhead>=3B<br>>=3B >=3B<=<br>> >> =3Bstyle>=3B<=3B!--<br>>=3B >=3B.hmmessage P<br>>=3B >=3B{<br>&=<br>> >> gt=3B >=3Bmargin:0px=3D3B<br>>=3B >=3Bpadding:0px<br>>=3B >=3B}<b=<br>> >> r>>=3B >=3Bbody.hmmessage<br>>=3B >=3B{<br>>=3B >=3Bfont-size: =<br>> >> 10pt=3D3B<br>>=3B >=3Bfont-family:Tahoma<br>>=3B >=3B}<br>>=3B &g=<br>> >> t=3B-->=3B<=3B/style>=3B<br>>=3B >=3B<=3B/head>=3B<br>>=3B =<br>> >> >=3B<=3Bbody class=3D3D'hmmessage'>=3B<br>>=3B >=3B&=3Bgt=3D3B=<br>> >> The following code from RTProcessC.c ensures that it is neeeded on e=3D<br=<br>> >>> >=3B >=3Bvery Unix except<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B /=<br>> >> * NOTE: Even userthreads now depends<=3BBR>=3B=3D<br>>=3B >=3B&=<br>> >> =3Bgt=3D3B &=3Bgt=3D3B * on availability of pthreads.<=3BBR>=3B&=<br>> >> =3Bgt=3D3B &=3Bgt=3D3B * This can be=3D<br>>=3B >=3B fixed if need b=<br>> >> e.<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B */<=3BBR>=3B<=3BBR>=<br>> >> =3B<br>>=3B >=3B&=3Bnbsp=3D3B<=3BBR>=3B<br>>=3B >=3BOk=3D2C =<br>> >> "we" should probably go ahead and fix that.<=3BBR>=3B<br>>=3B >=3BI=<br>> >> 'll try to=3D2C but no&=3Bnbsp=3D3Bpromises=3D2C sorry.<=3BBR>=3B<br=<br>> >>> >=3B >=3B&=3Bnbsp=3D3B<=3BBR>=3B<br>>=3B >=3B&=3Bnbsp=3D=<br>> >> 3B- Jay<=3BBR>=3B&=3Bnbsp=3D3B<=3BBR>=3B<br>>=3B >=3B&=3B=<br>> >> gt=3D3B From: hosking@cs.purdue.edu<=3BBR>=3B&=3Bgt=3D3B Date: Mon=<br>> >> =3D2C 18 Apr 2011 18:11=3D<br>>=3B >=3B:26 -0400<=3BBR>=3B&=3Bgt=<br>> >> =3D3B To: mika@async.caltech.edu<=3BBR>=3B&=3Bgt=3D3B CC: m3devel@el=<br>> >> egos=3D<br>>=3B >=3Boft.com<=3BBR>=3B&=3Bgt=3D3B Subject: Re: [M=<br>> >> 3devel] 5.8.6 LINUXLIBC6 breakage=3D2C kerne=3D<br>>=3B >=3Bl 2.6.23=3D=<br>> >> 2C glibc-2.6-4<=3BBR>=3B&=3Bgt=3D3B <=3BBR>=3B&=3Bgt=3D3B Pro=<br>> >> bably unnecessary=3D2C given =3D<br>>=3B >=3Bthat I think there is anot=<br>> >> her entry point to forking a process (I forget wh=3D<br>>=3B >=3Bere) i=<br>> >> n the thread-specific portion of m3core. In which case the necessary =3D<br=<br>> >>> >=3B >=3Bwork might be done there?<=3BBR>=3B&=3Bgt=3D3B <=3BB=<br>> >> R>=3B&=3Bgt=3D3B On Apr 18=3D2C 2011=3D2C at 2:45=3D<br>>=3B >=3B =<br>> >> PM=3D2C Mika Nystrom wrote:<=3BBR>=3B&=3Bgt=3D3B <=3BBR>=3B&=<br>> >> =3Bgt=3D3B &=3Bgt=3D3B Tony Hosking writes:=3D<br>>=3B >=3B<=3BBR&=<br>> >> gt=3B&=3Bgt=3D3B &=3Bgt=3D3B ...<=3BBR>=3B&=3Bgt=3D3B &=3Bg=<br>> >> t=3D3B&=3Bgt=3D3B pthread_atfork should not be n=3D<br>>=3B >=3Beede=<br>> >> d under user threads.<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B ...<=3BB=<br>> >> R>=3B&=3Bgt=3D3B &=3Bgt=3D3B <=3BBR>=3B&=3Bgt=3D3B =3D<br>&g=<br>> >> t=3B >=3B&=3Bgt=3D3B The following code from RTProcessC.c ensures that=<br>> >> it is neeeded on e=3D<br>>=3B >=3Bvery Unix except<=3BBR>=3B&=<br>> >> =3Bgt=3D3B &=3Bgt=3D3B FreeBSD before 6. The comment is from the=3D<br>&=<br>> >> gt=3B >=3B checked-in source file.<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=<br>> >> =3D3B <=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B /* NOTE: Even u=3D<br>>=<br>> >> =3B >=3Bserthreads now depends<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B=<br>> >> * on availability of pthreads.<=3BBR>=3B&=3B=3D<br>>=3B >=3Bgt=<br>> >> =3D3B &=3Bgt=3D3B * This can be fixed if need be.<=3BBR>=3B&=3Bgt=<br>> >> =3D3B &=3Bgt=3D3B */<=3BBR>=3B&=3Bgt=3D3B =3D<br>>=3B >=3B&am=<br>> >> p=3Bgt=3D3B <=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B INTEGER<=3BBR>=<br>> >> =3B&=3Bgt=3D3B &=3Bgt=3D3B __cdecl<=3BBR>=3B&=3Bgt=3D3B &=<br>> >> =3Bgt=3D3B =3D<br>>=3B >=3BRTProcess__RegisterForkHandlers(<=3BBR>=<br>> >> =3B&=3Bgt=3D3B &=3Bgt=3D3B ForkHandler prepare=3D2C<=3BBR=3D<br>>=<br>> >> =3B >=3B>=3B&=3Bgt=3D3B &=3Bgt=3D3B ForkHandler parent=3D2C<=3B=<br>> >> BR>=3B&=3Bgt=3D3B &=3Bgt=3D3B ForkHandler child)<=3BBR=3D<br>>=<br>> >> =3B >=3B>=3B&=3Bgt=3D3B &=3Bgt=3D3B {<=3BBR>=3B&=3Bgt=3D3B=<br>> >> &=3Bgt=3D3B /* FreeBSD &=3Blt=3D3B 6 lacks pthread_atfork.=3D<br>>=<br>> >> =3B >=3B Would be good to use autoconf.<=3BBR>=3B&=3Bgt=3D3B &=<br>> >> =3Bgt=3D3B * VMS lacks pthread_atfork=3D<br>>=3B >=3B? Would be good to=<br>> >> use autoconf.<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B * Win32 lacks pth=<br>> >> read_atf=3D<br>>=3B >=3Bork and fork. OK.<=3BBR>=3B&=3Bgt=3D3B &=<br>> >> amp=3Bgt=3D3B *<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B * As well=3D2C f=<br>> >> or all =3D<br>>=3B >=3BPosix systems=3D2C we could implement<=3BBR>=<br>> >> =3B&=3Bgt=3D3B &=3Bgt=3D3B * atfork ourselves=3D2C =3D<br>>=3B >=<br>> >> =3Bas long as we provide a fork()<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3=<br>> >> B * wrapper that code uses.<=3BB=3D<br>>=3B >=3BR>=3B&=3Bgt=3D3B=<br>> >> &=3Bgt=3D3B */<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B #if defined(_=<br>> >> WIN32) \<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B =3D<br>>=3B >=3B|| =<br>> >> defined(__vms) \<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B || (defined(__F=<br>> >> reeBSD__) /* &=3Bamp=3D3B&=3Bam=3D<br>>=3B >=3Bp=3D3B (__FreeBSD_=<br>> >> _ &=3Blt=3D3B 6)*/ )<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B return 0=<br>> >> =3D3B<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D<br>>=3B >=3B=3D3B #else&=<br>> >> lt=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B while (1)<=3BBR>=3B&=3Bgt=<br>> >> =3D3B &=3Bgt=3D3B {<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B in=3D<br>=<br>> >> >=3B >=3Bt i =3D3D pthread_atfork(prepare=3D2C parent=3D2C child)=3D3B&=<br>> >> lt=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B if (=3D<br>>=3B >=3Bi !=3D3D=<br>> >> EAGAIN)<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B return i=3D3B<=3BBR&g=<br>> >> t=3B&=3Bgt=3D3B &=3Bgt=3D3B sleep(0)=3D3B<=3BBR=3D<br>>=3B >=3B=<br>> >> >=3B&=3Bgt=3D3B &=3Bgt=3D3B }<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=<br>> >> =3D3B #endif<=3BBR>=3B&=3Bgt=3D3B &=3Bgt=3D3B }<=3BBR>=3B&=<br>> >> =3Bgt=3D3B <=3BBR>=3B =3D<br>>=3B >=3B <=3B/body>=3B<=<br>> >> br>>=3B >=3B<=3B/html>=3B=3D<br>>=3B >=3B<br>>=3B >=3B--_3d=<br>> >> d397d4-ac1d-4148-a9ff-059d27dd794a_--<br> </body><br>> >> </html>=<br>> >> <br>> >> --_0203aac2-5c8e-49c7-9074-6bc448ecf342_--<br>> <br> </body>
</html>