[M3devel] WeakRef mechanism alive and well? cm3-cvshead

[Dragiša Durić]

Is your work on GC covered somewhere else except source, comments and
this list?

dd

On Thu, 2008-03-13 at 18:35 -0400, Tony Hosking wrote:
> Ah, the fact that you needed to be aggressive suggests that  
> conservatism may be the culprit.  I keep meaning one day to make sure  
> that we do not can objects on pinned pages unless we know that object  
> is definitely alive.
> 
> On Mar 13, 2008, at 6:21 PM, Dragiša Durić wrote:
> 
> > I see... What bugs me now is a fact about my flow... I am already  
> > doing
> > this, and there is always few thousand objects uncleaned from of type
> > Message... I'll follow this through, to crosscheck everything again.
> >
> > What is nice, after I made this agressive RTCollector.Collect  
> > "policy",
> > my memory footprint remains same for many hours... It was swelling
> > before.
> >
> > Thanks,
> > dd
> >
> > On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote:
> >> No, when a page is pinned because one of its objects is directly
> >> referenced from a thread stack then all objects on that page, both
> >> live and dead, are retained.  The collector does not move objects to
> >> or from pinned pages.  The objects on the page can only be freed when
> >> the page is no longer pinned by a direct reference from any thread
> >> stack.  Moral of the story is to NIL references aggressively in local
> >> variables.
> >>
> >> On Mar 13, 2008, at 6:09 PM, Dragiša Durić wrote:
> >>
> >>> When object becomes unreachable, and it sits on page with other
> >>> reachable objects, it's pinned?
> >>>
> >>> What I asked is... As garbage collector moves objects around, would
> >>> it
> >>> pack reachable objects over unreachable ones in these pages? Thus
> >>> kicking them out of pages where they're pinned?
> >>>
> >>> dd
> >>>
> >>> On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote:
> >>>> I'm not sure I understand what you mean?
> >>>>
> >>>> On Mar 13, 2008, at 5:46 PM, Dragiša Durić wrote:
> >>>>
> >>>>> Are objects on pages prepacked/moved?
> >>>>>
> >>>>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote:
> >>>>>> The thing about conservatism is that even if you NIL *all*
> >>>>>> references
> >>>>>> to an object that just happens to lie on a pinned page, that  
> >>>>>> object
> >>>>>> (and objects reachable from it) will be retained until the page
> >>>>>> is no
> >>>>>> longer pinned.
> >>>>>>
> >>>>>> Antony Hosking | Associate Professor | Computer Science | Purdue
> >>>>>> University
> >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA
> >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mar 13, 2008, at 1:43 PM, Dragiša Durić wrote:
> >>>>>>
> >>>>>>> I understand transitiveness :), but I've made big effort to  
> >>>>>>> escape
> >>>>>>> that.
> >>>>>>>
> >>>>>>> Lifetime of thread equals lifetime of Client.T instance, and  
> >>>>>>> that
> >>>>>>> instance is kept in thread's closure, and also in local  
> >>>>>>> variable.
> >>>>>>> Both
> >>>>>>> are set to NIL before thread termination - all terminations and
> >>>>>>> joins
> >>>>>>> happen and are counted and shown.
> >>>>>>>
> >>>>>>> Pointers to Message objects are kept in two places. Hub.T keeps
> >>>>>>> pointer
> >>>>>>> to list tail, and each Client.T contains pointer to it's current
> >>>>>>> element. That pointer is forwarded as Message's are added to
> >>>>>>> end. No
> >>>>>>> temporary variable, not even WITH substitution is used. And  
> >>>>>>> these
> >>>>>>> threads are ones dying with  their Client.T's.
> >>>>>>>
> >>>>>>> "head" is new mechanism used to unlink all elements with id less
> >>>>>>> than
> >>>>>>> minimal id used in active Client.T instances. Unlinked totally  
> >>>>>>> and
> >>>>>>> all
> >>>>>>> as one free for garbage collector to finish them.
> >>>>>>>
> >>>>>>> I see no transition.
> >>>>>>>
> >>>>>>> dd
> >>>>>>>
> >>>>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote:
> >>>>>>>> Realizing that conservative roots collectors like ours can  
> >>>>>>>> retain
> >>>>>>>> data
> >>>>>>>> unnecessarily.  If a thread stack refers to a page that holds a
> >>>>>>>> reference that (transitively) refers to some otherwise dead
> >>>>>>>> object
> >>>>>>>> then that object will be retained.  We say that the page is
> >>>>>>>> pinned.
> >>>>>>>> Smartening up the collector to ignore dead objects in pinned
> >>>>>>>> pages
> >>>>>>>> is
> >>>>>>>> possible, but not currently implemented.
> >>>>>>>>
> >>>>>>>> Antony Hosking | Associate Professor | Computer Science |  
> >>>>>>>> Purdue
> >>>>>>>> University
> >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA
> >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Mar 13, 2008, at 12:28 PM, Dragiša Durić wrote:
> >>>>>>>>
> >>>>>>>>> [17:25:19] <Hubetta> 117 users, 1 ops and 1 unresolved;  
> >>>>>>>>> minId =
> >>>>>>>>> 18617; thrs = 120
> >>>>>>>>> [17:25:19] <Hubetta> started = 663; cleaned = 452; nilled =  
> >>>>>>>>> 543;
> >>>>>>>>> joined = 543
> >>>>>>>>> [17:25:19] <Hubetta> Dangling thrs-a-b-n = 1
> >>>>>>>>> [17:25:19] <Hubetta> Live queue end is 18621 (head = 18616),
> >>>>>>>>> maxClean = 18611, cleaned = 16583
> >>>>>>>>>
> >>>>>>>>> About one hour uptime, after I added RTCollector.Collect()  
> >>>>>>>>> call
> >>>>>>>>> every 20
> >>>>>>>>> seconds.
> >>>>>>>>>
> >>>>>>>>> dd
> >>>>>>>>>
> >>>>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote:
> >>>>>>>>>> Yes.  RTCollectorSRC.FinishCollection;
> >>>>>>>>>> RTCollectorSRC.StartCollection.  This will finish the current
> >>>>>>>>>> (possibly minor) collection, and start a major collection.
> >>>>>>>>>> The
> >>>>>>>>>> major
> >>>>>>>>>> collection must finish before you can guarantee garbage has
> >>>>>>>>>> been
> >>>>>>>>>> freed, so you may want to RTCollectorSRC.FinishCollection.
> >>>>>>>>>> This
> >>>>>>>>>> sequence is packaged as RTCollector.Collect.
> >>>>>>>>>>
> >>>>>>>>>> Antony Hosking | Associate Professor | Computer Science |
> >>>>>>>>>> Purdue
> >>>>>>>>>> University
> >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA
> >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragiša Durić wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Can I force "major collection" ?
> >>>>>>>>>>>
> >>>>>>>>>>> dd
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote:
> >>>>>>>>>>>> Indeed! Can you devise a testcase?  Note that the
> >>>>>>>>>>>> generational
> >>>>>>>>>>>> collector will retain old objects for some time until a
> >>>>>>>>>>>> major
> >>>>>>>>>>>> collection occurs,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer Science |
> >>>>>>>>>>>> Purdue
> >>>>>>>>>>>> University
> >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA
> >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragiša Durić wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> And not only that, their stackbase is also set to NIL,
> >>>>>>>>>>>>> meaning
> >>>>>>>>>>>>> their
> >>>>>>>>>>>>> stack is not regarded in any way during future
> >>>>>>>>>>>>> collections -
> >>>>>>>>>>>>> meaning
> >>>>>>>>>>>>> all
> >>>>>>>>>>>>> local variables are forgotten once apply method returns.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> That way, "spuriousness"  is not an issue, once thread
> >>>>>>>>>>>>> returns?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> dd
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote:
> >>>>>>>>>>>>>> They should get unlinked from the global ring.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer
> >>>>>>>>>>>>>> Science |
> >>>>>>>>>>>>>> Purdue
> >>>>>>>>>>>>>> University
> >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 |
> >>>>>>>>>>>>>> USA
> >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragiša Durić wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> What happens to thread stacks once threads are
> >>>>>>>>>>>>>>> Join-ed?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> dd
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>> This is probably a result of conservatism for
> >>>>>>>>>>>>>>>> references
> >>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>> thread
> >>>>>>>>>>>>>>>> stacks which results in spurious retention.  It
> >>>>>>>>>>>>>>>> may be
> >>>>>>>>>>>>>>>> necessary
> >>>>>>>>>>>>>>>> to be
> >>>>>>>>>>>>>>>> more careful about what references are held on the
> >>>>>>>>>>>>>>>> stack
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> threads implementation (and elsewhere in your
> >>>>>>>>>>>>>>>> code).
> >>>>>>>>>>>>>>>> Until we
> >>>>>>>>>>>>>>>> diagnose the place where objects are being
> >>>>>>>>>>>>>>>> retained it
> >>>>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>>>> hard to
> >>>>>>>>>>>>>>>> pinpoint.  You should realize that weak refs are a
> >>>>>>>>>>>>>>>> problem
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>> many
> >>>>>>>>>>>>>>>> language implementations, not just Modula-3.  I
> >>>>>>>>>>>>>>>> look
> >>>>>>>>>>>>>>>> forward
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> hearing more definitively from you.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer
> >>>>>>>>>>>>>>>> Science |
> >>>>>>>>>>>>>>>> Purdue
> >>>>>>>>>>>>>>>> University
> >>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN
> >>>>>>>>>>>>>>>> 47907 |
> >>>>>>>>>>>>>>>> USA
> >>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragiša Durić wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Problem in my case can be thoroughness of GC...
> >>>>>>>>>>>>>>>>> Because
> >>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>> linked
> >>>>>>>>>>>>>>>>> list of WeakRef-ed objects, it's always first
> >>>>>>>>>>>>>>>>> element in
> >>>>>>>>>>>>>>>>> list to
> >>>>>>>>>>>>>>>>> become
> >>>>>>>>>>>>>>>>> unreachable first - all others are reachable at
> >>>>>>>>>>>>>>>>> least
> >>>>>>>>>>>>>>>>> through
> >>>>>>>>>>>>>>>>> previous
> >>>>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I
> >>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>> observed
> >>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> because of some thousands of freeable objects,
> >>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> missed
> >>>>>>>>>>>>>>>>> too
> >>>>>>>>>>>>>>>>> often.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I have other object being cleaned and counted...
> >>>>>>>>>>>>>>>>> There
> >>>>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>> cleanup
> >>>>>>>>>>>>>>>>> invocation looks like it's lagging too much.
> >>>>>>>>>>>>>>>>> I'll
> >>>>>>>>>>>>>>>>> come
> >>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>> numbers
> >>>>>>>>>>>>>>>>> after I have more tests.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> dd
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>> Sounds like we need some sort of testcase.
> >>>>>>>>>>>>>>>>>> Would
> >>>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> able
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> devise one?  It will be hard to make it
> >>>>>>>>>>>>>>>>>> deterministic,
> >>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>> least we
> >>>>>>>>>>>>>>>>>> should see a non-zero cleanup count.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Antony Hosking | Associate Professor |
> >>>>>>>>>>>>>>>>>> Computer
> >>>>>>>>>>>>>>>>>> Science |
> >>>>>>>>>>>>>>>>>> Purdue
> >>>>>>>>>>>>>>>>>> University
> >>>>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN
> >>>>>>>>>>>>>>>>>> 47907 |
> >>>>>>>>>>>>>>>>>> USA
> >>>>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427
> >>>>>>>>>>>>>>>>>> 5484
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragiša Durić
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I have single linked list that I use to send
> >>>>>>>>>>>>>>>>>>> messages to
> >>>>>>>>>>>>>>>>>>> many
> >>>>>>>>>>>>>>>>>>> threads.
> >>>>>>>>>>>>>>>>>>> It's end is locked for addition, and current
> >>>>>>>>>>>>>>>>>>> end
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> given to
> >>>>>>>>>>>>>>>>>>> each
> >>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>> client connecting. This way, all client are
> >>>>>>>>>>>>>>>>>>> going
> >>>>>>>>>>>>>>>>>>> towards
> >>>>>>>>>>>>>>>>>>> end of
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> list so it's head becomes unreferenced and
> >>>>>>>>>>>>>>>>>>> goes
> >>>>>>>>>>>>>>>>>>> away
> >>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>> GC.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Except it does not.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential
> >>>>>>>>>>>>>>>>>>> id's
> >>>>>>>>>>>>>>>>>>> so I
> >>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> record
> >>>>>>>>>>>>>>>>>>> maximum
> >>>>>>>>>>>>>>>>>>> freed id for checking. No single cleanup
> >>>>>>>>>>>>>>>>>>> happens.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I'll cross check my code and count my
> >>>>>>>>>>>>>>>>>>> references
> >>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>> application
> >>>>>>>>>>>>>>>>>>> side,
> >>>>>>>>>>>>>>>>>>> but maybe someone else has similar problems?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> dd
> >>>>>>>>>>>>>>>>>>> -- 
> >>>>>>>>>>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -- 
> >>>>>>>>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -- 
> >>>>>>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> -- 
> >>>>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> -- 
> >>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> -- 
> >>>>>>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>> -- 
> >>>>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>>>
> >>>>>>
> >>>>> -- 
> >>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>
> >>> -- 
> >>> Dragiša Durić <dragisha at m3w.org>
> >>
> > -- 
> > Dragiša Durić <dragisha at m3w.org>
> 
-- 
Dragiša Durić <dragisha at m3w.org>