[M3devel] WeakRef mechanism alive and well? cm3-cvshead

[Dragiša Durić]

I see... What bugs me now is a fact about my flow... I am already doing
this, and there is always few thousand objects uncleaned from of type
Message... I'll follow this through, to crosscheck everything again.

What is nice, after I made this agressive RTCollector.Collect "policy",
my memory footprint remains same for many hours... It was swelling
before.

Thanks,
dd

On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote:
> No, when a page is pinned because one of its objects is directly  
> referenced from a thread stack then all objects on that page, both  
> live and dead, are retained.  The collector does not move objects to  
> or from pinned pages.  The objects on the page can only be freed when  
> the page is no longer pinned by a direct reference from any thread  
> stack.  Moral of the story is to NIL references aggressively in local  
> variables.
> 
> On Mar 13, 2008, at 6:09 PM, Dragiša Durić wrote:
> 
> >  When object becomes unreachable, and it sits on page with other
> > reachable objects, it's pinned?
> >
> >  What I asked is... As garbage collector moves objects around, would  
> > it
> > pack reachable objects over unreachable ones in these pages? Thus
> > kicking them out of pages where they're pinned?
> >
> >  dd
> >
> > On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote:
> >> I'm not sure I understand what you mean?
> >>
> >> On Mar 13, 2008, at 5:46 PM, Dragiša Durić wrote:
> >>
> >>> Are objects on pages prepacked/moved?
> >>>
> >>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote:
> >>>> The thing about conservatism is that even if you NIL *all*  
> >>>> references
> >>>> to an object that just happens to lie on a pinned page, that object
> >>>> (and objects reachable from it) will be retained until the page  
> >>>> is no
> >>>> longer pinned.
> >>>>
> >>>> Antony Hosking | Associate Professor | Computer Science | Purdue
> >>>> University
> >>>> 305 N. University Street | West Lafayette | IN 47907 | USA
> >>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Mar 13, 2008, at 1:43 PM, Dragiša Durić wrote:
> >>>>
> >>>>> I understand transitiveness :), but I've made big effort to escape
> >>>>> that.
> >>>>>
> >>>>> Lifetime of thread equals lifetime of Client.T instance, and that
> >>>>> instance is kept in thread's closure, and also in local variable.
> >>>>> Both
> >>>>> are set to NIL before thread termination - all terminations and
> >>>>> joins
> >>>>> happen and are counted and shown.
> >>>>>
> >>>>> Pointers to Message objects are kept in two places. Hub.T keeps
> >>>>> pointer
> >>>>> to list tail, and each Client.T contains pointer to it's current
> >>>>> element. That pointer is forwarded as Message's are added to  
> >>>>> end. No
> >>>>> temporary variable, not even WITH substitution is used. And these
> >>>>> threads are ones dying with  their Client.T's.
> >>>>>
> >>>>> "head" is new mechanism used to unlink all elements with id less
> >>>>> than
> >>>>> minimal id used in active Client.T instances. Unlinked totally and
> >>>>> all
> >>>>> as one free for garbage collector to finish them.
> >>>>>
> >>>>> I see no transition.
> >>>>>
> >>>>> dd
> >>>>>
> >>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote:
> >>>>>> Realizing that conservative roots collectors like ours can retain
> >>>>>> data
> >>>>>> unnecessarily.  If a thread stack refers to a page that holds a
> >>>>>> reference that (transitively) refers to some otherwise dead  
> >>>>>> object
> >>>>>> then that object will be retained.  We say that the page is
> >>>>>> pinned.
> >>>>>> Smartening up the collector to ignore dead objects in pinned  
> >>>>>> pages
> >>>>>> is
> >>>>>> possible, but not currently implemented.
> >>>>>>
> >>>>>> Antony Hosking | Associate Professor | Computer Science | Purdue
> >>>>>> University
> >>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA
> >>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mar 13, 2008, at 12:28 PM, Dragiša Durić wrote:
> >>>>>>
> >>>>>>> [17:25:19] <Hubetta> 117 users, 1 ops and 1 unresolved; minId =
> >>>>>>> 18617; thrs = 120
> >>>>>>> [17:25:19] <Hubetta> started = 663; cleaned = 452; nilled = 543;
> >>>>>>> joined = 543
> >>>>>>> [17:25:19] <Hubetta> Dangling thrs-a-b-n = 1
> >>>>>>> [17:25:19] <Hubetta> Live queue end is 18621 (head = 18616),
> >>>>>>> maxClean = 18611, cleaned = 16583
> >>>>>>>
> >>>>>>> About one hour uptime, after I added RTCollector.Collect() call
> >>>>>>> every 20
> >>>>>>> seconds.
> >>>>>>>
> >>>>>>> dd
> >>>>>>>
> >>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote:
> >>>>>>>> Yes.  RTCollectorSRC.FinishCollection;
> >>>>>>>> RTCollectorSRC.StartCollection.  This will finish the current
> >>>>>>>> (possibly minor) collection, and start a major collection.
> >>>>>>>> The
> >>>>>>>> major
> >>>>>>>> collection must finish before you can guarantee garbage has
> >>>>>>>> been
> >>>>>>>> freed, so you may want to RTCollectorSRC.FinishCollection.
> >>>>>>>> This
> >>>>>>>> sequence is packaged as RTCollector.Collect.
> >>>>>>>>
> >>>>>>>> Antony Hosking | Associate Professor | Computer Science |
> >>>>>>>> Purdue
> >>>>>>>> University
> >>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA
> >>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragiša Durić wrote:
> >>>>>>>>
> >>>>>>>>> Can I force "major collection" ?
> >>>>>>>>>
> >>>>>>>>> dd
> >>>>>>>>>
> >>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote:
> >>>>>>>>>> Indeed! Can you devise a testcase?  Note that the
> >>>>>>>>>> generational
> >>>>>>>>>> collector will retain old objects for some time until a
> >>>>>>>>>> major
> >>>>>>>>>> collection occurs,
> >>>>>>>>>>
> >>>>>>>>>> Antony Hosking | Associate Professor | Computer Science |
> >>>>>>>>>> Purdue
> >>>>>>>>>> University
> >>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA
> >>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragiša Durić wrote:
> >>>>>>>>>>
> >>>>>>>>>>> And not only that, their stackbase is also set to NIL,
> >>>>>>>>>>> meaning
> >>>>>>>>>>> their
> >>>>>>>>>>> stack is not regarded in any way during future
> >>>>>>>>>>> collections -
> >>>>>>>>>>> meaning
> >>>>>>>>>>> all
> >>>>>>>>>>> local variables are forgotten once apply method returns.
> >>>>>>>>>>>
> >>>>>>>>>>> That way, "spuriousness"  is not an issue, once thread
> >>>>>>>>>>> returns?
> >>>>>>>>>>>
> >>>>>>>>>>> dd
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote:
> >>>>>>>>>>>> They should get unlinked from the global ring.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Antony Hosking | Associate Professor | Computer
> >>>>>>>>>>>> Science |
> >>>>>>>>>>>> Purdue
> >>>>>>>>>>>> University
> >>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 |
> >>>>>>>>>>>> USA
> >>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragiša Durić wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> What happens to thread stacks once threads are
> >>>>>>>>>>>>> Join-ed?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> dd
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>> This is probably a result of conservatism for
> >>>>>>>>>>>>>> references
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>> thread
> >>>>>>>>>>>>>> stacks which results in spurious retention.  It
> >>>>>>>>>>>>>> may be
> >>>>>>>>>>>>>> necessary
> >>>>>>>>>>>>>> to be
> >>>>>>>>>>>>>> more careful about what references are held on the
> >>>>>>>>>>>>>> stack
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>> threads implementation (and elsewhere in your
> >>>>>>>>>>>>>> code).
> >>>>>>>>>>>>>> Until we
> >>>>>>>>>>>>>> diagnose the place where objects are being
> >>>>>>>>>>>>>> retained it
> >>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>> hard to
> >>>>>>>>>>>>>> pinpoint.  You should realize that weak refs are a
> >>>>>>>>>>>>>> problem
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>> many
> >>>>>>>>>>>>>> language implementations, not just Modula-3.  I
> >>>>>>>>>>>>>> look
> >>>>>>>>>>>>>> forward
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>> hearing more definitively from you.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer
> >>>>>>>>>>>>>> Science |
> >>>>>>>>>>>>>> Purdue
> >>>>>>>>>>>>>> University
> >>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN
> >>>>>>>>>>>>>> 47907 |
> >>>>>>>>>>>>>> USA
> >>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragiša Durić wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Problem in my case can be thoroughness of GC...
> >>>>>>>>>>>>>>> Because
> >>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>> linked
> >>>>>>>>>>>>>>> list of WeakRef-ed objects, it's always first
> >>>>>>>>>>>>>>> element in
> >>>>>>>>>>>>>>> list to
> >>>>>>>>>>>>>>> become
> >>>>>>>>>>>>>>> unreachable first - all others are reachable at
> >>>>>>>>>>>>>>> least
> >>>>>>>>>>>>>>> through
> >>>>>>>>>>>>>>> previous
> >>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I
> >>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>> observed
> >>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>> because of some thousands of freeable objects,
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> missed
> >>>>>>>>>>>>>>> too
> >>>>>>>>>>>>>>> often.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I have other object being cleaned and counted...
> >>>>>>>>>>>>>>> There
> >>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>> cleanup
> >>>>>>>>>>>>>>> invocation looks like it's lagging too much.
> >>>>>>>>>>>>>>> I'll
> >>>>>>>>>>>>>>> come
> >>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>> numbers
> >>>>>>>>>>>>>>> after I have more tests.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> dd
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>> Sounds like we need some sort of testcase.
> >>>>>>>>>>>>>>>> Would
> >>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>> able
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> devise one?  It will be hard to make it
> >>>>>>>>>>>>>>>> deterministic,
> >>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>> least we
> >>>>>>>>>>>>>>>> should see a non-zero cleanup count.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Antony Hosking | Associate Professor |
> >>>>>>>>>>>>>>>> Computer
> >>>>>>>>>>>>>>>> Science |
> >>>>>>>>>>>>>>>> Purdue
> >>>>>>>>>>>>>>>> University
> >>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN
> >>>>>>>>>>>>>>>> 47907 |
> >>>>>>>>>>>>>>>> USA
> >>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427
> >>>>>>>>>>>>>>>> 5484
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragiša Durić
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I have single linked list that I use to send
> >>>>>>>>>>>>>>>>> messages to
> >>>>>>>>>>>>>>>>> many
> >>>>>>>>>>>>>>>>> threads.
> >>>>>>>>>>>>>>>>> It's end is locked for addition, and current
> >>>>>>>>>>>>>>>>> end
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> given to
> >>>>>>>>>>>>>>>>> each
> >>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>> client connecting. This way, all client are
> >>>>>>>>>>>>>>>>> going
> >>>>>>>>>>>>>>>>> towards
> >>>>>>>>>>>>>>>>> end of
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> list so it's head becomes unreferenced and
> >>>>>>>>>>>>>>>>> goes
> >>>>>>>>>>>>>>>>> away
> >>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>> GC.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Except it does not.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential
> >>>>>>>>>>>>>>>>> id's
> >>>>>>>>>>>>>>>>> so I
> >>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>> record
> >>>>>>>>>>>>>>>>> maximum
> >>>>>>>>>>>>>>>>> freed id for checking. No single cleanup
> >>>>>>>>>>>>>>>>> happens.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I'll cross check my code and count my
> >>>>>>>>>>>>>>>>> references
> >>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>> application
> >>>>>>>>>>>>>>>>> side,
> >>>>>>>>>>>>>>>>> but maybe someone else has similar problems?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> dd
> >>>>>>>>>>>>>>>>> -- 
> >>>>>>>>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -- 
> >>>>>>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> -- 
> >>>>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> -- 
> >>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> -- 
> >>>>>>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>> -- 
> >>>>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>>>
> >>>>>>
> >>>>> -- 
> >>>>> Dragiša Durić <dragisha at m3w.org>
> >>>>>
> >>>>
> >>> -- 
> >>> Dragiša Durić <dragisha at m3w.org>
> >>
> > -- 
> > Dragiša Durić <dragisha at m3w.org>
> 
-- 
Dragiša Durić <dragisha at m3w.org>