[M3devel] WeakRef mechanism alive and well? cm3-cvshead
Tony Hosking
hosking at cs.purdue.edu
Thu Mar 13 23:35:00 CET 2008
Ah, the fact that you needed to be aggressive suggests that
conservatism may be the culprit. I keep meaning one day to make sure
that we do not can objects on pinned pages unless we know that object
is definitely alive.
On Mar 13, 2008, at 6:21 PM, Dragiša Durić wrote:
> I see... What bugs me now is a fact about my flow... I am already
> doing
> this, and there is always few thousand objects uncleaned from of type
> Message... I'll follow this through, to crosscheck everything again.
>
> What is nice, after I made this agressive RTCollector.Collect
> "policy",
> my memory footprint remains same for many hours... It was swelling
> before.
>
> Thanks,
> dd
>
> On Thu, 2008-03-13 at 18:15 -0400, Tony Hosking wrote:
>> No, when a page is pinned because one of its objects is directly
>> referenced from a thread stack then all objects on that page, both
>> live and dead, are retained. The collector does not move objects to
>> or from pinned pages. The objects on the page can only be freed when
>> the page is no longer pinned by a direct reference from any thread
>> stack. Moral of the story is to NIL references aggressively in local
>> variables.
>>
>> On Mar 13, 2008, at 6:09 PM, Dragiša Durić wrote:
>>
>>> When object becomes unreachable, and it sits on page with other
>>> reachable objects, it's pinned?
>>>
>>> What I asked is... As garbage collector moves objects around, would
>>> it
>>> pack reachable objects over unreachable ones in these pages? Thus
>>> kicking them out of pages where they're pinned?
>>>
>>> dd
>>>
>>> On Thu, 2008-03-13 at 18:03 -0400, Tony Hosking wrote:
>>>> I'm not sure I understand what you mean?
>>>>
>>>> On Mar 13, 2008, at 5:46 PM, Dragiša Durić wrote:
>>>>
>>>>> Are objects on pages prepacked/moved?
>>>>>
>>>>> On Thu, 2008-03-13 at 17:41 -0400, Tony Hosking wrote:
>>>>>> The thing about conservatism is that even if you NIL *all*
>>>>>> references
>>>>>> to an object that just happens to lie on a pinned page, that
>>>>>> object
>>>>>> (and objects reachable from it) will be retained until the page
>>>>>> is no
>>>>>> longer pinned.
>>>>>>
>>>>>> Antony Hosking | Associate Professor | Computer Science | Purdue
>>>>>> University
>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA
>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mar 13, 2008, at 1:43 PM, Dragiša Durić wrote:
>>>>>>
>>>>>>> I understand transitiveness :), but I've made big effort to
>>>>>>> escape
>>>>>>> that.
>>>>>>>
>>>>>>> Lifetime of thread equals lifetime of Client.T instance, and
>>>>>>> that
>>>>>>> instance is kept in thread's closure, and also in local
>>>>>>> variable.
>>>>>>> Both
>>>>>>> are set to NIL before thread termination - all terminations and
>>>>>>> joins
>>>>>>> happen and are counted and shown.
>>>>>>>
>>>>>>> Pointers to Message objects are kept in two places. Hub.T keeps
>>>>>>> pointer
>>>>>>> to list tail, and each Client.T contains pointer to it's current
>>>>>>> element. That pointer is forwarded as Message's are added to
>>>>>>> end. No
>>>>>>> temporary variable, not even WITH substitution is used. And
>>>>>>> these
>>>>>>> threads are ones dying with their Client.T's.
>>>>>>>
>>>>>>> "head" is new mechanism used to unlink all elements with id less
>>>>>>> than
>>>>>>> minimal id used in active Client.T instances. Unlinked totally
>>>>>>> and
>>>>>>> all
>>>>>>> as one free for garbage collector to finish them.
>>>>>>>
>>>>>>> I see no transition.
>>>>>>>
>>>>>>> dd
>>>>>>>
>>>>>>> On Thu, 2008-03-13 at 12:45 -0400, Tony Hosking wrote:
>>>>>>>> Realizing that conservative roots collectors like ours can
>>>>>>>> retain
>>>>>>>> data
>>>>>>>> unnecessarily. If a thread stack refers to a page that holds a
>>>>>>>> reference that (transitively) refers to some otherwise dead
>>>>>>>> object
>>>>>>>> then that object will be retained. We say that the page is
>>>>>>>> pinned.
>>>>>>>> Smartening up the collector to ignore dead objects in pinned
>>>>>>>> pages
>>>>>>>> is
>>>>>>>> possible, but not currently implemented.
>>>>>>>>
>>>>>>>> Antony Hosking | Associate Professor | Computer Science |
>>>>>>>> Purdue
>>>>>>>> University
>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA
>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mar 13, 2008, at 12:28 PM, Dragiša Durić wrote:
>>>>>>>>
>>>>>>>>> [17:25:19] <Hubetta> 117 users, 1 ops and 1 unresolved;
>>>>>>>>> minId =
>>>>>>>>> 18617; thrs = 120
>>>>>>>>> [17:25:19] <Hubetta> started = 663; cleaned = 452; nilled =
>>>>>>>>> 543;
>>>>>>>>> joined = 543
>>>>>>>>> [17:25:19] <Hubetta> Dangling thrs-a-b-n = 1
>>>>>>>>> [17:25:19] <Hubetta> Live queue end is 18621 (head = 18616),
>>>>>>>>> maxClean = 18611, cleaned = 16583
>>>>>>>>>
>>>>>>>>> About one hour uptime, after I added RTCollector.Collect()
>>>>>>>>> call
>>>>>>>>> every 20
>>>>>>>>> seconds.
>>>>>>>>>
>>>>>>>>> dd
>>>>>>>>>
>>>>>>>>> On Thu, 2008-03-13 at 10:54 -0400, Tony Hosking wrote:
>>>>>>>>>> Yes. RTCollectorSRC.FinishCollection;
>>>>>>>>>> RTCollectorSRC.StartCollection. This will finish the current
>>>>>>>>>> (possibly minor) collection, and start a major collection.
>>>>>>>>>> The
>>>>>>>>>> major
>>>>>>>>>> collection must finish before you can guarantee garbage has
>>>>>>>>>> been
>>>>>>>>>> freed, so you may want to RTCollectorSRC.FinishCollection.
>>>>>>>>>> This
>>>>>>>>>> sequence is packaged as RTCollector.Collect.
>>>>>>>>>>
>>>>>>>>>> Antony Hosking | Associate Professor | Computer Science |
>>>>>>>>>> Purdue
>>>>>>>>>> University
>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA
>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mar 13, 2008, at 10:22 AM, Dragiša Durić wrote:
>>>>>>>>>>
>>>>>>>>>>> Can I force "major collection" ?
>>>>>>>>>>>
>>>>>>>>>>> dd
>>>>>>>>>>>
>>>>>>>>>>> On Thu, 2008-03-13 at 10:08 -0400, Tony Hosking wrote:
>>>>>>>>>>>> Indeed! Can you devise a testcase? Note that the
>>>>>>>>>>>> generational
>>>>>>>>>>>> collector will retain old objects for some time until a
>>>>>>>>>>>> major
>>>>>>>>>>>> collection occurs,
>>>>>>>>>>>>
>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer Science |
>>>>>>>>>>>> Purdue
>>>>>>>>>>>> University
>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 | USA
>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mar 13, 2008, at 9:56 AM, Dragiša Durić wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> And not only that, their stackbase is also set to NIL,
>>>>>>>>>>>>> meaning
>>>>>>>>>>>>> their
>>>>>>>>>>>>> stack is not regarded in any way during future
>>>>>>>>>>>>> collections -
>>>>>>>>>>>>> meaning
>>>>>>>>>>>>> all
>>>>>>>>>>>>> local variables are forgotten once apply method returns.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That way, "spuriousness" is not an issue, once thread
>>>>>>>>>>>>> returns?
>>>>>>>>>>>>>
>>>>>>>>>>>>> dd
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, 2008-03-13 at 09:46 -0400, Tony Hosking wrote:
>>>>>>>>>>>>>> They should get unlinked from the global ring.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer
>>>>>>>>>>>>>> Science |
>>>>>>>>>>>>>> Purdue
>>>>>>>>>>>>>> University
>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN 47907 |
>>>>>>>>>>>>>> USA
>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mar 13, 2008, at 6:16 AM, Dragiša Durić wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What happens to thread stacks once threads are
>>>>>>>>>>>>>>> Join-ed?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> dd
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, 2008-03-12 at 11:12 -0400, Tony Hosking
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> This is probably a result of conservatism for
>>>>>>>>>>>>>>>> references
>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>>> stacks which results in spurious retention. It
>>>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>> necessary
>>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>> more careful about what references are held on the
>>>>>>>>>>>>>>>> stack
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> threads implementation (and elsewhere in your
>>>>>>>>>>>>>>>> code).
>>>>>>>>>>>>>>>> Until we
>>>>>>>>>>>>>>>> diagnose the place where objects are being
>>>>>>>>>>>>>>>> retained it
>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>> hard to
>>>>>>>>>>>>>>>> pinpoint. You should realize that weak refs are a
>>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>> language implementations, not just Modula-3. I
>>>>>>>>>>>>>>>> look
>>>>>>>>>>>>>>>> forward
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> hearing more definitively from you.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Antony Hosking | Associate Professor | Computer
>>>>>>>>>>>>>>>> Science |
>>>>>>>>>>>>>>>> Purdue
>>>>>>>>>>>>>>>> University
>>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN
>>>>>>>>>>>>>>>> 47907 |
>>>>>>>>>>>>>>>> USA
>>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427 5484
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mar 12, 2008, at 4:21 AM, Dragiša Durić wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Problem in my case can be thoroughness of GC...
>>>>>>>>>>>>>>>>> Because
>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>> linked
>>>>>>>>>>>>>>>>> list of WeakRef-ed objects, it's always first
>>>>>>>>>>>>>>>>> element in
>>>>>>>>>>>>>>>>> list to
>>>>>>>>>>>>>>>>> become
>>>>>>>>>>>>>>>>> unreachable first - all others are reachable at
>>>>>>>>>>>>>>>>> least
>>>>>>>>>>>>>>>>> through
>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>> (in list) WeakRef-ed object. So, behaviour I
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> observed
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> because of some thousands of freeable objects,
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> missed
>>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>>> often.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have other object being cleaned and counted...
>>>>>>>>>>>>>>>>> There
>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>> cleanup
>>>>>>>>>>>>>>>>> invocation looks like it's lagging too much.
>>>>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>>>>> come
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> numbers
>>>>>>>>>>>>>>>>> after I have more tests.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> dd
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, 2008-03-11 at 09:50 -0400, Tony Hosking
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> Sounds like we need some sort of testcase.
>>>>>>>>>>>>>>>>>> Would
>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> able
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> devise one? It will be hard to make it
>>>>>>>>>>>>>>>>>> deterministic,
>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> least we
>>>>>>>>>>>>>>>>>> should see a non-zero cleanup count.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Antony Hosking | Associate Professor |
>>>>>>>>>>>>>>>>>> Computer
>>>>>>>>>>>>>>>>>> Science |
>>>>>>>>>>>>>>>>>> Purdue
>>>>>>>>>>>>>>>>>> University
>>>>>>>>>>>>>>>>>> 305 N. University Street | West Lafayette | IN
>>>>>>>>>>>>>>>>>> 47907 |
>>>>>>>>>>>>>>>>>> USA
>>>>>>>>>>>>>>>>>> Office +1 765 494 6001 | Mobile +1 765 427
>>>>>>>>>>>>>>>>>> 5484
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mar 11, 2008, at 4:03 AM, Dragiša Durić
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I have single linked list that I use to send
>>>>>>>>>>>>>>>>>>> messages to
>>>>>>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>>> threads.
>>>>>>>>>>>>>>>>>>> It's end is locked for addition, and current
>>>>>>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> given to
>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>> client connecting. This way, all client are
>>>>>>>>>>>>>>>>>>> going
>>>>>>>>>>>>>>>>>>> towards
>>>>>>>>>>>>>>>>>>> end of
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> list so it's head becomes unreferenced and
>>>>>>>>>>>>>>>>>>> goes
>>>>>>>>>>>>>>>>>>> away
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>> GC.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Except it does not.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I've added WeakRef cleanup and sequential
>>>>>>>>>>>>>>>>>>> id's
>>>>>>>>>>>>>>>>>>> so I
>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> record
>>>>>>>>>>>>>>>>>>> maximum
>>>>>>>>>>>>>>>>>>> freed id for checking. No single cleanup
>>>>>>>>>>>>>>>>>>> happens.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'll cross check my code and count my
>>>>>>>>>>>>>>>>>>> references
>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>> application
>>>>>>>>>>>>>>>>>>> side,
>>>>>>>>>>>>>>>>>>> but maybe someone else has similar problems?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> dd
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Dragiša Durić <dragisha at m3w.org>
>>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> Dragiša Durić <dragisha at m3w.org>
>>>>>>>
>>>>>>
>>>>> --
>>>>> Dragiša Durić <dragisha at m3w.org>
>>>>
>>> --
>>> Dragiša Durić <dragisha at m3w.org>
>>
> --
> Dragiša Durić <dragisha at m3w.org>
More information about the M3devel
mailing list