[M3devel] Re: RTHooks CheckStoreTraced and CheckLoadTraced

[Darko]

Thanks, thats very useful. Garbage collection is a bit of a mystery,  
since I can't imagine myself changing it I've never studied it, but  
it's on my mind constantly. M3 is so safe whenever something weird  
happens with my code I know it's because I've trodden on a traced  
reference, or copied it into a untraced space and back or something.

I've relied on the pinning of references that appear in the stack  
extensively, and I think a lot of other code does too. It a bit of a  
worry to think that it's possible it might disappear. If we don't  
have this, short of stopping the collector, what do you do if you  
want your object to stay still?


On 19/04/2007, at 3:34 PM, Tony Hosking wrote:

> My immediate reaction is that you probably want to approach reads  
> slightly differently.  Instead of using your generic InTypeRead  
> routine to access the appropriate bits/bytes of the value, you  
> probably want to replace it with some sort of simple addressing  
> mechanism that computes the raw address of the field.  Traced  
> reference fields must always be aligned within the heap object they  
> are stored in so your addressing for these will not need bit  
> offsets.  Once you have an ADDRESS "addr" for the traced reference  
> field, so long as you use the idiom I gave earlier, you will get  
> the correct behavior for reads.  That is,
>
> WITH field = LOOPHOLE(addr, UNTRACED REF REFANY) DO
>   .. use field^ to access the traced reference stored at address  
> field ..
> END;
>
> The alternative is to invoke "RTHooks.CheckLoadTracedRef(r)" on the  
> traced reference "r" once you have loopholed it through the raw  
> field access into a REFANY, before you return it from InTypeReadRef:
>
> PROCEDURE InTypeReadRef (this: T; obj: REFANY; field: CARDINAL):  
> REFANY =
>   VAR v: REFANY;
>   BEGIN
>     this.check(field, Kind.Ref, obj);
>     this.read(obj, field, ADR(v), BYTESIZE(v));
>     RTHooks.CheckLoadTracedRef(v);
>     RETURN v;
>   END InTypeReadRef;
>
> This is safe for the current CM3 ambiguous roots collector, but  
> this code would probably break with a fully copying/relocating  
> collector (if one was ever implemented for Modula-3), since the raw  
> bytes of the reference might become stale between the point you  
> load them from memory and the point they are placed in the typed  
> REFANY variable "v" (effectively, you'd be hiding a reference from  
> the GC for a brief period, and it would not be able to update that  
> hidden reference in the face of its target being moved).
>
> I note, generally, that your field addressing mechanisms only work  
> anyway because we are using an ambiguous roots collector that pins  
> all objects referenced from the thread stacks (your "obj"  
> parameters in the accessor methods), which allows the use of  
> RTHeap.GetDataAdr to provide an address for the object that is  
> valid while the object reference is held on the stack.
>
> When writing references to object fields you will need to invoke  
> "RTHooks.CheckStoreTraced(o)" on the heap object "o" whose field is  
> being stored.  Thus, for InTypeWriteRef:
>
> PROCEDURE InTypeWriteRef(this: T; obj: REFANY; val: REFANY; field:  
> CARDINAL) =
>   BEGIN
>     this.check(field, Kind.Ref, obj);
>     WITH f = this.offs.get(field) DO
>       IF NOT ISTYPE(f, RefField) OR NOT RTType.IsSubtype(TYPECODE 
> (val), NARROW(f, RefField).tc) THEN RAISE WrongType END;
>     END;
>     this.write(obj, field, ADR(val), BYTESIZE(val));
>     RTHooks.CheckStoreTraced(obj);
>   END InTypeWriteRef;
>
> If you are performing multiple writes of reference fields on the  
> same object (e.g., to an array) then you can invoke  
> CheckStoreTraced once for the whole object.
>
> Hope this helps.
>
> On Apr 18, 2007, at 10:49 PM, Darko wrote:
>
>> Thanks, that's very helpful. I agree I shouldn't need it but it's  
>> exactly because I'm not using the compiler is why I need to know  
>> about it. The code is attached, it's still in development and  
>> unfinished but you'll get the idea.
>>
>> Basically the module allows you to programatically access  
>> structure fields. It allows for applications like being able to  
>> create indexes (using an adaptation of the Table code) of objects  
>> using arbitrary fields as keys. It makes dynamic programming a bit  
>> more dynamic. The code is intended to be completely safe, but  
>> there's more checks I need to do.
>>
>>
>>
>> <InType.m3>
>> <InType.i3>
>>
>>
>>
>>
>> On 19/04/2007, at 4:05 AM, Tony Hosking wrote:
>>
>>>
>>> On Apr 18, 2007, at 9:37 PM, Darko wrote:
>>>
>>>> Not actually using that code, but I did see your helpful commit  
>>>> on it some time ago. It seems that the problem I described  
>>>> earlier is due to another bug. The code I'm working on does a  
>>>> lot of reading and writing of traced references and I was hoping  
>>>> to get a better understanding of situations I need to be aware  
>>>> of and how to use those RTHooks calls more efficiently.
>>>
>>> You SHOULD NOT need to use these calls so long as you are  
>>> accessing using properly typed references (as in my example  
>>> earlier).  This is not an issue for SAFE code, only in UNSAFE  
>>> modules must you be careful how you use LOOPHOLE to cast references.
>>>
>>>> So say I have two traced objects and I am copying a traced  
>>>> reference from a field in one to the other. What set of calls  
>>>> would be normally required there?
>>>
>>> How are you copying the references?  I would like to see your  
>>> code for this.  So long as you properly type things then the  
>>> calls to RTHooks.CheckLoadTracedRef and RTHooks.CheckStoreTraced  
>>> will be generated by the compiler.
>>>
>>> Anyway...
>>>
>>> RTHooks.CheckLoadTracedRef is generated by the compiler whenever  
>>> you access a traced reference in memory (e.g., via dereference or  
>>> from a shared variable).  Its job is to prevent mutators from  
>>> acquiring references to "gray" objects (that are reachable but  
>>> still to be scanned for the traced references that they  
>>> contain).  The check turns the object from gray to black by  
>>> scanning its references and making sure none of them refer to  
>>> white objects (that are not known to be reachable by the collector).
>>>
>>> RTHooks.CheckStoreTraced is generated by the compiler whenever a  
>>> traced reference is stored (or might be stored, e.g., for VAR)  
>>> into a traced object.  The check makes sure that the generational  
>>> GC knows that the object has been modified.  It needs to know  
>>> this for objects in the older genration.
>>>
>>> Again, I reiterate that it is not my intention that programmers  
>>> actually call these runtime hooks explicitly!  I am certain I can  
>>> rewrite any code you might have so that you do not need to call  
>>> them explicitly.  Instead, properly typed references will result  
>>> in the checks being generated for you by the compiler.  Please  
>>> feel free to send me code snippets for review.
>>>
>>>> I'm sure you haven't got time to go into the minutiae of the  
>>>> runtime system, but any hints would be appreciated.
>>>>
>>>> Cheers,
>>>> Darko.
>>>>
>>>>
>>>> On 18/04/2007, at 4:34 PM, Tony Hosking wrote:
>>>>
>>>>> I assume you have an apply method for your RTTypeMap.Visitor  
>>>>> that takes "field: ADDRESS" and treats it as "REF REFANY".    
>>>>> This is wrong.  When reading a REF field you should use the  
>>>>> following idiom:
>>>>>
>>>>> WITH ref = LOOPHOLE(field, UNTRACED REF REFANY) DO
>>>>>   ... access field via ref^ ...
>>>>> END;
>>>>>
>>>>> This will automatically insert a call to the appropriate  
>>>>> runtime routines on accessing the reference field.
>>>>>
>>>>> There should be no need for you to call the runtime routines  
>>>>> directly.
>>>>>
>>>>> On Apr 17, 2007, at 9:51 PM, Darko wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Wondering if you can explain the use of these calls a little  
>>>>>> more. I'm currently using type maps to read and write fields  
>>>>>> from traced objects. Reading a traced reference from inside a  
>>>>>> traced object into a local variable is not working as it  
>>>>>> should. Should I use CheckLoadTraced and if so when and how?  
>>>>>> Looking at your changes to RTTypeMap, writing references into  
>>>>>> objects means you need to call CheckStoreTraced on the object  
>>>>>> written inside of, before it is written?
>>>>>>
>>>>>> Cheers,
>>>>>> Darko.
>>>>>
>>>
>>
>