[M3devel] compiler behavior under "low memory"?

[Daniel Alejandro Benavides D.]

Hi:
If it only had such small memory requirements I suppose less than 64 MB RAM I didn't get the number of interfaces is about the same (probably a 10 % larger by now with new added packages), probably the best way forward is not just to change the swap memory as its a common sense to double the RAM but in that case you would need such a thing for one process less the rest of operating system, i.e solution is not general, If I understand well Mark 's Alpha DEC machine had the same machine and probably with M3 running in top of it in 1995. I wonder how these years have expanded the compilers proficiency (as time is against space and vice-versa, that is, less time is more memory like to make a recursive program to unfold in a sequential one) in such a memory constrained environments could be solved I can remember now that there was a change in Amer Diwan Whole program optimizer to grab in a pickled AST to later unpickle it and compile to M3CG-machine assembly
 optimized see:
http://www.modula3.org/threads/1/#advanced
and also a proposed or to-do of DEC-SRC M3 days about Cache policy it mentioned see:
http://www.cs.arizona.edu/~collberg/Research/Modula-3/modula-3/html/todo.html
AST cache policy. The current driver keeps an in-memory cache of compiled interfaces, but never flushes entries, this refers to the simple problem of big heap which another application generated out of memory back in those days the ESC/Modula-3 checker available for Alphas see:
http://web.archive.org/web/20030704143617/http://research.compaq.com/SRC/esc/escm3/bin/Esc.alpha.bin.Z (which was user of Simplify theorem prover http://portal.acm.org/citation.cfm?id=1066100.1066102 
or doi:0.1145/1066100.1066102 as it should be cited and also used m3tk toolkit ) and they managed to solve by means of calling the collector to scan and throw away the uncollected things at least in Simplify, years later I retired the line with no problem in Simplify and all run in good benchmarks tests of Simplify with no problems see too http://qpq.csl.sri.com/downloads/simplify_benchmarks-tar.gz/view 
The result was that they created a tool to understand where the memory heap was being actually ran out of memory and they manage to solve I believe by managing the same way besides clearing stuff not needed something is useful in any garbage collected type language http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.49.3474&rep=rep1&type=pdf
see slides onhttp://labs.oracle.com/people/detlefs/papers/detlefs-coots95-slides.ps and extra figures http://labs.oracle.com/people/detlefs/papers/detlefs-coots95-slides2.ps
In our case in that case as it can't throw away AST representations before M3CG-machine tough there is separate compilation we don't manage to type check all m3tk in runtime eliminating when is compiled and not used code generation currently the problem might be solved in the compilation server written for pm3 (see:
http://arxiv.org/pdf/cs/0506035 )
or as Amer Diwan solved to manage for his particular purposes (see ftp://ftp.cs.umass.edu/pub/osl/papers/diwan-dissertation.ps.gz) because in any case of low resources limits it would be stored pickled or in swaped memory in the disc but in non-low memory limits it would be in both memory and disk as it is now (at some point you need to bring from copy from disk the needed bytes back to memory).
Besides that the implemented behavior on VM exhaustion should be not taken in false as guys didn't in those days
http://modula3.elegosoft.com/pm3/intro/questions/new.html
An implemented solution but actually not used not seen here:

http://web.archive.org/web/19980613063641/www.vlsi.polymtl.ca/m3/pkg/contrib/whenNEWfails/.ghindex.html

Thanks for any corrections in advance if so


 


--- El sáb, 17/7/10, Jay K <jay.krell at cornell.edu> escribió:

> De: Jay K <jay.krell at cornell.edu>
> Asunto: [M3devel] compiler behavior under "low memory"?
> Para: "m3devel" <m3devel at elegosoft.com>
> Fecha: sábado, 17 de julio, 2010 20:30
> 
> Mark had a machine with low memory.
> It has more swap now.
> 
> 
> Our behavior was:
> 
> 
> Compiling m3tk would make good progress, complete many
> files, then fail for
>  lack of memory.
> 
> 
> Manual retry would start over due to
>  "missing version information" or such.
> 
> 
> Thus progress was "impossible".
> By editing down m3makefile I was able to make progress.
>  Some stuff would fail due to missing interfaces, but
>  whatever I left in would get the version information
> saved.
>  
>  
> Perhaps we should write out the version information every
> "few" files?
>  For some definition of "few"? 10? 10% of the total?
>  
>  
> That would at least let such situations progress via manual
> retry.
>   (unless not even a "few" fit in memory)
> 
> 
> I also wonder about reducing memory used, or why indeed we
> run out,
>  but that is a thornier problem. You know, all we should
> need to
>  continue forward progress is to hold all interfaces in
> memory and
>  one implementation at a time. Now, there are a few
> unknowns here.
>  It is possible Mark's machine didn't have enough memory
> for
>  all the relevant interfaces, and that we only "load" them
> on-demand.
> 
>  
>  It is also likely, I should add, that virtual address
> space is the
>   problem, not working set. Some systems want to commit
> room
>   in swap for all allocated virtual memory, like, to be
> sure
>   ahead of time that everything can be paged out so other
>   code can progress. Something like that. I'm not sure
> here.
>   That is, I suspect we don't care. Virtual address space
> is probably
>   reasonable to deem cheap. No problem using almost 2GB of
> address space
>   as long as working set isn't that large (ie: to fit in
> 31bit address space).
>   If some OSes don't see it that way, oh well. This might
> just
>   be a design/policy thing in Tru64.
>   
>   
> (I'm well aware of virtual memory vs. physical memory vs.
> working set.
> But for whatever reason, we were failing due to memory
> use.
> All that *should* matter is that address space usage stays
> within
> around 2GB and that working set isn't huge. Working set
> I've recently
> come to understand can be managed like so:
>   - random access over set of memory that fits in physical
> memory
>   - sequential access over arbitrary memory (up to 2GB)
>   - hybrid of previous
>       ie. sequential accesses over multiple separate
> areas
> 
> As long as access is sequential, you shouldn't "thrash".
> It is only random access over data that doesn't fit in
> physical memory
>  that causes thrashing.)
> 
> 
>  - Jay
>     
>         
>           
>