[M3devel] compiler behavior under "low memory"?

[Olaf Wagner]

Perhaps a small limit was set on the system? (bash: ulimit, csh: limit)

As for a fix, perhaps we could catch the exception in the builder and
write out all known version information acquired so far (FINALLY block)?

Olaf

Quoting Jay K <jay.krell at cornell.edu>:

> Mark had a machine with low memory.
> It has more swap now.
>
> Our behavior was:
>
> Compiling m3tk would make good progress, complete many files, then fail for
>  lack of memory.
>
> Manual retry would start over due to
>  "missing version information" or such.
>
> Thus progress was "impossible".
> By editing down m3makefile I was able to make progress.
>  Some stuff would fail due to missing interfaces, but
>  whatever I left in would get the version information saved.
>  
> Perhaps we should write out the version information every "few" files?
>  For some definition of "few"? 10? 10% of the total?
>  
>  
> That would at least let such situations progress via manual retry.
>   (unless not even a "few" fit in memory)
>
>
> I also wonder about reducing memory used, or why indeed we run out,
>  but that is a thornier problem. You know, all we should need to
>  continue forward progress is to hold all interfaces in memory and
>  one implementation at a time. Now, there are a few unknowns here.
>  It is possible Mark's machine didn't have enough memory for
>  all the relevant interfaces, and that we only "load" them on-demand.
>
>  
>  It is also likely, I should add, that virtual address space is the
>   problem, not working set. Some systems want to commit room
>   in swap for all allocated virtual memory, like, to be sure
>   ahead of time that everything can be paged out so other
>   code can progress. Something like that. I'm not sure here.
>   That is, I suspect we don't care. Virtual address space is probably
>   reasonable to deem cheap. No problem using almost 2GB of address space
>   as long as working set isn't that large (ie: to fit in 31bit   
> address space).
>   If some OSes don't see it that way, oh well. This might just
>   be a design/policy thing in Tru64.
>  
>  
> (I'm well aware of virtual memory vs. physical memory vs. working set.
> But for whatever reason, we were failing due to memory use.
> All that *should* matter is that address space usage stays within
> around 2GB and that working set isn't huge. Working set I've recently
> come to understand can be managed like so:
>   - random access over set of memory that fits in physical memory
>   - sequential access over arbitrary memory (up to 2GB)
>   - hybrid of previous
>       ie. sequential accesses over multiple separate areas
>
> As long as access is sequential, you shouldn't "thrash".
> It is only random access over data that doesn't fit in physical memory
>  that causes thrashing.)
>
>
>  - Jay
>



-- 
Olaf Wagner -- elego Software Solutions GmbH
                Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
phone: +49 30 23 45 86 96  mobile: +49 177 2345 869  fax: +49 30 23 45 86 95
    http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin
Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194