[M3devel] compiler behavior under "low memory"?

[Jay K]

Mark had a machine with low memory.
It has more swap now.


Our behavior was:


Compiling m3tk would make good progress, complete many files, then fail for
 lack of memory.


Manual retry would start over due to
 "missing version information" or such.


Thus progress was "impossible".
By editing down m3makefile I was able to make progress.
 Some stuff would fail due to missing interfaces, but
 whatever I left in would get the version information saved.
 
 
Perhaps we should write out the version information every "few" files?
 For some definition of "few"? 10? 10% of the total?
 
 
That would at least let such situations progress via manual retry.
  (unless not even a "few" fit in memory)


I also wonder about reducing memory used, or why indeed we run out,
 but that is a thornier problem. You know, all we should need to
 continue forward progress is to hold all interfaces in memory and
 one implementation at a time. Now, there are a few unknowns here.
 It is possible Mark's machine didn't have enough memory for
 all the relevant interfaces, and that we only "load" them on-demand.

 
 It is also likely, I should add, that virtual address space is the
  problem, not working set. Some systems want to commit room
  in swap for all allocated virtual memory, like, to be sure
  ahead of time that everything can be paged out so other
  code can progress. Something like that. I'm not sure here.
  That is, I suspect we don't care. Virtual address space is probably
  reasonable to deem cheap. No problem using almost 2GB of address space
  as long as working set isn't that large (ie: to fit in 31bit address space).
  If some OSes don't see it that way, oh well. This might just
  be a design/policy thing in Tru64.
  
  
(I'm well aware of virtual memory vs. physical memory vs. working set.
But for whatever reason, we were failing due to memory use.
All that *should* matter is that address space usage stays within
around 2GB and that working set isn't huge. Working set I've recently
come to understand can be managed like so:
  - random access over set of memory that fits in physical memory
  - sequential access over arbitrary memory (up to 2GB)
  - hybrid of previous
      ie. sequential accesses over multiple separate areas

As long as access is sequential, you shouldn't "thrash".
It is only random access over data that doesn't fit in physical memory
 that causes thrashing.)


 - Jay