[M3devel] m3override

[Jay K]

 > I remember using an override somewhere ... but I don't remember where or 
 > why now.
 
 
So..it sounds like maybe you don't know what overrides are.
That is ok.
But you can't then reason about changes to them.
Including claiming that all changes are scary or bad.
Not that you did, but many people are prone to that way of thinking.
 
 
 > But I'd worry if this change broke something.
 
 
Sure. You should sorry about anything that breaks anything.
But this doesn't break anything.
And, if it did, you'd have to apply context-sensitive heuristics.
Does it break anything that matters? Matters much?
What is the cost of maintaining the code to preserve "everything" "working"?
Were the definitions of "working" correct in the first place?
Does widespread expectation make something correct?
 
 
Realize that for nearly any given change, I can tell you something that it breaks.
rmrec wasn't working. But I could easily theorize code that depended on it not working.
  It not working was a way to detect certain characteristics of the filesystem -- presence of symlinks, possible behaviors of readdir+unlink.
 
 
LONGINT didn't exist, in the base language. But I was using LONGINT as a type already, TYPE LONGINT = ARRAY [0..1] OF INTEGER.
  (No, I wasn't actually.)
Putting it in the language broke such code.
 
 
Time passing breaks code. Seriously. (Year2000, Year2038).
But you can't really stop it. Except by changing the time on your computer to be wrong.
 
 
In either case, I put the useless warning back and changed our master m3override to ignore it.
That is *not* evidence of the utility of the warning, that it made me change code to ignore.
There is not actual improvement to the code by avoiding the warning.
 
 
 
To understand this stuff, it helps to consider make and make install and chroot.
  Can I run stuff I haven't run "make install" on? Sometimes. Can I build my own libc.so without chroot? Sometimes.
Modula-3 merely reinveints those sorts of things itself.
Those things are flawed and Modula-3 fixes some but not all of the flaws.
while still leaving major problems present.
   It is this consistent not fixing all the problems that makes me consider it not worth being so different..
Ultimately chroot is the heavyweight reliable solution.
And/or strictly building things -- everything -- from the bottom of the dependency tree and up.
 
 
In the Modula-3 build system, there is a "package store".
e.g. /usr/local/cm3/pkg.
That contains packages, e.g. /usr/local/cm3/pkg/m3core.
 
 
When you say "import(m3core)", it has to find m3core somewhere.
The default is package store + package name -- e.g. /usr/local/cm3/pkg/m3core.
 
 
If you are building package foo, let's say at /usr/src/foo, and you want it to use some
custom m3core, which is at /usr/src/m3core then you create
 
/usr/src/foo/src/m3overrides that says override("m3core", "/usr/src")
 
something like that.
 
This is deceptive though. You sort of can't actually do this. Not so easily.
 You can only really do it if everything in the dependency tree between foo and m3core were also built using the same overrides.
   You need consistent definitions of types, interfaces, etc. But granted, if you only changed implementation, you can maybe do it.
   Of course, static linking interferes as well. Are you using stubgen? Should it pickup the new m3core? Then you have to build it too.
Which is to say, you can really only use entire alterate package stores. You can't really pick and choice packages from one place or another. Unless you are very careful. And none of our codebases are particularly large anyway. You might as well just make whatever source changes in any package and build it all to alternative package store. Even relying on good albeit buggy incrementality in cm3.
 
 
Another way to view this, the way we use it in Hudson et. al. is that there are precisely two package stores.
/usr/local/cm3/pkg
and the "uninstalled" or "unshipped" /usr/cvs/cm3/m3-libs/m3core, /usr/cvs/cm3/m3-sys/m3quake etc.
 
 
The m3overrides file points into the CVS checkout, which, regrettably, also contains all of our outputs.
 (see also: "read only source tree", which we sort of have, sort of are/were ahead of everyone else, but not really..)
 
 
cm3 -x uses the m3overrides files. Cm3 without -x does not.
 
 
And we use it as I described -- not to pick and chose packages from one place or another.
But to pick and chose all packages from one overall tree or another.
 
 
And then you run into the potential problem that the two directories have a different layout.
So stuff like rpath $ORIGIN/../lib doesn't work.
 
 
Ultimately, often people want binaries to work without running make install.
And they often do.
Ultimately chroot is how you really build custom stuff. But it seems onerous. I have almost never done it.
 
 
We can cheat, because we don't build ld.so, libc.so. We only need "chroot" from a certain point and on up.
 
I suggest instead of overrides, instead:
  1) have the "intermediate" (not installed) files go into a layout like the current install, not next to the source as currently 
    This is the .a/.so/.exe files, not the .o/.obj/.mo/.io files. .o/.obj/.mo/.io belong somewhere else.
  2) "install" is merely deleting one root and copying or renaming the other over it 
  3) no more notion of overrides 
 
 
If you don't want to build "everything", you can instead copy or hardlink-tree the original.
 (symlinks won't do, not used with $ORIGIN; though maybe we should abandon $ORIGIN? So many systems don't support it, it turns out).
 
 
There are tradeoffs either way.
 
 
In this scheme, rpath $ORIGIN/../lib works without install.
You would no longer have executables in the package store, only in cm3root/bin.
We could stop statically linking anything.
 
 
However you do lose "hierarchical naming".
 
 
Well, maybe. Maybe executables in the package store would use rpath $ORIGIN/../../lib.
Heck, maybe the model is everything is in the package store and bin and lib only contain symlinks?
 
 
(the warning still hasn't been explained, or maybe I'm behind on email)
 
 
 - Jay


> Date: Fri, 30 Jul 2010 17:50:51 -0400
> From: hendrik at topoi.pooq.com
> To: m3devel at elegosoft.com
> Subject: Re: [M3devel] [M3commit] CVS Update: cm3
> 
> On Fri, Jul 30, 2010 at 01:58:45AM +0000, Jay K wrote:
> > 
> > Randy, Without reading the code, what does it mean?
> > 
> > My first guess was I believe wrong and based on reading the code I think it is even more pointless than I previously thought.
> > 
> > It gets printed *all the time*, when using overrides. Look at any of the Hudson logs.
> > 
> > It wouldn't be printed so much in the older way of doing things, where overrides were applied haphazardly and incompletely. Now that they are used fairly consistently/correctly, it always triggers. I hope to eventually replace the overrides mechanism -- if we just put files into a designated root directory, with the same structure as "the install", then "override" is just setting that directory, optionally prepopulating with an entire copy of the current install -- depending on if you want to build up from scratch or selectively override. You could use hardlinks to speed it up, as long as our copy is delete + copy and not just copy (sometimes a copy will share with the original link(s), sometimes it will unshare, depending on the open flags (I think) or if you first delete). And "install" becomes just moving the install root, if the new one is complete.
> 
> I remember using an override somewhere ... but I don't remember where or 
> why now. But I'd worry if this change broke something.
> 
> -- hendrik
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20100804/c470ef45/attachment-0001.html>