[M3devel] m3override

[Jay K]

ps: another suggestion here:
 Don't use overrides in Hudson or package building.
   Instead either build from scratch or with a new install that is a copy of the current (ie. of m3core/libm3). I alread automated this in make-dist.py, which is a based on some other *.sh.
   And then copy/swap at the end, or after some testing.
   Except, er, um, either this is with $ORIGIN, or with relink upon install -- not just copy/swap.
 
 
The central sticking point in "all" this, in software configuration in general -- is "run path".
 
 
 And then, delete all the m3overrides files in the tree. And then optionally remove the entire mechanism, or leave it alone, unused, to bitrot.
 
 
To some extent this is: don't bother implementing a mechanism in cm3, just do it in scripts.
 
> M3 build system for large projects is based on the way the work
> of the single developers can be split up and isolated. And in large

 
Imagine a system composed of five pieces: bottom, almost bottom, middle, almost top, top.
The developer of "top" has it easy, he uses the "existing" bottom, almost bottom, middle, almost top.
What are the developers of any other piece to do?
There are at least two factors to consider:
  run path 
  public interfaces 
 
Even if public interfaces are unchanged, developer building a new "almost top", must either put it where "top" expects, or rebuild top to get it from the new location.
 
 
It has been said that LD_LIBRARY_PATH is for development scenarios. So maybe that is the answer.
 
 
To whatever extent systems are composed of just bottom and top and nobody changes bottom, which I think is actually pretty common, no problem, and people don't even understand the potential problem (I theorize). But that seems..wrong.
 
 
 - Jay
 


From: jay.krell at cornell.edu
To: wagner at elegosoft.com; m3devel at elegosoft.com
Date: Wed, 4 Aug 2010 10:56:13 +0000
Subject: Re: [M3devel] m3override




Specifically it is a warning about overriding the package you are building.
 
 
The thing is, isn't the "bits and pieces overridden" model prone to just not work?
Which pieces do you build vs. which do you leave alone? You need "everything" to be
"fairly" consistent, for some definition of "everything" and "fairly".
 
 
There is a larger unsolved problem here -- which parts of a system can you change
independently of which parts?
 
 
There are two extremes: one thing at a time, or everything at a time.
The truth is generally in between:  some things at a time. Which?
 
 
"Everything" at a time is more difficult the larger you define "everything".
"one" thing at a time, and again "one" possibly varies -- maybe you distribute .o files?
"one" thing at at time leads to a combinatorial explosion of combination of "things"
that are "supposed" to work. With some (enough?) runtime checking that a valid
composition has been made. (Certainly Modula-3 tries much harder here than C.)
 
 
 
I kind of espouse a model of "everything at time", but you can "pre-seed" the new package store/install
with a copy/link of the original (as long you delete before writing over links).
 
 
> work well in this setup (more or less). For one notable exception,
> which others have noticed (who were involved with the original design):
> linking has never been tried implement portably in a general way.

 
Do you mean like, regarding $ORIGIN? Or something else?
I'm too blame for giving us $ORIGIN, and I'm now very ambivalent about it.
libtool + relink upon install seems the better approach and is widely used in practise.
 e.g. OpenBSD.
 This has a side benefit that uninstalled/unshipped dynamic binaries do work.
 
 
I do suspect $ORIGIN was an improvement, since we don't relink upon install.
 
 
There is also a hybrid suggestion of what I have suggested and the current model.
You can still "build" into a new empty directory, with a hierarchy matching the install,
leaving open either later install in-place or "reroot".
 
 
I think, really, I have to back down from $ORIGIN, which restores "pick and chose"
to work when it can, we can still, as I said, build into an install-like structure, and install
by "reroot" or "delete and copy". I think the key is doing relink upon install.
Along with, no longer distributing fully built binaries.
There is a big task here but I think we have seen enough real problems and discussed
enough the theory, to end up at a conclusion -- no $ORIGIN, no binaries, distribute
C source, assembly source (or maybe .o files), and do final link on the target.
 
 
We can distribute some common binaries, like with hardcoded install location.
  The hudon builds might be tricky, since if we build binaries to install to /usr/local/cm3,
  we can't then install and test them. Unless maybe chroot is possible as non-root??
 
 
Possibly Hudson builds would install to $HOME/... and just the final distribution wouldn't.
Probably.
 
 
I kind of..can't think of the word...really don't like this conclusion. Am nervous.
How many people install binaries vs. compile/link?
This would be signing up for much additional headache all around? Compile/link would
fail for lots of people and we'd have to help them a lot more?
Or it's not that bad and we just have to do it?
 
 
 - Jay

 
> Date: Wed, 4 Aug 2010 08:49:30 +0200
> From: wagner at elegosoft.com
> To: m3devel at elegosoft.com
> Subject: Re: [M3devel] m3override
> 
> I wouldn't worry much about the warning Jay removed.
> It only said that an override wasn't used, which is the default
> behaviour of cm3 (without -x or -override option). One could argue
> that if one wanted this warning, one should use an option like
> -warn=overrides or -warn=all (which don't actually exist (yet)).
> 
> However, I'd still like others to think about the two perspectives
> one can take here, and what should be the direction we are tweaking
> our build system.
> 
> Traditionally, the focus was on the package view, i.e. the view of
> one developer, and not a whole system. The builder was designed to
> work well in this setup (more or less). For one notable exception,
> which others have noticed (who were involved with the original design):
> linking has never been tried implement portably in a general way.
> With this focus, overrides are the sole responsibility of the
> developer, and one could argue that he should know exactly what
> he does if he uses them.
> 
> In the CM3 distribution, I used the override feature to allow
> building the whole set of package in the workspace, which can be
> seen equivalent to use a completely different package pool.
> The scripts (which originally were only meant as a tool for the
> distribution maintainer) distinguish between build(local) and
> buildglobal (or buildship). In the first case, the global package
> store isn't involved at all.
> 
> Jay has suggested to optimize towards building the whole distribution,
> but I'm not sure yet if that's the correct direction. Obviously,
> we need to support both perspectives. Much of the charm of the
> M3 build system for large projects is based on the way the work
> of the single developers can be split up and isolated. And in large
> projects there are of course more developers than build or release
> engineers. But then, there seems to have been no real use of CM3 for this
> kind of development for many years; most (all?) active users seem to
> work on their projects alone. So perhaps Jay is right after all?
> 
> Olaf
> -- 
> Olaf Wagner -- elego Software Solutions GmbH
> Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95
> http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin
> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194
> 
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20100804/dd89636e/attachment-0002.html>