[M3devel] Some comments about package structure and package pools, was: Re: ROOT

[Olaf Wagner]

Quoting Randy Coleburn <rcolebur at scires.com>:

> Jay:
>
> Not sure I fully appreciate/understand your last 3 posts below.
>
> I'd also like to hear what Tony and Olaf think about all this.

I think Jay chiefly opts for a long unimplemented wish of mine:
a hierarchy of repositories that is searched by the M3 builder.
Imagine that you can have one global repository, which contains
some standard and base packages and tools, and several project
repositories, each of which is used by teams whose members all have
private repositories for their own test and development purposes.

A note on terminology: I don't really like to use the term
`repository' here, as it is usually used in connection with version
control systems. In ComPact, a tool developed and used years ago by
Elego, we called these `repositories' which contained pre-built packages
`package pools' or just `pools'. So I'll use that term instead of
repository from now on.

The CM3 builder needs to be extended to search a set of pools
instead of just one (INSTALL_ROOT). This could be expressed by
a CM3_POOL_PATH, similar to the Java class path:

   CM3_POOL_PATH=/home/wagner/cm3:/home/projects/banana/cm3:/usr/local/cm3

To satisfy package imports, the builder would start searching in the
first pool and continue until it has found an appropriate package.
This way local or project changes could easily be tested and integrated
before being `promoted' to a higher level.

For backward compatibility reasons, there should be a default pool
(/usr/local/cm3) which is used when nothing else is defined.

Shipping of packages would by default use the first pool found in the
CM3_POOL_PATH.

The second extension needed would be the `promote' operation, which
ships a package from one pool to another pool. Of course CM3 would
need to make sure that all dependencies are correct and the shipped
package does actually work within the new context (the destination
pool's packages).

If the package structure is the same in the workspace and in package
pools, standard shipping and promoting operations would just be
file copies (after appropriate correctness checks) as Jay says.

Also, as Jay noticed, the `hack' of m3overrides wouldn't be
necessary any more.

I don't think this would be too difficult to implement. I just haven't
found the time to do it yet ;-)

Some further remarks to topics touched below:

Shared libraries weren't in wide-spread use when the M3 package
systme was designed. Only few of the DEC SRC targets actually
contained support for them.

IMO it's OK to ship them all to one directory, but to make things
work for multi-architecture pools, we'd need to distinguish the
library and binary paths there, too. So we'd have something like

root/pkg/pkg-a/src
               /NT386
               /I386_DARWIN
         /pkg-b/src
               /NT386
               /I386_DARWIN
     /man
     /bin/NT386
         /I386_DARWIN
         ...
     /lib/NT386
         /I386_DARWIN
         ...

Symbolic links were used for dynamic libraries by the related quake
code as long as I remember.

The variable ROOT is legacy, I think it was used for overrides even
by DEC SRC. I've got no objection to rename it, but, as described above,
we should rather make it vanish.

Olaf

> Regards,
> Randy
>
>>>> <jay.krell at cornell.edu> 7/2/2009 9:57 PM >>>
>
> Ps this would also obsolete m3overrides, providing similar but   
> better functionality, no need to encode the source structure in all   
> those little extra files.
>
>  - Jay (phone)
>
> On Jul 2, 2009, at 6:50 PM, jayk123 at hotmail.com wrote:
>
>
> In particular I would argue that the directory layout should be made  
>  right at link time, into an alternate root. That alternate root   
> possibly be prepopulated with links to or copies of the current good  
>  shared repository. Or the compiler should have notion of multiple   
> roots to probe. However running the stuff implies hardlinks that are  
>  broken upon write. This scheme works better if source/interfaces  
> are  not needed in repository, just to cut down on the size. Are  
> shipped  source/interfaces used by compiler, or just the  
> .m3exports/.m3x  files? Anyway this is all thinking for a future  
> release, not this  year. 'ship' becomes just recursive copy or  
> move.or change roots,  plus maybe shipping source/interfaces too,  
> kind of a wart.
>
>  - Jay (phone)
>
> On Jul 2, 2009, at 6:36 PM, jayk123 at hotmail.com wrote:
>
>
> The system used/uses symlinks under the covers. I don't think cm3   
> historically supported shared libs on hpux probably because the   
> bundled compiler does not. Granted my exposure to hpux is only in   
> recent times. 'standalone' as you define is useful but I think that   
> reason isn't applicable to anything 'within cm3 itself'. Maybe more   
> to say later. In particular I think this design we have is flawed.   
> It's goals are good but can be better achieved slightly differently.  
>  In particular the unshipped and shipped directory layout should be   
> 'the same' but just differently rooted. Not in this release though.   
> That let's $origin work better, among other advantages. Related and   
> possibly solved same IMHO we don't adequately separate source and   
> output - it should separate across multiple packages not just one at  
>  a time. But again, not on this release.
>
>  - Jay (phone)
>
> On Jul 2, 2009, at 4:14 PM, "Randy Coleburn" <rcolebur at scires.com> wrote:
>
>
> I thought it might be helpful to highlight some of what I've always   
> understood as the design for the CM3 package system.  So, I pulled   
> out my old documentation from Critical Mass as reference.
>
> I offer the following information to see if this discussion thread   
> concurs and/or wants to make any changes going forward, esp. as we   
> prepare for a new CM3 release.
>
> 1.  CM3 takes care to separate source and derived files because   
> doing so (a) isolates source files for backup, revision control, and  
>  searching; and (b) enables sharing the same source tree across   
> operating systems and architectures, without confusing object files   
> from different platforms.
>
> 2.  Each package resides in a directory, with sources in a source   
> subdirectory ("src"), and generated files in a derived subdirectory   
> named to denote the platform on which the sources were built, e.g.,   
> "ALPHA_OSF", "HPPA", "NT386", "SPARC", etc.
>
> 3.  Each developer can have multiple private packages, each stored   
> wherever desired in the filesystem.  A private package named "foo"   
> would have the following filesystem structure:
>    +---foo
>    |   +---src
>    |   +---NT386
>    |   +---HPPA
>    |   \---(...)
>
> 4.  For each CM3 installation, there is one public package   
> repository that is available to all developers.  When you ship a   
> [private] package, you make that package available to other packages  
>  (and developers) via this public package repository.  The idea is   
> that as a developer you would test your package in your private   
> repository before shipping it to the public repository.  (Sometimes   
> m3overrides were needed when testing several related private   
> packages before shipping them.)
>
> 5.  A typical CM3 installation is rooted at a given point in the   
> file system and contains the following folder structure:
>    CM3
>    +---bin
>    +---doc
>    |   +---help
>    |   +---pics
>    |   +---reference
>    |   +---src_reports
>    |   \---tutorial
>    +---examples
>    +---lib
>    +---pkg
>    |   +---bitvector
>    |   |   +---src
>    |   |   +---NT386
>    |   |   +---HPPA
>    |   |   \---(...)
>    |   +---cm3ide
>    |   |   +---src
>    |   |   +---NT386
>    |   |   +---HPPA
>    |   |   \---(...)
>    |   +---(...)
>    |   |   +---src
>    |   |   +---NT386
>    |   |   +---HPPA
>    |   |   \---(...)
>
> 6.  In the past, the environment variable INSTALL_ROOT pointed to   
> the root of the CM3 tree in the file system.  I think this variable   
> is ambiguously named and that we should change it to   
> CM3_INSTALL_ROOT.  Using this approach, if one had multiple CM3   
> installations (perhaps for different releases), one would simply   
> need to change the CM3_INSTALL_ROOT variable to point to the desired  
>  installation for use at any particular point in time.  For this to   
> work, all other variables cue off of CM3_INSTALL_ROOT.  I know that   
> for the old cm3.cfg file, this was indeed the behavior.  Then, if   
> someone had a special situation, they could override the "cueing"   
> behavior for any particular variable simply by changing its   
> definition on the fly.
>
> 7.  Now, I also understand some (but not all) of what Jay is saying   
> about the library paths.  Back in the old cm3 v4.1 days, I had both   
> HPPA and NT386 derived files for the same set of sources.  For   
> Windows, the shipped exe and dll files went into the public   
> repository and you needed to have the CM3/bin folder on your path.    
> For private exe and dlls, you typically just ran them out of the   
> private package's derived NT386 folder.  On HPPA, there were some   
> places in the file system that contained static libraries and shared  
>  libraries, plus then you had the libraries built using CM3.  Again,  
>  the CM3 libraries went to the public repository and there were   
> environment variables to facilitate finding the rest, including   
> LD_LIBRARY_PATH.  Now, from what Jay is saying, the variety of   
> operating environments seems to cloud up all this.  I know I'm a bit  
>  rusty wrt all the various unix variants out there, but I recall  
> that  v4.1 worked out-of-the-box for both NT386 and HPPA with  
> respect to  all this library path stuff and I didn't have to make  
> any symbolic  links nor any hard links to make it work.  IMO, links  
> are bad and  too easily broken.
>
> 8.  As for "build_standalone", I know there are various   
> build/bootstrap reasons why some parts of CM3 had to be built this   
> way.  But, for me, I've often used this feature for utility-type   
> programs to make it easier to distribute them.  I can simply   
> distribute the one executable file without having to worry that the   
> target system might not have the right DLLs or the right shared   
> libraries in the right locations.  For production code, I've always   
> built an installation program or an installation script that   
> installed all my executables and shared libraries in a folder   
> structure rooted at a particular point chosen by the end user.    
> Then, my programs always launched and used relative paths from the   
> install location to find everything they needed.
>
> Hope this info is helpful.
>
> Regards,
> Randy Coleburn



-- 
Olaf Wagner -- elego Software Solutions GmbH
                Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
phone: +49 30 23 45 86 96  mobile: +49 177 2345 869  fax: +49 30 23 45 86 95
    http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin
Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194