[M3devel] Some comments about package structure and package pools, was: Re: ROOT

[Jay]

[didn't really finish editing this..ran out of energe..]

 

 

Cool.

 

 

 > For backward compatibility reasons, there should be a default pool
 > (/usr/local/cm3) which is used when nothing else is defined.


 

"I THINK:" (ie: I'm not sure, winging it..)

 

 

The last pool searched would correspond to where the compiler is.

The first one might be ROOT/pkg.

I'd kind of like it to be outside ROOT entirely, but having it be one toplevel

directory in ROOT is progress.

?

 

An initial version that only supports two pools doesn't rqeuire another setting.

?

 

You know, do-cm3-all realclean would be rm -rf firstpool and optionally

"clone" or "populate" firstpool from secondpool.

Clone/populate might be separate step, such as if I'm cleaning now,

but not building till later.

?

 

 

A system with multiple pools requires more thought than just two pools.

Like, well, you need more parameters.

In a two pool system, promote is always from "private" to "public".

In a multi pool system, you need to specify earlier pool and later pool,

and all the "in between" pools need to be updated as well, or possibly deleted.

 

 

Actually a multi pool system..I'm not sure this works well.

In particular because of $origin.

In particular because the search for shared libraries doesn't necessarily

follow analogously to cm3's search for packages.

 

 

Well, ok, I admit, what probably works well is

 CM3_POOLs=pool1;pool2;pool3 

 LD_LIBRARY_PATH=pool1/lib;pool2/lib:pool3/lib 

 

Then you can have an arbitrary number of pools and they can all

be sparse and you can imagine promoting just a package at a time.

 

If you use $origin, pool/lib can't be sparse.

Perhaps just pool/lib, but all the rest can be sparse.

 

 

Do you promote packages or pools?

I think you actually promote entire pools.

Aha, see, this solves another problem.

This solves cm3 being dynamically linked.

Because you promote an entire pool at a time, multiple packages simultaneously,

you don't have a new cm3 using an old m3core, because you would build both

before any promote.

This also fixes the problem where cm3 can't ship itself, because, something like,

you never run anything in the first pool.

(Similarly it removes the need to build things build_standalone because they

run "unshipped". In fact, it makes it so runpath is just $origin/../lib is ok,

it removes the problem of unshipped binaries living in a directory layout

vs. shipped binaries).

 

 

That defers to the problem. cm3 might be unable to promote itself.

However, first of all, one many systems even NT you can rename and replace.

Subsequent invocations can look for and delete the renamed copies.

 

 

Also, promote might be a wrapper .sh/.cmd, runs the executable, which does

most of the work, and leaves behind another .sh/.cmd for the wrapper to run

as its last step.

 


Any build output that is "shipped" today would instead be written to the first pool.

Non-shipped build outputs, not sure.

 .o/.mo/.io files.

There would probably be another "root" to indicated where unshipped outputs go.

That way the entire source tree is kept clean, not just individual packages.

 

 

> IMO it's OK to ship them all to one directory, but to make things
> work for multi-architecture pools, we'd need to distinguish the
> library and binary paths there, too. So we'd have something like


 

This is a long-standing dilemna of mine.

If multi-architecture pools are the goal, or you just use different roots.

Or at least if you have multi-architecture host pools, or only multi-targets.

 

 

Specifically, bin and lib historically lack architecture, while pkg has it.

This should suffice for such a pool to target multiple architectures but not

host on multiple architectures.

 

 

I've long though that maybe root/bin should only contain .cmd/.py/.sh files

that determine their host architecture and then run root/pkg/foo/target/foo.

Note that this breaks $origin.

You could use root/bin/target/foo, root/lib/target/foo and $origin/../../lib

but I worry when there are too many ".." in $origin, you run the risk

of "escaping" away from "your files" and out into whatever is in the file system.

 

 

This is a constant dilemna for me because I mostly use a host that

supports three Modula-3 hosts.

 

 

My installation is at c:\cm3 and sometimes c:\cm3\bin\cm3.exe

is Cygwin, sometimes it is NT386, and Interix uses c:\cm3\bin\cm3 with

no extension. I don't have a clean workflow here.

 

 

Either I should have c:\cm3 (native), c:\cm3.cygwin, c:\cm3.interix,

or I should have c:\cm3\bin\nt386\cm3, c:\cm3\bin\cygwin\cm3,

c:\cm3\bin\interix\cm3, or somesuch. Again, though, $origin

is a factor here.

 

 

c:\cm3.cygwin et. al. requires no changes and no concert wrt $origin.

Olaf below advocates something more like c:\cm3\bin\nt386\cm3 and

I don't disagree, I'm just not sure.

 

 

"Repository" has many uses.

It is "a place you put things".

Often it is more like "well structured part of the file system",

or deliberately structured.

A pool is more like "a collection from which to draw stuff".

 

 

 - Jay


 
> Date: Fri, 3 Jul 2009 11:22:05 +0200
> From: wagner at elegosoft.com
> To: m3devel at elegosoft.com
> Subject: [M3devel] Some comments about package structure and package pools, was: Re: ROOT
> 
> Quoting Randy Coleburn <rcolebur at scires.com>:
> 
> > Jay:
> >
> > Not sure I fully appreciate/understand your last 3 posts below.
> >
> > I'd also like to hear what Tony and Olaf think about all this.
> 
> I think Jay chiefly opts for a long unimplemented wish of mine:
> a hierarchy of repositories that is searched by the M3 builder.
> Imagine that you can have one global repository, which contains
> some standard and base packages and tools, and several project
> repositories, each of which is used by teams whose members all have
> private repositories for their own test and development purposes.
> 
> A note on terminology: I don't really like to use the term
> `repository' here, as it is usually used in connection with version
> control systems. In ComPact, a tool developed and used years ago by
> Elego, we called these `repositories' which contained pre-built packages
> `package pools' or just `pools'. So I'll use that term instead of
> repository from now on.
> 
> The CM3 builder needs to be extended to search a set of pools
> instead of just one (INSTALL_ROOT). This could be expressed by
> a CM3_POOL_PATH, similar to the Java class path:
> 
> CM3_POOL_PATH=/home/wagner/cm3:/home/projects/banana/cm3:/usr/local/cm3
> 
> To satisfy package imports, the builder would start searching in the
> first pool and continue until it has found an appropriate package.
> This way local or project changes could easily be tested and integrated
> before being `promoted' to a higher level.
> 
> For backward compatibility reasons, there should be a default pool
> (/usr/local/cm3) which is used when nothing else is defined.
> 
> Shipping of packages would by default use the first pool found in the
> CM3_POOL_PATH.
> 
> The second extension needed would be the `promote' operation, which
> ships a package from one pool to another pool. Of course CM3 would
> need to make sure that all dependencies are correct and the shipped
> package does actually work within the new context (the destination
> pool's packages).
> 
> If the package structure is the same in the workspace and in package
> pools, standard shipping and promoting operations would just be
> file copies (after appropriate correctness checks) as Jay says.
> 
> Also, as Jay noticed, the `hack' of m3overrides wouldn't be
> necessary any more.
> 
> I don't think this would be too difficult to implement. I just haven't
> found the time to do it yet ;-)
> 
> Some further remarks to topics touched below:
> 
> Shared libraries weren't in wide-spread use when the M3 package
> systme was designed. Only few of the DEC SRC targets actually
> contained support for them.
> 
> IMO it's OK to ship them all to one directory, but to make things
> work for multi-architecture pools, we'd need to distinguish the
> library and binary paths there, too. So we'd have something like
> 
> root/pkg/pkg-a/src
> /NT386
> /I386_DARWIN
> /pkg-b/src
> /NT386
> /I386_DARWIN
> /man
> /bin/NT386
> /I386_DARWIN
> ...
> /lib/NT386
> /I386_DARWIN
> ...
> 
> Symbolic links were used for dynamic libraries by the related quake
> code as long as I remember.
> 
> The variable ROOT is legacy, I think it was used for overrides even
> by DEC SRC. I've got no objection to rename it, but, as described above,
> we should rather make it vanish.
> 
> Olaf
> 
> > Regards,
> > Randy
> >
> >>>> <jay.krell at cornell.edu> 7/2/2009 9:57 PM >>>
> >
> > Ps this would also obsolete m3overrides, providing similar but 
> > better functionality, no need to encode the source structure in all 
> > those little extra files.
> >
> > - Jay (phone)
> >
> > On Jul 2, 2009, at 6:50 PM, jayk123 at hotmail.com wrote:
> >
> >
> > In particular I would argue that the directory layout should be made 
> > right at link time, into an alternate root. That alternate root 
> > possibly be prepopulated with links to or copies of the current good 
> > shared repository. Or the compiler should have notion of multiple 
> > roots to probe. However running the stuff implies hardlinks that are 
> > broken upon write. This scheme works better if source/interfaces 
> > are not needed in repository, just to cut down on the size. Are 
> > shipped source/interfaces used by compiler, or just the 
> > .m3exports/.m3x files? Anyway this is all thinking for a future 
> > release, not this year. 'ship' becomes just recursive copy or 
> > move.or change roots, plus maybe shipping source/interfaces too, 
> > kind of a wart.
> >
> > - Jay (phone)
> >
> > On Jul 2, 2009, at 6:36 PM, jayk123 at hotmail.com wrote:
> >
> >
> > The system used/uses symlinks under the covers. I don't think cm3 
> > historically supported shared libs on hpux probably because the 
> > bundled compiler does not. Granted my exposure to hpux is only in 
> > recent times. 'standalone' as you define is useful but I think that 
> > reason isn't applicable to anything 'within cm3 itself'. Maybe more 
> > to say later. In particular I think this design we have is flawed. 
> > It's goals are good but can be better achieved slightly differently. 
> > In particular the unshipped and shipped directory layout should be 
> > 'the same' but just differently rooted. Not in this release though. 
> > That let's $origin work better, among other advantages. Related and 
> > possibly solved same IMHO we don't adequately separate source and 
> > output - it should separate across multiple packages not just one at 
> > a time. But again, not on this release.
> >
> > - Jay (phone)
> >
> > On Jul 2, 2009, at 4:14 PM, "Randy Coleburn" <rcolebur at scires.com> wrote:
> >
> >
> > I thought it might be helpful to highlight some of what I've always 
> > understood as the design for the CM3 package system. So, I pulled 
> > out my old documentation from Critical Mass as reference.
> >
> > I offer the following information to see if this discussion thread 
> > concurs and/or wants to make any changes going forward, esp. as we 
> > prepare for a new CM3 release.
> >
> > 1. CM3 takes care to separate source and derived files because 
> > doing so (a) isolates source files for backup, revision control, and 
> > searching; and (b) enables sharing the same source tree across 
> > operating systems and architectures, without confusing object files 
> > from different platforms.
> >
> > 2. Each package resides in a directory, with sources in a source 
> > subdirectory ("src"), and generated files in a derived subdirectory 
> > named to denote the platform on which the sources were built, e.g., 
> > "ALPHA_OSF", "HPPA", "NT386", "SPARC", etc.
> >
> > 3. Each developer can have multiple private packages, each stored 
> > wherever desired in the filesystem. A private package named "foo" 
> > would have the following filesystem structure:
> > +---foo
> > | +---src
> > | +---NT386
> > | +---HPPA
> > | \---(...)
> >
> > 4. For each CM3 installation, there is one public package 
> > repository that is available to all developers. When you ship a 
> > [private] package, you make that package available to other packages 
> > (and developers) via this public package repository. The idea is 
> > that as a developer you would test your package in your private 
> > repository before shipping it to the public repository. (Sometimes 
> > m3overrides were needed when testing several related private 
> > packages before shipping them.)
> >
> > 5. A typical CM3 installation is rooted at a given point in the 
> > file system and contains the following folder structure:
> > CM3
> > +---bin
> > +---doc
> > | +---help
> > | +---pics
> > | +---reference
> > | +---src_reports
> > | \---tutorial
> > +---examples
> > +---lib
> > +---pkg
> > | +---bitvector
> > | | +---src
> > | | +---NT386
> > | | +---HPPA
> > | | \---(...)
> > | +---cm3ide
> > | | +---src
> > | | +---NT386
> > | | +---HPPA
> > | | \---(...)
> > | +---(...)
> > | | +---src
> > | | +---NT386
> > | | +---HPPA
> > | | \---(...)
> >
> > 6. In the past, the environment variable INSTALL_ROOT pointed to 
> > the root of the CM3 tree in the file system. I think this variable 
> > is ambiguously named and that we should change it to 
> > CM3_INSTALL_ROOT. Using this approach, if one had multiple CM3 
> > installations (perhaps for different releases), one would simply 
> > need to change the CM3_INSTALL_ROOT variable to point to the desired 
> > installation for use at any particular point in time. For this to 
> > work, all other variables cue off of CM3_INSTALL_ROOT. I know that 
> > for the old cm3.cfg file, this was indeed the behavior. Then, if 
> > someone had a special situation, they could override the "cueing" 
> > behavior for any particular variable simply by changing its 
> > definition on the fly.
> >
> > 7. Now, I also understand some (but not all) of what Jay is saying 
> > about the library paths. Back in the old cm3 v4.1 days, I had both 
> > HPPA and NT386 derived files for the same set of sources. For 
> > Windows, the shipped exe and dll files went into the public 
> > repository and you needed to have the CM3/bin folder on your path. 
> > For private exe and dlls, you typically just ran them out of the 
> > private package's derived NT386 folder. On HPPA, there were some 
> > places in the file system that contained static libraries and shared 
> > libraries, plus then you had the libraries built using CM3. Again, 
> > the CM3 libraries went to the public repository and there were 
> > environment variables to facilitate finding the rest, including 
> > LD_LIBRARY_PATH. Now, from what Jay is saying, the variety of 
> > operating environments seems to cloud up all this. I know I'm a bit 
> > rusty wrt all the various unix variants out there, but I recall 
> > that v4.1 worked out-of-the-box for both NT386 and HPPA with 
> > respect to all this library path stuff and I didn't have to make 
> > any symbolic links nor any hard links to make it work. IMO, links 
> > are bad and too easily broken.
> >
> > 8. As for "build_standalone", I know there are various 
> > build/bootstrap reasons why some parts of CM3 had to be built this 
> > way. But, for me, I've often used this feature for utility-type 
> > programs to make it easier to distribute them. I can simply 
> > distribute the one executable file without having to worry that the 
> > target system might not have the right DLLs or the right shared 
> > libraries in the right locations. For production code, I've always 
> > built an installation program or an installation script that 
> > installed all my executables and shared libraries in a folder 
> > structure rooted at a particular point chosen by the end user. 
> > Then, my programs always launched and used relative paths from the 
> > install location to find everything they needed.
> >
> > Hope this info is helpful.
> >
> > Regards,
> > Randy Coleburn
> 
> 
> 
> -- 
> Olaf Wagner -- elego Software Solutions GmbH
> Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95
> http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin
> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20090703/c3e70eff/attachment-0002.html>