<html>
<head>
<style>
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
</style>
</head>
<body class='hmmessage'>
[didn't really finish editing this..ran out of energe..]<BR>
<BR>
<BR>
Cool.<BR>
<BR>
<BR>
> For backward compatibility reasons, there should be a default pool<BR> > (/usr/local/cm3) which is used when nothing else is defined.<BR><BR>
<BR>
"I THINK:" (ie: I'm not sure, winging it..)<BR>
<BR>
<BR>
The last pool searched would correspond to where the compiler is.<BR>
The first one might be ROOT/pkg.<BR>
I'd kind of like it to be outside ROOT entirely, but having it be one toplevel<BR>
directory in ROOT is progress.<BR>
?<BR>
<BR>
An initial version that only supports two pools doesn't rqeuire another setting.<BR>
?<BR>
<BR>
You know, do-cm3-all realclean would be rm -rf firstpool and optionally<BR>
"clone" or "populate" firstpool from secondpool.<BR>
Clone/populate might be separate step, such as if I'm cleaning now,<BR>
but not building till later.<BR>
?<BR>
<BR>
<BR>
A system with multiple pools requires more thought than just two pools.<BR>
Like, well, you need more parameters.<BR>
In a two pool system, promote is always from "private" to "public".<BR>
In a multi pool system, you need to specify earlier pool and later pool,<BR>
and all the "in between" pools need to be updated as well, or possibly deleted.<BR>
<BR>
<BR>
Actually a multi pool system..I'm not sure this works well.<BR>
In particular because of $origin.<BR>
In particular because the search for shared libraries doesn't necessarily<BR>
follow analogously to cm3's search for packages.<BR>
<BR>
<BR>
Well, ok, I admit, what probably works well is<BR>
CM3_POOLs=pool1;pool2;pool3 <BR>
LD_LIBRARY_PATH=pool1/lib;pool2/lib:pool3/lib <BR>
<BR>
Then you can have an arbitrary number of pools and they can all<BR>
be sparse and you can imagine promoting just a package at a time.<BR>
<BR>
If you use $origin, pool/lib can't be sparse.<BR>
Perhaps just pool/lib, but all the rest can be sparse.<BR>
<BR>
<BR>
Do you promote packages or pools?<BR>
I think you actually promote entire pools.<BR>
Aha, see, this solves another problem.<BR>
This solves cm3 being dynamically linked.<BR>
Because you promote an entire pool at a time, multiple packages simultaneously,<BR>
you don't have a new cm3 using an old m3core, because you would build both<BR>
before any promote.<BR>
This also fixes the problem where cm3 can't ship itself, because, something like,<BR>
you never run anything in the first pool.<BR>
(Similarly it removes the need to build things build_standalone because they<BR>
run "unshipped". In fact, it makes it so runpath is just $origin/../lib is ok,<BR>
it removes the problem of unshipped binaries living in a directory layout<BR>
vs. shipped binaries).<BR>
<BR>
<BR>
That defers to the problem. cm3 might be unable to promote itself.<BR>
However, first of all, one many systems even NT you can rename and replace.<BR>
Subsequent invocations can look for and delete the renamed copies.<BR>
<BR>
<BR>
Also, promote might be a wrapper .sh/.cmd, runs the executable, which does<BR>
most of the work, and leaves behind another .sh/.cmd for the wrapper to run<BR>
as its last step.<BR>
<BR>
<BR>Any build output that is "shipped" today would instead be written to the first pool.<BR>
Non-shipped build outputs, not sure.<BR>
.o/.mo/.io files.<BR>
There would probably be another "root" to indicated where unshipped outputs go.<BR>
That way the entire source tree is kept clean, not just individual packages.<BR>
<BR>
<BR>
> IMO it's OK to ship them all to one directory, but to make things<BR>> work for multi-architecture pools, we'd need to distinguish the<BR>> library and binary paths there, too. So we'd have something like<BR><BR>
<BR>
This is a long-standing dilemna of mine.<BR>
If multi-architecture pools are the goal, or you just use different roots.<BR>
Or at least if you have multi-architecture host pools, or only multi-targets.<BR>
<BR>
<BR>
Specifically, bin and lib historically lack architecture, while pkg has it.<BR>
This should suffice for such a pool to target multiple architectures but not<BR>
host on multiple architectures.<BR>
<BR>
<BR>
I've long though that maybe root/bin should only contain .cmd/.py/.sh files<BR>
that determine their host architecture and then run root/pkg/foo/target/foo.<BR>
Note that this breaks $origin.<BR>
You could use root/bin/target/foo, root/lib/target/foo and $origin/../../lib<BR>
but I worry when there are too many ".." in $origin, you run the risk<BR>
of "escaping" away from "your files" and out into whatever is in the file system.<BR>
<BR>
<BR>
This is a constant dilemna for me because I mostly use a host that<BR>
supports three Modula-3 hosts.<BR>
<BR>
<BR>
My installation is at c:\cm3 and sometimes c:\cm3\bin\cm3.exe<BR>
is Cygwin, sometimes it is NT386, and Interix uses c:\cm3\bin\cm3 with<BR>
no extension. I don't have a clean workflow here.<BR>
<BR>
<BR>
Either I should have c:\cm3 (native), c:\cm3.cygwin, c:\cm3.interix,<BR>
or I should have c:\cm3\bin\nt386\cm3, c:\cm3\bin\cygwin\cm3,<BR>
c:\cm3\bin\interix\cm3, or somesuch. Again, though, $origin<BR>
is a factor here.<BR>
<BR>
<BR>
c:\cm3.cygwin et. al. requires no changes and no concert wrt $origin.<BR>
Olaf below advocates something more like c:\cm3\bin\nt386\cm3 and<BR>
I don't disagree, I'm just not sure.<BR>
<BR>
<BR>
"Repository" has many uses.<BR>
It is "a place you put things".<BR>
Often it is more like "well structured part of the file system",<BR>
or deliberately structured.<BR>
A pool is more like "a collection from which to draw stuff".<BR>
<BR>
<BR>
- Jay<BR><BR>
<BR>> Date: Fri, 3 Jul 2009 11:22:05 +0200<BR>> From: wagner@elegosoft.com<BR>> To: m3devel@elegosoft.com<BR>> Subject: [M3devel] Some comments about package structure and package pools, was: Re: ROOT<BR>> <BR>> Quoting Randy Coleburn <rcolebur@scires.com>:<BR>> <BR>> > Jay:<BR>> ><BR>> > Not sure I fully appreciate/understand your last 3 posts below.<BR>> ><BR>> > I'd also like to hear what Tony and Olaf think about all this.<BR>> <BR>> I think Jay chiefly opts for a long unimplemented wish of mine:<BR>> a hierarchy of repositories that is searched by the M3 builder.<BR>> Imagine that you can have one global repository, which contains<BR>> some standard and base packages and tools, and several project<BR>> repositories, each of which is used by teams whose members all have<BR>> private repositories for their own test and development purposes.<BR>> <BR>> A note on terminology: I don't really like to use the term<BR>> `repository' here, as it is usually used in connection with version<BR>> control systems. In ComPact, a tool developed and used years ago by<BR>> Elego, we called these `repositories' which contained pre-built packages<BR>> `package pools' or just `pools'. So I'll use that term instead of<BR>> repository from now on.<BR>> <BR>> The CM3 builder needs to be extended to search a set of pools<BR>> instead of just one (INSTALL_ROOT). This could be expressed by<BR>> a CM3_POOL_PATH, similar to the Java class path:<BR>> <BR>> CM3_POOL_PATH=/home/wagner/cm3:/home/projects/banana/cm3:/usr/local/cm3<BR>> <BR>> To satisfy package imports, the builder would start searching in the<BR>> first pool and continue until it has found an appropriate package.<BR>> This way local or project changes could easily be tested and integrated<BR>> before being `promoted' to a higher level.<BR>> <BR>> For backward compatibility reasons, there should be a default pool<BR>> (/usr/local/cm3) which is used when nothing else is defined.<BR>> <BR>> Shipping of packages would by default use the first pool found in the<BR>> CM3_POOL_PATH.<BR>> <BR>> The second extension needed would be the `promote' operation, which<BR>> ships a package from one pool to another pool. Of course CM3 would<BR>> need to make sure that all dependencies are correct and the shipped<BR>> package does actually work within the new context (the destination<BR>> pool's packages).<BR>> <BR>> If the package structure is the same in the workspace and in package<BR>> pools, standard shipping and promoting operations would just be<BR>> file copies (after appropriate correctness checks) as Jay says.<BR>> <BR>> Also, as Jay noticed, the `hack' of m3overrides wouldn't be<BR>> necessary any more.<BR>> <BR>> I don't think this would be too difficult to implement. I just haven't<BR>> found the time to do it yet ;-)<BR>> <BR>> Some further remarks to topics touched below:<BR>> <BR>> Shared libraries weren't in wide-spread use when the M3 package<BR>> systme was designed. Only few of the DEC SRC targets actually<BR>> contained support for them.<BR>> <BR>> IMO it's OK to ship them all to one directory, but to make things<BR>> work for multi-architecture pools, we'd need to distinguish the<BR>> library and binary paths there, too. So we'd have something like<BR>> <BR>> root/pkg/pkg-a/src<BR>> /NT386<BR>> /I386_DARWIN<BR>> /pkg-b/src<BR>> /NT386<BR>> /I386_DARWIN<BR>> /man<BR>> /bin/NT386<BR>> /I386_DARWIN<BR>> ...<BR>> /lib/NT386<BR>> /I386_DARWIN<BR>> ...<BR>> <BR>> Symbolic links were used for dynamic libraries by the related quake<BR>> code as long as I remember.<BR>> <BR>> The variable ROOT is legacy, I think it was used for overrides even<BR>> by DEC SRC. I've got no objection to rename it, but, as described above,<BR>> we should rather make it vanish.<BR>> <BR>> Olaf<BR>> <BR>> > Regards,<BR>> > Randy<BR>> ><BR>> >>>> <jay.krell@cornell.edu> 7/2/2009 9:57 PM >>><BR>> ><BR>> > Ps this would also obsolete m3overrides, providing similar but <BR>> > better functionality, no need to encode the source structure in all <BR>> > those little extra files.<BR>> ><BR>> > - Jay (phone)<BR>> ><BR>> > On Jul 2, 2009, at 6:50 PM, jayk123@hotmail.com wrote:<BR>> ><BR>> ><BR>> > In particular I would argue that the directory layout should be made <BR>> > right at link time, into an alternate root. That alternate root <BR>> > possibly be prepopulated with links to or copies of the current good <BR>> > shared repository. Or the compiler should have notion of multiple <BR>> > roots to probe. However running the stuff implies hardlinks that are <BR>> > broken upon write. This scheme works better if source/interfaces <BR>> > are not needed in repository, just to cut down on the size. Are <BR>> > shipped source/interfaces used by compiler, or just the <BR>> > .m3exports/.m3x files? Anyway this is all thinking for a future <BR>> > release, not this year. 'ship' becomes just recursive copy or <BR>> > move.or change roots, plus maybe shipping source/interfaces too, <BR>> > kind of a wart.<BR>> ><BR>> > - Jay (phone)<BR>> ><BR>> > On Jul 2, 2009, at 6:36 PM, jayk123@hotmail.com wrote:<BR>> ><BR>> ><BR>> > The system used/uses symlinks under the covers. I don't think cm3 <BR>> > historically supported shared libs on hpux probably because the <BR>> > bundled compiler does not. Granted my exposure to hpux is only in <BR>> > recent times. 'standalone' as you define is useful but I think that <BR>> > reason isn't applicable to anything 'within cm3 itself'. Maybe more <BR>> > to say later. In particular I think this design we have is flawed. <BR>> > It's goals are good but can be better achieved slightly differently. <BR>> > In particular the unshipped and shipped directory layout should be <BR>> > 'the same' but just differently rooted. Not in this release though. <BR>> > That let's $origin work better, among other advantages. Related and <BR>> > possibly solved same IMHO we don't adequately separate source and <BR>> > output - it should separate across multiple packages not just one at <BR>> > a time. But again, not on this release.<BR>> ><BR>> > - Jay (phone)<BR>> ><BR>> > On Jul 2, 2009, at 4:14 PM, "Randy Coleburn" <rcolebur@scires.com> wrote:<BR>> ><BR>> ><BR>> > I thought it might be helpful to highlight some of what I've always <BR>> > understood as the design for the CM3 package system. So, I pulled <BR>> > out my old documentation from Critical Mass as reference.<BR>> ><BR>> > I offer the following information to see if this discussion thread <BR>> > concurs and/or wants to make any changes going forward, esp. as we <BR>> > prepare for a new CM3 release.<BR>> ><BR>> > 1. CM3 takes care to separate source and derived files because <BR>> > doing so (a) isolates source files for backup, revision control, and <BR>> > searching; and (b) enables sharing the same source tree across <BR>> > operating systems and architectures, without confusing object files <BR>> > from different platforms.<BR>> ><BR>> > 2. Each package resides in a directory, with sources in a source <BR>> > subdirectory ("src"), and generated files in a derived subdirectory <BR>> > named to denote the platform on which the sources were built, e.g., <BR>> > "ALPHA_OSF", "HPPA", "NT386", "SPARC", etc.<BR>> ><BR>> > 3. Each developer can have multiple private packages, each stored <BR>> > wherever desired in the filesystem. A private package named "foo" <BR>> > would have the following filesystem structure:<BR>> > +---foo<BR>> > | +---src<BR>> > | +---NT386<BR>> > | +---HPPA<BR>> > | \---(...)<BR>> ><BR>> > 4. For each CM3 installation, there is one public package <BR>> > repository that is available to all developers. When you ship a <BR>> > [private] package, you make that package available to other packages <BR>> > (and developers) via this public package repository. The idea is <BR>> > that as a developer you would test your package in your private <BR>> > repository before shipping it to the public repository. (Sometimes <BR>> > m3overrides were needed when testing several related private <BR>> > packages before shipping them.)<BR>> ><BR>> > 5. A typical CM3 installation is rooted at a given point in the <BR>> > file system and contains the following folder structure:<BR>> > CM3<BR>> > +---bin<BR>> > +---doc<BR>> > | +---help<BR>> > | +---pics<BR>> > | +---reference<BR>> > | +---src_reports<BR>> > | \---tutorial<BR>> > +---examples<BR>> > +---lib<BR>> > +---pkg<BR>> > | +---bitvector<BR>> > | | +---src<BR>> > | | +---NT386<BR>> > | | +---HPPA<BR>> > | | \---(...)<BR>> > | +---cm3ide<BR>> > | | +---src<BR>> > | | +---NT386<BR>> > | | +---HPPA<BR>> > | | \---(...)<BR>> > | +---(...)<BR>> > | | +---src<BR>> > | | +---NT386<BR>> > | | +---HPPA<BR>> > | | \---(...)<BR>> ><BR>> > 6. In the past, the environment variable INSTALL_ROOT pointed to <BR>> > the root of the CM3 tree in the file system. I think this variable <BR>> > is ambiguously named and that we should change it to <BR>> > CM3_INSTALL_ROOT. Using this approach, if one had multiple CM3 <BR>> > installations (perhaps for different releases), one would simply <BR>> > need to change the CM3_INSTALL_ROOT variable to point to the desired <BR>> > installation for use at any particular point in time. For this to <BR>> > work, all other variables cue off of CM3_INSTALL_ROOT. I know that <BR>> > for the old cm3.cfg file, this was indeed the behavior. Then, if <BR>> > someone had a special situation, they could override the "cueing" <BR>> > behavior for any particular variable simply by changing its <BR>> > definition on the fly.<BR>> ><BR>> > 7. Now, I also understand some (but not all) of what Jay is saying <BR>> > about the library paths. Back in the old cm3 v4.1 days, I had both <BR>> > HPPA and NT386 derived files for the same set of sources. For <BR>> > Windows, the shipped exe and dll files went into the public <BR>> > repository and you needed to have the CM3/bin folder on your path. <BR>> > For private exe and dlls, you typically just ran them out of the <BR>> > private package's derived NT386 folder. On HPPA, there were some <BR>> > places in the file system that contained static libraries and shared <BR>> > libraries, plus then you had the libraries built using CM3. Again, <BR>> > the CM3 libraries went to the public repository and there were <BR>> > environment variables to facilitate finding the rest, including <BR>> > LD_LIBRARY_PATH. Now, from what Jay is saying, the variety of <BR>> > operating environments seems to cloud up all this. I know I'm a bit <BR>> > rusty wrt all the various unix variants out there, but I recall <BR>> > that v4.1 worked out-of-the-box for both NT386 and HPPA with <BR>> > respect to all this library path stuff and I didn't have to make <BR>> > any symbolic links nor any hard links to make it work. IMO, links <BR>> > are bad and too easily broken.<BR>> ><BR>> > 8. As for "build_standalone", I know there are various <BR>> > build/bootstrap reasons why some parts of CM3 had to be built this <BR>> > way. But, for me, I've often used this feature for utility-type <BR>> > programs to make it easier to distribute them. I can simply <BR>> > distribute the one executable file without having to worry that the <BR>> > target system might not have the right DLLs or the right shared <BR>> > libraries in the right locations. For production code, I've always <BR>> > built an installation program or an installation script that <BR>> > installed all my executables and shared libraries in a folder <BR>> > structure rooted at a particular point chosen by the end user. <BR>> > Then, my programs always launched and used relative paths from the <BR>> > install location to find everything they needed.<BR>> ><BR>> > Hope this info is helpful.<BR>> ><BR>> > Regards,<BR>> > Randy Coleburn<BR>> <BR>> <BR>> <BR>> -- <BR>> Olaf Wagner -- elego Software Solutions GmbH<BR>> Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany<BR>> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95<BR>> http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin<BR>> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194<BR>> <BR></body>
</html>