[M3devel] ship vs. where build outputs go, etc.?

[Olaf Wagner]

Quoting Jay <jay.krell at cornell.edu>:

>
> The point is it would not always be the "public" repositoy.
>
>
> Today we have two places, the source tree and the repository.
> They have different structures.
>
>
> Instead, you could just have multiple repositories.
> They would all have the same structure.
>
>
> "Install" or "ship" is just recursive copy from a "private"
> repository to the "public" repository, instead of
> arbitrary rearrangement of files.

This does only work reliably if you keep exact version info for
all files in the derived package pools (I'd like to reserve the term
repository for source repository). A repository contains the source code
(complete with history), while package pools contain derived files
(intermediate results). A third (complementing) notion would be a
deployable package (an installable archive or package).

CM3 checks the consistency of one build wrt. a package pool and allows
shipping into this pool only if everything is OK. Thus any override must
inhibit any shipping, as it is not guaranteed that the overriden packages
are available in the pool (in this version).

If you ship packages from one pool to the other, there is no consistency
check at all, unless you know exactly what versions of packages are
in both pools.

> OR merely "blessing" a private repository and making it the new
> public repository, such as by setting and an environment variable
> or possibly delete/rename.
>
> You could get something closer to the current experience by having
> two environment variables, CM3_PUBLIC, CM3_PRIVATE.
> cm3 -buildship would output to CM3_PUBLIC.
> cm3 -build would output to CM3_PRIVATE.
> Ideally it would do no extra writes vs. the current scheme.

I've got no problem with multiple package pools, but we need to
consider carefully what operations will be needed and which ones
are safe.

> "Upgrade" would look like:
>
>  set CM3_PRIVATE= some new temporary place
>  build everything
>  if successful
>    rm -rf CM3_PUBLIC
>    mv CM3_PUBLIC CM3_PRIVATE
>
>  See, today "upgrade" writes over the live system and is therefore
>   "dangerous". In this new scheme, "ship" would lose much of its
>   danger. Rather than making changes as it goes, the final "commit"
>   would not occur until the end, and would be easier to keep backups
>   and such. Granted, if you look at what make-dist does, this is
>   already possible today, by twiddling with CM3_INSTALL and always
>   shipping. Though that costs extra I/O and you do have to prepopulate
>   CM3_INSTALL..at least with a compiler, eh, just two files.
>   Also with m3core/libm3 in some bootstrapping scenarios.
>    (make-dist assumes so, though it usually is not the case).

You mix concepts from the base CM3 builder with concepts from the
maintenance scripts. I never intended these scripts to be used as
a general build system for the everyday M3 user. Indeed it does not
keep any of the guarantees that M3 gives.

>  Along with an alternate root for intermediate outputs, this would
>   enable multiple concurrent builds as well.
>
>  A similar proposal is that "CM3_PUBLIC" could be colon or semicolon
>  delimited -- a "search path" for reads, write to the first location.
>  This would probably be confusing if it contained more than two paths though.
>  publlic/private is "limited" in that it only supports two, but you can
>   always use recursive copy to merge.

I could imagine the following extensions (without deep consideration of
the consequences):

  o allow to ship to more than one package pool (but keep it consistent
    with the builds)

  o allow multiple package pools for imports (e.g. by using a package pool
    path)

  o allow shipping only if all imports come from the destination package
    pool (as it is now)

>  Fixing clean would not take away realclean as an option if that is necessary
>  for compat (I forget if compiler even implements realclean; the scripts
>  implement it and I always use that version). I find it hard to believe
>  that anyone depends on that clean leaves a few files around.
>  Which files does it even leave around?
>
>  I'm not sure what is documented or not. Could be that a lot is?

cm3 -clean should be correct insofar as it removes all derived files
(that the compiler knows of). It only works if all imports can be found,
which also means that cleaning must be done in the correct order. As
the builder currently doesn't work on package sets but only on one
package this turns out to be little useful for large projects. Thus
the maintenance scripts introduced the realclean command. Again, this
was not intended as a general end user solution.

Olaf
-- 
Olaf Wagner -- elego Software Solutions GmbH
                Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
phone: +49 30 23 45 86 96  mobile: +49 177 2345 869  fax: +49 30 23 45 86 95
    http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin
Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194