[M3devel] Hudson structure?

[Olaf Wagner]

Quoting Jay K <jay.krell at cornell.edu>:

>
>> I thought I had explained that some time ago, but maybe I forgot. So here
>> is the gist:
>>
>> o current, last-ok, prev-ok are all complete package pools + compiler that
>> contain a complete cm3 installation
>> o for builds, current is initialized from last-ok; then packages
>> are built and shipped to current
>> o if a build succeeds:
>> - prev-ok is replaced by last-ok
>> - last-ok is replaced by current
>> Thus we should always have a backup for recovery in case something
>> goes wrong. And last-ok should always be exactly what its name
>> indicates (the last working version) (it seems that this invariant
>> sometimes does not hold :-/)
>> o tests usually just use last-ok
>
> Do we need all this?
> I guess it is ok. There should be just two when idle and three   
> during a build.
>  But there are more than that currently, more than what you list.

test and build jobs may run in parallel, I think test-all-pkgs
uses its own current.

> But we don't need them to be complete.
> They can all just contain cm3, cm3cg, mklib (for a few targets),   
> m3core, libm3.
>
> More generally I thinking of digging into the Hudson stuff.
> I think I know what to do about cm3cg, at least.
>
> There are two kinds of "ships" or installs.
> There is ship/install on top of the "currently running" install,
> and there is install to other than it, possibly to a directory that   
> started empty (recently).
>
> Shipping of cm3 doesn't do anything.
> This could be viewed as a limitation -- the comments indicate it is   
> at least partly
> due to shipping in-use files doesn't work on some operating systems   
> (not just Windows,
> in fact, it is easy to do on Windows -- rename away first).
> However it is also a good thing.
>
> Shipping of m3cc/cm3g does do something.

Yes. That's not consequent. I seem to remember that there were
objections to _not_ ship cm3cg to bin though.

> I assert that shipping either of cm3 or cm3cg on top of the   
> "currently running"
> install is wrong.
>
> Shipping into other, a new empty directory, is perfectly fine.
> In fact, a good metric might be, in both cases, to ship as long as   
> the destination doesn't yet exist.
>
> For the "currently running" install, ship should do nothing, and   
> should instead be done
> "outside the usual system" via sh/python.
> Exactly as we do for cm3 already. But that should also be how cm3cg   
> is dealt with.
> And the config files (I suspect also already the case).
>
> That way they can be updated together.  *** the very important point **

Yes.

> Related question: now that we have far less frequent builds, can we   
> merge the m3cc and build Hudson tasks?

I'd rather avoid that. The m3cc builds takes hours on some systems,
and _usually_ (unless someone is working on the backend) this should
not need to be built at all.

On the other hand: you may be right. We could easily just disable all
the m3cc jobs for the time being and always build m3cc together with
the other core packages in the -build- jobs.

> I'd like there to be as few of tasks as possible.
> "Multiplication is bad", so to speak. Multiplying work, combinations  
>  to test, etc.
> Though granularity is also nice -- if some things are broken, still   
> nice to see some tasks succeeding.
>
> Related: we currently keep all snapshots on the build machines.
> We shouldn't keep so many.

Do we? We shouldn't keep any on the build machine. Snapshots are
(should be?) copied to birch.elegosoft.com and kept there for some
time.

Olaf
-- 
Olaf Wagner -- elego Software Solutions GmbH
                Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
phone: +49 30 23 45 86 96  mobile: +49 177 2345 869  fax: +49 30 23 45 86 95
    http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin
Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194