[M3devel] Looking for CM3 Tinderbox regression test participants, was: Re: M3 concerns

[Tony Hosking]

Score a big win for the regression test framework!  I would not have  
devoted the time to finding the recently fixed race condition for  
mutex/condition initialization in the pthreads-based thread  
implementation if I had not been told about the problem with p007 on  
FreeBSD4.  The problem only revealed itself in a repeatable way on  
FreeBSD4, and not repeatably enough to be debuggable on all the other  
systems I use (I386_DARWIN, PPC_DARWIN, LINUXLIBC6, or Solaris --  
except for a clue in an error report from Alex Bochanek).  Regression  
testing on multiple platforms is practically the only way to shake  
out these sorts of insidious bugs.  Yay!

Good work guys!

On Jan 31, 2008, at 6:43 AM, Olaf Wagner wrote:

> Quoting Olaf Wagner <wagner at elegosoft.com>:
>>  o I think we have agreed on the fact that automatic regression  
>> testing
>>    would help and should be introduced. Tests should be run daily/ 
>> nightly
>>    on as many platforms as possible, and results should be collected
>>    and made available via WWW (elego offers to host these  
>> services, as
>>    it does for CVS repositories and CM3 web access).
>
> As you've surely all seen in the Status section of
>
>   http://modula3.elegosoft.com/cm3/
>
> the Elego Tinderbox regression test framework for CM3 has now been
> working for about two weeks. Though it's still not as complete and
> stable as I'd like it to be, I think it is now the right time for  
> others
> to join the test framework and run the prepared tests in regular
> intervals on their favourite platforms. The results can now be  
> transfered
> to Elego via your ssh account you also use for repository commits.
>
> Currently the tests are running on a Debian Linux system and a  
> FreeBSD 6
> system at Elego, and now and then I'm starting a run on my MacOS X
> laptop. The latter is not ideal for this purpose though.
>
> I'm now looking for other who would be willing to setup nightly tests
> on their own servers. The following systems are of interest
>
>  o PPC_DARWIN on MacOS X 10.4 or 10.5
>  o I386_DARWIN
>  o SOLgnu on any Solaris version
>  o SOLsun "
>  o LINUXLIBC6 on RedHat, Fedora, Ubuntu, Suse Linux (have I forgotten
>    some variants?)
>  o NT386* on Windows 2000/XP/Vista
>  o NetBSD2_i386
>  o FreeBSD4 on older FreeBSD systems (4.x, 5.x) and on current (7.0)
>  o PPC_LINUX (though this seems to be broken for some time)
>  o ALPHA_OSF -- is anybody still using Alphas?
>
> I think the other targets are more or less unused, but would of  
> course not
> object to test results for them.
>
> There's a short description by Kaspar Schleiser of how to  
> participate in
> cm3/scripts/regression/README. Basically you need to checkout the
> scripts, install 5.4.0 manually once, and setup a cron job.
>
> We're mostly interested in the results of `tinderbox-build.sh  
> cm3.build',
> which is the complete bootstrap and release build based on 5.4.0,
> as I think the lastok build need not run on all platforms. It's mainly
> there to detect incompatible changes.
>
> The release build includes the package and m3tests tests.
>
> If you want to participate, I'd suggest to setup everything and run
> it once without transferring your results, then enable the transfer
> in cm3.build:
>
> tinderbox_mailer() {
>         true    # needed if function is emtpy without this...
> # to report to the elego tinderbox host, check README and uncomment  
> this:
> #       cat "$1" | ssh tinderbox.elego.de "sudo -u tinderbox /usr/ 
> local/tinderbox/tinderbox-cgi/processmail_builds.cgi"
> }
>
> That's all. Please let me know in advance if you setup the tests,
>
> Olaf
> -- 
> Olaf Wagner -- elego Software Solutions GmbH
>                Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin,  
> Germany
> phone: +49 30 23 45 86 96  mobile: +49 177 2345 869  fax: +49 30 23  
> 45 86 95
>    http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz:  
> Berlin
> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr:  
> DE163214194
>