[M3devel] Integration of package-local tests

[Olaf Wagner]

As some of you may have noticed, the regression test scope is
increasing steadily though slowly. During recent days I tried to
integrate the existing tests that are local to some packages.
It's working this way:

During a complete package build run with

   ./scripts/do-cm3-all.sh -report build

all pacakges are compiled and the results reported on the web
server (see http://www.opencm3.net/logs/package-status.html).
For an easy overview, packages that compile OK are coloured green,
packages with errors are red, and those unsupported on a certain
platform are marked yellow.

Recently a fourth column has been added which reports the results
of tests if they are present. Tests are assumed if in the package
root a sub directory test or tests exists, and if beneath that
src/m3makefile is found. The fourth column will be yellow if this
is not the case.

If tests are present, they are compiled and (probably) run with

   cm3 -build -override -DROOT=$ROOT -DRUN

A sample tests/src/m3makefile from package arithmetic looks like this:

   import("libm3")
   import("tempfiles")
   import("arithmetic")

   implementation("TestWordEx")
   implementation("TestBits")
   %[more test sources omitted]
   implementation("TestFunctional")

   Module("Test")
   program("test")

   if defined("RUN")
     r = q_exec( "." & SL & "test" )
     write( CR, "program returned ", r, CR )
   end

The last four lines will run the compiled test executable. Of course,
more complex setups are possible there.
The test result column will be red if the tests do not compile or
another fatal error is detected, orange, if something is output to
stderr, and green in all other cases.

At least for the actual test candidate an appropriate override
should be defined like this:

   override("arithmetic", PACKAGE_DIR & SL & ".." & SL & "..")

As usual, this should be done in the src/m3overrides file.

To give a slightly more complex example, the tests for patternmatching
are performed thus:

   if defined("RUN")
     r = q_exec( "." & SL & "test-regex -unix < .." & SL & "tests.input" )
     write( CR, "program returned ", r, CR )
     r = q_exec( "." & SL & "test-regex -unix < .." & SL &  
"testsdiffer.input" )
     write( CR, "program returned ", r, CR )
   end

Here, two test runs are made with different input files.

I'd like to encourage all contributors to adapt or extend their packages
accordingly, so that more things can be automatically validated every
night on miscellaneous platforms.

Of course it would also be great if the encountered errors get fixed :-)

Thanks for your attention and cooperation,

Olaf

PS: I'll attach one current package report file for those too lazy
     to follow the links ;-)
PPS: Of course you can run the tests and generate the reports locally,
     too. During development I've used something like

       HTML_REPORT=/var/tmp/cm3-pkg-report-FreeBSD4-2008-02-17.html \
       time ./scripts/do-cm3-all.sh -report build 2>&1 | tee plog

     to generate a complete report with a fixed name and log all
     actions in plog.
PPPS: The actual implementation of report generation is in
     scripts/pkgmap.sh. So you can use -report with almost all
     build scripts.
-- 
Olaf Wagner -- elego Software Solutions GmbH
                Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
phone: +49 30 23 45 86 96  mobile: +49 177 2345 869  fax: +49 30 23 45 86 95
    http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin
Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20080217/5ce9a3b1/attachment-0001.html>