[M3devel] M3 concerns / automated regression tests

Sun Jan 6 20:39:03 CET 2008

Quoting Randy Coleburn <rcoleburn at scires.com>:

> Question:  As various people make contributions and changes, is   
> there any group of folks who is verifying that the changes don't   
> break existing code or core principles?  Another idea would be to   
> have a series of test batteries that have to be passed before   
> changes can be accepted.  I know of a number of projects where many   
> people contribute, but the contributions have to be vetted by   
> passing some tests.
>
> Without these types of controls, I fear that good forward progress   
> will inevitably be hampered or compromised by changes made by   
> someone who hasn't completely thought out and tested the   
> ramifications of those changes.

As announced in other mails, I'd like to get into some details
of the regression test standardization I suggest. As you may have
noticed, several scripts have been committed to cm3/scripts/regression/.

There are some scripts for the integration of tests into the elego
tinderbox framework, which are still work in progress. Kaspar Schleiser
is working on this; it may take some days before everything works
as expected.

The main functions for test standardization are contained in the
file defs.sh. This is a shell script which can be sourced and used
in different contexts. It contains functions for CVS checkout,
binary installation, and the actual tests.

In my opinion, the most important thing that should be ensured is
that the current sources can be compiled with the latest official
release of the system. This can get tricky in case of incompatible
changes in the compiler runtime, frontend, or configuration (cm3.cfg).
So the main part of test_build_core_rel() is an upgrade performed
by scripts/upgrade.sh based on the last release. In order to use
this, it must of course be installed. This can be done with
install_bin_dist().

The test framework uses three installations of CM3, located in
INSTBASE, which defaults to ${HOME}/work/cm3-inst. There are

   rel-${LASTREL}, e.g. rel-5.4.0
   current
   last-ok

rel-x.x.x should contain at least a minimal installation of the last
official release. It may have to be installed manually, as the installer
was interactive until recently. (This has been changed in d5.5.1.)

current always contains the CM3 installation which is used for actual
shipping during test builds and is usually based on the current sources.
last-ok is always updated with the last successfully built version.

The workspaces which are used for checkout and compilation contain
date stamps in their name, eg. ~/work/cm3-ws-2008-01-04-20-05-06/.
They can be set with the variables DS and WS:

   DS=${DS:-`date -u +'%Y-%m-%d-%H-%M-%S' | tr -d '\\n'`}
   WS=${WS:-${HOME}/work/cm3-ws-${DS}}

Apart from building with the latest release, there are currently
the following tests automated:

  o building the core system with the last-ok version. This will work
    until anybody checks in an incompatible change for the runtime,
    frontend, or configuration. It may be used as an indicator that
    a full upgrade procedure is necessary from this point in time.
    As this is expected to fail eventually, it should not be considered
    fatal. This is performed by test_build_core_lastok().

  o building the complete set of standard packages with the last-ok /
    current version. This excludes the compiler itself, and will give
    an indication if the second step of the bootstrap after initial
    installation works (installing all standard packages with the
    minimal system). This is done by test_build_std_lastok().

  o building of a minimal binary installation archive. This will
    ensure that we can actually build a new release for the tested
    platform. It is done by test_make_bin_dist().
    The produced archives can be distributed as current snapshots.

  o testall() and main() simply combine all tests into one run which
    might be done nightly. This is currently not compatible with
    the intended tinderbox integration though.

Needed extensions:

  o The actual compiler regression tests (m3tests) need to be included.
    I've imported them 2003 from the PM3 distribution, but they are not
    in a good shape. I've started to work on them, but will need
    help of others; you will have noticed my messages.

  o There are many more package-related regression tests, which need
    to be added, too. I don't know about the status of most though,
    so this will be a tedious task, too.

    We could distinguish between tests for core packages

        work/cm3/m3-libs/binIO/test
      work/cm3/m3-libs/bitvector/test
      work/cm3/m3-libs/libm3/tests
      work/cm3/m3-libs/libm3/tests/rw/autotest
      work/cm3/m3-libs/m3core/tests
      work/cm3/m3-libs/patternmatching/tests
      work/cm3/m3-libs/slisp/tests
      work/cm3/m3-libs/sortedtableextras/tests

    and others

       work/cm3/caltech-parser/drawcontext/test
       work/cm3/caltech-parser/m3tmplhack/test
       work/cm3/caltech-parser/parserlib/parserlib/test
       work/cm3/caltech-parser/parserlib/parserlib/test_stdin
       work/cm3/m3-comm/events/tests
       work/cm3/m3-comm/events/tests/test
       work/cm3/m3-comm/netobj/tests
       work/cm3/m3-comm/rdwr/test
       work/cm3/m3-comm/serial/test
       work/cm3/m3-comm/sharedobj/tests
       work/cm3/m3-comm/sharedobj/tests/netobjtest
       work/cm3/m3-comm/sharedobjgen/test
       work/cm3/m3-comm/udp/test
       work/cm3/m3-db/db/test
       work/cm3/m3-db/odbc/test
       work/cm3/m3-db/pgodbc/test
       work/cm3/m3-db/postgres95/test
       work/cm3/m3-db/stable/test
       work/cm3/m3-libs/dps/test
       work/cm3/m3-libs/synthesizer/test
       work/cm3/m3-libs/tcl/test
       work/cm3/m3-libs/deepcopy/test
       work/cm3/m3-libs/plplot/test
       work/cm3/m3-libs/arithmetic/test
       work/cm3/m3-libs/fftw/test
       work/cm3/m3-libs/lapack/test
       work/cm3/m3-libs/unittest
       work/cm3/m3-libs/unittest-numeric
       work/cm3/m3-libs/wellfett/test
       work/cm3/m3-sys/m3modinittest
       work/cm3/m3-ui/juno-2/juno-app/juno-src/test
       work/cm3/m3-ui/juno-2/juno-app/juno-src/testRedundant
       work/cm3/m3-ui/juno-2/juno-compiler/tests
       work/cm3/m3-ui/ui/test
       work/cm3/m3-ui/motif/tests
       work/cm3/m3-ui/ui-tests
       work/cm3/m3-ui/vbtkit/src/vtext/testing

I'll try these after m3tests are working. Of course, contributions
of others are welcome for all these.

As a start, it may be a good idea if those interested in running
regular tests could try the core test framework. What needs to be
done is

   o ( . ./scripts/regression/defs.sh; install_bin_dist )
     once, potentially manuallly, or with overwriting the cminstall
     program to use something current:

     ( export CMINSTALL=...; . ./scripts/regression/defs.sh; install_bin_dist )

     This expects the latest binary installation archive in your home
     directory.

   o Now tests should be able to run on a regular basis like this:

     ( . ./scripts/regression/defs.sh; main )

     This will create a new workspace, checkout the current sources,
     and perform all the tests described above.

I've tried this now on a FreeBSD 6.2 and Debian Linux system. It may
well be that more adaptions and extensions to the scripts are
necessary, but we have to try.

If the above works for others, too, we can turn to the tinderbox
integration to gather all the results in a database and display
them. You may have a look at the results at

   http://tinderbox.elegosoft.com/
   http://tinderbox.elegosoft.com/cm3/status.html

Keep in mind that this work is not finished yet.

Please let me know of any problems, improvements, or just comments.

Olaf
-- 
Olaf Wagner -- elego Software Solutions GmbH
                Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
phone: +49 30 23 45 86 96  mobile: +49 177 2345 869  fax: +49 30 23 45 86 95
    http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin
Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194