[M3devel] www.opencm3.net/m3tests

[Jay]

The bogus commands are actually a "simulation" of what the code is doing, I think.
It isn't so dumb as to run little rm commands here it and there. It deletes files without running anything and prints stuff that if run will approximate what it did. It is indeed worthwhile I think, though it isn't "-commands" per se.
That's my vague understanding not having read the code closely.
The "strange" thing is to see "rm" commands on NT386.
 
Planning for success is arguably wrong.
There will be errors. They need to be debugged.
You can't always rerun the scenario.
 
I think a log file with a little more information is a reasonable compromise.
 
Even that is gray. For example, the log file should not contain the output of -trace.
That's just too low level, too voluminous, and fails too rarely to be interesting.
 
Echoing command lines always into a log seems good on the assumption that command lines are expensive, so the i/o to put them in a log should be ok. This isn't entirely true, what with stuff like mkdir and rm.
On the assumption that command lines are running "big" things that "often" fail.
Again this is mixed. Most systems that show command lines still hide many.
ie: invocations of gcc are often shown, but invocations of cc1 and as not so much.
 
Personally I debug a lot of stuff and I err in favor of debuggability, be it live or post-mortem.
Debugging is hard enough, I don't like the obvious easy powers taken away from me.
I know logging and printf is incredibly crude vs. stepping through code in a debugger, but I also had it be fantastically successful and fast too many times to discount. Esp. in data-intensive code where the same code runs successfully many times before the problem occurs.
 
I will grant that in my Modula-3 use, debugging has mostly been easy enough, the repro cases available enough, that optimizing for post-mortem debugging of one and only one instance of a problem is probably overkill.
But still, a log file seems reasonable..
(The debugging has actually been not easy, quite difficult at times, the NT386GNU X windows problem, the double aligment, the stat overflow...but the repro cases and ability to peel the onion in run after run after run has been good; like cm3cg -y for example..and the logging we are talking about only helps in the easiest of cases...)
 
 - Jay


CC: wagner at elegosoft.com; m3devel at elegosoft.comFrom: hosking at cs.purdue.eduTo: jayk123 at hotmail.comSubject: Re: [M3devel] www.opencm3.net/m3testsDate: Tue, 6 May 2008 10:05:25 -0400

On May 6, 2008, at 6:57 AM, Jay wrote:


Olaf, I don't entirely agree.The logs should be fairly informative without intervention.Just not too verbose. I'd would rather remove the lines that say "compiling" and show the cm3cg invocations.But more than one line per source file is too much.-commands doesn't really work. Lots of bogus extra commands are echoed (rm .tmp), and not all the commands that are run are echoed.

Not bogus - it does exactly what one would expect - showing all commands executed.   I'm  with  Olaf on this one.


I understand that stability for tests is also a goal, but information for the interactive user and for post mortem debugging via a log is also valuable, and these are contrary to stability-for-tests. Somehow contradictory goals need to be met.

I prefer terse output for tests  that complete normally.  On detailed  investigation l can always turn on a verbose flag.

How about a compromise?How about we always log more stuff to TARGET/cm3.log? And then exactly what you want to stdout/stderr?While, as I said, more than one line per file is overkill, as part of this compromise I'd be willing to include the cm3cg and as invocation. Have folks used build.exe in the Windows NT DDK?It's model is close to what you get here.It has three sets of information:  to the console/stdout   build.log, build.exe is a wrapper over make, and this includes all the make output -- all the command lines run   build.err, a small subset of console/stdout output My "design" here has been to avoid over-verbosity. Link and mt run on the order of once, or a small number of times, per directory, so showing the full command line (response files...) isn't so verbose -- vs. say something shown once per compiled file. There's another "problem" here that I have partly fixed.Underlying errors are not shown on console/stdout/stderr.They are often in like TARGET/m3core.lst. I changed stuff to refer to that file, after having hunted the errors down myself slowly the first time. We could change things to alway says "see TARGET/cm3.log" for more information, or somesuch.  - Jay

> Date: Tue, 6 May 2008 10:07:11 +0200> From: wagner at elegosoft.com> To: m3devel at elegosoft.com> Subject: Re: [M3devel] www.opencm3.net/m3tests> > Quoting Jay <jayk123 at hotmail.com>:> > > The "problem" here, a really really small one, is just that the link > > and mt commands got echoed.> > Olaf made them not echoed. I then made them conditionally echoed.> > I made m3-sys\m3tests define M3TESTS so that NT386.common doesn't echo.> > It's not a big deal either way.> > Aha -- tests in other directories would have a problem, and I think > > there are some.> >> > I like more echoing usually, so the system explains what is going on,> > instead of the vaguer "linking foo" sort of message.> > Granted, nobody bothers watching gcc run assembler commands, so I > > guess it is just quite gray.> > Jay, cm3 needs to behave the same on all platforms. By default> commands should _not_ be echoed to stdout or stderr; that's what the> -commands switch is for: this gives you exactly the commands the> compiler issues to invoke other tools.> > For more verbose output, you may also depend on the -verbose or the> -debug switches.> > Please adapt the configuration files that bey default all echoing> of commands is off.> > Olaf> > > I don't know how to run the Tinderbox either yet, sorry.> > I tried.> >> > For adding tests, well, there is m3-sys\m3tests.> > That is where a lot of the tests are, but not all.> > I am not sure where tests belong. I added a small number there.> > They are named with just a letter and a number.> > The letters have some meaning, explained in the m3makefile.> > "p" is programs that are run and their stdout/stderr compared > > against expected> > "e" is programs build and the errors from the compiler checked to be > > reasonable.> > I don't think in general they are expected to successfully compile > > or run, though the case of "warnings" may be unclear.> > "c" is programs built so a human can look at the generated code.> > There is another case I think for human checking.> > The numbers are just 001, 002, 003, etc.> > Each hundred tests are in a separate directory, like p0\p001, > > p0\p002, p1\p100, p1\101, etc.> >> > Something like this.> > Wherever I have details wrong, just look in m3-sys\m3tests. It's > > pretty simple, obvious, and well commented.> >> > The output is a little clearer if you have a working diff.exe on the path.> > Then what you do is search for "@@" in the output.> >> > - Jay> >> >> > Date: Tue, 6 May 2008 03:46:27 +0200From: dabenavidesd at yahoo.esTo: > > m3devel at elegosoft.comSubject: [M3devel] www.opencm3.net/m3testsDear > > developers:Does the recent tests on NT386 seem broken because a > > recent change on the m3-sys tree, or is the HTML bad generated, I > > mean can you check the last tests Sunday, May 4th (p001 to p042) has > > a red background > > http://www.opencm3.net/m3tests/m3tests-NT386-2008-04-23-13-30-57.html</pre></td></tr> <tr class="bgred tl"><td> p002 </td><td class="small" width="45%"> Text</td><td class="small"><pre>Comparing files P0\P002\stdout.build and ..\SRC\P0\P002\STDOUT.BUILD***** P0\P002\stdout.buildlink @_m3responsefile0.txt 2>&1 > pgm.lstmt /nologo /manifest pgm.exe.manifest /outputresource:pgm.exe;1***** ..\SRC\P0\P002\STDOUT.BUILD*****Comparing files P0\P002\stderr.build and ..\SRC\P0\P002\STDERR.BUILDFC: no differences encounteredComparing files P0\P002\stdout.pgm and ..\SRC\P0\P002\STDOUT.PGMFC: no differences encounteredComparing files P0\P002\stderr.pgm and ..\SRC\P0\P002\STDERR.PGMFC: no differences encounteredand almost the same pattern in the above tests.I would suggest if it is thinkable using NT386 variant with a complete dedicated machine/system, I can try to set up one this week and send the data back (the machine is behind a proxy), but I don't remember the mail explaining the set up, and also want to know if there is a chance of run the tests with one run script.Also what is the best natural way to put a new tests, and the standard name it should > > have?Thanks> >> >> > Enviado desde Correo Yahoo!La bandeja de entrada más inteligente.> > > > -- > Olaf Wagner -- elego Software Solutions GmbH> Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95> http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20080506/48ba41d0/attachment-0002.html>