[M3devel] cm3 command line interface and output; was: RE: www.opencm3.net/m3tests

[Jay]

Well, Olaf, I see your point, but I don't entirely agree.
I believe there are contradictory goals.
Which is partly to say, there will always be "problems" here, though I think producing a more detailed log file is a good compromise.
 
In particular, consider the role of "remote debugging", or even local debugging.
Wouldn't it be great if many situations could be debugged from merely looking at "m3.log" from the first and only run that failed? What if the problem is intermittent? You can't always rely on running again with different switches. I have to post-mortem debug lots of stuff and logs are indispensible. They don't always work, and they can't always work, but they can be extremely helpful.
 
To a large extent, you need to plan for failure. There will be errors. When there is an error, what is the design that allows quickest easiest diagnosis, some large percentage of the time? A nice friendly abstract stdout that hides details, or something with more information? Or somethin in between like a friendly stdout and a verbose log file?
 
The need for a cross-platform user experience is unclear -- I want file names printed in a form my editor understands, and my editors unfortunately are not consistent. Perhaps, as you allude, the "success" ui should be cross platform, and in the face of an error, the experience can devolve and contain platform specific details. Here you can see the tests already having problems, as the paths have forward or backward slashes, and "crashes" have different appearances. So far this has been mostly papered over (which reminds me, I need to get the darwin and bsd variants; currently only NT and Linux are handled).
 
For regression tests, yes I understand.
However, stdout/stderr is a very crude thing.
It'd be nice if the code could make, uh, like, "function calls" to indicate what happened instead of random text that is meant to be both human and machine consumable.
But I know that's just too high tech to happen any time soon..
 
(Similarly, "function calls" to drive a compiler are a good idea to. "Command lines" are also very crude and type-unsafe..the whole problem with spaces....)
 
A good example I think of something that doesn't work well today is what happens when there are link errors.
The problem here isn't so much the echoing of commands, but the appearance of the linker output.
All you get is cm3 checks the exit code and/or the existance of the output file and stdout indicates a boolean success/faliure. User has to dig into another file to find the actual error. Now, again, I am talking out of both sides. I argue for a log for the verbose information, and now complain about having to dig into that log. I would at least like to have there be one such log, not one for the link output and nothing else. That would be a little better. The inconvenience of having to dig into a log is a more difficult problem, I know very well, related again to the crudeness of stdout. Because the linker has no structured way to report errors, other than an exit code, whatever wraps it is left to some crude device like attempting to "parse" the stdout. This is a well traveled path, and it can work quite well, but it is also frought with peril, as different linkers and different versions of linkers output errors differently...
 
As well, on the other end of things, you know, if you assume success, you really want less output.
make-dist.cmd is really good this way, though make-dist.py is not yet.
In there I suppress a LOT of output, but I do keep it in logs in case of failure..
 
I don't have a conclusion.
I understand the goals.
I also believe logging, to stdout or to a file, greatly aids debugging.
I think producing a log file in the output directory is a good compromise, and one I have seen successfully used.
  Though you don't want the easy out of having a log file to lead to the log file becoming too verbose. I have seen log files that take a while for some editors to open..and people throwing in "verbose" switches and not realize the damage they are doing. I did it myself once. I threw in /verbose on the Visual C++ linker and the output of that is so voluminous that it slows things down. It's great for debugging certain problems, but too expensive to put in all the time "just in case".
 
 - Jay


> Date: Tue, 6 May 2008 15:33:18 +0200> From: wagner at elegosoft.com> To: jayk123 at hotmail.com> CC: m3devel at elegosoft.com> Subject: cm3 command line interface and output; was: RE: [M3devel] www.opencm3.net/m3tests> > Quoting Jay <jayk123 at hotmail.com>:> > > When I say show cm3cg/as invocations, I mean in this new TARGET/cm3.log.> > As I said, I propose exactly the stdout/stderr you want, but then > > some other output always, a log file.> > Also, about -commands, I have also claimed that it does work -- > > talking out of both sides of my mouth. The extra commands it prints > > are obviously harmless. I forget what doesn't print, maybe exec vs. > > q_exec? Also the '@' character wasn't working with q_exec, but that > > isn't necessarily relevant here..I forgot where I use exec vs. > > q_exec..> > And there plenty of other ways to debug than -commands and it is > > "just" a debugging facility..> > I've got a rather definite opinion about the output behaviour of> cm3. It's like this, and I would like others to comment, too:> > o cm3 without any switches (-commands, -debug, -verbose) should> not output any called commands, neither to stdout nor stderr.> It may print what it is doing in an abstract, target independent> (short) way. This output may refer to compilation units, interfaces,> modules, and compilation phases, but not to target specific commands> and files (unless there are errors, of course) Even then I'd like> to abstract from differences.> > o cm3 -silent should suppress the normal output, except for warnings> and error messages.> > o cm3 -commands should output exactly all invokations of externals> commands so that they can be re-executed from the command line> if necessary.> > I realize that it might not be this way in every aspect, so that> we may need to adapt the code here and there. But in my experience> it has worked this way quite well. Here are some points to consider:> > o We need consistent behaviour and output across all target platforms.> This is not just a requirement of regression testing, but also of> a consistent user interface. I don't want different behaviour on> different platforms.> > o If -commands does not work as expected for some use case,> we need to fix it.> > o If q_exec or whatever quake function is still deficient in this> respect, we need to fix/adapt it. It should not be much effort.> > o I find the options -commands, -verbose, -debug, -silent a very> good categorization of information types, intuitively understandable> and usable in practive (as I've never had any problems with it, but> have used them efficiently for years). So I don't see the point> in changing anything, unless there is a better concept.> > o I don't think it is a good idea to write any log files by default> (apart from the stdout and stderr streams).> > This is of course just my opinion, but unless there are others in> favour of changes (and I'd like some motivation for them) I'd insist> that we keep the existing design.> > Olaf> > > From: jayk123 at hotmail.comTo: wagner at elegosoft.com; > > m3devel at elegosoft.comDate: Tue, 6 May 2008 10:57:01 +0000Subject: > > Re: [M3devel] www.opencm3.net/m3tests> >> >> > Olaf, I don't entirely agree.The logs should be fairly informative > > without intervention.Just not too verbose. I'd would rather remove > > the lines that say "compiling" and show the cm3cg invocations.But > > more than one line per source file is too much.-commands doesn't > > really work. Lots of bogus extra commands are echoed (rm .tmp), and > > not all the commands that are run are echoed. I understand that > > stability for tests is also a goal, but information for the > > interactive user and for post mortem debugging via a log is also > > valuable, and these are contrary to stability-for-tests. Somehow > > contradictory goals need to be met. How about a compromise?How about > > we always log more stuff to TARGET/cm3.log? And then exactly what > > you want to stdout/stderr?While, as I said, more than one line per > > file is overkill, as part of this compromise I'd be willing to > > include the cm3cg and as invocation. Have folks used build.exe in > > the Windows NT DDK?It's model is close to what you get here.It has > > three sets of information: to the console/stdout build.log, > > build.exe is a wrapper over make, and this includes all the make > > output -- all the command lines run build.err, a small subset of > > console/stdout output My "design" here has been to avoid > > over-verbosity. Link and mt run on the order of once, or a small > > number of times, per directory, so showing the full command line > > (response files...) isn't so verbose -- vs. say something shown once > > per compiled file. There's another "problem" here that I have > > partly fixed.Underlying errors are not shown on > > console/stdout/stderr.They are often in like TARGET/m3core.lst. I > > changed stuff to refer to that file, after having hunted the errors > > down myself slowly the first time. We could change things to alway > > says "see TARGET/cm3.log" for more information, or somesuch. - Jay> >> >> Date: Tue, 6 May 2008 10:07:11 +0200> From: wagner at elegosoft.com> > >> To: m3devel at elegosoft.com> Subject: Re: [M3devel] > >> www.opencm3.net/m3tests> > Quoting Jay <jayk123 at hotmail.com>:> > > > >> The "problem" here, a really really small one, is just that the > >> link > > and mt commands got echoed.> > Olaf made them not echoed. > >> I then made them conditionally echoed.> > I made m3-sys\m3tests > >> define M3TESTS so that NT386.common doesn't echo.> > It's not a big > >> deal either way.> > Aha -- tests in other directories would have a > >> problem, and I think > > there are some.> >> > I like more echoing > >> usually, so the system explains what is going on,> > instead of > >> the vaguer "linking foo" sort of message.> > Granted, nobody > >> bothers watching gcc run assembler commands, so I > > guess it is > >> just quite gray.> > Jay, cm3 needs to behave the same on all > >> platforms. By default> commands should _not_ be echoed to stdout > >> or stderr; that's what the> -commands switch is for: this gives > >> you exactly the commands the> compiler issues to invoke other > >> tools.> > For more verbose output, you may also depend on the > >> -verbose or the> -debug switches.> > Please adapt the > >> configuration files that bey default all echoing> of commands is > >> off.> > Olaf> > > I don't know how to run the Tinderbox either > >> yet, sorry.> > I tried.> >> > For adding tests, well, there is > >> m3-sys\m3tests.> > That is where a lot of the tests are, but not > >> all.> > I am not sure where tests belong. I added a small number > >> there.> > They are named with just a letter and a number.> > The > >> letters have some meaning, explained in the m3makefile.> > "p" is > >> programs that are run and their stdout/stderr compared > > against > >> expected> > "e" is programs build and the errors from the compiler > >> checked to be > > reasonable.> > I don't think in general they are > >> expected to successfully compile > > or run, though the case of > >> "warnings" may be unclear.> > "c" is programs built so a human can > >> look at the generated code.> > There is another case I think for > >> human checking.> > The numbers are just 001, 002, 003, etc.> > > >> Each hundred tests are in a separate directory, like p0\p001, > > > >> p0\p002, p1\p100, p1\101, etc.> >> > Something like this.> > > >> Wherever I have details wrong, just look in m3-sys\m3tests. It's > > >> > pretty simple, obvious, and well commented.> >> > The output is > >> a little clearer if you have a working diff.exe on the path.> > > >> Then what you do is search for "@@" in the output.> >> > - Jay> >> > >> >> > Date: Tue, 6 May 2008 03:46:27 +0200From: > >> dabenavidesd at yahoo.esTo: > > m3devel at elegosoft.comSubject: > >> [M3devel] www.opencm3.net/m3testsDear > > developers:Does the > >> recent tests on NT386 seem broken because a > > recent change on > >> the m3-sys tree, or is the HTML bad generated, I > > mean can you > >> check the last tests Sunday, May 4th (p001 to p042) has > > a red > >> background > > > >> http://www.opencm3.net/m3tests/m3tests-NT386-2008-04-23-13-30-57.html</pre></td></tr> <tr class="bgred tl"><td> p002 </td><td class="small" width="45%"> Text</td><td class="small"><pre>Comparing files P0\P002\stdout.build and ..\SRC\P0\P002\STDOUT.BUILD***** P0\P002\stdout.buildlink @_m3responsefile0.txt 2>&1 > pgm.lstmt /nologo /manifest pgm.exe.manifest /outputresource:pgm.exe;1***** ..\SRC\P0\P002\STDOUT.BUILD*****Comparing files P0\P002\stderr.build and ..\SRC\P0\P002\STDERR.BUILDFC: no differences encounteredComparing files P0\P002\stdout.pgm and ..\SRC\P0\P002\STDOUT.PGMFC: no differences encounteredComparing files P0\P002\stderr.pgm and ..\SRC\P0\P002\STDERR.PGMFC: no differences encounteredand almost the same pattern in the above tests.I would suggest if it is thinkable using NT386 variant with a complete dedicated machine/system, I can try to set up one this week and send the data back (the machine is behind a proxy), but I don't remember the mail explaining the set up, and also want to know if there is a chance of run the tests with one run script.Also what is the best natural way to put a new tests, and the standard name it should > > have?Thanks> >> >> > Enviado desde Correo Yahoo!La bandeja de entrada más inteligente.> > > > -- > Olaf Wagner -- elego Software Solutions GmbH> Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95> http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: > >> DE163214194>> > > > -- > Olaf Wagner -- elego Software Solutions GmbH> Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95> http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20080506/025d5e37/attachment-0002.html>