[M3devel] cm3 command line interface and output; was: RE: www.opencm3.net/m3tests

[Tony Hosking]

I'm am strongly with Jay on this one.

On May 6, 2008, at 9:33 AM, Olaf Wagner wrote:

> Quoting Jay <jayk123 at hotmail.com>:
>
>> When I say show cm3cg/as invocations, I mean in this new TARGET/ 
>> cm3.log.
>> As I said, I propose exactly the stdout/stderr you want, but then   
>> some other output always, a log file.
>> Also, about -commands, I have also claimed that it does work --   
>> talking out of both sides of my mouth. The extra commands it  
>> prints  are obviously harmless. I forget what doesn't print, maybe  
>> exec vs.  q_exec? Also the '@' character wasn't working with  
>> q_exec, but that  isn't necessarily relevant here..I forgot where I  
>> use exec vs.  q_exec..
>> And there plenty of other ways to debug than -commands and it is   
>> "just" a debugging facility..
>
> I've got a rather definite opinion about the output behaviour of
> cm3. It's like this, and I would like others to comment, too:
>
> o cm3 without any switches (-commands, -debug, -verbose) should
>   not output any called commands, neither to stdout nor stderr.
>   It may print what it is doing in an abstract, target independent
>   (short) way. This output may refer to compilation units, interfaces,
>   modules, and compilation phases, but not to target specific commands
>   and files (unless there are errors, of course) Even then I'd like
>   to abstract from differences.
>
> o cm3 -silent should suppress the normal output, except for warnings
>   and error messages.
>
> o cm3 -commands should output exactly all invokations of externals
>   commands so that they can be re-executed from the command line
>   if necessary.
>
> I realize that it might not be this way in every aspect, so that
> we may need to adapt the code here and there. But in my experience
> it has worked this way quite well. Here are some points to consider:
>
> o We need consistent behaviour and output across all target platforms.
>   This is not just a requirement of regression testing, but also of
>   a consistent user interface. I don't want different behaviour on
>   different platforms.
>
> o If -commands does not work as expected for some use case,
>   we need to fix it.
>
> o If q_exec or whatever quake function is still deficient in this
>   respect, we need to fix/adapt it. It should not be much effort.
>
> o I find the options -commands, -verbose, -debug, -silent a very
>   good categorization of information types, intuitively understandable
>   and usable in practive (as I've never had any problems with it, but
>   have used them efficiently for years). So I don't see the point
>   in changing anything, unless there is a better concept.
>
> o I don't think it is a good idea to write any log files by default
>   (apart from the stdout and stderr streams).
>
> This is of course just my opinion, but unless there are others in
> favour of changes (and I'd like some motivation for them) I'd insist
> that we keep the existing design.
>
> Olaf
>
>> From: jayk123 at hotmail.comTo: wagner at elegosoft.com;  m3devel at elegosoft.comDate 
>> : Tue, 6 May 2008 10:57:01 +0000Subject:  Re: [M3devel] www.opencm3.net/m3tests
>>
>>
>> Olaf, I don't entirely agree.The logs should be fairly informative   
>> without intervention.Just not too verbose. I'd would rather remove   
>> the lines that say "compiling" and show the cm3cg invocations.But   
>> more than one line per source file is too much.-commands doesn't   
>> really work. Lots of bogus extra commands are echoed (rm .tmp),  
>> and  not all the commands that are run are echoed. I understand  
>> that  stability for tests is also a goal, but information for the   
>> interactive user and for post mortem debugging via a log is also   
>> valuable, and these are contrary to stability-for-tests. Somehow   
>> contradictory goals need to be met. How about a compromise?How  
>> about  we always log more stuff to TARGET/cm3.log? And then exactly  
>> what  you want to stdout/stderr?While, as I said, more than one  
>> line per  file is overkill, as part of this compromise I'd be  
>> willing to  include the cm3cg and as invocation. Have folks used  
>> build.exe in  the Windows NT DDK?It's model is close to what you  
>> get here.It has  three sets of information:  to the console/ 
>> stdout   build.log,  build.exe is a wrapper over make, and this  
>> includes all the make  output -- all the command lines run    
>> build.err, a small subset of  console/stdout output My "design"  
>> here has been to avoid  over-verbosity. Link and mt run on the  
>> order of once, or a small  number of times, per directory, so  
>> showing the full command line  (response files...) isn't so verbose  
>> -- vs. say something shown once  per compiled file. There's another  
>> "problem" here that I have partly  fixed.Underlying errors are not  
>> shown on console/stdout/stderr.They  are often in like TARGET/ 
>> m3core.lst. I changed stuff to refer to  that file, after having  
>> hunted the errors down myself slowly the  first time. We could  
>> change things to alway says "see  TARGET/cm3.log" for more  
>> information, or somesuch.  - Jay
>>
>>> Date: Tue, 6 May 2008 10:07:11 +0200> From: wagner at elegosoft.com>   
>>> To: m3devel at elegosoft.com> Subject: Re: [M3devel]  www.opencm3.net/m3tests 
>>> > > Quoting Jay <jayk123 at hotmail.com>:> > >  The "problem" here, a  
>>> really really small one, is just that the  link > > and mt  
>>> commands got echoed.> > Olaf made them not echoed.  I then made  
>>> them conditionally echoed.> > I made m3-sys\m3tests  define  
>>> M3TESTS so that NT386.common doesn't echo.> > It's not a big  deal  
>>> either way.> > Aha -- tests in other directories would have a   
>>> problem, and I think > > there are some.> >> > I like more  
>>> echoing  usually, so the system explains what is going on,> >  
>>> instead of the  vaguer "linking foo" sort of message.> > Granted,  
>>> nobody bothers  watching gcc run assembler commands, so I > >  
>>> guess it is just  quite gray.> > Jay, cm3 needs to behave the same  
>>> on all platforms.  By default> commands should _not_ be echoed to  
>>> stdout or stderr;  that's what the> -commands switch is for: this  
>>> gives you exactly  the commands the> compiler issues to invoke  
>>> other tools.> > For  more verbose output, you may also depend on  
>>> the -verbose or the>  -debug switches.> > Please adapt the  
>>> configuration files that bey  default all echoing> of commands is  
>>> off.> > Olaf> > > I don't know  how to run the Tinderbox either  
>>> yet, sorry.> > I tried.> >> > For  adding tests, well, there is m3- 
>>> sys\m3tests.> > That is where a lot  of the tests are, but not  
>>> all.> > I am not sure where tests belong.  I added a small number  
>>> there.> > They are named with just a letter  and a number.> > The  
>>> letters have some meaning, explained in the  m3makefile.> > "p" is  
>>> programs that are run and their stdout/stderr  compared > >  
>>> against expected> > "e" is programs build and the  errors from the  
>>> compiler checked to be > > reasonable.> > I don't  think in  
>>> general they are expected to successfully compile > > or  run,  
>>> though the case of "warnings" may be unclear.> > "c" is  programs  
>>> built so a human can look at the generated code.> > There  is  
>>> another case I think for human checking.> > The numbers are just   
>>> 001, 002, 003, etc.> > Each hundred tests are in a separate   
>>> directory, like p0\p001, > > p0\p002, p1\p100, p1\101, etc.> >> >   
>>> Something like this.> > Wherever I have details wrong, just look  
>>> in  m3-sys\m3tests. It's > > pretty simple, obvious, and well   
>>> commented.> >> > The output is a little clearer if you have a   
>>> working diff.exe on the path.> > Then what you do is search for   
>>> "@@" in the output.> >> > - Jay> >> >> > Date: Tue, 6 May 2008   
>>> 03:46:27 +0200From: dabenavidesd at yahoo.esTo: > >  m3devel at elegosoft.comSubject 
>>> : [M3devel] www.opencm3.net/m3testsDear  > > developers:Does the  
>>> recent tests on NT386 seem broken because a  > > recent change on  
>>> the m3-sys tree, or is the HTML bad generated,  I > > mean can you  
>>> check the last tests Sunday, May 4th (p001 to  p042) has > > a red  
>>> background > >  http://www.opencm3.net/m3tests/m3tests-NT386-2008-04-23-13-30-57.html 
>>> </pre></td></tr> <tr class="bgred tl"><td> p002 </td><td  
>>> class="small" width="45%"> Text</td><td  
>>> class="small"><pre>Comparing files P0\P002\stdout.build and ..\SRC 
>>> \P0\P002\STDOUT.BUILD***** P0\P002\stdout.buildlink  
>>> @_m3responsefile0.txt 2>&1 > pgm.lstmt /nologo /manifest  
>>> pgm.exe.manifest /outputresource:pgm.exe;1***** ..\SRC 
>>> \P0\P002\STDOUT.BUILD*****Comparing files P0\P002\stderr.build  
>>> and ..\SRC\P0\P002\STDERR.BUILDFC: no differences  
>>> encounteredComparing files P0\P002\stdout.pgm and ..\SRC 
>>> \P0\P002\STDOUT.PGMFC: no differences encounteredComparing files  
>>> P0\P002\stderr.pgm and ..\SRC\P0\P002\STDERR.PGMFC: no differences  
>>> encounteredand almost the same pattern in the above tests.I would  
>>> suggest if it is thinkable using NT386 variant with a complete  
>>> dedicated machine/system, I can try to set up one this week and  
>>> send the data back (the machine is behind a proxy), but I don't  
>>> remember the mail explaining the set up, and also want to know if  
>>> there is a chance of run the tests with one run script.Also what  
>>> is the best natural way to put a new tests, and the standard name  
>>> it should > > have?Thanks> >> >> > Enviado desde Correo Yahoo!La  
>>> bandeja de entrada más inteligente.> > > > -- > Olaf Wagner --  
>>> elego Software Solutions GmbH> Gustav-Meyer-Allee 25 / Gebäude 12,  
>>> 13355 Berlin, Germany> phone: +49 30 23 45 86 96 mobile: +49 177  
>>> 2345 869 fax: +49 30 23 45 86 95> http://www.elegosoft.com |  
>>> Geschäftsführer: Olaf Wagner | Sitz: Berlin> Handelregister:  
>>> Amtsgericht Charlottenburg HRB 77719 | USt-IdNr:  DE163214194>
>
>
>
> -- 
> Olaf Wagner -- elego Software Solutions GmbH
>               Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin,  
> Germany
> phone: +49 30 23 45 86 96  mobile: +49 177 2345 869  fax: +49 30 23  
> 45 86 95
>   http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz:  
> Berlin
> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr:  
> DE163214194
>