[M3devel] cm3 command line interface and output; was: RE: www.opencm3.net/m3tests

[Olaf Wagner]

Quoting Jay <jayk123 at hotmail.com>:

> When I say show cm3cg/as invocations, I mean in this new TARGET/cm3.log.
> As I said, I propose exactly the stdout/stderr you want, but then   
> some other output always, a log file.
> Also, about -commands, I have also claimed that it does work --   
> talking out of both sides of my mouth. The extra commands it prints   
> are obviously harmless. I forget what doesn't print, maybe exec vs.   
> q_exec? Also the '@' character wasn't working with q_exec, but that   
> isn't necessarily relevant here..I forgot where I use exec vs.   
> q_exec..
> And there plenty of other ways to debug than -commands and it is   
> "just" a debugging facility..

I've got a rather definite opinion about the output behaviour of
cm3. It's like this, and I would like others to comment, too:

  o cm3 without any switches (-commands, -debug, -verbose) should
    not output any called commands, neither to stdout nor stderr.
    It may print what it is doing in an abstract, target independent
    (short) way. This output may refer to compilation units, interfaces,
    modules, and compilation phases, but not to target specific commands
    and files (unless there are errors, of course) Even then I'd like
    to abstract from differences.

  o cm3 -silent should suppress the normal output, except for warnings
    and error messages.

  o cm3 -commands should output exactly all invokations of externals
    commands so that they can be re-executed from the command line
    if necessary.

I realize that it might not be this way in every aspect, so that
we may need to adapt the code here and there. But in my experience
it has worked this way quite well. Here are some points to consider:

  o We need consistent behaviour and output across all target platforms.
    This is not just a requirement of regression testing, but also of
    a consistent user interface. I don't want different behaviour on
    different platforms.

  o If -commands does not work as expected for some use case,
    we need to fix it.

  o If q_exec or whatever quake function is still deficient in this
    respect, we need to fix/adapt it. It should not be much effort.

  o I find the options -commands, -verbose, -debug, -silent a very
    good categorization of information types, intuitively understandable
    and usable in practive (as I've never had any problems with it, but
    have used them efficiently for years). So I don't see the point
    in changing anything, unless there is a better concept.

  o I don't think it is a good idea to write any log files by default
    (apart from the stdout and stderr streams).

This is of course just my opinion, but unless there are others in
favour of changes (and I'd like some motivation for them) I'd insist
that we keep the existing design.

Olaf

> From: jayk123 at hotmail.comTo: wagner at elegosoft.com;   
> m3devel at elegosoft.comDate: Tue, 6 May 2008 10:57:01 +0000Subject:   
> Re: [M3devel] www.opencm3.net/m3tests
>
>
> Olaf, I don't entirely agree.The logs should be fairly informative   
> without intervention.Just not too verbose. I'd would rather remove   
> the lines that say "compiling" and show the cm3cg invocations.But   
> more than one line per source file is too much.-commands doesn't   
> really work. Lots of bogus extra commands are echoed (rm .tmp), and   
> not all the commands that are run are echoed. I understand that   
> stability for tests is also a goal, but information for the   
> interactive user and for post mortem debugging via a log is also   
> valuable, and these are contrary to stability-for-tests. Somehow   
> contradictory goals need to be met. How about a compromise?How about  
>  we always log more stuff to TARGET/cm3.log? And then exactly what   
> you want to stdout/stderr?While, as I said, more than one line per   
> file is overkill, as part of this compromise I'd be willing to   
> include the cm3cg and as invocation. Have folks used build.exe in   
> the Windows NT DDK?It's model is close to what you get here.It has   
> three sets of information:  to the console/stdout   build.log,   
> build.exe is a wrapper over make, and this includes all the make   
> output -- all the command lines run   build.err, a small subset of   
> console/stdout output My "design" here has been to avoid   
> over-verbosity. Link and mt run on the order of once, or a small   
> number of times, per directory, so showing the full command line   
> (response files...) isn't so verbose -- vs. say something shown once  
>  per compiled file. There's another "problem" here that I have  
> partly  fixed.Underlying errors are not shown on  
> console/stdout/stderr.They  are often in like TARGET/m3core.lst. I  
> changed stuff to refer to  that file, after having hunted the errors  
> down myself slowly the  first time. We could change things to alway  
> says "see  TARGET/cm3.log" for more information, or somesuch.  - Jay
>
>> Date: Tue, 6 May 2008 10:07:11 +0200> From: wagner at elegosoft.com>   
>> To: m3devel at elegosoft.com> Subject: Re: [M3devel]   
>> www.opencm3.net/m3tests> > Quoting Jay <jayk123 at hotmail.com>:> > >   
>> The "problem" here, a really really small one, is just that the   
>> link > > and mt commands got echoed.> > Olaf made them not echoed.   
>> I then made them conditionally echoed.> > I made m3-sys\m3tests   
>> define M3TESTS so that NT386.common doesn't echo.> > It's not a big  
>>  deal either way.> > Aha -- tests in other directories would have a  
>>  problem, and I think > > there are some.> >> > I like more echoing  
>>  usually, so the system explains what is going on,> > instead of  
>> the  vaguer "linking foo" sort of message.> > Granted, nobody  
>> bothers  watching gcc run assembler commands, so I > > guess it is  
>> just  quite gray.> > Jay, cm3 needs to behave the same on all  
>> platforms.  By default> commands should _not_ be echoed to stdout  
>> or stderr;  that's what the> -commands switch is for: this gives  
>> you exactly  the commands the> compiler issues to invoke other  
>> tools.> > For  more verbose output, you may also depend on the  
>> -verbose or the>  -debug switches.> > Please adapt the  
>> configuration files that bey  default all echoing> of commands is  
>> off.> > Olaf> > > I don't know  how to run the Tinderbox either  
>> yet, sorry.> > I tried.> >> > For  adding tests, well, there is  
>> m3-sys\m3tests.> > That is where a lot  of the tests are, but not  
>> all.> > I am not sure where tests belong.  I added a small number  
>> there.> > They are named with just a letter  and a number.> > The  
>> letters have some meaning, explained in the  m3makefile.> > "p" is  
>> programs that are run and their stdout/stderr  compared > > against  
>> expected> > "e" is programs build and the  errors from the compiler  
>> checked to be > > reasonable.> > I don't  think in general they are  
>> expected to successfully compile > > or  run, though the case of  
>> "warnings" may be unclear.> > "c" is  programs built so a human can  
>> look at the generated code.> > There  is another case I think for  
>> human checking.> > The numbers are just  001, 002, 003, etc.> >  
>> Each hundred tests are in a separate  directory, like p0\p001, > >  
>> p0\p002, p1\p100, p1\101, etc.> >> >  Something like this.> >  
>> Wherever I have details wrong, just look in  m3-sys\m3tests. It's >  
>> > pretty simple, obvious, and well  commented.> >> > The output is  
>> a little clearer if you have a  working diff.exe on the path.> >  
>> Then what you do is search for  "@@" in the output.> >> > - Jay> >>  
>> >> > Date: Tue, 6 May 2008  03:46:27 +0200From:  
>> dabenavidesd at yahoo.esTo: > >  m3devel at elegosoft.comSubject:  
>> [M3devel] www.opencm3.net/m3testsDear  > > developers:Does the  
>> recent tests on NT386 seem broken because a  > > recent change on  
>> the m3-sys tree, or is the HTML bad generated,  I > > mean can you  
>> check the last tests Sunday, May 4th (p001 to  p042) has > > a red  
>> background > >   
>> http://www.opencm3.net/m3tests/m3tests-NT386-2008-04-23-13-30-57.html</pre></td></tr> <tr class="bgred tl"><td> p002 </td><td class="small" width="45%"> Text</td><td class="small"><pre>Comparing files P0\P002\stdout.build and ..\SRC\P0\P002\STDOUT.BUILD***** P0\P002\stdout.buildlink @_m3responsefile0.txt 2>&1 > pgm.lstmt /nologo /manifest pgm.exe.manifest /outputresource:pgm.exe;1***** ..\SRC\P0\P002\STDOUT.BUILD*****Comparing files P0\P002\stderr.build and ..\SRC\P0\P002\STDERR.BUILDFC: no differences encounteredComparing files P0\P002\stdout.pgm and ..\SRC\P0\P002\STDOUT.PGMFC: no differences encounteredComparing files P0\P002\stderr.pgm and ..\SRC\P0\P002\STDERR.PGMFC: no differences encounteredand almost the same pattern in the above tests.I would suggest if it is thinkable using NT386 variant with a complete dedicated machine/system, I can try to set up one this week and send the data back (the machine is behind a proxy), but I don't remember the mail explaining the set up, and also want to know if there is a chance of run the tests with one run script.Also what is the best natural way to put a new tests, and the standard name it should > > have?Thanks> >> >> > Enviado desde Correo Yahoo!La bandeja de entrada más inteligente.> > > > -- > Olaf Wagner -- elego Software Solutions GmbH> Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95> http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr:   
>> DE163214194>



-- 
Olaf Wagner -- elego Software Solutions GmbH
                Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
phone: +49 30 23 45 86 96  mobile: +49 177 2345 869  fax: +49 30 23 45 86 95
    http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin
Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194