[M3devel] Assertions (Was FW: thread.fork getting nil on nt386gnu?)

[Rodney M. Bates]

Jay wrote:
> I find there are bugs /everywhere/. Code that is trying to find bugs is 
> no more immune from bugs than the code it is attempting to check. More 
> code leads to more bugs. More asserts lead to more incorrect asserts. 
> More runtime checks lead to more incorrect runtime checks. etc. Of 
> course, checks are still good, and like everything else they must be 
> written carefully and like everything else, when things go apparently 
> wrong, pretty much all code should be assumed wrong until proven 
> otherwise (read it, run it, read what it depends on, etc.). Code is 
> guilty until proven innocent, many times. One rotten apple spoils the 
> bunch....
>  
> 

Well, here is my documented experience.  For more than 20 years, I
have kept detailed logs of every bug in most programming projects of
significant size.  This is from a project in Modula-3 that I have
worked on and off on for years, with around 105K SLOC,

 From about 15% of the work for which I kept compile-time error data,
75% of all programming errors were found at compile time.  This is a
count of required changes to the code, not of compile time messages,
which, of course, usually greatly outnumber the number of actual
errors.  Where an equivalent change needed to be made systematically
in many places, I count this as one change.  No doubt records from
earlier in the project would show a higher percentage of compile-time
errors.

I have kept logs for all of the errors discovered at runtime.  From
about 40% of the more recent of these logs that are already classified
and tallied, the breakdown of how the errors manifested is;

Type of error                         % of all       % of errors found
                                       errors         at runtime

Compile time                          75    %        N/A

Failures of correct assertions        10    %        40    %

Runtime errors                         6.5  %        26    %

Observed bad output/behavior           6.5  %        26    %

Runaway loops/recursion                1.03 %         4.10 %

Bad internal values, observed in a      .42 %         1.68 %
debugger or debug output

Overzealous assertions                  .37 %         1.47 %

Uncaught exceptions                     .34 %         1.36 %

Synchronization/deadlock, etc.          .11 %          .42 %

By "overzealous assertions", I mean assertions that assert something
that they should not, and thus crash the program unnecessarily.
Instances of assertions finding real bugs outnumber overzealous
assertions more than 25 to one.  A very good tradeoff, I think.

The valid assertion failures were more numerous earlier in the
project.

Finding bugs by assertion failure also has the big advantage that it
does not require verifying correct output/behavior, or even deciding
manually what is correct.

Of course, such results depend on the programmer's style of deciding
what to assert.  The count of assertions coded in this program is 652.
YMMV.


-- 
-------------------------------------------------------------
Rodney M. Bates, retired assistant professor
Dept. of Computer Science, Wichita State University
Wichita, KS 67260-0083
316-978-3922
rodney.bates at wichita.edu