[M3devel] CM3 report compiler compilation

Sun Oct 9 22:22:53 CEST 2016

While there is plenty of room for improvement in CM3, perhaps it has
provoked for you, a crisis of rising expectations, leading you to
apply a higher standard than to other build systems.  The Modula-3
language and the CM3 implementation of it are already far ahead of the
prevailing standard for software build systems.

For *within* a package, CM3 automatically detects and carries out any
recompilation necessary due to changes, and on a declaration
granularity.  If an interface you import has been changed only in
declarations you do not actually reference, your code need not be
recompiled.  Compare this to C/C++/Make, where, if the file you depend
on has been touched in any way, even just editing of comments or just
changing its recorded date, your file has to be recompiled.

Moreover, the dependencies have to be manually written into a
Makefile, or the recompilation will not occur when it is necessary.
There are side tools around that will automatically generate Makefile
dependencies, but they can only judge on the basis of #include lines,
not actual uses.  They have to be rerun along with every
recompilation, or else, inevitably, the Makefile will get out of date.
That can lead to very difficult-to-diagnose bugs that often don't show
up for a long time later, after many unrelated changes and rebuilds.

Such tools cannot help with the very difficult task of discovering
when a #include has become unnecessary.  This is extremely difficult
to find, and requires combing through the entire closure of #includes,
in order, not just the direct ones, looking for referenced
declarations.  CM3, in contrast will warn of an IMPORT that is not
needed.  BTW, in Modula-3,only the direct IMPORTs need to be examined.

Between packages, CM3 always detects incompatibilities.  In most
systems, the closest you get to this is via a package manager, e.g.,
APT or RPM.  Again, for these to work, somebody has to manually note
the dependencies in the package.  These are even coarser than
file-granularity.  They can only use library names, with embedded
version numbers, to detect problems.  There is nothing to check
whether the version numbers correctly reflect incompatibilities.  What
detection does happen is delayed from link time to install time.  And
you have to manually rebuild the library to fix it.

CM3 will always detect, at declaration granularity, at link time, when
such an incompatibility exists, with no extra manual effort beyond the
actual modification itself.  The worst you can criticize is that the
messages are not very informative (but still much better), and as
above, you have to ask for a recompile of your package.

But the situation that brought this all up goes even deeper.  There,
package cm3ide links in file ConnFD.i3, in package tcp, with object
code in libm3tcp.so, which needs to be recompiled because it links in
ThreadPThread.m3, in package m3core, compiled into libm3core.so.  This
is (non-trivial) *transitive* dependency in the link closure of
libraries.  CM3 detected this, but has to be asked manually to do the
necessary recompilation.  I am far from knowledgeable about all build
systems in existence, but I would be be very surprised if more than
one or two others do even this detection.

I am not even sure one would want a build system to automatically
transitively rebuild across multiple libraries.  It might work
when all the libraries are in a single repository like Modula3-cm3.
But often, programs use libraries from very different projects,
with source code in far-flung places, if available at all.

I suppose one could write a Makefile that notes when a linked-in
library depends on another linked-in library.  I doubt this has been
done very often.  Even so, it would at best, detect only at whole
library granularity.

On 10/08/2016 01:22 AM, Wolfgang Keller wrote:
> The "upgrade" process claims to update a target installation system (perhaps also called the "work installation"). In my test case this was in " /home/user/Projekte/M3-Test2/". This is also the system which is made available in global environment paths.
>
> 1) If - after creation of the target system - compiling "cm3" within "cm3ide" of CM3 (git) fails then the claim to having built a valid upgrade has also failed (if we assume that cm3ide holds a valid composition of sources).
>
> 2) The compile process reports taking reference to m3bundle which appears to imply it is an essential part of the core compiler.
>
> "/home/user/Projekte/M3-Test2/bin/m3bundle -name CM3_IDE_Bundle -F/tmp/qk
> new source -> compiling Buf.i3
> new source -> compiling Buf.m3
> new source -> compiling ErrLog.i3
> new source -> compiling ErrLog.m3
>
> Fatal Error: bad version stamps: ConnFD.i3"
>
> 3) If we look at the /bin directory of the target system, we find some updated executables with the exception of "cm3ide" and "m3bundle" which still date to 12 July 2010.
>
> 4) Suspicion arises that omitting compilation or copy of m3bundle causes the failed compile process.
>
> 5) Obviously the "cm3" folder in CM3 (git) is not a sufficient source for executing the compiler. Hence I recommend there should be a target "build" system inside the git working repository as a result of any "upgrade" or "build" script. This build area should be the primary target of all scripts targeting at ensuring a workable compiler installation. Then in a secondary step, which may also be scripted but may also be omitted, copy from this area can be taken to update, upgrade or build new any other installation. The "build" area in git can be set to .gitignore so it doesn't litter up the tracking system. Is that an idea to follow?
>
> - Wolfgang
>
> Am 08.10.2016 um 01:24 schrieb Rodney M. Bates:
>>
>>
>> On 10/07/2016 01:26 PM, Wolfgang Keller wrote:
>>>
>>>
>>> Am 07.10.2016 um 18:17 schrieb Rodney M. Bates:
>>>>
>>>>
>>>> On 10/07/2016 03:53 AM, Wolfgang Keller wrote:
>>>>> I attempted at compiler compilation and found a few peculiar things.
>>>>>
>>>>> 1. The Result
>>>>>
>>>>> Critical Mass Modula-3 version d5.10.0
>>>>>    last updated: 2015-05-21
>>>>>    compiled: 2016-10-07 03:58:31
>>>>>    configuration: /home/user/Projekte/M3-Test2/bin/cm3.cfg
>>>>>    host: AMD64_LINUX
>>>>>    target: AMD64_LINUX
>>>>>
>>>>> This is how cm3 identifies.
>>>>>
>>>>> a) "last updated" - since you modified the sources, should there be a younger date?
>>>>
>>>> Yes.  This is now manually updated in scripts/version.quake.
>>>>
>>>> m3devel list:  We need to do this with every commit, or else automate it.
>>>>
>>>>>
>>>>> b) compilation results (executables) as well as configuration file have been put into CM3 (Git version) folders and into the "independent" installation of the older version of CM3, i.e. 5.8.6. I see this critical! The independent installation should stay unmodified in its functionality. Users might wish to continue to use it, e.g. when the newer version proves buggy or otherwise undesirable.
>>>>>
>>>>
>>>> Yes, we have been working on this.  You can recreate a clean release installation at
>>>> any time, but it's a real pain for intermediate versions during development.  I am
>>>> very conservative about saving existing versions before touching the compiler or its
>>>> core libraries, either by copying critical executable file and library files, or making
>>>> whole copies of /usr/local/cm3.
>>>>
>>>>> c) Erroneous execution. When calling "cm3" in the "cm3ide" directory it does not compile and the text result is
>>>>>
>>>>> -- begin --
>>>>> --- building in AMD64_LINUX ---
>>>>>
>>>>> ignoring ../src/m3overrides
>>>>>
>>>>> Fatal Error: bad version stamps: ConnFD.i3
>>>>>
>>>>> inconsistent opaque object info for type _t1541f475
>>>>> super type: _te422a136  data: (size: 64, align: 64)  method: (size: 0)
>>>>>   => ConnFD.i3
>>>>> super type: _te422a136  data: (size: 192, align: 64) method: (size: 0)
>>>>>   => ThreadPThread.m3
>>>>>
>>>>> -- end --
>>>>>
>>>>
>>>> This is almost certainly a need for complete recompile of cm3ide. Try, in
>>>> the cm3ide directory:
>>>>
>>>> cm3 -realclean
>>>> cm3
>>>>
>>>>
>>>> Within a package, the compiler infers from source code what needs to be recompiled
>>>> and does so. Between packages, and for certain kinds of changes in machine-level
>>>> types, it can only detect incompatibilities.  This is rare. The messages are
>>>> highly uninformative, but fixing them requires changes to the compiler's internal
>>>> intermediate representation, which entails widespread, but mundane code changes.
>>>> I have undertaken some of this previously, but it hasn't had the highest priority.
>>>
>>> Taking your advice it results in the following:
>>>
>>> -- begin --
>>> --- building in AMD64_LINUX ---
>>>
>>> ignoring ../src/m3overrides
>>>
>>> /home/user/Projekte/M3-Test2/bin/m3bundle -name CM3_IDE_Bundle -F/tmp/qk
>>> new source -> compiling Buf.i3
>>> new source -> compiling Buf.m3
>>> new source -> compiling ErrLog.i3
>>> new source -> compiling ErrLog.m3
>>>
>>> Fatal Error: bad version stamps: ConnFD.i3
>>>
>>> inconsistent opaque object info for type _t1541f475
>>> super type: _te422a136  data: (size: 64, align: 64)  method: (size: 0)
>>>   => ConnFD.i3
>>> super type: _te422a136  data: (size: 192, align: 64)  method: (size: 0)
>>>   => ThreadPThread.m3
>>> -- end --
>>>
>>> Obviously compilation takes reference into the "other" installation and resulting in inconsistencies. Why does it do so? This is the type of mixed up configurations that I have been speaking of! Up to now I don't quite understand how people are working with this system. Does it work for you?
>>>
>>
>> It's the other way around.  ConnFD.i3 has an old, already compiled version around that
>> depends on something in ThreadPThread, which has changed and been recently recompiled.
>> ConnFD.i3 now needs to be recompiled to adapt, now that a new libm3core is installed
>> (which ThreadPThread is part of).
>>
>> I should have looked at where ConnFD.i3 is located.  It's in package group m3-comm, which
>> upgrade.sh apparently didn't rebuild.  So try this, as I just did:
>>
>> in directory scripts:
>>
>> ./do-cm3-comm.sh realclean
>> ,/do-cm3-comm.sh buildship
>>
>> Then try rebuilding cm3ide.  There could be more things that haven't been rebuilt
>> by update.sh.  I would predict that m3-ui and m3-db haven't, though I doubt cm3ide
>> needs m3-ui.  There are do-cm3-*.sh scripts for a lot of these groups.
>>
>> I have worked in compilation systems for modular languages that have a two-level
>> automatic rebuild system.  Modula-3 does not.  The automatic rebuild only works
>> within a package.  It would be nice to have one.
>>
>>> Theoretically, the missing file attributes may have had an impact on the results of the upgrade. Who knows?
>>>
>>>
>>>
>>>
>>
>
>

-- 
Rodney Bates
rodney.m.bates at acm.org