[M3devel] CM3 report compiler compilation

[Rodney M. Bates]

On 10/11/2016 02:48 AM, Wolfgang Keller wrote:
>
>
> Am 10.10.2016 um 19:16 schrieb Rodney M. Bates:
>>
>>
>> On 10/09/2016 03:22 PM, Rodney M. Bates wrote:
>>> While there is plenty of room for improvement in CM3, perhaps it has
>>> provoked for you, a crisis of rising expectations, leading you to
>>> apply a higher standard than to other build systems.  The Modula-3
>>> language and the CM3 implementation of it are already far ahead of the
>>> prevailing standard for software build systems.
>
> The latter may be true when compared to COBOL and C. The C development system in antiquity of engineering and has only survived today because of its inviting openness for all sorts of unmaintainability manifestations. Many programmers and developers still measure their personal importance by means of length and difficulty of their work, not to forget! And some do get paid for. - For a reason, Pascal and Ada have been developed to allow for a better controlled building and development process, and these are the more suitable languages to compare to!
>
>
>>> For *within* a package, CM3 automatically detects and carries out any
>>> recompilation necessary due to changes, and on a declaration
>>> granularity.  If an interface you import has been changed only in
>>> declarations you do not actually reference, your code need not be
>>> recompiled.
>
>
>>> Compare this to C/C++/Make, where, if the file you depend
>>> on has been touched in any way, even just editing of comments or just
>>> changing its recorded date, your file has to be recompiled.
>>>
>>> Moreover, the dependencies have to be manually written into a
>>> Makefile, or the recompilation will not occur when it is necessary.
>>> There are side tools around that will automatically generate Makefile
>>> dependencies, but they can only judge on the basis of #include lines,
>>> not actual uses.  They have to be rerun along with every
>>> recompilation, or else, inevitably, the Makefile will get out of date.
>>> That can lead to very difficult-to-diagnose bugs that often don't show
>>> up for a long time later, after many unrelated changes and rebuilds.
>>>
>>> Such tools cannot help with the very difficult task of discovering
>>> when a #include has become unnecessary.  This is extremely difficult
>>> to find, and requires combing through the entire closure of #includes,
>>> in order, not just the direct ones, looking for referenced
>>> declarations.  CM3, in contrast will warn of an IMPORT that is not
>>> needed.  BTW, in Modula-3,only the direct IMPORTs need to be examined.
>>>
>>> Between packages, CM3 always detects incompatibilities.  In most
>>> systems, the closest you get to this is via a package manager, e.g.,
>>> APT or RPM.  Again, for these to work, somebody has to manually note
>>> the dependencies in the package.  These are even coarser than
>>> file-granularity.  They can only use library names, with embedded
>>> version numbers, to detect problems.  There is nothing to check
>>> whether the version numbers correctly reflect incompatibilities.  What
>>> detection does happen is delayed from link time to install time.  And
>>> you have to manually rebuild the library to fix it.
>>>
>>> CM3 will always detect, at declaration granularity, at link time, when
>>> such an incompatibility exists, with no extra manual effort beyond the
>>> actual modification itself.  The worst you can criticize is that the
>>> messages are not very informative (but still much better), and as
>>> above, you have to ask for a recompile of your package.
>
> That is one point, yes, that
>
>      Fatal Error: bad version stamps: ConnFD.i3"
>
> is not informative above the core type of error. It should name all involved modules and result with a recommendation of action to be taken, if such an inference can be made. The current user feedback is not very practical!
>
>>>
>>> But the situation that brought this all up goes even deeper. There,
>>> package cm3ide links in file ConnFD.i3, in package tcp, with object
>>> code in libm3tcp.so, which needs to be recompiled because it links in
>>> ThreadPThread.m3, in package m3core, compiled into libm3core.so.  This
>>> is (non-trivial) *transitive* dependency in the link closure of
>>> libraries.  CM3 detected this, but has to be asked manually to do the
>>> necessary recompilation.  I am far from knowledgeable about all build
>>> systems in existence, but I would be be very surprised if more than
>>> one or two others do even this detection.
>>
>> Actually, I described this wrongly, or at oversimplifiedly.
>> What I am sure really happed here is cm3ide links in m3core directly,
>> and also links in tcp, which, the last time it was compiled,
>> linked in a different (earlier, in this case, but not necessarily,
>> in general) version of m3core, which is what CM3
>> complained about.  It detected this directly from type
>> definitions, without needing version numbers, file names,
>> dates, etc.  It does however, rely on having the compiler-generated
>> .M3WEB file for m3core properly installed beside the library.
>>
>
> What ever one may say about the depth of transitive recompiling, on thing is certain: a script inside CM3 which claims to build a target system must make sure it correctly does so. Can we agree on that? There may be a need to organise packages and compilations in a way that ensures a consistent rebuild. That is what the "make" files are traditionally good for. As it seems from your report, cm3 is not free for the need of such.
>
> The objective appears quite simple: make sure only packages are referenced which are of the same compiler version-id. If this is not the case and the source is available -> recompile, otherwise break with an error msg. If package + source are only available remote, consider to recompile to a local copy (not replacing the remote binary). Such behaviour can be made dependent on pragmas and compiler switches. There could be a problem with cyclic package references, but I assume you can also solve that during linkage. That's, I admit, my guess here. ;)
>

I has just occurred to me that you probably view an IDE as fundamental
part of the development system.  I have been thinking of it as an
independent tool, similar to the many other executables in the repository,
since I had never tried it until just recently.  Apparently, not many
others have either, or the looping would have been noticed earlier.

BTW, I have tracked at least one instance of the looping to a LONGREAL
arithmetic bug in ThreadPThread itself, so it's not cm3ide's fault.
So far, it's not so clear what to do about it.  It involves busy waiting
for network traffic, and that's a delicate balance.

>>>
>>> I am not even sure one would want a build system to automatically
>>> transitively rebuild across multiple libraries.  It might work
>>> when all the libraries are in a single repository like Modula3-cm3.
>>> But often, programs use libraries from very different projects,
>>> with source code in far-flung places, if available at all.
>>>
>>> I suppose one could write a Makefile that notes when a linked-in
>>> library depends on another linked-in library.  I doubt this has been
>>> done very often.  Even so, it would at best, detect only at whole
>>> library granularity.
>>>
>>>
>>>
>>> On 10/08/2016 01:22 AM, Wolfgang Keller wrote:
>>>> The "upgrade" process claims to update a target installation system (perhaps also called the "work installation"). In my test case this was in " /home/user/Projekte/M3-Test2/". This is also the system which is made available in global environment paths.
>>>>
>>>> 1) If - after creation of the target system - compiling "cm3" within "cm3ide" of CM3 (git) fails then the claim to having built a valid upgrade has also failed (if we assume that cm3ide holds a valid composition of sources).
>>>>
>>>> 2) The compile process reports taking reference to m3bundle which appears to imply it is an essential part of the core compiler.
>>>>
>>>> "/home/user/Projekte/M3-Test2/bin/m3bundle -name CM3_IDE_Bundle -F/tmp/qk
>>>> new source -> compiling Buf.i3
>>>> new source -> compiling Buf.m3
>>>> new source -> compiling ErrLog.i3
>>>> new source -> compiling ErrLog.m3
>>>>
>>>> Fatal Error: bad version stamps: ConnFD.i3"
>>>>
>>>> 3) If we look at the /bin directory of the target system, we find some updated executables with the exception of "cm3ide" and "m3bundle" which still date to 12 July 2010.
>>>>
>>>> 4) Suspicion arises that omitting compilation or copy of m3bundle causes the failed compile process.
>>>>
>>>> 5) Obviously the "cm3" folder in CM3 (git) is not a sufficient source for executing the compiler. Hence I recommend there should be a target "build" system inside the git working repository as a result of any "upgrade" or "build" script. This build area should be the primary target of all scripts targeting at ensuring a workable compiler installation. Then in a secondary step, which may also be scripted but may also be omitted, copy from this area can be taken to update, upgrade or build new any other installation. The "build" area in git can be set to .gitignore so it doesn't litter up the tracking system. Is that an idea to follow?
>>>>
>>>> - Wolfgang
>>>>
>>>> Am 08.10.2016 um 01:24 schrieb Rodney M. Bates:
>>>>>
>>>>>
>>>>> On 10/07/2016 01:26 PM, Wolfgang Keller wrote:
>>>>>>
>>>>>>
>>>>>> Am 07.10.2016 um 18:17 schrieb Rodney M. Bates:
>>>>>>>
>>>>>>>
>>>>>>> On 10/07/2016 03:53 AM, Wolfgang Keller wrote:
>>>>>>>> I attempted at compiler compilation and found a few peculiar things.
>>>>>>>>
>>>>>>>> 1. The Result
>>>>>>>>
>>>>>>>> Critical Mass Modula-3 version d5.10.0
>>>>>>>>    last updated: 2015-05-21
>>>>>>>>    compiled: 2016-10-07 03:58:31
>>>>>>>>    configuration: /home/user/Projekte/M3-Test2/bin/cm3.cfg
>>>>>>>>    host: AMD64_LINUX
>>>>>>>>    target: AMD64_LINUX
>>>>>>>>
>>>>>>>> This is how cm3 identifies.
>>>>>>>>
>>>>>>>> a) "last updated" - since you modified the sources, should there be a younger date?
>>>>>>>
>>>>>>> Yes.  This is now manually updated in scripts/version.quake.
>>>>>>>
>>>>>>> m3devel list:  We need to do this with every commit, or else automate it.
>>>>>>>
>>>>>>>>
>>>>>>>> b) compilation results (executables) as well as configuration file have been put into CM3 (Git version) folders and into the "independent" installation of the older version of CM3, i.e. 5.8.6. I see this critical! The independent installation should stay unmodified in its functionality. Users might wish to continue to use it, e.g. when the newer version proves buggy or otherwise undesirable.
>>>>>>>>
>>>>>>>
>>>>>>> Yes, we have been working on this.  You can recreate a clean release installation at
>>>>>>> any time, but it's a real pain for intermediate versions during development.  I am
>>>>>>> very conservative about saving existing versions before touching the compiler or its
>>>>>>> core libraries, either by copying critical executable file and library files, or making
>>>>>>> whole copies of /usr/local/cm3.
>>>>>>>
>>>>>>>> c) Erroneous execution. When calling "cm3" in the "cm3ide" directory it does not compile and the text result is
>>>>>>>>
>>>>>>>> -- begin --
>>>>>>>> --- building in AMD64_LINUX ---
>>>>>>>>
>>>>>>>> ignoring ../src/m3overrides
>>>>>>>>
>>>>>>>> Fatal Error: bad version stamps: ConnFD.i3
>>>>>>>>
>>>>>>>> inconsistent opaque object info for type _t1541f475
>>>>>>>> super type: _te422a136  data: (size: 64, align: 64) method: (size: 0)
>>>>>>>>   => ConnFD.i3
>>>>>>>> super type: _te422a136  data: (size: 192, align: 64) method: (size: 0)
>>>>>>>>   => ThreadPThread.m3
>>>>>>>>
>>>>>>>> -- end --
>>>>>>>>
>>>>>>>
>>>>>>> This is almost certainly a need for complete recompile of cm3ide. Try, in
>>>>>>> the cm3ide directory:
>>>>>>>
>>>>>>> cm3 -realclean
>>>>>>> cm3
>>>>>>>
>>>>>>>
>>>>>>> Within a package, the compiler infers from source code what needs to be recompiled
>>>>>>> and does so. Between packages, and for certain kinds of changes in machine-level
>>>>>>> types, it can only detect incompatibilities.  This is rare. The messages are
>>>>>>> highly uninformative, but fixing them requires changes to the compiler's internal
>>>>>>> intermediate representation, which entails widespread, but mundane code changes.
>>>>>>> I have undertaken some of this previously, but it hasn't had the highest priority.
>>>>>>
>>>>>> Taking your advice it results in the following:
>>>>>>
>>>>>> -- begin --
>>>>>> --- building in AMD64_LINUX ---
>>>>>>
>>>>>> ignoring ../src/m3overrides
>>>>>>
>>>>>> /home/user/Projekte/M3-Test2/bin/m3bundle -name CM3_IDE_Bundle -F/tmp/qk
>>>>>> new source -> compiling Buf.i3
>>>>>> new source -> compiling Buf.m3
>>>>>> new source -> compiling ErrLog.i3
>>>>>> new source -> compiling ErrLog.m3
>>>>>>
>>>>>> Fatal Error: bad version stamps: ConnFD.i3
>>>>>>
>>>>>> inconsistent opaque object info for type _t1541f475
>>>>>> super type: _te422a136  data: (size: 64, align: 64) method: (size: 0)
>>>>>>   => ConnFD.i3
>>>>>> super type: _te422a136  data: (size: 192, align: 64) method: (size: 0)
>>>>>>   => ThreadPThread.m3
>>>>>> -- end --
>>>>>>
>>>>>> Obviously compilation takes reference into the "other" installation and resulting in inconsistencies. Why does it do so? This is the type of mixed up configurations that I have been speaking of! Up to now I don't quite understand how people are working with this system. Does it work for you?
>>>>>>
>>>>>
>>>>> It's the other way around.  ConnFD.i3 has an old, already compiled version around that
>>>>> depends on something in ThreadPThread, which has changed and been recently recompiled.
>>>>> ConnFD.i3 now needs to be recompiled to adapt, now that a new libm3core is installed
>>>>> (which ThreadPThread is part of).
>>>>>
>>>>> I should have looked at where ConnFD.i3 is located.  It's in package group m3-comm, which
>>>>> upgrade.sh apparently didn't rebuild.  So try this, as I just did:
>>>>>
>>>>> in directory scripts:
>>>>>
>>>>> ./do-cm3-comm.sh realclean
>>>>> ,/do-cm3-comm.sh buildship
>>>>>
>>>>> Then try rebuilding cm3ide.  There could be more things that haven't been rebuilt
>>>>> by update.sh.  I would predict that m3-ui and m3-db haven't, though I doubt cm3ide
>>>>> needs m3-ui.  There are do-cm3-*.sh scripts for a lot of these groups.
>>>>>
>>>>> I have worked in compilation systems for modular languages that have a two-level
>>>>> automatic rebuild system.  Modula-3 does not.  The automatic rebuild only works
>>>>> within a package.  It would be nice to have one.
>>>>>
>>>>>> Theoretically, the missing file attributes may have had an impact on the results of the upgrade. Who knows?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
>

-- 
Rodney Bates
rodney.m.bates at acm.org