[M3devel] CM3 report compiler compilation

[Darko Volaric]

Wolfgang - are you subscribed to the list? I can't see any of your original
emails, just those quoted.

I've been thinking about this what seems to be missing is a "project" based
approach. I emulate this by using include_dir instead of import and
makefile overrides (for technical reasons), but right now makefiles don't
let you nominate what libraries/packages are part of the active project and
what are a part of another project.

When a large project, such as the compiler, involves several separate
libraries/packages, you ought to be able to nominate them for rebuilding as
required from the client package, instead of just importing whatever was
last shipped. Instead we have the build scripts which are an unreliable and
ugly hack. You may not want to have the compiler try to divine what should
be rebuilt across arbitary packages, but you should be able to nominate
what packages belong to the same project and have the compiler manage the
builds.

On Mon, Oct 10, 2016 at 7:16 PM, Rodney M. Bates <rodney_bates at lcwb.coop>
wrote:

>
>
> On 10/09/2016 03:22 PM, Rodney M. Bates wrote:
>
>> While there is plenty of room for improvement in CM3, perhaps it has
>> provoked for you, a crisis of rising expectations, leading you to
>> apply a higher standard than to other build systems.  The Modula-3
>> language and the CM3 implementation of it are already far ahead of the
>> prevailing standard for software build systems.
>>
>> For *within* a package, CM3 automatically detects and carries out any
>> recompilation necessary due to changes, and on a declaration
>> granularity.  If an interface you import has been changed only in
>> declarations you do not actually reference, your code need not be
>> recompiled.  Compare this to C/C++/Make, where, if the file you depend
>> on has been touched in any way, even just editing of comments or just
>> changing its recorded date, your file has to be recompiled.
>>
>> Moreover, the dependencies have to be manually written into a
>> Makefile, or the recompilation will not occur when it is necessary.
>> There are side tools around that will automatically generate Makefile
>> dependencies, but they can only judge on the basis of #include lines,
>> not actual uses.  They have to be rerun along with every
>> recompilation, or else, inevitably, the Makefile will get out of date.
>> That can lead to very difficult-to-diagnose bugs that often don't show
>> up for a long time later, after many unrelated changes and rebuilds.
>>
>> Such tools cannot help with the very difficult task of discovering
>> when a #include has become unnecessary.  This is extremely difficult
>> to find, and requires combing through the entire closure of #includes,
>> in order, not just the direct ones, looking for referenced
>> declarations.  CM3, in contrast will warn of an IMPORT that is not
>> needed.  BTW, in Modula-3,only the direct IMPORTs need to be examined.
>>
>> Between packages, CM3 always detects incompatibilities.  In most
>> systems, the closest you get to this is via a package manager, e.g.,
>> APT or RPM.  Again, for these to work, somebody has to manually note
>> the dependencies in the package.  These are even coarser than
>> file-granularity.  They can only use library names, with embedded
>> version numbers, to detect problems.  There is nothing to check
>> whether the version numbers correctly reflect incompatibilities.  What
>> detection does happen is delayed from link time to install time.  And
>> you have to manually rebuild the library to fix it.
>>
>> CM3 will always detect, at declaration granularity, at link time, when
>> such an incompatibility exists, with no extra manual effort beyond the
>> actual modification itself.  The worst you can criticize is that the
>> messages are not very informative (but still much better), and as
>> above, you have to ask for a recompile of your package.
>>
>> But the situation that brought this all up goes even deeper.  There,
>> package cm3ide links in file ConnFD.i3, in package tcp, with object
>> code in libm3tcp.so, which needs to be recompiled because it links in
>> ThreadPThread.m3, in package m3core, compiled into libm3core.so.  This
>> is (non-trivial) *transitive* dependency in the link closure of
>> libraries.  CM3 detected this, but has to be asked manually to do the
>> necessary recompilation.  I am far from knowledgeable about all build
>> systems in existence, but I would be be very surprised if more than
>> one or two others do even this detection.
>>
>
> Actually, I described this wrongly, or at oversimplifiedly.
> What I am sure really happed here is cm3ide links in m3core directly,
> and also links in tcp, which, the last time it was compiled,
> linked in a different (earlier, in this case, but not necessarily,
> in general) version of m3core, which is what CM3
> complained about.  It detected this directly from type
> definitions, without needing version numbers, file names,
> dates, etc.  It does however, rely on having the compiler-generated
> .M3WEB file for m3core properly installed beside the library.
>
>
>
>> I am not even sure one would want a build system to automatically
>> transitively rebuild across multiple libraries.  It might work
>> when all the libraries are in a single repository like Modula3-cm3.
>> But often, programs use libraries from very different projects,
>> with source code in far-flung places, if available at all.
>>
>> I suppose one could write a Makefile that notes when a linked-in
>> library depends on another linked-in library.  I doubt this has been
>> done very often.  Even so, it would at best, detect only at whole
>> library granularity.
>>
>>
>>
>> On 10/08/2016 01:22 AM, Wolfgang Keller wrote:
>>
>>> The "upgrade" process claims to update a target installation system
>>> (perhaps also called the "work installation"). In my test case this was in
>>> " /home/user/Projekte/M3-Test2/". This is also the system which is made
>>> available in global environment paths.
>>>
>>> 1) If - after creation of the target system - compiling "cm3" within
>>> "cm3ide" of CM3 (git) fails then the claim to having built a valid upgrade
>>> has also failed (if we assume that cm3ide holds a valid composition of
>>> sources).
>>>
>>> 2) The compile process reports taking reference to m3bundle which
>>> appears to imply it is an essential part of the core compiler.
>>>
>>> "/home/user/Projekte/M3-Test2/bin/m3bundle -name CM3_IDE_Bundle
>>> -F/tmp/qk
>>> new source -> compiling Buf.i3
>>> new source -> compiling Buf.m3
>>> new source -> compiling ErrLog.i3
>>> new source -> compiling ErrLog.m3
>>>
>>> Fatal Error: bad version stamps: ConnFD.i3"
>>>
>>> 3) If we look at the /bin directory of the target system, we find some
>>> updated executables with the exception of "cm3ide" and "m3bundle" which
>>> still date to 12 July 2010.
>>>
>>> 4) Suspicion arises that omitting compilation or copy of m3bundle causes
>>> the failed compile process.
>>>
>>> 5) Obviously the "cm3" folder in CM3 (git) is not a sufficient source
>>> for executing the compiler. Hence I recommend there should be a target
>>> "build" system inside the git working repository as a result of any
>>> "upgrade" or "build" script. This build area should be the primary target
>>> of all scripts targeting at ensuring a workable compiler installation. Then
>>> in a secondary step, which may also be scripted but may also be omitted,
>>> copy from this area can be taken to update, upgrade or build new any other
>>> installation. The "build" area in git can be set to .gitignore so it
>>> doesn't litter up the tracking system. Is that an idea to follow?
>>>
>>> - Wolfgang
>>>
>>> Am 08.10.2016 um 01:24 schrieb Rodney M. Bates:
>>>
>>>>
>>>>
>>>> On 10/07/2016 01:26 PM, Wolfgang Keller wrote:
>>>>
>>>>>
>>>>>
>>>>> Am 07.10.2016 um 18:17 schrieb Rodney M. Bates:
>>>>>
>>>>>>
>>>>>>
>>>>>> On 10/07/2016 03:53 AM, Wolfgang Keller wrote:
>>>>>>
>>>>>>> I attempted at compiler compilation and found a few peculiar things.
>>>>>>>
>>>>>>> 1. The Result
>>>>>>>
>>>>>>> Critical Mass Modula-3 version d5.10.0
>>>>>>>    last updated: 2015-05-21
>>>>>>>    compiled: 2016-10-07 03:58:31
>>>>>>>    configuration: /home/user/Projekte/M3-Test2/bin/cm3.cfg
>>>>>>>    host: AMD64_LINUX
>>>>>>>    target: AMD64_LINUX
>>>>>>>
>>>>>>> This is how cm3 identifies.
>>>>>>>
>>>>>>> a) "last updated" - since you modified the sources, should there be
>>>>>>> a younger date?
>>>>>>>
>>>>>>
>>>>>> Yes.  This is now manually updated in scripts/version.quake.
>>>>>>
>>>>>> m3devel list:  We need to do this with every commit, or else automate
>>>>>> it.
>>>>>>
>>>>>>
>>>>>>> b) compilation results (executables) as well as configuration file
>>>>>>> have been put into CM3 (Git version) folders and into the "independent"
>>>>>>> installation of the older version of CM3, i.e. 5.8.6. I see this critical!
>>>>>>> The independent installation should stay unmodified in its functionality.
>>>>>>> Users might wish to continue to use it, e.g. when the newer version proves
>>>>>>> buggy or otherwise undesirable.
>>>>>>>
>>>>>>>
>>>>>> Yes, we have been working on this.  You can recreate a clean release
>>>>>> installation at
>>>>>> any time, but it's a real pain for intermediate versions during
>>>>>> development.  I am
>>>>>> very conservative about saving existing versions before touching the
>>>>>> compiler or its
>>>>>> core libraries, either by copying critical executable file and
>>>>>> library files, or making
>>>>>> whole copies of /usr/local/cm3.
>>>>>>
>>>>>> c) Erroneous execution. When calling "cm3" in the "cm3ide" directory
>>>>>>> it does not compile and the text result is
>>>>>>>
>>>>>>> -- begin --
>>>>>>> --- building in AMD64_LINUX ---
>>>>>>>
>>>>>>> ignoring ../src/m3overrides
>>>>>>>
>>>>>>> Fatal Error: bad version stamps: ConnFD.i3
>>>>>>>
>>>>>>> inconsistent opaque object info for type _t1541f475
>>>>>>> super type: _te422a136  data: (size: 64, align: 64)  method: (size:
>>>>>>> 0)
>>>>>>>   => ConnFD.i3
>>>>>>> super type: _te422a136  data: (size: 192, align: 64) method: (size:
>>>>>>> 0)
>>>>>>>   => ThreadPThread.m3
>>>>>>>
>>>>>>> -- end --
>>>>>>>
>>>>>>>
>>>>>> This is almost certainly a need for complete recompile of cm3ide.
>>>>>> Try, in
>>>>>> the cm3ide directory:
>>>>>>
>>>>>> cm3 -realclean
>>>>>> cm3
>>>>>>
>>>>>>
>>>>>> Within a package, the compiler infers from source code what needs to
>>>>>> be recompiled
>>>>>> and does so. Between packages, and for certain kinds of changes in
>>>>>> machine-level
>>>>>> types, it can only detect incompatibilities.  This is rare. The
>>>>>> messages are
>>>>>> highly uninformative, but fixing them requires changes to the
>>>>>> compiler's internal
>>>>>> intermediate representation, which entails widespread, but mundane
>>>>>> code changes.
>>>>>> I have undertaken some of this previously, but it hasn't had the
>>>>>> highest priority.
>>>>>>
>>>>>
>>>>> Taking your advice it results in the following:
>>>>>
>>>>> -- begin --
>>>>> --- building in AMD64_LINUX ---
>>>>>
>>>>> ignoring ../src/m3overrides
>>>>>
>>>>> /home/user/Projekte/M3-Test2/bin/m3bundle -name CM3_IDE_Bundle
>>>>> -F/tmp/qk
>>>>> new source -> compiling Buf.i3
>>>>> new source -> compiling Buf.m3
>>>>> new source -> compiling ErrLog.i3
>>>>> new source -> compiling ErrLog.m3
>>>>>
>>>>> Fatal Error: bad version stamps: ConnFD.i3
>>>>>
>>>>> inconsistent opaque object info for type _t1541f475
>>>>> super type: _te422a136  data: (size: 64, align: 64)  method: (size: 0)
>>>>>   => ConnFD.i3
>>>>> super type: _te422a136  data: (size: 192, align: 64)  method: (size: 0)
>>>>>   => ThreadPThread.m3
>>>>> -- end --
>>>>>
>>>>> Obviously compilation takes reference into the "other" installation
>>>>> and resulting in inconsistencies. Why does it do so? This is the type of
>>>>> mixed up configurations that I have been speaking of! Up to now I don't
>>>>> quite understand how people are working with this system. Does it work for
>>>>> you?
>>>>>
>>>>>
>>>> It's the other way around.  ConnFD.i3 has an old, already compiled
>>>> version around that
>>>> depends on something in ThreadPThread, which has changed and been
>>>> recently recompiled.
>>>> ConnFD.i3 now needs to be recompiled to adapt, now that a new libm3core
>>>> is installed
>>>> (which ThreadPThread is part of).
>>>>
>>>> I should have looked at where ConnFD.i3 is located.  It's in package
>>>> group m3-comm, which
>>>> upgrade.sh apparently didn't rebuild.  So try this, as I just did:
>>>>
>>>> in directory scripts:
>>>>
>>>> ./do-cm3-comm.sh realclean
>>>> ,/do-cm3-comm.sh buildship
>>>>
>>>> Then try rebuilding cm3ide.  There could be more things that haven't
>>>> been rebuilt
>>>> by update.sh.  I would predict that m3-ui and m3-db haven't, though I
>>>> doubt cm3ide
>>>> needs m3-ui.  There are do-cm3-*.sh scripts for a lot of these groups.
>>>>
>>>> I have worked in compilation systems for modular languages that have a
>>>> two-level
>>>> automatic rebuild system.  Modula-3 does not.  The automatic rebuild
>>>> only works
>>>> within a package.  It would be nice to have one.
>>>>
>>>> Theoretically, the missing file attributes may have had an impact on
>>>>> the results of the upgrade. Who knows?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
> --
> Rodney Bates
> rodney.m.bates at acm.org
> _______________________________________________
> M3devel mailing list
> M3devel at elegosoft.com
> https://m3lists.elegosoft.com/mailman/listinfo/m3devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20161011/84a476a1/attachment-0001.html>