[M3devel] replace build system??

[Rodney M. Bates]

On 06/03/2016 02:51 PM, Jay K wrote:
> Moving a module from one package to another is a problem I have hit.
> I moved some path manipulation module a few years ago and incremental builds failed.
>

Yes, intra-package dependencies are dealt with very poorly.  There is no
automatic rebuilding at all (do we really want it, given that a library may have
multiple client packages?)  Detection of problems doesn't happen until link time,
and then with virtually useless error messages that contain only hash codes of
language constructs.  I'm not all confident of how complete the detection is either.

I fixed just one such message to be slightly more helpful a couple of years
ago, but several IR operators need additional operands to do anything decent.

> Not tracking optimization can be seen as a feature, or a bug.
>    I might just want to not optimize a few files that I'm debugging.
>
> WIDECHAR certainly.
>
> Choice of C backend is another. :)
> Nested functions have a different interface.
> But appending "c" to BUILD_DIR is maybe a good workaround. Maybe.
> I wish I could be link compatible but I don't see how.
> gcc puts the static link often in a special register dedicated to the purpose -- ABIs actually designed and optimized with this in mind!
> I put it as the last parameter.
>
>   - Jay
>
>
> ----------------------------------------
>> Date: Fri, 3 Jun 2016 09:44:12 -0500
>> From: rodney_bates at lcwb.coop
>> To: m3devel at elegosoft.com
>> Subject: Re: [M3devel] replace build system??
>>
>> BTW, 'though the current build system is considerably sophisticated,
>> it does have some bugs, which we should fix. I have the exact setup &
>> symptom for one of them written down somewhere. I think it's when adding
>> a procedure signature to an interface when a procedure body for it was
>> already present in a compiled exporting module, things don't get recompiled
>> right.
>>
>> Also, I occasionally get recompile failures that I don't think are valid,
>> and when I clean the package and recompile that, all is well. I haven't
>> ever tracked down how to reproduce this or noted its exact symptom.
>>
>> We also don't keep track of compile options that affect generated code,
>> so as to recompile when only the options change. E.g., optimization level,
>> debug info presence and format. I did do this for the size of WIDECHAR,
>> since mismatches here would be disastrous.
>>
>> On 06/03/2016 02:22 AM, Jay K wrote:
>>> I understand the C model very well, but I'm not certain of all
>>> the ramifications for Modula-3.
>>>
>>>
>>> I understand C++ has limited checking via name mangling.
>>>
>>>
>>> And that other systems carry names/signatures forward to runtime
>>> and check them there. Even dynamically linked C++. And Modula-3.
>>>
>>>
>>> However function names aren't everything -- i.e. they don't include
>>> structural hashes of types. It is name-based, not structure-based.
>>>
>>>
>>> The most common mistake in the loose C systems is not doing things
>>> wrong per se, but accidentally mixing old and new versions,
>>> and the result is undefined/crash.
>>>
>>>
>>> We have the same problem with our "extern" mechanism -- how many times
>>> I debugged stat corrupting stack until I put in the extra layer.
>>>
>>>
>>> C++ systems are getting better and check for "ODR violations" at link time.
>>> But checking at link time kinda contradicts my question.
>>> Goal being to fit into "generic" build systems but C++ only starts
>>> winning when the linker knows about it -- when the system is C++ specific.
>>> If C++ requires C++ linker/driver, we shouldn't feel too bad
>>> requiring a Modula-3 linker/driver.
>>>
>>>
>>>> Typical front end compilation actual work is similar to
>>>> or less than executable program startup/shutdown overhead.
>>>
>>>
>>> Ideally this is not the case. Ideally process startup is just referencing
>>> pages already in file system cache. Demand paging, unified file system cache
>>> and memory manager and all that (all systems NT, Linux, etc. are the same
>>> these days). But I realize in reality there is alway some amount of other CPU
>>> to burn at each process start. The ideal is not quite reached.
>>>
>>>> m3cc is a separate executable..has to start up for every .[im]3 file. B
>>>
>>> I wonder if we can fix this btw.
>>> If we can start it just once per library.
>>> Process create really is fast on typical Posix systems, so people maybe
>>> don't notice.
>>>
>>> There are a few methods:
>>> - alway keep the m3cg ir files
>>> - if any out of date, read them all into one large IR and write out one large assembly file
>>> - I want do that anyway, to achieve a rich/poor man's LTCG/LTO.
>>> poor: No need to worry about linker/driver support.
>>> rich: We can afford the memory these days. Though at some point it wouldn't scale.
>>> - I want do similar for C backend, as it seems noticable either the compiler startup
>>> or the few #includes I sometimes use (though C++ modules promise some relief..)
>>>
>>>
>>> Or, really, more simply, just accept a bunch of .mc/ms file pairs and process them
>>> separately as usual, but within one m3cg invocation.
>>>
>>>
>>>> Does/can it retain its incrementality?
>>>
>>> I think the best but not great we could do is make would decide pessmisically
>>> to run the compiler, but the compiler could figure out that the old/existing
>>> output is still correct. The world to decide up-to-date is significant but
>>> perhaps somewhat less than the work to just do all the work.
>>>
>>>> see no benefit and almost certainly losses, in putting these in different executables.
>>>
>>> Less code to maintain. Easier to redistribute.
>>> I like the model of distribution of C source.
>>> wget source && tar xf && configure && make && make install
>>>
>>> I want something like that.
>>> I am uncertain of "redistribution" and "active development" must/should be different.
>>> i.e. We do have something like this, and the two ways of working cooexist.
>>>
>>>
>>> - Jay
>>>
>>>
>>>
>>>
>>>
>>> ----------------------------------------
>>>> Date: Wed, 1 Jun 2016 17:08:34 -0500
>>>> From: rodney_bates at lcwb.coop
>>>> To: m3devel at elegosoft.com
>>>> Subject: Re: [M3devel] replace build system??
>>>>
>>>>
>>>>
>>>> On 06/01/2016 04:30 AM, Jay K wrote:
>>>>> We need to understand if and how well Modula-3 fits in the traditional and widespread C build infrastructure.
>>>>
>>>> It doesn't, not very well. A deep principle, if you can call it that, of C is that
>>>> each compilation has no knowledge of any other compilation. Likewise, when linking
>>>> them together, it's all just binary stuff, addresses, constants, etc. To get a
>>>> semblance of type-safe compilation, all coders have to correctly follow some variant
>>>> of a header file/#include idiom and create a matching makefile that correctly reflects
>>>> the actual interdependencies. There is no checking that these are done correctly, and
>>>> from direct experience, I can say they are routinely done wrong.
>>>>
>>>> All the modular languages have direct linguistic support for type-safe separate
>>>> compilation. Implementing this correctly while using the C compilation model lies
>>>> somewhere between very kludgy and impossible.
>>>>
>>>>>
>>>>> Does/can it retain its build speed if you invoke cm3 per .i3 and per .m3 file?
>>>>
>>>> It couldn't possibly. Typical front end compilation actual work is similar to
>>>> or less than executable program startup/shutdown overhead. This effect
>>>> might be partially lost in the fact that m3cc is a separate executable and it
>>>> has to start up for every .[im]3 file. But it would still about double the
>>>> number of program executions and startup overheads.
>>>>
>>>>> Does/can it retain its incrementality?
>>>>
>>>> I don't think this is possible all. No matter what is in the makefile, Make
>>>> only understands dependencies among whole source files and must rebuild if
>>>> a depended-on file has been touched, even no changes to content. Our current
>>>> build system works on declaration granularity. (I presume this is what you
>>>> mean by incrementality." I am not aware of any language-independent build
>>>> infrastructure that keeps track of dependencies on other that source file
>>>> granularity.
>>>>
>>>>>
>>>>> Or do we really need to be more of the "driver" and do a lot of stuff at the lib/link level?
>>>>>
>>>>
>>>> In a sense, we already have this. Cm3 contains a sophisticated driver, a compiler
>>>> frontend, and language-specific link-time functions in one executable. I see no
>>>> benefit and almost certainly losses, in putting these in different executables.
>>>>
>>>>> - Jay
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ----------------------------------------
>>>>>> From: jay.krell at cornell.edu
>>>>>> To: mika at async.caltech.edu; estellnb at elstel.org; m3devel at elegosoft.com; dabenavidesd at yahoo.es
>>>>>> Date: Tue, 31 May 2016 17:18:46 +0000
>>>>>> Subject: Re: [M3devel] replace build system??
>>>>>>
>>>>>> We should probably learn how to get the number of processors and optionally system load and make this more automatic.
>>>>>> Posix should standardize more of this. :(
>>>>>>
>>>>>> - Jay
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ----------------------------------------
>>>>>>> To: estellnb at elstel.org; jay.krell at cornell.edu; m3devel at elegosoft.com
>>>>>>> To: dabenavidesd at yahoo.es
>>>>>>> Date: Tue, 31 May 2016 10:16:05 -0700
>>>>>>> From: mika at async.caltech.edu
>>>>>>> Subject: Re: [M3devel] replace build system??
>>>>>>>
>>>>>>> I mentioned this before a few times on this mailing list.... CM3 is already fairly
>>>>>>> parallel if you turn on the right options. Every back-end invocation can be done
>>>>>>> in parallel.
>>>>>>>
>>>>>>> Set M3_PARALLEL_BACK in the config to 10 or 20 and watch it go...
>>>>>>>
>>>>>>> "Daniel Alejandro Benavides D." writes:
>>>>>>> ...
>>>>>>>> A make-based build solution would have other advantages as well like=20
>>>>>>>> f.i. parallel build by make --jobs=3D4. That way CM3 could build up to=20
>>>>>>>> four times faster.
>>>>>>> ...
>>>>>>> _______________________________________________
>>>>>>> M3devel mailing list
>>>>>>> M3devel at elegosoft.com
>>>>>>> https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel
>>>>>>
>>>>>> _______________________________________________
>>>>>> M3devel mailing list
>>>>>> M3devel at elegosoft.com
>>>>>> https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel
>>>>>
>>>>> _______________________________________________
>>>>> M3devel mailing list
>>>>> M3devel at elegosoft.com
>>>>> https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel
>>>>>
>>>>
>>>> --
>>>> Rodney Bates
>>>> rodney.m.bates at acm.org
>>>> _______________________________________________
>>>> M3devel mailing list
>>>> M3devel at elegosoft.com
>>>> https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel
>>>
>>> _______________________________________________
>>> M3devel mailing list
>>> M3devel at elegosoft.com
>>> https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel
>>>
>>
>> --
>> Rodney Bates
>> rodney.m.bates at acm.org
>> _______________________________________________
>> M3devel mailing list
>> M3devel at elegosoft.com
>> https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel
>   		 	   		
> _______________________________________________
> M3devel mailing list
> M3devel at elegosoft.com
> https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel
>

-- 
Rodney Bates
rodney.m.bates at acm.org