[M3devel] replace build system??

Fri Jun 3 21:48:25 CEST 2016

> like this idea, but we don't want to force serialization where we now do parallel
> runs of m3cg.

Good point. We could use pthreads in there though, maybe.
Except I'm not sure we own enough of the code..and..er...oh..nevermind.
I believe gcc is replete with globals. As is the frontend.
We could still partition the work. If we have say 40 files to compile, we could run 4 m3cg's in parallel that each compile 10 files...

 > OTOH, it has puzzled me why, when compiling m3cc or m3gdb, the C compile commands
 > seem to scroll by slower than the unit compiles when running cm3, despite the fact

I believe the C compiler is effectively slow. The C compiler is actually very fast, but it has to do the same or almost the same work very many times. Compiling a thousand or so lines is very quick, but #include windows.h or stdio.h or vector and you have really a ton of code, and it redoes the work for every source files. Precompiled headers can help. Though at least with the Microsoft compiler, they change then language very slightly. And you have to work through each compiler's command line options (maybe cmake/automake??) C++ modules will be the salvation, maybe soon.
 - Jay

----------------------------------------
> Date: Fri, 3 Jun 2016 09:53:16 -0500
> From: rodney_bates at lcwb.coop
> To: m3devel at elegosoft.com
> Subject: Re: [M3devel] replace build system??
>
>
>
> On 06/03/2016 02:22 AM, Jay K wrote:
>> I understand the C model very well, but I'm not certain of all
>> the ramifications for Modula-3.
>>
>>
>> I understand C++ has limited checking via name mangling.
>>
>>
>> And that other systems carry names/signatures forward to runtime
>> and check them there. Even dynamically linked C++. And Modula-3.
>>
>>
>> However function names aren't everything -- i.e. they don't include
>> structural hashes of types. It is name-based, not structure-based.
>>
>>
>> The most common mistake in the loose C systems is not doing things
>> wrong per se, but accidentally mixing old and new versions,
>> and the result is undefined/crash.
>>
>>
>> We have the same problem with our "extern" mechanism -- how many times
>> I debugged stat corrupting stack until I put in the extra layer.
>>
>>
>> C++ systems are getting better and check for "ODR violations" at link time.
>> But checking at link time kinda contradicts my question.
>> Goal being to fit into "generic" build systems but C++ only starts
>> winning when the linker knows about it -- when the system is C++ specific.
>> If C++ requires C++ linker/driver, we shouldn't feel too bad
>> requiring a Modula-3 linker/driver.
>>
>>
>>> Typical front end compilation actual work is similar to
>>> or less than executable program startup/shutdown overhead.
>>
>>
>> Ideally this is not the case. Ideally process startup is just referencing
>> pages already in file system cache. Demand paging, unified file system cache
>> and memory manager and all that (all systems NT, Linux, etc. are the same
>> these days). But I realize in reality there is alway some amount of other CPU
>> to burn at each process start. The ideal is not quite reached.
>>
>>> m3cc is a separate executable..has to start up for every .[im]3 file. B
>>
>> I wonder if we can fix this btw.
>> If we can start it just once per library.
>> Process create really is fast on typical Posix systems, so people maybe
>> don't notice.
>>
>> There are a few methods:
>> - alway keep the m3cg ir files
>> - if any out of date, read them all into one large IR and write out one large assembly file
>> - I want do that anyway, to achieve a rich/poor man's LTCG/LTO.
>> poor: No need to worry about linker/driver support.
>> rich: We can afford the memory these days. Though at some point it wouldn't scale.
>> - I want do similar for C backend, as it seems noticable either the compiler startup
>> or the few #includes I sometimes use (though C++ modules promise some relief..)
>>
>>
>> Or, really, more simply, just accept a bunch of .mc/ms file pairs and process them
>> separately as usual, but within one m3cg invocation.
>>
>
> I like this idea, but we don't want to force serialization where we now do parallel
> runs of m3cg.
>
> OTOH, it has puzzled me why, when compiling m3cc or m3gdb, the C compile commands
> seem to scroll by slower than the unit compiles when running cm3, despite the fact
> that the latter include a main program invocation of m3cc. Surely it's not merely
> smaller units? And we are also running separately too.
>
>>
>>> Does/can it retain its incrementality?
>>
>> I think the best but not great we could do is make would decide pessmisically
>> to run the compiler, but the compiler could figure out that the old/existing
>> output is still correct. The world to decide up-to-date is significant but
>> perhaps somewhat less than the work to just do all the work.
>>
>>> see no benefit and almost certainly losses, in putting these in different executables.
>>
>> Less code to maintain. Easier to redistribute.
>> I like the model of distribution of C source.
>> wget source && tar xf && configure && make && make install
>>
>
> We could just wrap what we have in makefile that, almost trivially invokes cm3 with
> a few odd options. Or is that cheating?
>
>> I want something like that.
>> I am uncertain of "redistribution" and "active development" must/should be different.
>> i.e. We do have something like this, and the two ways of working cooexist.
>>
>>
>> - Jay
>>
>>
>>
>>
>>
>> ----------------------------------------
>>> Date: Wed, 1 Jun 2016 17:08:34 -0500
>>> From: rodney_bates at lcwb.coop
>>> To: m3devel at elegosoft.com
>>> Subject: Re: [M3devel] replace build system??
>>>
>>>
>>>
>>> On 06/01/2016 04:30 AM, Jay K wrote:
>>>> We need to understand if and how well Modula-3 fits in the traditional and widespread C build infrastructure.
>>>
>>> It doesn't, not very well. A deep principle, if you can call it that, of C is that
>>> each compilation has no knowledge of any other compilation. Likewise, when linking
>>> them together, it's all just binary stuff, addresses, constants, etc. To get a
>>> semblance of type-safe compilation, all coders have to correctly follow some variant
>>> of a header file/#include idiom and create a matching makefile that correctly reflects
>>> the actual interdependencies. There is no checking that these are done correctly, and
>>> from direct experience, I can say they are routinely done wrong.
>>>
>>> All the modular languages have direct linguistic support for type-safe separate
>>> compilation. Implementing this correctly while using the C compilation model lies
>>> somewhere between very kludgy and impossible.
>>>
>>>>
>>>> Does/can it retain its build speed if you invoke cm3 per .i3 and per .m3 file?
>>>
>>> It couldn't possibly. Typical front end compilation actual work is similar to
>>> or less than executable program startup/shutdown overhead. This effect
>>> might be partially lost in the fact that m3cc is a separate executable and it
>>> has to start up for every .[im]3 file. But it would still about double the
>>> number of program executions and startup overheads.
>>>
>>>> Does/can it retain its incrementality?
>>>
>>> I don't think this is possible all. No matter what is in the makefile, Make
>>> only understands dependencies among whole source files and must rebuild if
>>> a depended-on file has been touched, even no changes to content. Our current
>>> build system works on declaration granularity. (I presume this is what you
>>> mean by incrementality." I am not aware of any language-independent build
>>> infrastructure that keeps track of dependencies on other that source file
>>> granularity.
>>>
>>>>
>>>> Or do we really need to be more of the "driver" and do a lot of stuff at the lib/link level?
>>>>
>>>
>>> In a sense, we already have this. Cm3 contains a sophisticated driver, a compiler
>>> frontend, and language-specific link-time functions in one executable. I see no
>>> benefit and almost certainly losses, in putting these in different executables.
>>>
>>>> - Jay
>>>>
>>>>
>>>>
>>>>
>>>> ----------------------------------------
>>>>> From: jay.krell at cornell.edu
>>>>> To: mika at async.caltech.edu; estellnb at elstel.org; m3devel at elegosoft.com; dabenavidesd at yahoo.es
>>>>> Date: Tue, 31 May 2016 17:18:46 +0000
>>>>> Subject: Re: [M3devel] replace build system??
>>>>>
>>>>> We should probably learn how to get the number of processors and optionally system load and make this more automatic.
>>>>> Posix should standardize more of this. :(
>>>>>
>>>>> - Jay
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ----------------------------------------
>>>>>> To: estellnb at elstel.org; jay.krell at cornell.edu; m3devel at elegosoft.com
>>>>>> To: dabenavidesd at yahoo.es
>>>>>> Date: Tue, 31 May 2016 10:16:05 -0700
>>>>>> From: mika at async.caltech.edu
>>>>>> Subject: Re: [M3devel] replace build system??
>>>>>>
>>>>>> I mentioned this before a few times on this mailing list.... CM3 is already fairly
>>>>>> parallel if you turn on the right options. Every back-end invocation can be done
>>>>>> in parallel.
>>>>>>
>>>>>> Set M3_PARALLEL_BACK in the config to 10 or 20 and watch it go...
>>>>>>
>>>>>> "Daniel Alejandro Benavides D." writes:
>>>>>> ...
>>>>>>> A make-based build solution would have other advantages as well like=20
>>>>>>> f.i. parallel build by make --jobs=3D4. That way CM3 could build up to=20
>>>>>>> four times faster.
>>>>>> ...
>>>>>> _______________________________________________
>>>>>> M3devel mailing list
>>>>>> M3devel at elegosoft.com
>>>>>> https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel
>>>>>
>>>>> _______________________________________________
>>>>> M3devel mailing list
>>>>> M3devel at elegosoft.com
>>>>> https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel
>>>>
>>>> _______________________________________________
>>>> M3devel mailing list
>>>> M3devel at elegosoft.com
>>>> https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel
>>>>
>>>
>>> --
>>> Rodney Bates
>>> rodney.m.bates at acm.org
>>> _______________________________________________
>>> M3devel mailing list
>>> M3devel at elegosoft.com
>>> https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel
>>
>> _______________________________________________
>> M3devel mailing list
>> M3devel at elegosoft.com
>> https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel
>>
>
> --
> Rodney Bates
> rodney.m.bates at acm.org
> _______________________________________________
> M3devel mailing list
> M3devel at elegosoft.com
> https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel