<div dir="ltr">Jay I can see the complexity of the task but can we have an option for users who cannot invest time in this?<div><br></div><div>What would be great is if you could set up the scripts in a static environment available to all that builds various platforms and back ends that are requested or that you feel like maintaining. Having it so it could auto build based on a particular version from github would be optimal.</div><div><br></div><div>Surely this would help with developing and debugging your scripts and backend and would solve the problem of other people getting the environment right.</div><div><br></div><div>I have dozens of servers (x86 and ARM) with ample memory and storage plus PDUs that allow powering the servers up and down over the net. If you want to set up the environments and scripts I can setup the access and install Mac OS and whatever Unixes on some of them. You can have carte blanche as to what you do with them, I don't care as long as working compilers pop out somewhere. </div><div><br></div><div>Maybe others are interested in helping with this too.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 11, 2015 at 11:56 AM, Jay K <span dir="ltr"><<a href="mailto:jay.krell@cornell.edu" target="_blank">jay.krell@cornell.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div dir="ltr"><div> The problem with cross compiling is you need: <br> a working cross-compiler for C/C++ <br> a working cross-assembler <br> a working cross-linker </div><div><br></div><div><br> If you have those, it is fairly easy. * <br> Getting those is often not easy. <br> Esp. the C/C+ compiler as you need headers/libraries/startup code. "sysroot" or "buildroot"<br> it is often called.</div><div><br></div><div><br> * fairly easy: <br> for NT386 -- nothing extra to do, the backend is in cm3 <br> There are endianness bugs in mklib, but it works on the majority of systems -- little endian hosts. I can fix it. </div><div><br></div><div> For the C backend -- nothing extra to do, the backend is in cm3. </div><div><br></div><div> For the gcc-based backend -- fairly easy -- you need to rebuild a different cm3cg <br> and point the config at it. It kind of is already setup to work. </div><div><br></div><div><br> For the LLVM backend, probably also easy. </div><div><br></div><div><br> I think on Debian and Gentoo, cross C compilers at least to every Linux architecture are readily available. <br> But I don't know about otherwise. </div><div><br></div><div><br> In as much as the compiler is gcc, the assembler GNU as, and the linker GNU ld, that gets you far, <br> but you still need the headers/libraries/startup code. </div><div><br></div><div><br> If your C compiler is LLVM, that might also help for compiler/assembler/linker.<br> But again, the headers/libraries/startup code.</div><div><br></div><div><br> There are various systems out there that try to automate this but I don't have much experience<br> with them, and they often are slightly different than what we want.</div><div><br></div><div><br> Otherwise, what we have setup is we cross compile to assembly files, copy them to the target, <br> which typically does have the compiler/assembler/linker, and finish there.</div><div><br></div><div><br> What remains there is that I only have this automated to build cm3. <br> It should also at least also build m3core and libm3 statically. And therefore reach<br> approximately parity with the 3.6 "boot" archives and install process. </div><div><br></div><div><br> Has anyone here installed 3.6? And consider there method a decent goal and stopping point?<br> Today is slightly different since quake has been rewritten from C to Modula-3 in the 4.0 timeframe.</div><div><br></div><div><br> And *possibly* boot archives should contain everything -- going beyond what 3.6 did.</div><div><br></div><div><br></div><div><br></div><div> This is just a matter of work in scripts/python/pylib.py. <br> Copying stuff into sub directories and generating a hierarhical recursive or not make system in there. <br> My make skills aren't great, and I've been hung up on details like depending on GNU make or trying to<br> be more portable. </div><div><br></div><div><br>On the threading area, I believe there is a simple almost ideal design.</div><div><br></div><div> - parse and mostly "execute" all the quake code almost unchanged, single thread </div><div><br></div><div> - difference starts at about the last line of quake where it says library or program </div><div><br></div><div> - possibly though queueing every file to a "prefetcher" thread, it just reads<br> every file into a reused buffer and throws out the data, populating the operating<br> system's file system cache; or possibly mmaping every file and keeping it mapped,<br> and touching every page; doing this in a somewhat thread aware/safe way for the<br> upcoming actual accesses</div><div><br></div><div> - once quake is done, one of two choices:</div><div><br></div><div> simple but not optimal: do a single threaded parse of every interface<br> less simple: parsing here can be in parallel, depending on the dependency graph;<br> you'd just start up a thread per interface, but block on parsing its dependencies;<br> as you get away from the root (RT0.i3), the tree should get ever wider</div><div><br></div><div><br> You would either manually throttle the number of threads, or rely on an underlying threadpool.</div><div><br></div><div><br> NOTE: Modula-3 used to have a thread pool on Win32 but I removed it to be simplar.<br> Maybe that wasn't good. Simpler as in, including, thread id is now just the underlying<br> Win32 thread id. It wasn't before. </div><div><br></div><div> Win32 has had a thread pool since circa Windows 2000. That might be profitable to use.</div><div><br></div><div> - once parsing interfaces is done, I believe codegen of every interface, and parsing and codegen<br> of every module can proceed with arbitrary parallelism</div><div><br></div><div><br> Fetching interface/module contents might synchronize with the prefetcher depending<br> on which of the two approaches -- if the prefetcher is just prefetch and discard,<br> populating the file system cache, then later compilation just refetches oblibviously/unchanged.<br> If the prefetcher thread mmaps and keeps everything, then serialize with it.</div><div><br></div><div><br> Also, you might have multiple prefetcher threads. What you really want is..difficult.<br> You want a prefetcher thread per spindle. How to count them up?<br> If you have an SSD, then prefetching might not buy much and just forget about it.<br> The thing is, if you have spindles, this might be the most gain, and if you have an SSD,<br> the cost might be negligible or slightly beneficial.</div><div><br></div><div><br> I/O is so often the bottleneck, at least in the days of rotating storage. </div><div><br></div><div><br> In the event that the file system cache is small, or the source for a package<br> really large, prefetching can be counterproductive -- moving stuff through the cache<br> only to flush it before it is used and fetching it a second time.</div><div><br></div><div><br> I expect on most systems this is not a concern though.</div><div><br></div><div><br> One of the pieces of work here, I believe, is to move most globals in m3front<br> to be in Module.T. This would actually make things cleaner imho.<br> Yes, it consolidates knowledge that you might want distributed.<br> And accessing context'ed data is slightly slower than globals.<br> But imho globals are bad and everything should be placed in *some* context<br> other than the process or thread.</div><div><br></div><div><br> I would not advocate compiling functions within a module in parallel, only modules.</div><div><br></div><div><br> Similarly, code generators must have no globals, and put all the state in the cg object.</div><div><br></div><div><br> - Jay<br><br><br><br><br></div><div><hr>Date: Tue, 11 Aug 2015 10:20:26 -0700<br>From: <a href="mailto:lists@darko.org" target="_blank">lists@darko.org</a><br>To: <a href="mailto:wagner@elegosoft.com" target="_blank">wagner@elegosoft.com</a><br>CC: <a href="mailto:m3devel@elegosoft.com" target="_blank">m3devel@elegosoft.com</a>; <a href="mailto:jay.krell@cornell.edu" target="_blank">jay.krell@cornell.edu</a><span class=""><br>Subject: Re: [M3devel] multi-threaded m3front?<br><br></span><div><div class="h5"><div dir="ltr">I couldn't agree with you more. I think being able to compile or cross compile the system without spending hours (or days) hacking scripts/environments would be a huge step forward for the project. Or having compiler binaries for more than two platforms (four if you count different word sizes). Meanwhile I've never heard anyone complain that the compiler is too slow.<div><br></div><div>But of course everyone has different priorities, different interests and different opinions in a volunteer project so any discussion on this subject will inevitably boil down to "he who writes the code determines the priorities."</div><div><br></div><div><div><br></div><div><br></div></div></div><div><br><div>On Tue, Aug 11, 2015 at 12:13 AM, Olaf Wagner <span dir="ltr"><<a href="mailto:wagner@elegosoft.com" target="_blank">wagner@elegosoft.com</a>></span> wrote:<br><blockquote style="padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><span>On Mon, 10 Aug 2015 22:00:17 -0700<br>
Darko Volaric <<a href="mailto:lists@darko.org" target="_blank">lists@darko.org</a>> wrote:<br>
<br>
> A more fruitful approach might be pipelining the compiler:<br>
><br>
> - files enter the pipeline in reverse dependency order<br>
> - have a thread (stage) to read those files into memory<br>
> - have a thread to tokenize and parse the files and form the data structure<br>
> - have a thread to do intermediary processing<br>
> - have a thread to generate the output representation for the back end<br>
><br>
> You'll only get a maximum 4x speedup and you'll only be as fast as the<br>
> slowest thread, but you can reliably tune the divisions based on simple<br>
> benchmarking of each thread. You're very likely to get somewhat close to<br>
> the 4x speedup. This works best when there is a 1:1 between pipeline stages<br>
> and cores. If you were ambitious you could attempt an 8 stage pipeline for<br>
> 8 core processors.<br>
><br>
><br>
> On Sat, Aug 8, 2015 at 8:05 PM, Jay K <<a href="mailto:jay.krell@cornell.edu" target="_blank">jay.krell@cornell.edu</a>> wrote:<br>
><br>
> > Is anyone interested in updating m3front to be multi-threaded?<br>
> ><br>
> > I haven't seen a single core system in a while.<br>
> ><br>
> > Surely each module can be compiled separately, possibly with some<br>
> > serialization around compiling interfaces?<br>
<br>
</span>While this is all correct, I'd like to remark that in my expecience<br>
optimizing for performance has always got in the way of clear,<br>
understandable and maintainable code.<br>
<br>
So while there are semantic refactorings and feature additions in the<br>
pipeline I would not like to see anybody rework the whole compiler<br>
with the intention of making it faster. I would rather see the few<br>
active programmers interested in M3 concentrate on more backends, better<br>
intermediate code, better code optimization, better maintainability<br>
etc.<br>
<span><font color="#888888"><br>
Olaf<br>
--<br>
Olaf Wagner -- elego Software Solutions GmbH -- <a href="http://www.elegosoft.com" rel="noreferrer" target="_blank">http://www.elegosoft.com</a><br>
Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany<br>
phone: <a>+49 30 23 45 86 96</a> mobile: <a>+49 177 2345 869</a> fax: <a>+49 30 23 45 86 95</a><br>
Geschäftsführer: Olaf Wagner | Sitz: Berlin<br>
Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194<br>
</font></span></blockquote></div><br></div>
<br></div></div><span class="">_______________________________________________
M3devel mailing list
<a href="mailto:M3devel@elegosoft.com" target="_blank">M3devel@elegosoft.com</a>
<a href="https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel" target="_blank">https://mail.elegosoft.com/cgi-bin/mailman/listinfo/m3devel</a></span></div> </div></div>
</blockquote></div><br></div>