From dabenavidesd at yahoo.es Mon Jul 3 22:34:44 2017 From: dabenavidesd at yahoo.es (Daniel Alejandro Benavides D.) Date: Mon, 3 Jul 2017 20:34:44 +0000 (UTC) Subject: [M3devel] Another backend? References: <1192403415.6727858.1499114084342.ref@mail.yahoo.com> Message-ID: <1192403415.6727858.1499114084342@mail.yahoo.com> http://dada.cs.washington.edu/htbin-post/unrestricted/cecil/cvsweb.cgi/vortex-extensions/M3/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Tue Jul 4 08:34:03 2017 From: jayk123 at hotmail.com (Jay K) Date: Tue, 4 Jul 2017 06:34:03 +0000 Subject: [M3devel] compiler switch "reduce target variation"? In-Reply-To: References: Message-ID: Trying to resume work here.. Here, I've commited this to a branch: https://github.com/modula3/cm3/commit/0d59546d11641afe3772b73f226c2e90b960d7fc There are some unrelated changes I will tease out and commit separately. I still believe we should implement this. It is a variation of what people call "deterministic builds". "deterministic builds" are kind of an overloaded term, or common sense at first, but then controversial as details emerge. For examples: - Compilers should not generate random numbers, at least not base their output on them. - Optimizers should not "give up" after some elapsed time or consumed memory -- leading to different results on similar-but-different hardware -- though if memory can be accounted precisely and controlled by a switch, maybe But then again, compilers should make good use of their host machines -- you can't have it both ways. - full paths should occur nowhere in the output see also debugging information, that might work better with full paths -- no need to tell debugger some such search paths - varying a source file's full path should not change how it compiles; but see constructs like #include ../foo.c and printf(__FILE__) or printf("%d\n", sizeof(__FILE__)). And is __FILE__ a full path or not? (Modula-3 has the analogous Compiler.ThisLine and Compiler.ThisFile so please pardon my C++-ness, it still is relevant Perhaps compiler can trim prefixes of paths. - adding a comment in a heavily used header should not change any output; but see for example, in such header inline void foo() { printf("%d\n", __LINE__); - adding comments anywhere, really, should not change any output -- but again __LINE__ - recompiling all the same files with the same compiler and command lines should produce the same output every time -- but see for examples printf("%d\n", __TIME__ or __DATE__ or __TIMESTAMP__) or things like Modula-3 date-based-version; burden is perhaps on the programer to not use these problematic constructs, and compiler can maybe get switches to warn/error for them. So, then, I add to this mix, Linux/x86, NetBSD/x86, OpenBSD/x86, Solaris/x86 should generally have identical output. But again, things like printing targetname -- including as comments from the C backend. - Jay ________________________________ From: M3devel on behalf of Jay K Sent: Wednesday, March 15, 2017 8:37 AM To: m3devel Subject: [M3devel] compiler switch "reduce target variation"? Tangent: Should we start use git branches and pull requests? Are people here reasonable able to view textual diffs? It pushes my limits personally. But I know some developers work this way. This is some old stuff I was working on. I kinda think it should be the default, but almost everyone here protests almost all of my changes, so here is a new switch instead. The idea is to remove somewhat gratituous target variation. One would have to argue over each difference and if it is gratuitous and if it should be removed by default, or under a switch, and which switch. So I'm proposing generally one switch to put everything under. The differences I had noticed in my investigation months ago is the name of the TARGET appearing, for example in paths. Such as maybe in generic instantiations, in debug symbols, in output logs. I actually think debug symbols need full paths, and so for generics, they end up with target. But for now, I desist on that agenda in favor of rougly the opposite. The goal here, as I mentioned before, is to establish that I386_LINUX == I386_FREEBSD == I386_NETBSD == I386_OPENBSD == I386_SOLARIS == I386_DARWIN == I386_CYGWIN == I386_SOLARIS maybe == I386_NT ditto AMD64, SPARC32, SPARC64, PPC32, PPC64, etc. So that we might collapse targets down. A soon/later step would establish even fewer equivalence classes: PP32be == SPARC32, PPC64be == SPARC64, => BigEndian64 or PosixBigEndian64 I386 == ARM => LittleEndian32 AMD64 == ARM64 == PPC64le => LittleEndian64 etc. and then eventually we could have one or a small number of C-based distributions. I have to go back and build and test all this anew. Reasonable? Maybe the default? Maybe add a switch for restore-target-difference or preserve-target-differences? diff --git a/m3-sys/cm3/src/Main.m3 b/m3-sys/cm3/src/Main.m3 index 5d753e6..61b3f18 100644 --- a/m3-sys/cm3/src/Main.m3 +++ b/m3-sys/cm3/src/Main.m3 @@ -5,7 +5,7 @@ MODULE Main; IMPORT M3Timers, Pathname, Process, Quake; IMPORT RTCollector, RTParams, RTutils, Thread, Wr; -IMPORT TextTextTbl; +IMPORT TextTextTbl, Target; IMPORT Builder, Dirs, M3Build, M3Options, Makefile, Msg, Utils, WebFile; IMPORT MxConfig(*, M3Config, CMKey, CMCurrent *); @@ -18,6 +18,8 @@ VAR build_dir : TEXT := NIL; mach : Quake.Machine := NIL; +CONST BoolToText = ARRAY BOOLEAN OF TEXT{"FALSE", "TRUE"}; + PROCEDURE DefineIfNotDefined (qmachine: Quake.Machine; symbol, value: TEXT) RAISES {Quake.Error} = BEGIN @@ -73,6 +75,7 @@ VAR defs: TextTextTbl.T; (* DefineIfNotDefined (mach, "THREAD_LIBRARY", Version.ThreadLibrary); *) (* DefineIfNotDefined (mach, "WINDOW_LIBRARY", Version.WindowLibrary); *) DefineIfNotDefined (mach, "WORD_SIZE", MxConfig.HOST_WORD_SIZE); + DefineIfNotDefined (mach, "REDUCE_TARGET_VARIATION", BoolToText[Target.ReduceTargetVariation]); (* Even if the config file overrides the defaults, such as to do a cross build, the host characteristics are still available. *) diff --git a/m3-sys/cm3/src/Makefile.m3 b/m3-sys/cm3/src/Makefile.m3 index 64e169e..f2e1c7b 100644 --- a/m3-sys/cm3/src/Makefile.m3 +++ b/m3-sys/cm3/src/Makefile.m3 @@ -5,7 +5,7 @@ MODULE Makefile; IMPORT FS, M3File, M3Timers, OSError, Params, Process, Text, Thread, Wr; IMPORT Arg, M3Build, M3Options, M3Path, Msg, Utils, TextSeq, TextTextTbl; -IMPORT MxConfig, Dirs, Version; +IMPORT MxConfig, Dirs, Version, Target; TYPE NK = M3Path.Kind; @@ -267,6 +267,9 @@ PROCEDURE ConvertOption (VAR s: State; arg: TEXT; arg_len: INTEGER) | 'r' => IF Text.Equal(arg, "-realclean") THEN ok := TRUE; (* mode set during the pre-scan *) s.found_work := TRUE; + ELSIF Text.Equal(arg, "-reduce-target-variation") THEN + ok := TRUE; + Target.ReduceTargetVariation := TRUE; END; | 's' => IF Text.Equal (arg, "-silent") THEN @@ -707,6 +710,9 @@ CONST " -group-writable \"", " -pb allow parallelism in running back-end (experimental)", " -no-m3ship-resolution use quake variables in .M3SHIP (experimental)", + " -reduce-target-variation omit target in some minor places such as", + " current working directory in debug information", + " for internal development purposes (showing target equivalence)", "", "environment variables:", " M3CONFIG platform dependent configuration file to use (cm3.cfg)", diff --git a/m3-sys/cminstall/src/config-no-install/cm3cfg.common b/m3-sys/cminstall/src/config-no-install/cm3cfg.common index 7334252..42db06f 100644 --- a/m3-sys/cminstall/src/config-no-install/cm3cfg.common +++ b/m3-sys/cminstall/src/config-no-install/cm3cfg.common @@ -458,8 +458,14 @@ proc GetM3Back() is else if equal(HOST, TARGET) m3back = INSTALL_ROOT & "/bin/" end end - + m3back = "@" & m3back & "cm3cg " & GetM3BackFlags() + + if defined ("REDUCE_TARGET_VARIATION") + if REDUCE_TARGET_VARIATION + m3back = m3back & " -freduce-target-variation" + end + end return m3back end diff --git a/m3-sys/m3back/src/M3C.m3 b/m3-sys/m3back/src/M3C.m3 index f6740c7..d584fea 100644 --- a/m3-sys/m3back/src/M3C.m3 +++ b/m3-sys/m3back/src/M3C.m3 @@ -29,7 +29,7 @@ VAR CaseDefaultAssertFalse := FALSE; (* Taken together, these help debugging, as you get more lines in the C and the error messages reference C line numbers *) - CONST output_line_directives = TRUE; + VAR output_line_directives := TRUE; CONST output_extra_newlines = FALSE; CONST inline_extract = FALSE; @@ -2197,7 +2197,9 @@ BEGIN END; IF (*self.suppress_line_directive < 1 AND*) text_last_char = '\n' THEN - Wr.PutText(self.c, self.line_directive); + IF NOT Target.ReduceTargetVariation THEN + Wr.PutText(self.c, self.line_directive); + END; self.width := 0; self.last_char_was_newline := TRUE; RETURN; @@ -2205,7 +2207,9 @@ BEGIN IF Text.FindChar(text, '\n') # -1 THEN self.width := 0; (* roughly *) - Wr.PutText(self.c, self.nl_line_directive); + IF NOT Target.ReduceTargetVariation THEN + Wr.PutText(self.c, self.nl_line_directive); + END; self.last_char_was_newline := TRUE; RETURN; END; @@ -2217,10 +2221,12 @@ BEGIN END; self.width := 0; - IF self.last_char_was_newline THEN - Wr.PutText(self.c, self.line_directive); - ELSE - Wr.PutText(self.c, self.nl_line_directive); + IF NOT Target.ReduceTargetVariation THEN + IF self.last_char_was_newline THEN + Wr.PutText(self.c, self.line_directive); + ELSE + Wr.PutText(self.c, self.nl_line_directive); + END; END; self.last_char_was_newline := TRUE; END print; @@ -2339,9 +2345,10 @@ END set_error_handler; PROCEDURE Prefix_Print(self: T; multipass: Multipass_t) = BEGIN self.comment("begin unit"); - self.comment("M3_TARGET = ", Target.System_name); - (* This is an unnecessary target-specific output. *) - (* self.comment("M3_TARGET = ", Target.System_name); *) + output_line_directives := output_line_directives AND NOT Target.ReduceTargetVariation; + IF NOT Target.ReduceTargetVariation THEN + self.comment("M3_TARGET = ", Target.System_name); + END; self.comment("M3_WORDSIZE = ", IntToDec(Target.Word.size)); self.static_link_id := M3ID.Add("_static_link"); self.alloca_id := M3ID.Add("alloca"); @@ -4836,7 +4843,8 @@ BEGIN & " ok2=" & BoolToText[ok2] & "\n" ); RTIO.Flush(); - <* ASSERT FALSE *> + <* ASSERT ok1 *> + <* ASSERT ok2 *> END; IF type = CGType.Int32 AND TInt.EQ(i, TInt.Min32) THEN RETURN "-" & intLiteralPrefix[type] & TInt.ToText(TInt.Max32) & intLiteralSuffix[type] & "-1"; diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c index dc52576..33f8844 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c +++ b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c @@ -1065,8 +1065,11 @@ dbxout_init (const char *input_file_name) labels. */ ASM_GENERATE_INTERNAL_LABEL (ltext_label_name, "Ltext", 0); - /* Put the current working directory in an N_SO symbol. */ - if (use_gnu_debug_info_extensions && !NO_DBX_MAIN_SOURCE_DIRECTORY) + /* Limit paths in debug output, to limit target variation. */ + if (!reduce_target_variation) + { + /* Put the current working directory in an N_SO symbol. */ + if (use_gnu_debug_info_extensions && !NO_DBX_MAIN_SOURCE_DIRECTORY) { static const char *cwd; @@ -1087,6 +1090,7 @@ dbxout_init (const char *input_file_name) used_ltext_label_name = true; #endif /* no DBX_OUTPUT_MAIN_SOURCE_DIRECTORY */ } + } mapped_name = remap_debug_filename (input_file_name); #ifdef DBX_OUTPUT_MAIN_SOURCE_FILENAME diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c index 6f60742..291aea3 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c +++ b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c @@ -15450,15 +15450,18 @@ add_gnat_descriptive_type_attribute (dw_die_ref die, tree type, static void add_comp_dir_attribute (dw_die_ref die) { + /* Limit paths in debug output, to limit target variation. */ + if (reduce_target_variation) + return; + const char *wd = get_src_pwd (); - char *wd1; if (wd == NULL) return; if (DWARF2_DIR_SHOULD_END_WITH_SEPARATOR) { - int const wdlen = (int)strlen (wd); + size_t const wdlen = strlen (wd); char * const wd1 = (char *) ggc_alloc_atomic (wdlen + 2); memcpy (wd1, wd, wdlen); wd1 [wdlen] = DIR_SEPARATOR; diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c index 63e4b92..d468632 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c @@ -217,11 +217,8 @@ const char * get_src_pwd (void) { if (! src_pwd) - { - src_pwd = getpwd (); - if (!src_pwd) - src_pwd = "."; - } + if (reduce_target_variation || !(src_pwd = getpwd ())) + src_pwd = "."; return src_pwd; } diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h index 588cfdb..f4f7cc7 100644 --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h @@ -80,4 +80,6 @@ extern bool set_src_pwd (const char *); extern HOST_WIDE_INT get_random_seed (bool); extern const char *set_random_seed (const char *); +extern bool reduce_target_variation; + #endif /* ! GCC_TOPLEV_H */ diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt index 3bd0469..cb0189f 100644 --- a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt +++ b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt @@ -28,9 +28,6 @@ m3cg Language M3CG -y -m3cg M3CG - fopcodes-trace m3cg M3CG Trace opcodes @@ -59,10 +56,17 @@ ftypes-trace m3cg M3CG Trace types +reduce-target-variation +m3cg M3CG +Reduce target variation somewhat, such as by omitting current working +directory from debug info. Many necessary target variations remain. + v m3cg M3CG +print version y m3cg M3CG +Trace opcodes ; This comment is to ensure we retain the blank line above. diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c index 03417ed..071bbc8 100644 --- a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c +++ b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c @@ -246,6 +246,7 @@ build_case_label (tree low_value, tree high_value, tree label_decl) /*======================================================= OPTION HANDLING ===*/ static int option_trace_all; +bool reduce_target_variation; /*===========================================================================*/ @@ -6364,6 +6365,10 @@ m3_handle_option (size_t code, PCSTR /*arg*/, int /*value*/) case OPT_ftypes_trace: option_trace_all += 1; break; + + case OPT_reduce_target_variation: + reduce_target_variation = true; + break; } return 1; diff --git a/m3-sys/m3front/src/misc/Coverage.m3 b/m3-sys/m3front/src/misc/Coverage.m3 index c04c902..73fff21 100644 I don't remember what is going on here, but coverage isn't used much.. I suspect it might just have to do with forward vs. backward slashes, and that this distinction does not really matter -- Windows accepts forward slashes so we should just use them everywhere. --- a/m3-sys/m3front/src/misc/Coverage.m3 +++ b/m3-sys/m3front/src/misc/Coverage.m3 @@ -77,8 +77,9 @@ PROCEDURE NoteProcedure (v: Value.T) = PROCEDURE GenerateTables () = VAR nLines := MAX (0, maxLine - minLine) + 1; + fname := Host.FileTail (Host.filename); l_header := TLen (Header); - l_fname := TLen (Host.filename); + l_fname := TLen (fname); l_trailer := TLen (Trailer); size : INTEGER; p : ProcHead; @@ -124,10 +125,10 @@ PROCEDURE GenerateTables () = (* CG.Init_int (size, Target.Integer.size, TInt.Zero, FALSE); *) INC (size, Target.Integer.size); (*timestamp*) - CG.Init_intt (size, Target.Integer.size, Text.Length (Host.filename), FALSE); + CG.Init_intt (size, Target.Integer.size, Text.Length (fname), FALSE); INC (size, Target.Integer.size); (*fileLen*) - CG.Init_chars (size, Host.filename, FALSE); + CG.Init_chars (size, fname, FALSE); INC (size, l_fname * Target.Char.size); (*file*) CG.Init_intt (size, Target.Integer.size, minLine, FALSE); diff --git a/m3-sys/m3front/src/misc/Host.i3 b/m3-sys/m3front/src/misc/Host.i3 index c71489f..af6a89a 100644 --- a/m3-sys/m3front/src/misc/Host.i3 +++ b/m3-sys/m3front/src/misc/Host.i3 @@ -75,4 +75,7 @@ PROCEDURE OpenUnit (name: M3ID.T; interface, generic: BOOLEAN; PROCEDURE CloseFile (rd: File.T); +PROCEDURE FileTail (path: TEXT): TEXT; + (* returns the 'tail' of 'path' -- after any slashes or even spaces *) + END Host. diff --git a/m3-sys/m3front/src/misc/Host.m3 b/m3-sys/m3front/src/misc/Host.m3 index 962d3c6..79042e3 100644 --- a/m3-sys/m3front/src/misc/Host.m3 +++ b/m3-sys/m3front/src/misc/Host.m3 @@ -9,7 +9,7 @@ MODULE Host; -IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler; +IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler, Target; PROCEDURE Initialize (READONLY options: ARRAY OF TEXT): BOOLEAN = BEGIN @@ -192,5 +192,24 @@ PROCEDURE CloseFile (rd: File.T) = END; END CloseFile; +PROCEDURE FileTail (path: TEXT): TEXT = + VAR c: CHAR; + BEGIN + IF NOT Target.ReduceTargetVariation THEN RETURN path; END; + + IF (path = NIL) THEN RETURN NIL END; + + (* search for the last slash or blank in the string *) + FOR x := Text.Length (path) - 1 TO 0 BY -1 DO + c := Text.GetChar (path, x); + IF (c = '/') OR (c = ' ') OR (c = '\\') THEN + RETURN Text.Sub (path, x+1); + END; + END; + + (* no slashes *) + RETURN path; + END FileTail; + BEGIN END Host. diff --git a/m3-sys/m3front/src/misc/M3Header.m3 b/m3-sys/m3front/src/misc/M3Header.m3 index 1e4decf..877d77c 100644 --- a/m3-sys/m3front/src/misc/M3Header.m3 +++ b/m3-sys/m3front/src/misc/M3Header.m3 @@ -104,7 +104,7 @@ PROCEDURE PushGeneric (VAR s: State) = IF (s.generic = NIL) THEN s.failed := TRUE; RETURN; END; (* build a synthetic file name & start reading *) - filename := old_filename & " => " & filename; + filename := Host.FileTail(old_filename) & " => " & filename; Scanner.Push (filename, s.generic, is_main := Scanner.in_main); (* make sure we got what we wanted *) diff --git a/m3-sys/m3front/src/misc/Scanner.m3 b/m3-sys/m3front/src/misc/Scanner.m3 index 7470374..e1dc024 100644 --- a/m3-sys/m3front/src/misc/Scanner.m3 +++ b/m3-sys/m3front/src/misc/Scanner.m3 @@ -228,13 +228,16 @@ PROCEDURE Here (VAR file: TEXT; VAR line: INTEGER) = BEGIN file := files [offset DIV MaxLines]; line := offset MOD MaxLines; + IF Target.ReduceTargetVariation THEN + file := Host.FileTail(file); + END; END Here; PROCEDURE LocalHere (VAR file: TEXT; VAR line: INTEGER) = VAR fnum := offset DIV MaxLines; BEGIN IF (local_files[fnum] = NIL) THEN - local_files[fnum] := files[fnum]; + local_files[fnum] := Host.FileTail(files[fnum]); END; file := local_files [fnum]; line := offset MOD MaxLines; diff --git a/m3-sys/m3front/src/values/Module.m3 b/m3-sys/m3front/src/values/Module.m3 index a085eab..576c857 100644 --- a/m3-sys/m3front/src/values/Module.m3 +++ b/m3-sys/m3front/src/values/Module.m3 @@ -421,7 +421,7 @@ PROCEDURE PushGeneric (t: T; VAR rd: File.T): M3ID.T = END; (* build a synthetic file name & start reading *) - filename := old_filename & " => " & filename; + filename := Host.FileTail(old_filename) & " => " & filename; Scanner.Push (filename, rd, is_main := Scanner.in_main); t.genericFile := filename; diff --git a/m3-sys/m3middle/src/Target.i3 b/m3-sys/m3middle/src/Target.i3 index fe198d2..cd7acee 100644 --- a/m3-sys/m3middle/src/Target.i3 +++ b/m3-sys/m3middle/src/Target.i3 @@ -529,4 +529,8 @@ VAR (*CONST*) test for nested procedures passed as parameters must be more elaborate (e.g. HPPA). *) + (* This removes some unnecessary target variation in the output, + * such as current working directory in debug output. *) + ReduceTargetVariation: BOOLEAN; + END Target. diff --git a/scripts/python/pylib.py b/scripts/python/pylib.py index 73e622f..487ad17 100755 --- a/scripts/python/pylib.py +++ b/scripts/python/pylib.py @@ -388,7 +388,7 @@ def _GetAllTargets(): _CBackend = "c" in sys.argv or "C" in sys.argv _BuildDirC = ["", "c"][_CBackend] -_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why"] +_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why", "reduce-target-variation", "reducetargetvariation"] _SkipGccFlags = ["nogcc", "skipgcc", "omitgcc"] _PossiblePylibFlags = ["noclean", "nocleangcc", "c", "C"] + _SkipGccFlags + _PossibleCm3Flags @@ -1610,9 +1610,9 @@ def Boot(): Makefile.close() if vms or nt: - _MakeZip(BootDir[2:]) + pass#_MakeZip(BootDir[2:]) else: - _MakeTGZ(BootDir[2:]) + pass#_MakeTGZ(BootDir[2:]) #----------------------------------------------------------------------------- # map action names to code and possibly other data -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Tue Jul 4 09:08:44 2017 From: jayk123 at hotmail.com (Jay K) Date: Tue, 4 Jul 2017 07:08:44 +0000 Subject: [M3devel] Fw: [M3commit] [modula3/cm3] 0d5954: reduce-target-variation In-Reply-To: <595b33d19292d_77853fb988581c2819783d@hookshot-fe-6dbb0c4.cp1-iad.github.net.mail> References: <595b33d19292d_77853fb988581c2819783d@hookshot-fe-6dbb0c4.cp1-iad.github.net.mail> Message-ID: I'd really like an *option* to make builds far more "deterministic" and "consistent" and "repeatable" from machine to machine, maybe similar-target to similar-target (OpenBSD/x86==Linux/x86==NetBSD/x86==Solaris/x86). That is: I want the default to have full paths all over the place, but an option to reduce paths to something the same across all machines. I want the C backend to by default have a bunch of line numbers to aid debugging, but an option (same option) to omit them. Paths like Compiler.ThisFIle, if they have any slashes, should probably just always be forward. This might break some workflows around copy/paste them into file.open dialogs on Windows though. Much of Windows supports forward slashes, and much does not. Maybe "-very-deterministic". Yes this is half joke -- deterministic is thought to be a boolean, and everything should be deterministic, but the definition of deterministic is actually unclear. The "target" in reduce-target-variation isn't clearly the right name for this. Maybe just reduce-variation or reduce-noise or more-reproducible. This is a step toward a distribution form that isn't target-specific. One run of the C backend could generate something redistributable to all or many targets. - Jay ________________________________ From: M3commit on behalf of GitHub Sent: Tuesday, July 4, 2017 6:21 AM To: m3commit at elegosoft.com Subject: [M3commit] [modula3/cm3] 0d5954: reduce-target-variation Branch: refs/heads/reduce-target-variation Home: https://github.com/modula3/cm3 [https://avatars3.githubusercontent.com/u/7759860?v=3&s=400] GitHub - modula3/cm3: Critical Mass Modula-3 github.com cm3 - Critical Mass Modula-3 ... Clone with HTTPS Use Git or checkout with SVN using the web URL. Commit: 0d59546d11641afe3772b73f226c2e90b960d7fc https://github.com/modula3/cm3/commit/0d59546d11641afe3772b73f226c2e90b960d7fc [https://avatars0.githubusercontent.com/u/1635728?v=3&s=200] reduce-target-variation ? modula3/cm3 at 0d59546 github.com cm3 - Critical Mass Modula-3 Author: jaykrell Date: 2017-07-03 (Mon, 03 Jul 2017) Changed paths: M m3-sys/cm3/src/Main.m3 M m3-sys/cm3/src/Makefile.m3 M m3-sys/cminstall/src/config-no-install/cm3cfg.common M m3-sys/m3back/src/M3C.m3 M m3-sys/m3cc/gcc-4.7/gcc/dbxout.c M m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c M m3-sys/m3cc/gcc-4.7/gcc/toplev.c M m3-sys/m3cc/gcc-4.7/gcc/toplev.h M m3-sys/m3cc/gcc/gcc/m3cg/lang.opt M m3-sys/m3cc/gcc/gcc/m3cg/parse.c M m3-sys/m3front/src/misc/Coverage.m3 M m3-sys/m3front/src/misc/Host.i3 M m3-sys/m3front/src/misc/Host.m3 M m3-sys/m3front/src/misc/M3Header.m3 M m3-sys/m3front/src/misc/Scanner.m3 M m3-sys/m3front/src/values/Module.m3 M m3-sys/m3middle/src/Target.i3 M scripts/python/pylib.py Log Message: ----------- reduce-target-variation -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ATT00001.txt URL: From jayk123 at hotmail.com Tue Jul 4 09:27:21 2017 From: jayk123 at hotmail.com (Jay K) Date: Tue, 4 Jul 2017 07:27:21 +0000 Subject: [M3devel] Thread.HardwareConcurrency()? Message-ID: Clearing out more backlog... I suggest, for example, the cm3 compiler should use a thread per "processor" for the backend. I acknowledge that "processor" isn't well defined, such as with Intel hyperthreading and in general with various resources (registers, integer ALU, floating point ALU, branch predictor memory, cache) not being equally duplicated. I acknowledge that hogging all the CPU is deleterious to interactive performance, or overall system performance if anything else is running. Adapting to overall system load and being good citizen is quite difficult unless you are the semi-omniscient operating system kernel. I believe C++11 exposes this notion and Boost did before it. (C++11 finally acknowledged the existance of threads and has a memory model and library for them, 20 years late) I propose that we can freely integrate Boost code into our runtime. That its license is close in spirit to ours. edit m3-libs/m3core/src/thread/Common/BoostThread.cpp #ifndef _WIN32 // Copyright (C) 2001-2003 // William E. Kempf // Copyright (C) 2007-8 Anthony Williams // (C) Copyright 2011-2012 Vicente J. Botet Escriba // // Distributed under the Boost Software License, Version 1.0. (See accompanying // file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) #ifdef __GLIBC__ #include #elif defined(__APPLE__) || defined(__FreeBSD__) #include #include #else // if defined BOOST_HAS_UNISTD_H #include #endif #endif extern "C" unsigned m3_thread_hardware_concurrency() { #if defined(PTW32_VERSION) || defined(__hpux) return pthread_num_processors_np(); #elif defined(__APPLE__) || defined(__FreeBSD__) int count; size_t size = sizeof(count); return sysctlbyname("hw.ncpu", &count, &size, NULL, 0) ? 0 : count; #elif defined(BOOST_HAS_UNISTD_H) && defined(_SC_NPROCESSORS_ONLN) int const count = sysconf(_SC_NPROCESSORS_ONLN); return (count > 0) ? count : 0; #elif defined(__GLIBC__) return get_nprocs(); #else return 0; #endif } #else #include // Distributed under the Boost Software License, Version 1.0. (See // accompanying file LICENSE_1_0.txt or copy at // http://www.boost.org/LICENSE_1_0.txt) // (C) Copyright 2007 Anthony Williams // (C) Copyright 2007 David Deakins // (C) Copyright 2011-2013 Vicente J. Botet Escriba extern "C" unsigned m3_m3_thread_hardware_concurrency() { SYSTEMINFO info; GetSystemInfo(&info); return info.dwNumberOfProcessors; } #endif and then something in Thread.i3. <* external m3_thread_hardware_concurrent *> PROCEDURE Thread.HardwareConcurrency(): INTEGER; ? - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Tue Jul 4 09:52:29 2017 From: jayk123 at hotmail.com (Jay K) Date: Tue, 4 Jul 2017 07:52:29 +0000 Subject: [M3devel] Aligned_procedures? Message-ID: Aligned_procedures I'm sure I've mentioned this before...but I'm clearing out my backlog of lingering diffs. In my bid to make more of the targets look more the same, I suggest making Aligned_procedures always be false. This slightly pessimises mainstream targets: x86 and amd64. I believe it slightly bloats all calls through function pointers. (including object methods? Maybe, but I don't think those can be closures, so that could/should be fixed -- though the idea of a method being a closure is a good one...) It has no affect on PowerPC, ARM, SPARC, MIPS, Alpha, etc. -- 32bit or 64bit. I believe the difference is that when calling a function pointer, on x86/amd64, we just read it for a pointer-size integer, and compare to -1. If Aligned_procedures is left as always false, that check would first see if the pointer is aligned on a pointer-size, and if not, skip the check for -1. This is because most architectures will issue an alignment fault for the unaligned read, and we know such unaligned values are not closures. x86/amd64 do not care much about alignment. I have proposed, somewhat the opposite, that this check actually be always be 4 bytes, not a full pointer. That would likely allow it to always be TRUE. Closures would still be pointer-aligned, but we'd only check for 4 bytes -1 instead of a full pointer. The idea is that all functions are 4-aligned on all targets that care about integer alignment. Even if they aren't 8-aligned on 64bit targets. I believe that would not work for ARM32-Thumb and I can't bring myself to rule out such targets. Another option would be to make this only be for the C backend. It isn't clearly useful given the gcc backend -- unless maybe redistributing same IR across multiple targets. - Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From hendrik at topoi.pooq.com Tue Jul 4 16:52:08 2017 From: hendrik at topoi.pooq.com (Hendrik Boom) Date: Tue, 4 Jul 2017 10:52:08 -0400 Subject: [M3devel] compiler switch "reduce target variation"? In-Reply-To: References: Message-ID: <20170704145208.GC19542@topoi.pooq.com> On Tue, Jul 04, 2017 at 06:34:03AM +0000, Jay K wrote: > Trying to resume work here.. > > Here, I've commited this to a branch: > > > https://github.com/modula3/cm3/commit/0d59546d11641afe3772b73f226c2e90b960d7fc > > > There are some unrelated changes I will tease out and commit separately. > > > I still believe we should implement this. > > > It is a variation of what people call "deterministic builds". > Whn the source code explicitly uses features whose whole purpose is to vary between builds, then it's quite reasonable for the object code to vary, even if one asks for a deterministic build. -- hendrik From jayk123 at hotmail.com Tue Jul 4 17:54:52 2017 From: jayk123 at hotmail.com (Jay K) Date: Tue, 4 Jul 2017 15:54:52 +0000 Subject: [M3devel] compiler switch "reduce target variation"? In-Reply-To: <20170704145208.GC19542@topoi.pooq.com> References: , <20170704145208.GC19542@topoi.pooq.com> Message-ID: Perhaps warnings or errors should be requestable though. For example in C a warning for __TIME__. There is a goal in some parts that take a source tree and be able to build it again on another computer and get identical bytes. It is good for determining "provenance" -- this is the actual source to those binaries (or at least one of them -- if you consider comments, there are infinite source variations that might map to the same binaries). It is good for build systems with aggressive caching -- change comments, recompile those sources, get the same objects, "cut off" the graph walk, everything is correct in the cache, just add another hash for source that maps to object. - Jay ________________________________ From: M3devel on behalf of Hendrik Boom Sent: Tuesday, July 4, 2017 2:52 PM To: m3devel at elegosoft.com Subject: Re: [M3devel] compiler switch "reduce target variation"? On Tue, Jul 04, 2017 at 06:34:03AM +0000, Jay K wrote: > Trying to resume work here.. > > Here, I've commited this to a branch: > > > https://github.com/modula3/cm3/commit/0d59546d11641afe3772b73f226c2e90b960d7fc [https://avatars0.githubusercontent.com/u/1635728?v=3&s=200] reduce-target-variation ? modula3/cm3 at 0d59546 github.com cm3 - Critical Mass Modula-3 > > > There are some unrelated changes I will tease out and commit separately. > > > I still believe we should implement this. > > > It is a variation of what people call "deterministic builds". > Whn the source code explicitly uses features whose whole purpose is to vary between builds, then it's quite reasonable for the object code to vary, even if one asks for a deterministic build. -- hendrik _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Wed Jul 5 17:48:03 2017 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 5 Jul 2017 10:48:03 -0500 Subject: [M3devel] Another backend? In-Reply-To: <1192403415.6727858.1499114084342@mail.yahoo.com> References: <1192403415.6727858.1499114084342.ref@mail.yahoo.com> <1192403415.6727858.1499114084342@mail.yahoo.com> Message-ID: Looks unmaintained for ~10 years. It would probably take significant work to update and integrate. Lots of nice-sounding optimizations. I think we'd have to get it working and try out lots of cases to see how it compared with llvm. On 07/03/2017 03:34 PM, Daniel Alejandro Benavides D. wrote: > http://dada.cs.washington.edu/htbin-post/unrestricted/cecil/cvsweb.cgi/vortex-extensions/M3/ > > > > > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Wed Jul 5 18:07:57 2017 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 5 Jul 2017 11:07:57 -0500 Subject: [M3devel] compiler switch "reduce target variation"? In-Reply-To: References: Message-ID: <1b1e8f67-1637-cedf-18a3-731fea761627@lcwb.coop> The last I knew (a loooong time ago), gcc internally inserted a compilation date & time into an object and/or executable file, without being asked by any source code. The rebuild-twice-and-check-identical-output scripts contained hard-coded displacements of where these are in the object/executable and patched them to zeros or something before diffing. Just another source of nondeterminism in builds. With our own gcc-derived backend, it should not be hard to eliminate, but somebody has to do it. I have the impression the x86 integrated back end does this too. I would expect any stock C or C++ compiler would do it too, but I guess diffing the C source output of the C backend would be where you would check it, rather than the final object/executable. Our embryonic llvm back end so far feeds stock llvm optimization and code generation, which probably does the same thing. I have hopes we can avoid making mods to stock llvm, but that's a very fragile goal. I suppose for this one, we could dump and diff the textual llvm IR code. And, we could still have an after-the-fact patch script/program, as above, but the location to patch probably depends on the backend/compiler used and is also subject to change without notice. On 07/04/2017 01:34 AM, Jay K wrote: > Trying to resume work here.. > > Here, I've commited this to a branch: > > > https://github.com/modula3/cm3/commit/0d59546d11641afe3772b73f226c2e90b960d7fc > > > There are some unrelated changes I will tease out and commit separately. > > > I still believe we should implement this. > > > It is a variation of what people call "deterministic builds". > > > "deterministic builds" are kind of an overloaded term, or common sense at first, but then controversial as details emerge. > > > For examples: > > - Compilers should not generate random numbers, at least not base their output on them. > > > - Optimizers should not "give up" after some elapsed time or consumed memory -- leading to different results on similar-but-different hardware -- though if memory can be accounted precisely and controlled by a switch, maybe > > But then again, compilers should make good use of their host machines -- you can't have it both ways. > > > - full paths should occur nowhere in the output > > see also debugging information, that might work better with full paths -- no need to tell debugger some such search paths > > > - varying a source file's full path should not change how it compiles; but see constructs like #include ../foo.c and printf(__FILE__) or printf("%d\n", sizeof(__FILE__)). And is __FILE__ a full path or not? (Modula-3 has the analogous Compiler.ThisLine and Compiler.ThisFile so please pardon my C++-ness, it still is relevant > > Perhaps compiler can trim prefixes of paths. > > > - adding a comment in a heavily used header should not change any output; but see for example, in such header inline void foo() { printf("%d\n", __LINE__); > > > - adding comments anywhere, really, should not change any output -- but again __LINE__ > > > - recompiling all the same files with the same compiler and command lines should produce the same output every time -- but see for examples printf("%d\n", __TIME__ or __DATE__ or __TIMESTAMP__) or things like Modula-3 date-based-version; burden is perhaps on the programer to not use these problematic constructs, and compiler can maybe get switches to warn/error for them. > > > So, then, I add to this mix, Linux/x86, NetBSD/x86, OpenBSD/x86, Solaris/x86 should generally have identical output. > > But again, things like printing targetname -- including as comments from the C backend. > > > > - Jay > > > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > *From:* M3devel on behalf of Jay K > *Sent:* Wednesday, March 15, 2017 8:37 AM > *To:* m3devel > *Subject:* [M3devel] compiler switch "reduce target variation"? > > > Tangent: Should we start use git branches and pull requests? > Are people here reasonable able to view textual diffs? It pushes my limits personally. > But I know some developers work this way. > > > This is some old stuff I was working on. > > > I kinda think it should be the default, but almost everyone here protests > almost all of my changes, so here is a new switch instead. > > The idea is to remove somewhat gratituous target variation. > One would have to argue over each difference and if it is gratuitous > and if it should be removed by default, or under a switch, and which switch. > > So I'm proposing generally one switch to put everything under. > > The differences I had noticed in my investigation months ago > is the name of the TARGET appearing, for example in paths. > Such as maybe in generic instantiations, in debug symbols, in output logs. > > I actually think debug symbols need full paths, and so for generics, > they end up with target. But for now, I desist on that agenda in favor > of rougly the opposite. > > The goal here, as I mentioned before, is to establish that > I386_LINUX == I386_FREEBSD == I386_NETBSD == I386_OPENBSD == I386_SOLARIS > == I386_DARWIN == I386_CYGWIN == I386_SOLARIS maybe == I386_NT > ditto AMD64, SPARC32, SPARC64, PPC32, PPC64, etc. > > So that we might collapse targets down. > A soon/later step would establish even fewer equivalence classes: > PP32be == SPARC32, PPC64be == SPARC64, => BigEndian64 or PosixBigEndian64 > I386 == ARM => LittleEndian32 > AMD64 == ARM64 == PPC64le => LittleEndian64 > etc. > > and then eventually we could have one or a small number of C-based distributions. > > I have to go back and build and test all this anew. > > Reasonable? > > Maybe the default? > Maybe add a switch for restore-target-difference or preserve-target-differences? > > > diff --git a/m3-sys/cm3/src/Main.m3 b/m3-sys/cm3/src/Main.m3 > index 5d753e6..61b3f18 100644 > --- a/m3-sys/cm3/src/Main.m3 > +++ b/m3-sys/cm3/src/Main.m3 > @@ -5,7 +5,7 @@ MODULE Main; > > IMPORT M3Timers, Pathname, Process, Quake; > IMPORT RTCollector, RTParams, RTutils, Thread, Wr; > -IMPORT TextTextTbl; > +IMPORT TextTextTbl, Target; > > IMPORT Builder, Dirs, M3Build, M3Options, Makefile, Msg, Utils, WebFile; > IMPORT MxConfig(*, M3Config, CMKey, CMCurrent *); > @@ -18,6 +18,8 @@ VAR > build_dir : TEXT := NIL; > mach : Quake.Machine := NIL; > > +CONST BoolToText = ARRAY BOOLEAN OF TEXT{"FALSE", "TRUE"}; > + > PROCEDURE DefineIfNotDefined (qmachine: Quake.Machine; > symbol, value: TEXT) RAISES {Quake.Error} = > BEGIN > @@ -73,6 +75,7 @@ VAR defs: TextTextTbl.T; > (* DefineIfNotDefined (mach, "THREAD_LIBRARY", Version.ThreadLibrary); *) > (* DefineIfNotDefined (mach, "WINDOW_LIBRARY", Version.WindowLibrary); *) > DefineIfNotDefined (mach, "WORD_SIZE", MxConfig.HOST_WORD_SIZE); > + DefineIfNotDefined (mach, "REDUCE_TARGET_VARIATION", BoolToText[Target.ReduceTargetVariation]); > > (* Even if the config file overrides the defaults, such as to do > a cross build, the host characteristics are still available. *) > diff --git a/m3-sys/cm3/src/Makefile.m3 b/m3-sys/cm3/src/Makefile.m3 > index 64e169e..f2e1c7b 100644 > --- a/m3-sys/cm3/src/Makefile.m3 > +++ b/m3-sys/cm3/src/Makefile.m3 > @@ -5,7 +5,7 @@ MODULE Makefile; > > IMPORT FS, M3File, M3Timers, OSError, Params, Process, Text, Thread, Wr; > IMPORT Arg, M3Build, M3Options, M3Path, Msg, Utils, TextSeq, TextTextTbl; > -IMPORT MxConfig, Dirs, Version; > +IMPORT MxConfig, Dirs, Version, Target; > > TYPE > NK = M3Path.Kind; > @@ -267,6 +267,9 @@ PROCEDURE ConvertOption (VAR s: State; arg: TEXT; arg_len: INTEGER) > | 'r' => IF Text.Equal(arg, "-realclean") THEN > ok := TRUE; (* mode set during the pre-scan *) > s.found_work := TRUE; > + ELSIF Text.Equal(arg, "-reduce-target-variation") THEN > + ok := TRUE; > + Target.ReduceTargetVariation := TRUE; > END; > > | 's' => IF Text.Equal (arg, "-silent") THEN > @@ -707,6 +710,9 @@ CONST > " -group-writable \"", > " -pb allow parallelism in running back-end (experimental)", > " -no-m3ship-resolution use quake variables in .M3SHIP (experimental)", > + " -reduce-target-variation omit target in some minor places such as", > + " current working directory in debug information", > + " for internal development purposes (showing target equivalence)", > "", > "environment variables:", > " M3CONFIG platform dependent configuration file to use (cm3.cfg)", > diff --git a/m3-sys/cminstall/src/config-no-install/cm3cfg.common b/m3-sys/cminstall/src/config-no-install/cm3cfg.common > index 7334252..42db06f 100644 > --- a/m3-sys/cminstall/src/config-no-install/cm3cfg.common > +++ b/m3-sys/cminstall/src/config-no-install/cm3cfg.common > @@ -458,8 +458,14 @@ proc GetM3Back() is > else if equal(HOST, TARGET) > m3back = INSTALL_ROOT & "/bin/" > end end > - > + > m3back = "@" & m3back & "cm3cg " & GetM3BackFlags() > + > + if defined ("REDUCE_TARGET_VARIATION") > + if REDUCE_TARGET_VARIATION > + m3back = m3back & " -freduce-target-variation" > + end > + end > return m3back > end > > diff --git a/m3-sys/m3back/src/M3C.m3 b/m3-sys/m3back/src/M3C.m3 > index f6740c7..d584fea 100644 > --- a/m3-sys/m3back/src/M3C.m3 > +++ b/m3-sys/m3back/src/M3C.m3 > @@ -29,7 +29,7 @@ VAR CaseDefaultAssertFalse := FALSE; > > (* Taken together, these help debugging, as you get more lines in the > C and the error messages reference C line numbers *) > - CONST output_line_directives = TRUE; > + VAR output_line_directives := TRUE; > CONST output_extra_newlines = FALSE; > CONST inline_extract = FALSE; > > @@ -2197,7 +2197,9 @@ BEGIN > END; > > IF (*self.suppress_line_directive < 1 AND*) text_last_char = '\n' THEN > - Wr.PutText(self.c, self.line_directive); > + IF NOT Target.ReduceTargetVariation THEN > + Wr.PutText(self.c, self.line_directive); > + END; > self.width := 0; > self.last_char_was_newline := TRUE; > RETURN; > @@ -2205,7 +2207,9 @@ BEGIN > > IF Text.FindChar(text, '\n') # -1 THEN > self.width := 0; (* roughly *) > - Wr.PutText(self.c, self.nl_line_directive); > + IF NOT Target.ReduceTargetVariation THEN > + Wr.PutText(self.c, self.nl_line_directive); > + END; > self.last_char_was_newline := TRUE; > RETURN; > END; > @@ -2217,10 +2221,12 @@ BEGIN > END; > > self.width := 0; > - IF self.last_char_was_newline THEN > - Wr.PutText(self.c, self.line_directive); > - ELSE > - Wr.PutText(self.c, self.nl_line_directive); > + IF NOT Target.ReduceTargetVariation THEN > + IF self.last_char_was_newline THEN > + Wr.PutText(self.c, self.line_directive); > + ELSE > + Wr.PutText(self.c, self.nl_line_directive); > + END; > END; > self.last_char_was_newline := TRUE; > END print; > @@ -2339,9 +2345,10 @@ END set_error_handler; > PROCEDURE Prefix_Print(self: T; multipass: Multipass_t) = > BEGIN > self.comment("begin unit"); > - self.comment("M3_TARGET = ", Target.System_name); > - (* This is an unnecessary target-specific output. *) > - (* self.comment("M3_TARGET = ", Target.System_name); *) > + output_line_directives := output_line_directives AND NOT Target.ReduceTargetVariation; > + IF NOT Target.ReduceTargetVariation THEN > + self.comment("M3_TARGET = ", Target.System_name); > + END; > self.comment("M3_WORDSIZE = ", IntToDec(Target.Word.size)); > self.static_link_id := M3ID.Add("_static_link"); > self.alloca_id := M3ID.Add("alloca"); > @@ -4836,7 +4843,8 @@ BEGIN > & " ok2=" & BoolToText[ok2] & "\n" > ); > RTIO.Flush(); > - <* ASSERT FALSE *> > + <* ASSERT ok1 *> > + <* ASSERT ok2 *> > END; > IF type = CGType.Int32 AND TInt.EQ(i, TInt.Min32) THEN > RETURN "-" & intLiteralPrefix[type] & TInt.ToText(TInt.Max32) & intLiteralSuffix[type] & "-1"; > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > index dc52576..33f8844 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > +++ b/m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > @@ -1065,8 +1065,11 @@ dbxout_init (const char *input_file_name) > labels. */ > ASM_GENERATE_INTERNAL_LABEL (ltext_label_name, "Ltext", 0); > > - /* Put the current working directory in an N_SO symbol. */ > - if (use_gnu_debug_info_extensions && !NO_DBX_MAIN_SOURCE_DIRECTORY) > + /* Limit paths in debug output, to limit target variation. */ > + if (!reduce_target_variation) > + { > + /* Put the current working directory in an N_SO symbol. */ > + if (use_gnu_debug_info_extensions && !NO_DBX_MAIN_SOURCE_DIRECTORY) > { > static const char *cwd; > > @@ -1087,6 +1090,7 @@ dbxout_init (const char *input_file_name) > used_ltext_label_name = true; > #endif /* no DBX_OUTPUT_MAIN_SOURCE_DIRECTORY */ > } > + } > > mapped_name = remap_debug_filename (input_file_name); > #ifdef DBX_OUTPUT_MAIN_SOURCE_FILENAME > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > index 6f60742..291aea3 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > +++ b/m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > @@ -15450,15 +15450,18 @@ add_gnat_descriptive_type_attribute (dw_die_ref die, tree type, > static void > add_comp_dir_attribute (dw_die_ref die) > { > + /* Limit paths in debug output, to limit target variation. */ > + if (reduce_target_variation) > + return; > + > const char *wd = get_src_pwd (); > - char *wd1; > > if (wd == NULL) > return; > > if (DWARF2_DIR_SHOULD_END_WITH_SEPARATOR) > { > - int const wdlen = (int)strlen (wd); > + size_t const wdlen = strlen (wd); > char * const wd1 = (char *) ggc_alloc_atomic (wdlen + 2); > memcpy (wd1, wd, wdlen); > wd1 [wdlen] = DIR_SEPARATOR; > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c > index 63e4b92..d468632 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.c > +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.c > @@ -217,11 +217,8 @@ const char * > get_src_pwd (void) > { > if (! src_pwd) > - { > - src_pwd = getpwd (); > - if (!src_pwd) > -src_pwd = "."; > - } > + if (reduce_target_variation || !(src_pwd = getpwd ())) > + src_pwd = "."; > > return src_pwd; > } > diff --git a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h > index 588cfdb..f4f7cc7 100644 > --- a/m3-sys/m3cc/gcc-4.7/gcc/toplev.h > +++ b/m3-sys/m3cc/gcc-4.7/gcc/toplev.h > @@ -80,4 +80,6 @@ extern bool set_src_pwd (const char *); > extern HOST_WIDE_INT get_random_seed (bool); > extern const char *set_random_seed (const char *); > > +extern bool reduce_target_variation; > + > #endif /* ! GCC_TOPLEV_H */ > diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > index 3bd0469..cb0189f 100644 > --- a/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > +++ b/m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > @@ -28,9 +28,6 @@ m3cg > Language > M3CG > > -y > -m3cg M3CG > - > fopcodes-trace > m3cg M3CG > Trace opcodes > @@ -59,10 +56,17 @@ ftypes-trace > m3cg M3CG > Trace types > > +reduce-target-variation > +m3cg M3CG > +Reduce target variation somewhat, such as by omitting current working > +directory from debug info. Many necessary target variations remain. > + > v > m3cg M3CG > +print version > > y > m3cg M3CG > +Trace opcodes > > ; This comment is to ensure we retain the blank line above. > diff --git a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c > index 03417ed..071bbc8 100644 > --- a/m3-sys/m3cc/gcc/gcc/m3cg/parse.c > +++ b/m3-sys/m3cc/gcc/gcc/m3cg/parse.c > @@ -246,6 +246,7 @@ build_case_label (tree low_value, tree high_value, tree label_decl) > /*======================================================= OPTION HANDLING ===*/ > > static int option_trace_all; > +bool reduce_target_variation; > > /*===========================================================================*/ > > @@ -6364,6 +6365,10 @@ m3_handle_option (size_t code, PCSTR /*arg*/, int /*value*/) > case OPT_ftypes_trace: > option_trace_all += 1; > break; > + > + case OPT_reduce_target_variation: > + reduce_target_variation = true; > + break; > } > > return 1; > diff --git a/m3-sys/m3front/src/misc/Coverage.m3 b/m3-sys/m3front/src/misc/Coverage.m3 > index c04c902..73fff21 100644 > > I don't remember what is going on here, but coverage isn't used much.. > I suspect it might just have to do with forward vs. backward slashes, > and that this distinction does not really matter -- Windows accepts forward slashes > so we should just use them everywhere. > > --- a/m3-sys/m3front/src/misc/Coverage.m3 > +++ b/m3-sys/m3front/src/misc/Coverage.m3 > @@ -77,8 +77,9 @@ PROCEDURE NoteProcedure (v: Value.T) = > PROCEDURE GenerateTables () = > VAR > nLines := MAX (0, maxLine - minLine) + 1; > + fname := Host.FileTail (Host.filename); > l_header := TLen (Header); > - l_fname := TLen (Host.filename); > + l_fname := TLen (fname); > l_trailer := TLen (Trailer); > size : INTEGER; > p : ProcHead; > @@ -124,10 +125,10 @@ PROCEDURE GenerateTables () = > (* CG.Init_int (size, Target.Integer.size, TInt.Zero, FALSE); *) > INC (size, Target.Integer.size); (*timestamp*) > > - CG.Init_intt (size, Target.Integer.size, Text.Length (Host.filename), FALSE); > + CG.Init_intt (size, Target.Integer.size, Text.Length (fname), FALSE); > INC (size, Target.Integer.size); (*fileLen*) > > - CG.Init_chars (size, Host.filename, FALSE); > + CG.Init_chars (size, fname, FALSE); > INC (size, l_fname * Target.Char.size); (*file*) > > CG.Init_intt (size, Target.Integer.size, minLine, FALSE); > diff --git a/m3-sys/m3front/src/misc/Host.i3 b/m3-sys/m3front/src/misc/Host.i3 > index c71489f..af6a89a 100644 > --- a/m3-sys/m3front/src/misc/Host.i3 > +++ b/m3-sys/m3front/src/misc/Host.i3 > @@ -75,4 +75,7 @@ PROCEDURE OpenUnit (name: M3ID.T; interface, generic: BOOLEAN; > > PROCEDURE CloseFile (rd: File.T); > > +PROCEDURE FileTail (path: TEXT): TEXT; > + (* returns the 'tail' of 'path' -- after any slashes or even spaces *) > + > END Host. > diff --git a/m3-sys/m3front/src/misc/Host.m3 b/m3-sys/m3front/src/misc/Host.m3 > index 962d3c6..79042e3 100644 > --- a/m3-sys/m3front/src/misc/Host.m3 > +++ b/m3-sys/m3front/src/misc/Host.m3 > @@ -9,7 +9,7 @@ > > MODULE Host; > > -IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler; > +IMPORT File, Text, (*ETimer, M3Timers,*) M3ID, M3Compiler, Target; > > PROCEDURE Initialize (READONLY options: ARRAY OF TEXT): BOOLEAN = > BEGIN > @@ -192,5 +192,24 @@ PROCEDURE CloseFile (rd: File.T) = > END; > END CloseFile; > > +PROCEDURE FileTail (path: TEXT): TEXT = > + VAR c: CHAR; > + BEGIN > + IF NOT Target.ReduceTargetVariation THEN RETURN path; END; > + > + IF (path = NIL) THEN RETURN NIL END; > + > + (* search for the last slash or blank in the string *) > + FOR x := Text.Length (path) - 1 TO 0 BY -1 DO > + c := Text.GetChar (path, x); > + IF (c = '/') OR (c = ' ') OR (c = '\\') THEN > + RETURN Text.Sub (path, x+1); > + END; > + END; > + > + (* no slashes *) > + RETURN path; > + END FileTail; > + > BEGIN > END Host. > diff --git a/m3-sys/m3front/src/misc/M3Header.m3 b/m3-sys/m3front/src/misc/M3Header.m3 > index 1e4decf..877d77c 100644 > --- a/m3-sys/m3front/src/misc/M3Header.m3 > +++ b/m3-sys/m3front/src/misc/M3Header.m3 > @@ -104,7 +104,7 @@ PROCEDURE PushGeneric (VAR s: State) = > IF (s.generic = NIL) THEN s.failed := TRUE; RETURN; END; > > (* build a synthetic file name & start reading *) > - filename := old_filename & " => " & filename; > + filename := Host.FileTail(old_filename) & " => " & filename; > Scanner.Push (filename, s.generic, is_main := Scanner.in_main); > > (* make sure we got what we wanted *) > diff --git a/m3-sys/m3front/src/misc/Scanner.m3 b/m3-sys/m3front/src/misc/Scanner.m3 > index 7470374..e1dc024 100644 > --- a/m3-sys/m3front/src/misc/Scanner.m3 > +++ b/m3-sys/m3front/src/misc/Scanner.m3 > @@ -228,13 +228,16 @@ PROCEDURE Here (VAR file: TEXT; VAR line: INTEGER) = > BEGIN > file := files [offset DIV MaxLines]; > line := offset MOD MaxLines; > + IF Target.ReduceTargetVariation THEN > + file := Host.FileTail(file); > + END; > END Here; > > PROCEDURE LocalHere (VAR file: TEXT; VAR line: INTEGER) = > VAR fnum := offset DIV MaxLines; > BEGIN > IF (local_files[fnum] = NIL) THEN > - local_files[fnum] := files[fnum]; > + local_files[fnum] := Host.FileTail(files[fnum]); > END; > file := local_files [fnum]; > line := offset MOD MaxLines; > diff --git a/m3-sys/m3front/src/values/Module.m3 b/m3-sys/m3front/src/values/Module.m3 > index a085eab..576c857 100644 > --- a/m3-sys/m3front/src/values/Module.m3 > +++ b/m3-sys/m3front/src/values/Module.m3 > @@ -421,7 +421,7 @@ PROCEDURE PushGeneric (t: T; VAR rd: File.T): M3ID.T = > END; > > (* build a synthetic file name & start reading *) > - filename := old_filename & " => " & filename; > + filename := Host.FileTail(old_filename) & " => " & filename; > Scanner.Push (filename, rd, is_main := Scanner.in_main); > t.genericFile := filename; > > diff --git a/m3-sys/m3middle/src/Target.i3 b/m3-sys/m3middle/src/Target.i3 > index fe198d2..cd7acee 100644 > --- a/m3-sys/m3middle/src/Target.i3 > +++ b/m3-sys/m3middle/src/Target.i3 > @@ -529,4 +529,8 @@ VAR (*CONST*) > test for nested procedures passed as parameters must be more > elaborate (e.g. HPPA). *) > > + (* This removes some unnecessary target variation in the output, > + * such as current working directory in debug output. *) > + ReduceTargetVariation: BOOLEAN; > + > END Target. > diff --git a/scripts/python/pylib.py b/scripts/python/pylib.py > index 73e622f..487ad17 100755 > --- a/scripts/python/pylib.py > +++ b/scripts/python/pylib.py > @@ -388,7 +388,7 @@ def _GetAllTargets(): > > _CBackend = "c" in sys.argv or "C" in sys.argv > _BuildDirC = ["", "c"][_CBackend] > -_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why"] > +_PossibleCm3Flags = ["boot", "keep", "override", "commands", "verbose", "why", "reduce-target-variation", "reducetargetvariation"] > _SkipGccFlags = ["nogcc", "skipgcc", "omitgcc"] > _PossiblePylibFlags = ["noclean", "nocleangcc", "c", "C"] + _SkipGccFlags + _PossibleCm3Flags > > @@ -1610,9 +1610,9 @@ def Boot(): > Makefile.close() > > if vms or nt: > - _MakeZip(BootDir[2:]) > + pass#_MakeZip(BootDir[2:]) > else: > - _MakeTGZ(BootDir[2:]) > + pass#_MakeTGZ(BootDir[2:]) > > #----------------------------------------------------------------------------- > # map action names to code and possibly other data > > > > > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Wed Jul 5 18:13:53 2017 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 5 Jul 2017 11:13:53 -0500 Subject: [M3devel] Fw: [M3commit] [modula3/cm3] 0d5954: reduce-target-variation In-Reply-To: References: <595b33d19292d_77853fb988581c2819783d@hookshot-fe-6dbb0c4.cp1-iad.github.net.mail> Message-ID: <221824bc-5c00-069a-68e9-eb280fe7d337@lcwb.coop> On 07/04/2017 02:08 AM, Jay K wrote: > I'd really like an *option* to make builds far more "deterministic" and "consistent" and "repeatable" from machine to machine, maybe similar-target to similar-target (OpenBSD/x86==Linux/x86==NetBSD/x86==Solaris/x86). > > > That is: I want the default to have full paths all over the place, but an option to reduce paths to something the same across all machines. > > > I want the C backend to by default have a bunch of line numbers to aid debugging, but an option (same option) to omit them. > > > Paths like Compiler.ThisFIle, if they have any slashes, should probably just always be forward. > > This might break some workflows around copy/paste them into file.open dialogs on Windows though. > > Much of Windows supports forward slashes, and much does not. > > > Maybe "-very-deterministic". Yes this is half joke -- deterministic is thought to be a boolean, and everything should be deterministic, but the definition of deterministic is actually unclear. > > > The "target" in reduce-target-variation isn't clearly the right name for this. > > Maybe just reduce-variation or reduce-noise or more-reproducible. > > > This is a step toward a distribution form that isn't target-specific. > > One run of the C backend could generate something redistributable to all or many targets. In this case, we would be taking it on faith that the final C compiler didn't introduce any nondeterminism that we cared about. But perhaps determinism in our own code is all we are concerned with. > > > - Jay > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > *From:* M3commit on behalf of GitHub > *Sent:* Tuesday, July 4, 2017 6:21 AM > *To:* m3commit at elegosoft.com > *Subject:* [M3commit] [modula3/cm3] 0d5954: reduce-target-variation > > Branch: refs/heads/reduce-target-variation > Home: https://github.com/modula3/cm3 > > > GitHub - modula3/cm3: Critical Mass Modula-3 > github.com > cm3 - Critical Mass Modula-3 ... Clone with HTTPS Use Git or checkout with SVN using the web URL. > > > > Commit: 0d59546d11641afe3772b73f226c2e90b960d7fc > https://github.com/modula3/cm3/commit/0d59546d11641afe3772b73f226c2e90b960d7fc > > > reduce-target-variation ? modula3/cm3 at 0d59546 > github.com > cm3 - Critical Mass Modula-3 > > > > Author: jaykrell > Date: 2017-07-03 (Mon, 03 Jul 2017) > > Changed paths: > M m3-sys/cm3/src/Main.m3 > M m3-sys/cm3/src/Makefile.m3 > M m3-sys/cminstall/src/config-no-install/cm3cfg.common > M m3-sys/m3back/src/M3C.m3 > M m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > M m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > M m3-sys/m3cc/gcc-4.7/gcc/toplev.c > M m3-sys/m3cc/gcc-4.7/gcc/toplev.h > M m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > M m3-sys/m3cc/gcc/gcc/m3cg/parse.c > M m3-sys/m3front/src/misc/Coverage.m3 > M m3-sys/m3front/src/misc/Host.i3 > M m3-sys/m3front/src/misc/Host.m3 > M m3-sys/m3front/src/misc/M3Header.m3 > M m3-sys/m3front/src/misc/Scanner.m3 > M m3-sys/m3front/src/values/Module.m3 > M m3-sys/m3middle/src/Target.i3 > M scripts/python/pylib.py > > Log Message: > ----------- > reduce-target-variation > > > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Wed Jul 5 18:16:57 2017 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 5 Jul 2017 11:16:57 -0500 Subject: [M3devel] Thread.HardwareConcurrency()? In-Reply-To: References: Message-ID: <82dbe8df-fc3d-04fb-9cf0-07c5d47daf74@lcwb.coop> On 07/04/2017 02:27 AM, Jay K wrote: > Clearing out more backlog... > > > I suggest, for example, the cm3 compiler should use > a thread per "processor" for the backend. > > > I acknowledge that "processor" isn't well defined, such as with Intel > hyperthreading and in general with various resources (registers, integer ALU, > floating point ALU, branch predictor memory, cache) not being equally duplicated. > > > I acknowledge that hogging all the CPU is deleterious to interactive > performance, or overall system performance if anything else is running. > When I have the option, I usually use one fewer threads than processors, so I can read email, etc. while waiting. The compiler could alternatively be set up to do this. > > Adapting to overall system load and being good citizen is quite difficult > unless you are the semi-omniscient operating system kernel. > > > I believe C++11 exposes this notion and Boost did before it. > (C++11 finally acknowledged the existance of threads and has > a memory model and library for them, 20 years late) > > > I propose that we can freely integrate Boost code into our runtime. > That its license is close in spirit to ours. > > > edit m3-libs/m3core/src/thread/Common/BoostThread.cpp > > > #ifndef _WIN32 > // Copyright (C) 2001-2003 > // William E. Kempf > // Copyright (C) 2007-8 Anthony Williams > // (C) Copyright 2011-2012 Vicente J. Botet Escriba > // > // Distributed under the Boost Software License, Version 1.0. (See accompanying > // file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) > #ifdef __GLIBC__ > #include > #elif defined(__APPLE__) || defined(__FreeBSD__) > #include > #include > #else // if defined BOOST_HAS_UNISTD_H > #include > #endif > #endif > > > extern "C" > unsigned m3_thread_hardware_concurrency() > { > #if defined(PTW32_VERSION) || defined(__hpux) > return pthread_num_processors_np(); > #elif defined(__APPLE__) || defined(__FreeBSD__) > int count; > size_t size = sizeof(count); > return sysctlbyname("hw.ncpu", &count, &size, NULL, 0) ? 0 : count; > #elif defined(BOOST_HAS_UNISTD_H) && defined(_SC_NPROCESSORS_ONLN) > int const count = sysconf(_SC_NPROCESSORS_ONLN); > return (count > 0) ? count : 0; > #elif defined(__GLIBC__) > return get_nprocs(); > #else > return 0; > #endif > } > #else > #include > // Distributed under the Boost Software License, Version 1.0. (See > // accompanying file LICENSE_1_0.txt or copy at > // http://www.boost.org/LICENSE_1_0.txt) > // (C) Copyright 2007 Anthony Williams > // (C) Copyright 2007 David Deakins > // (C) Copyright 2011-2013 Vicente J. Botet Escriba > extern "C" > unsigned m3_m3_thread_hardware_concurrency() > { > SYSTEMINFO info; > GetSystemInfo(&info); > return info.dwNumberOfProcessors; > } > #endif > > > > and then something in Thread.i3. > > > <* external m3_thread_hardware_concurrent *> > PROCEDURE Thread.HardwareConcurrency(): INTEGER; > > > ? > > > - Jay > > > > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Wed Jul 5 21:22:53 2017 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Wed, 5 Jul 2017 14:22:53 -0500 Subject: [M3devel] Aligned_procedures? In-Reply-To: References: Message-ID: <599509ee-248d-6ab1-e7b7-e56e9da2932a@lcwb.coop> On 07/04/2017 02:52 AM, Jay K wrote: > Aligned_procedures > > > I'm sure I've mentioned this before...but I'm clearing out my backlog of lingering diffs. > > > In my bid to make more of the targets look more the same, > I suggest making Aligned_procedures always be false. > > > This slightly pessimises mainstream targets: x86 and amd64. > > > I believe it slightly bloats all calls through function pointers. > (including object methods? Maybe, but I don't think those can be closures, > so that could/should be fixed -- though the idea of a method being a closure > is a good one...) > Only calls through a formal parameter of procedure type (not a variable, field, etc.) and assignments other than passing things to a VALUE or READONLY formal need to do a closure-check. Other cases just use/copy the pointer value. > > It has no affect on PowerPC, ARM, SPARC, MIPS, Alpha, etc. -- 32bit or 64bit. > Is this because these targets require all procedures to be have the same alignment as integer anyway? So code is always as if Aligned_procedures were true, i.e., no alignment check is ever necessary? > > I believe the difference is that when calling a function pointer, on x86/amd64, > we just read it for a pointer-size integer, and compare to -1. > > > If Aligned_procedures is left as always false, that check would first > see if the pointer is aligned on a pointer-size, and if not, skip the check for -1. > > > This is because most architectures will issue an alignment fault for the > unaligned read, and we know such unaligned values are not closures. > x86/amd64 do not care much about alignment. > > > I have proposed, somewhat the opposite, that this check actually be always be 4 bytes, > not a full pointer. That would likely allow it to always be TRUE. Closures would still > be pointer-aligned, but we'd only check for 4 bytes -1 instead of a full pointer. > > The idea is that all functions are 4-aligned on all targets that care about integer alignment. > Even if they aren't 8-aligned on 64bit targets. > So no alignment check is ever required. We still have to pad function starts to 4-bytes. I would call this Aligned_procedures=true on32-bit targets and 64-bit targets that do not otherwise require 8-byte alignment of functions, and somewhere partway between false and true for 64-bit targets that do not otherwise require 8-byte alignment of functions, since functions are only partially aligned, and still no alignment check is required. We did once have the discussion whether there exists or could someday exist, a target where 4 bytes or 8 bytes of all one-bits would be valid machine code at the start of a function, or anywhere at all. The only conclusion I recall is that it is unlikely. But this scheme would be slightly weaker in this regard in that it would take a mere 4 bytes of -1 as valid code, to be mistaken for a closure. > > I believe that would not work for ARM32-Thumb and I can't bring myself to rule > out such targets. > What are the relevant properties of ARM32-Thumb? > > Another option would be to make this only be for the C backend. > > It isn't clearly useful given the gcc backend -- unless maybe redistributing > same IR across multiple targets. > > - Jay > > I like the idea of just unconditionally integer-aligning all procedures on all targets. No runtime alignment check would ever be necessary, reducing the time bloat, at the cost of extra code size bloat on those targets where aligning every procedure would not otherwise be required. I like that size/time tradeoff better. The code sequence for closure checks looks pretty gross right now. It is poorly optimized. I have looked at improving it, but some combination of the alignment check, the nil check, and the -1 check are produced at nicely-abstracted different places in CG that don't know about each other, so it would take some rework to do it. Maybe even a raised-level CG IR operator "closure_check". Actually, the unaligned checks increase code size as well as execution time for closure checks, which could partially compensate or even overcompensate for the alignment padding. OTOH, probably many programs have no cases that require closure checks at all, so for those, it would be pure size loss for the extra alignment pad bytes. > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org From jayk123 at hotmail.com Thu Jul 6 01:42:47 2017 From: jayk123 at hotmail.com (Jay K) Date: Wed, 5 Jul 2017 23:42:47 +0000 Subject: [M3devel] Aligned_procedures? In-Reply-To: <599509ee-248d-6ab1-e7b7-e56e9da2932a@lcwb.coop> References: , <599509ee-248d-6ab1-e7b7-e56e9da2932a@lcwb.coop> Message-ID: - Jay _____________________________ From: Rodney M. Bates > Sent: Wednesday, July 5, 2017 12:24 PM Subject: Re: [M3devel] Aligned_procedures? To: > 1. Because we are not allowed to store function pointers, lest they are closures, and become stale? 2. I might have made a mistake: 32 bit targets mights also be true. 3. The rationale is a little confusing because there are multiple factors. The factors are the alignment of normal functions and the alignment requirements of the architecture to read an INTEGER (which I suggest should be instead a 4 byte integer at least on most architectures, in this context). The generated code reads an integer from a function pointer, compares to -1. It assumes -1 can't be a valid instruction, at least as the first in a function. This is of dubious portability both because -1 is not well known to me as invalid code, and because not all systems allow reading code bytes. If -1, it is assumed to be a closure and reads the function pointer and static link from the subsequent words. The "problem" is that, closures are guaranteed to be at least word-aligned, and the read to check for -1 guaranteed not to trigger an alignment fault. But, on some systems, other function pointers have no such alignment guarantee. So an alignment check is optionally inserted to avoid the alignment fault. We could also unconditionally insert the alignment check. It is never wrong. It is code bloat if not needed but arguably it is a nice optimization. We could also leave the choice to the backend. x86/amd64 have no alignment requirement for integers or instructions or functions. So the check is not needed. PowerPC, MIPS, Alpha, Sparc, arm64 I believe all have fixed size 4 byte 4-aligned instructions. Reading a 4 byte integer should be ok, unconditionally through a function pointer, but not an 8 byte integer. Arm32 is wierd. I believe instructions are either 2 or 4 bytes, and aligned to only 2?? The low bit indicates the size: 0 for 4, 1 for 2. The alignment check is needed, or clear the low 2 bits and read. Clear the low bits and read is also a portable approach. IA64 bundles up to 3 instructions in 128 bits with..41 bits per instruction and 5 bit template. I don't know their alignment. I haven't been able to think of another solution, that doesn't use runtime codegen..until recently, but the other solution I know of..generates closures slowly and with OS and processor porting work. - Jay On 07/04/2017 02:52 AM, Jay K wrote: > Aligned_procedures > > > I'm sure I've mentioned this before...but I'm clearing out my backlog of lingering diffs. > > > In my bid to make more of the targets look more the same, > I suggest making Aligned_procedures always be false. > > > This slightly pessimises mainstream targets: x86 and amd64. > > > I believe it slightly bloats all calls through function pointers. > (including object methods? Maybe, but I don't think those can be closures, > so that could/should be fixed -- though the idea of a method being a closure > is a good one...) > Only calls through a formal parameter of procedure type (not a variable, field, etc.) and assignments other than passing things to a VALUE or READONLY formal need to do a closure-check. Other cases just use/copy the pointer value. > > It has no affect on PowerPC, ARM, SPARC, MIPS, Alpha, etc. -- 32bit or 64bit. > Is this because these targets require all procedures to be have the same alignment as integer anyway? So code is always as if Aligned_procedures were true, i.e., no alignment check is ever necessary? > > I believe the difference is that when calling a function pointer, on x86/amd64, > we just read it for a pointer-size integer, and compare to -1. > > > If Aligned_procedures is left as always false, that check would first > see if the pointer is aligned on a pointer-size, and if not, skip the check for -1. > > > This is because most architectures will issue an alignment fault for the > unaligned read, and we know such unaligned values are not closures. > x86/amd64 do not care much about alignment. > > > I have proposed, somewhat the opposite, that this check actually be always be 4 bytes, > not a full pointer. That would likely allow it to always be TRUE. Closures would still > be pointer-aligned, but we'd only check for 4 bytes -1 instead of a full pointer. > > The idea is that all functions are 4-aligned on all targets that care about integer alignment. > Even if they aren't 8-aligned on 64bit targets. > So no alignment check is ever required. We still have to pad function starts to 4-bytes. I would call this Aligned_procedures=true on32-bit targets and 64-bit targets that do not otherwise require 8-byte alignment of functions, and somewhere partway between false and true for 64-bit targets that do not otherwise require 8-byte alignment of functions, since functions are only partially aligned, and still no alignment check is required. We did once have the discussion whether there exists or could someday exist, a target where 4 bytes or 8 bytes of all one-bits would be valid machine code at the start of a function, or anywhere at all. The only conclusion I recall is that it is unlikely. But this scheme would be slightly weaker in this regard in that it would take a mere 4 bytes of -1 as valid code, to be mistaken for a closure. > > I believe that would not work for ARM32-Thumb and I can't bring myself to rule > out such targets. > What are the relevant properties of ARM32-Thumb? > > Another option would be to make this only be for the C backend. > > It isn't clearly useful given the gcc backend -- unless maybe redistributing > same IR across multiple targets. > > - Jay > > I like the idea of just unconditionally integer-aligning all procedures on all targets. No runtime alignment check would ever be necessary, reducing the time bloat, at the cost of extra code size bloat on those targets where aligning every procedure would not otherwise be required. I like that size/time tradeoff better. The code sequence for closure checks looks pretty gross right now. It is poorly optimized. I have looked at improving it, but some combination of the alignment check, the nil check, and the -1 check are produced at nicely-abstracted different places in CG that don't know about each other, so it would take some rework to do it. Maybe even a raised-level CG IR operator "closure_check". Actually, the unaligned checks increase code size as well as execution time for closure checks, which could partially compensate or even overcompensate for the alignment padding. OTOH, probably many programs have no cases that require closure checks at all, so for those, it would be pure size loss for the extra alignment pad bytes. > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayk123 at hotmail.com Thu Jul 6 01:53:10 2017 From: jayk123 at hotmail.com (Jay K) Date: Wed, 5 Jul 2017 23:53:10 +0000 Subject: [M3devel] Fw: [M3commit] [modula3/cm3] 0d5954: reduce-target-variation In-Reply-To: <221824bc-5c00-069a-68e9-eb280fe7d337@lcwb.coop> References: <595b33d19292d_77853fb988581c2819783d@hookshot-fe-6dbb0c4.cp1-iad.github.net.mail> , <221824bc-5c00-069a-68e9-eb280fe7d337@lcwb.coop> Message-ID: I'd like the IR or C to be deterministic and that is under our control. I'd like the assembly / object / executable to be deterministic too but we can't really control that. Determinism allows caching. - Jay ________________________________ From: M3devel on behalf of Rodney M. Bates Sent: Wednesday, July 5, 2017 9:13:53 AM To: m3devel at elegosoft.com Subject: Re: [M3devel] Fw: [M3commit] [modula3/cm3] 0d5954: reduce-target-variation On 07/04/2017 02:08 AM, Jay K wrote: > I'd really like an *option* to make builds far more "deterministic" and "consistent" and "repeatable" from machine to machine, maybe similar-target to similar-target (OpenBSD/x86==Linux/x86==NetBSD/x86==Solaris/x86). > > > That is: I want the default to have full paths all over the place, but an option to reduce paths to something the same across all machines. > > > I want the C backend to by default have a bunch of line numbers to aid debugging, but an option (same option) to omit them. > > > Paths like Compiler.ThisFIle, if they have any slashes, should probably just always be forward. > > This might break some workflows around copy/paste them into file.open dialogs on Windows though. > > Much of Windows supports forward slashes, and much does not. > > > Maybe "-very-deterministic". Yes this is half joke -- deterministic is thought to be a boolean, and everything should be deterministic, but the definition of deterministic is actually unclear. > > > The "target" in reduce-target-variation isn't clearly the right name for this. > > Maybe just reduce-variation or reduce-noise or more-reproducible. > > > This is a step toward a distribution form that isn't target-specific. > > One run of the C backend could generate something redistributable to all or many targets. In this case, we would be taking it on faith that the final C compiler didn't introduce any nondeterminism that we cared about. But perhaps determinism in our own code is all we are concerned with. > > > - Jay > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > *From:* M3commit on behalf of GitHub > *Sent:* Tuesday, July 4, 2017 6:21 AM > *To:* m3commit at elegosoft.com > *Subject:* [M3commit] [modula3/cm3] 0d5954: reduce-target-variation > > Branch: refs/heads/reduce-target-variation > Home: https://github.com/modula3/cm3 > > > GitHub - modula3/cm3: Critical Mass Modula-3 > github.com > cm3 - Critical Mass Modula-3 ... Clone with HTTPS Use Git or checkout with SVN using the web URL. > > > > Commit: 0d59546d11641afe3772b73f226c2e90b960d7fc > https://github.com/modula3/cm3/commit/0d59546d11641afe3772b73f226c2e90b960d7fc > > > reduce-target-variation ? modula3/cm3 at 0d59546 > github.com > cm3 - Critical Mass Modula-3 > > > > Author: jaykrell > Date: 2017-07-03 (Mon, 03 Jul 2017) > > Changed paths: > M m3-sys/cm3/src/Main.m3 > M m3-sys/cm3/src/Makefile.m3 > M m3-sys/cminstall/src/config-no-install/cm3cfg.common > M m3-sys/m3back/src/M3C.m3 > M m3-sys/m3cc/gcc-4.7/gcc/dbxout.c > M m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c > M m3-sys/m3cc/gcc-4.7/gcc/toplev.c > M m3-sys/m3cc/gcc-4.7/gcc/toplev.h > M m3-sys/m3cc/gcc/gcc/m3cg/lang.opt > M m3-sys/m3cc/gcc/gcc/m3cg/parse.c > M m3-sys/m3front/src/misc/Coverage.m3 > M m3-sys/m3front/src/misc/Host.i3 > M m3-sys/m3front/src/misc/Host.m3 > M m3-sys/m3front/src/misc/M3Header.m3 > M m3-sys/m3front/src/misc/Scanner.m3 > M m3-sys/m3front/src/values/Module.m3 > M m3-sys/m3middle/src/Target.i3 > M scripts/python/pylib.py > > Log Message: > ----------- > reduce-target-variation > > > > > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > -- Rodney Bates rodney.m.bates at acm.org _______________________________________________ M3devel mailing list M3devel at elegosoft.com https://m3lists.elegosoft.com/mailman/listinfo/m3devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodney_bates at lcwb.coop Thu Jul 6 17:23:02 2017 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Thu, 6 Jul 2017 10:23:02 -0500 Subject: [M3devel] Fw: [M3commit] [modula3/cm3] 0d5954: reduce-target-variation In-Reply-To: References: <595b33d19292d_77853fb988581c2819783d@hookshot-fe-6dbb0c4.cp1-iad.github.net.mail> <221824bc-5c00-069a-68e9-eb280fe7d337@lcwb.coop> Message-ID: <564b33c9-cd29-9cc4-2e93-a3c9003d022e@lcwb.coop> All agreed. On 07/05/2017 06:53 PM, Jay K wrote: > I'd like the IR or C to be deterministic > and that is under our control. > > I'd like the assembly / object / executable to be deterministic too but we can't really control that. > > Determinism allows caching. > > - Jay > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > *From:* M3devel on behalf of Rodney M. Bates > *Sent:* Wednesday, July 5, 2017 9:13:53 AM > *To:* m3devel at elegosoft.com > *Subject:* Re: [M3devel] Fw: [M3commit] [modula3/cm3] 0d5954: reduce-target-variation > > > > On 07/04/2017 02:08 AM, Jay K wrote: >> I'd really like an *option* to make builds far more "deterministic" and "consistent" and "repeatable" from machine to machine, maybe similar-target to similar-target (OpenBSD/x86==Linux/x86==NetBSD/x86==Solaris/x86). >> >> >> That is: I want the default to have full paths all over the place, but an option to reduce paths to something the same across all machines. >> >> >> I want the C backend to by default have a bunch of line numbers to aid debugging, but an option (same option) to omit them. >> >> >> Paths like Compiler.ThisFIle, if they have any slashes, should probably just always be forward. >> >> This might break some workflows around copy/paste them into file.open dialogs on Windows though. >> >> Much of Windows supports forward slashes, and much does not. >> >> >> Maybe "-very-deterministic". Yes this is half joke -- deterministic is thought to be a boolean, and everything should be deterministic, but the definition of deterministic is actually unclear. >> >> >> The "target" in reduce-target-variation isn't clearly the right name for this. >> >> Maybe just reduce-variation or reduce-noise or more-reproducible. >> >> >> This is a step toward a distribution form that isn't target-specific. >> >> One run of the C backend could generate something redistributable to all or many targets. > > In this case, we would be taking it on faith that the final C compiler didn't introduce any > nondeterminism that we cared about. But perhaps determinism in our own code is all we are > concerned with. > >> >> >> - Jay >> >> >> >> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ >> *From:* M3commit on behalf of GitHub >> *Sent:* Tuesday, July 4, 2017 6:21 AM >> *To:* m3commit at elegosoft.com >> *Subject:* [M3commit] [modula3/cm3] 0d5954: reduce-target-variation >> >> Branch: refs/heads/reduce-target-variation >> Home: https://github.com/modula3/cm3 >> >> >> GitHub - modula3/cm3: Critical Mass Modula-3 >> github.com >> cm3 - Critical Mass Modula-3 ... Clone with HTTPS Use Git or checkout with SVN using the web URL. >> >> >> >> Commit: 0d59546d11641afe3772b73f226c2e90b960d7fc >> https://github.com/modula3/cm3/commit/0d59546d11641afe3772b73f226c2e90b960d7fc >> >> >> reduce-target-variation ? modula3/cm3 at 0d59546 >> github.com >> cm3 - Critical Mass Modula-3 >> >> >> >> Author: jaykrell >> Date: 2017-07-03 (Mon, 03 Jul 2017) >> >> Changed paths: >> M m3-sys/cm3/src/Main.m3 >> M m3-sys/cm3/src/Makefile.m3 >> M m3-sys/cminstall/src/config-no-install/cm3cfg.common >> M m3-sys/m3back/src/M3C.m3 >> M m3-sys/m3cc/gcc-4.7/gcc/dbxout.c >> M m3-sys/m3cc/gcc-4.7/gcc/dwarf2out.c >> M m3-sys/m3cc/gcc-4.7/gcc/toplev.c >> M m3-sys/m3cc/gcc-4.7/gcc/toplev.h >> M m3-sys/m3cc/gcc/gcc/m3cg/lang.opt >> M m3-sys/m3cc/gcc/gcc/m3cg/parse.c >> M m3-sys/m3front/src/misc/Coverage.m3 >> M m3-sys/m3front/src/misc/Host.i3 >> M m3-sys/m3front/src/misc/Host.m3 >> M m3-sys/m3front/src/misc/M3Header.m3 >> M m3-sys/m3front/src/misc/Scanner.m3 >> M m3-sys/m3front/src/values/Module.m3 >> M m3-sys/m3middle/src/Target.i3 >> M scripts/python/pylib.py >> >> Log Message: >> ----------- >> reduce-target-variation >> >> >> >> >> _______________________________________________ >> M3devel mailing list >> M3devel at elegosoft.com >> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > > -- > Rodney Bates > rodney.m.bates at acm.org > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Thu Jul 6 17:58:13 2017 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Thu, 6 Jul 2017 10:58:13 -0500 Subject: [M3devel] Aligned_procedures? In-Reply-To: References: <599509ee-248d-6ab1-e7b7-e56e9da2932a@lcwb.coop> Message-ID: <8a9a1469-ec09-6e90-5fe6-62545787e60c@lcwb.coop> On 07/05/2017 06:42 PM, Jay K wrote: > > > - Jay > _____________________________ > From: Rodney M. Bates > > Sent: Wednesday, July 5, 2017 12:24 PM > Subject: Re: [M3devel] Aligned_procedures? > To: > > > 1. Because we are not allowed to store function pointers, lest they are closures, and become stale? > Yes. A pointed-to nested procedure can return and its activation record disappear, while the pointer to the AR dangles. This is a fundamental language semantics problem and is addressed in varying ways. The Modula-3 rule is not the weakest possible rule to prevent this, but it does allow both stored pointers to top-level procedures and formal parameters that identify nested procedures, which are probably covers most of the real use cases. > > 2. I might have made a mistake: 32 bit targets mights also be true. > > 3. The rationale is a little confusing because there are multiple factors. > > The factors are the alignment of normal functions and the alignment requirements of the architecture to read an INTEGER (which I suggest should be instead a 4 byte integer at least on most architectures, in this context). > > The generated code reads an integer from a function pointer, compares to -1. It assumes -1 can't be a valid instruction, at least as the first in a function. This is of dubious portability both because -1 is not well known to me as invalid code, and because not all systems allow reading code bytes. > > If -1, it is assumed to be a closure and reads the function pointer and static link from the subsequent words. > > The "problem" is that, closures are guaranteed to be at least word-aligned, and the read to check for -1 guaranteed not to trigger an alignment fault. But, on some systems, other function pointers have no such alignment guarantee. > > So an alignment check is optionally inserted to avoid the alignment fault. > > We could also unconditionally insert the alignment check. It is never wrong. It is code bloat if not needed but arguably it is a nice optimization. > In the night, I realized my proposal (always integer-align procedure entry code) has a serious flaw. We can control procedure alignment only for procedures compiled by our back ends. Code written in C or produced by the C backend goes through a stock C compiler, and llvm IR goes through stock llvm, all of which we can't control. The only reason I can think of for the, to my knowledge, unique, system we have, where a function pointer can, dynamically, be either a closure or direct code pointer is that it works for functions written and compiled in C. So for this to work, on x86/amd64, we have to keep the alignment check as first part of the closure check. (Or do we? see below.) We can still omit the alignment check on machines that require procedures to be integer aligned anyway, (or 4-byte aligned) but that is target variation, which you are trying to minimize. It would not be nondeterminism build-to-build, though. > We could also leave the choice to the backend. > Yes. This would require changing CG to not lower the code so far and create a closure_check CG IR operator. > x86/amd64 have no alignment requirement for integers or instructions or functions. So the check is not needed. > Really? not even for integers? I presume misaligned integer access would be slower, though, possibly a lot. If it's not much slower than explicitly coded alignment check, we could omit the check and just let the misaligned access happen. Presumably, when the pointer does turn out to be integer-aligned, there would be no time penalty at all. > PowerPC, MIPS, Alpha, Sparc, arm64 I believe all have fixed size 4 byte 4-aligned instructions. Reading a 4 byte integer should be ok, unconditionally through a function pointer, but not an 8 byte integer. > > Arm32 is wierd. I believe instructions are either 2 or 4 bytes, and aligned to only 2?? The low bit indicates the size: 0 for 4, 1 for 2. The alignment check is needed, or clear > the low 2 bits and read. > > Clear the low bits and read is also a portable approach. > > IA64 bundles up to 3 instructions in 128 bits with..41 bits per instruction and 5 bit template. I don't know their alignment. > > I haven't been able to think of another solution, that doesn't use runtime codegen..until recently, but the other solution I know of..generates closures slowly and with OS and processor porting work. > > - Jay > > On 07/04/2017 02:52 AM, Jay K wrote: >> Aligned_procedures >> >> >> I'm sure I've mentioned this before...but I'm clearing out my backlog of lingering diffs. >> >> >> In my bid to make more of the targets look more the same, >> I suggest making Aligned_procedures always be false. >> >> >> This slightly pessimises mainstream targets: x86 and amd64. >> >> >> I believe it slightly bloats all calls through function pointers. >> (including object methods? Maybe, but I don't think those can be closures, >> so that could/should be fixed -- though the idea of a method being a closure >> is a good one...) >> > > Only calls through a formal parameter of procedure type (not a variable, field, etc.) > and assignments other than passing things to a VALUE or READONLY formal need to do a > closure-check. Other cases just use/copy the pointer value. > >> >> It has no affect on PowerPC, ARM, SPARC, MIPS, Alpha, etc. -- 32bit or 64bit. >> > > Is this because these targets require all procedures to be have the same alignment as > integer anyway? So code is always as if Aligned_procedures were true, i.e., no > alignment check is ever necessary? > >> >> I believe the difference is that when calling a function pointer, on x86/amd64, >> we just read it for a pointer-size integer, and compare to -1. >> >> >> If Aligned_procedures is left as always false, that check would first >> see if the pointer is aligned on a pointer-size, and if not, skip the check for -1. >> >> >> This is because most architectures will issue an alignment fault for the >> unaligned read, and we know such unaligned values are not closures. >> x86/amd64 do not care much about alignment. >> >> >> I have proposed, somewhat the opposite, that this check actually be always be 4 bytes, >> not a full pointer. That would likely allow it to always be TRUE. Closures would still >> be pointer-aligned, but we'd only check for 4 bytes -1 instead of a full pointer. >> >> The idea is that all functions are 4-aligned on all targets that care about integer alignment. >> Even if they aren't 8-aligned on 64bit targets. >> > > So no alignment check is ever required. We still have to pad function starts > to 4-bytes. I would call this Aligned_procedures=true on32-bit targets and 64-bit > targets that do not otherwise require 8-byte alignment of functions, and somewhere > partway between false and true for 64-bit targets that do not otherwise require > 8-byte alignment of functions, since functions are only partially aligned, and still > no alignment check is required. > > We did once have the discussion whether there exists or could someday exist, a target > where 4 bytes or 8 bytes of all one-bits would be valid machine code at the > start of a function, or anywhere at all. The only conclusion I recall is that it > is unlikely. But this scheme would be slightly weaker in this regard in that it > would take a mere 4 bytes of -1 as valid code, to be mistaken for a closure. > >> >> I believe that would not work for ARM32-Thumb and I can't bring myself to rule >> out such targets. >> > > What are the relevant properties of ARM32-Thumb? > >> >> Another option would be to make this only be for the C backend. >> >> It isn't clearly useful given the gcc backend -- unless maybe redistributing >> same IR across multiple targets. >> >> - Jay >> >> > > I like the idea of just unconditionally integer-aligning all procedures on all > targets. No runtime alignment check would ever be necessary, reducing the time > bloat, at the cost of extra code size bloat on those targets where aligning every > procedure would not otherwise be required. I like that size/time tradeoff better. > > The code sequence for closure checks looks pretty gross right now. It is > poorly optimized. I have looked at improving it, but some combination of the > alignment check, the nil check, and the -1 check are produced at nicely-abstracted > different places in CG that don't know about each other, so it would take some rework > to do it. Maybe even a raised-level CG IR operator "closure_check". > > Actually, the unaligned checks increase code size as well as execution time for > closure checks, which could partially compensate or even overcompensate for the > alignment padding. OTOH, probably many programs have no cases that require closure > checks at all, so for those, it would be pure size loss for the extra alignment pad > bytes. > > >> >> _______________________________________________ >> M3devel mailing list >> M3devel at elegosoft.com >> https://m3lists.elegosoft.com/mailman/listinfo/m3devel >> > > -- > Rodney Bates > rodney.m.bates at acm.org > _______________________________________________ > M3devel mailing list > M3devel at elegosoft.com > https://m3lists.elegosoft.com/mailman/listinfo/m3devel > > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Wed Jul 19 03:02:36 2017 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Tue, 18 Jul 2017 20:02:36 -0500 Subject: [M3devel] M3CG_LLVM.m3, commit 7eb3628 Message-ID: <7c12e044-5c17-68d2-dcd3-ddc0219f944d@lcwb.coop> Peter, I don't understand the rationale for the change in commit 7eb3628. For llvm, the front end is set to pull nested procedures out and place them completely after their containing static ancestor. So all uses of a display index will occur after the end of the procedure declaring the uplevel-referenced variable, and the computation of cumUplevelRefdCt, at the end of the outer procedure will be soon enough. If it were not so, even this change would not be early enough, because, with Modula-3 having no declare-before-use rule, an uplevel reference can be to a variable declared after the referencing nested procedure. Have I missed something? -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Thu Jul 20 17:07:09 2017 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Thu, 20 Jul 2017 10:07:09 -0500 Subject: [M3devel] M3CG_LLVM.m3, commit 7eb3628 In-Reply-To: References: <7c12e044-5c17-68d2-dcd3-ddc0219f944d@lcwb.coop> Message-ID: On 07/19/2017 10:39 PM, Peter McKinna wrote: > Hi Rodney, > > It's been a while since I did that change and had to refresh my memory. I was getting segv's on some testcases. I'm not entirely sure that change is the correct way to do things and I dont quite understand the theory. Best if I show a small sample of code. > > Take this snippet > -- snip --- >> > END TestNested; > > the llvm code produced is (I have pointed out a couple of problems) Is this llvm code before or after your change? > > (excuse the wordiness - we should do something about those long names sometime) > > > I like long names, consistent names, and lots of explicitness in general. I end up doing a lot of maintenance work on large code bodies, and usually wish for even more of this kind of stuff than is there. I often put in a things that would have saved me a lot of time tracking down info that is not locally obvious. > -- snip --- > So I think GetAdrofUpLevelVar is getting a value from cumUplevelRefdCt which is always zero since it does not get > set until end_procedure and then subtracting the stack size leads to negative values. > My theory is that GetAdrOfUpLevelVar should not be called until after end_procedure. I just did a fairly thorough code review, and it looks like this should be true. > I'm not sure if I have fixed the problem completely but my testcases seem to work. Feel free to change/fix whatever. I certainly think the nested proc side of things was the most complex part of the whole exercise. > I regard this display idea, which is not quite like the usual meaning of the term as I understand it, as a messy hack, but it seemed the easiest way at the time. It does not support debugging cleanly at all, since a debugger user can do uplevel references to variables that are not uplevel referenced in the code. I'm not sure there is an alternate way for a debugger at all. I have two plans for a rework some day. One is to package the entire activation record in an explicit llvm struct and use displacements within for all addressing, compiled and debugger-initiated, uplevel or not. Then the "static link" would be a real static link as usually meant by the term. This would also fit reasonably well with using CG's already done allocation, instead of letting llvm do it all over again with llvm alloca instructions. An llvm developer on the llvm mailing list once recommended rather emphatically to me to *not* llvm do the layout. The other way is, sometime more recently, a newer version of llvm added an intrinsic or something that looked like it would allow to get back the displacements that llvm created for alloca's, so we could compute differences and use them in debug info. This would involve going to a newer llvm and thus redoing the DIBuilder binding, re. which, see below ... > Are you getting back into generating dwarf debug? I wrote a swig module that simplifies generating the di bindings which seemed to work ok. I might put them into git sometime. > I am trying to get back to it now. Creating the DIBuilder binding I did was by far the absolutely most painful programming task I have ever done, so I am not eager to repeat it. Automated binding generation would be a huge relief. Please make whatever you have done in this area available. For what version of llvm are the bindings you did? > On a separate note, you wouldn't happen to know how to get the gcc backend to generate LTO. It says it's not configured for it but I can't seem to build it with it LTO support. I am trying to build an interface to the AVX builtin functions but I would prefer if they were inlined into M3 code rather than the overhead of a call to a C wrapper. > I know very little about this, but I know Jay removed large amounts of unused stuff from the M3 gcc backend, so maybe the needed code is not there. > Regards Peter > > > On Wed, Jul 19, 2017 at 11:02 AM, Rodney M. Bates > wrote: > > Peter, I don't understand the rationale for the change in commit 7eb3628. For llvm, the front end > is set to pull nested procedures out and place them completely after their containing static ancestor. > So all uses of a display index will occur after the end of the procedure declaring the > uplevel-referenced variable, and the computation of cumUplevelRefdCt, at the end of the > outer procedure will be soon enough. > > If it were not so, even this change would not be early enough, because, with Modula-3 having no > declare-before-use rule, an uplevel reference can be to a variable declared after the referencing > nested procedure. > > Have I missed something? > -- > Rodney Bates > rodney.m.bates at acm.org > > -- Rodney Bates rodney.m.bates at acm.org From rodney_bates at lcwb.coop Thu Jul 20 23:02:44 2017 From: rodney_bates at lcwb.coop (Rodney M. Bates) Date: Thu, 20 Jul 2017 16:02:44 -0500 Subject: [M3devel] M3CG_LLVM.m3, commit 7eb3628 In-Reply-To: References: <7c12e044-5c17-68d2-dcd3-ddc0219f944d@lcwb.coop> Message-ID: <113c7c19-1377-5007-d060-b060dfb9be0c@lcwb.coop> On 07/20/2017 10:07 AM, Rodney M. Bates wrote: > > > On 07/19/2017 10:39 PM, Peter McKinna wrote: >> Hi Rodney, >> >>> >> (excuse the wordiness - we should do something about those long names sometime) >> >> > > > I like long names, consistent names, and lots of explicitness in general. I end up > doing a lot of maintenance work on large code bodies, and usually wish for even > more of this kind of stuff than is there. I often put in a things that would > have saved me a lot of time tracking down info that is not locally obvious. > OK, after spending time reading this llvm code, I agree, the names are a mess. Not only too long, but they don't follow a consistent pattern, don't contain some useful info and contain too much that is locally redundant. I'll work on that in parallel with fixing the display offsets, > > -- Rodney Bates rodney.m.bates at acm.org