[M3devel] m3cgc1: fatal error: *** bad M3CG version stamp (0x100), expected 0x110

[Rodney M. Bates]

On 05/29/2015 05:31 PM, Jay K wrote:
>   cm3/cm3cg are more special than the others, because cm3 is statically linked.
>   There is no m3core.dll, that cm3 uses, or m3quake.dll, etc.
>   There is m3core.dll, but cm3 doesn't uses it.
>   Shipping it doesn't affect the in-use cm3.
>
>
>   If/when we fix this, in fact, we can stop doing that static linking.
>   It makes sense for m3quake, etc. as there are few/no other users.
>   But for libm3/m3core, it'll provide a nice size optimization.
>
>
>   The mutual incompatibilty between old/new cm3/cm3cg is perfectly ok.
>   It is meant to be dealt with, not avoided.
>
>
>   I believe upgrade.py did deal with this correctly, but this environment
>   variable required to quash shipping of cm3cg broke it.
>
>
>   The environment variable change really should imho be removed from cm3cg.
>   Please.
>

Jay, I am not sure what you are proposing.  When I first looked at the commit for
this change, I was quite confused by my own change.  While the comments say it
is to _not_ prematurely ship cm3cg, the code appears to add code to conditionally
ship if INSTALL_CM3_IN_BIN=yes.  After the n-th look, I realized that before
the change, it _always_ shipped, regardless of the variable.  So it really does
provide for less premature shipping.

If I just revert it, do-cm3-front.sh will be unconditionally and unusably broken.
As is, it is at least usable if you don't set the variable.  Even if there is
some use case where one would want it to ship, (none occurs to me), as is, you
can get this behavior by just setting the variable.

Moreover, the current behavior for cm3cg is consistent with the current and
long-standing behavior for cm3, i.e., ship iff the variable is set.

It would make more sense to me to remove the variable check by unconditionally
_not_ shipping.  At least do-cm3-front.sh would be usable.  But then, what
would be the purpose of the inconsistent use of the variable for cm3?



>
>   And then we can switch to "destdir" and remove all the in place updating, perhaps.
>
>
>   Or I need to update upgrade.py to ship cm3cg itself.
>   I guess I'll do that.
>
>
> upgrade.py may not have worked for you, but I have used it many many times.
> It worked. But currently it is probably broken by the m3cc/src/m3makefile change.
> I'll update it to accommodate that change..
>
>
> And upgrade.sh I believe was in heavy use by Hudson.
>
>
>   - Jay
>
>
>
>  > Date: Fri, 29 May 2015 16:10:41 -0500
>  > From: rodney_bates at lcwb.coop
>  > CC: m3devel at elegosoft.com
>  > To:
>  > Subject: Re: [M3devel] m3cgc1: fatal error: *** bad M3CG version stamp (0x100), expected 0x110
>  >
>  > Yes, we could have a better organization. I agree, the principal way
>  > should be to build a complete set of everything that matters to the
>  > implementation of the language, them replace them all at once. A
>  > separate DESTDIR or such is needed.
>  >
>  > The set, as Jay points out, is larger than just cm3 and m3cc. It includes
>  > at least m3core, probably libm3, and cm3, the executable depends on several
>  > separately built packages--cm3, m3middle, m3front, m3quake, m3makefile, etc.
>  >
>  > In general, we can't really assume anything about the kinds of dependencies
>  > of one package on old or new versions of another. Right now, cm3 and m3cc
>  > are known mutually incompatible between 5.8 and the head, because the numbering
>  > in M3CG_Binary.i3 has changed. We could probably establish some coding rules and
>  > order-of-install rules that would allow a newer m3cc to work with an older cm3,
>  > but that's only one pattern.
>  >
>  > I once thought do-cm3-*.sh build would do what we are talking about, but it
>  > if I remember right, puts every package into immediate use as it is built
>  > and also uses overrides, which keeps things inside the source tree, but makes
>  > it unshippable. So I always use buildship. This has never, to my knowledge
>  > run afoul of any compatibility problem other than between cm3 and m3cc, but it
>  > does ship every package besides these two as it is built, so the potential
>  > is there.
>  >
>  > I haven't had the energy to even try to understand just what the upgrade
>  > scripts do, but they have not treated me well. If I had done different
>  > patterns of incompatible changes, they would probably have been better
>  > than the procedure I use.
>  >
>  > On 05/29/2015 01:06 PM, Jay K wrote:
>  > > > I don't see an easy way to make the variable work, as there is no
>  > > > constraint on the order that cm3 and cm3cg are built. do-cm3-front.sh
>  > > > happens to do cm3cg first, but that is not necessarily required.
>  > >
>  > > [please pardon my editing...]
>  > >
>  > >
>  > > I see what you mean that cm3cg is an independent node in the graph and can be built in any order,
>  > > but, I guess, the story is/was, you have to know the safe order to ship things.
>  > >
>  > >
>  > > It is encoded in pkginfo.txt and the Python scripts honor it..though they don't always use pkginfo.txt for their ordering..
>  > > And encoding the dependency graph there isn't ideal. It is really encoded in the import declarations
>  > > in the m3makefiles, usually, but not in this case...
>  > >
>  > >
>  > > The way that works is to build but don't ship cm3cg.
>  > > Then ship cm3cg.
>  > > Then copy cm3.
>  > > Then use the new cm3.
>  > > But perhaps it isn't strictly enforced enough.
>  > >
>  > >
>  > >
>  > > I used do this fairly often.
>  > > Though breaking changes in cm3cg were rare, granted.
>  > > But I'd also occasionally bootstrap from an older release.
>  > > I'll have to try to do that again soon -- actually make the current
>  > > tree easily buildable from the last release.
>  > >
>  > >
>  > > update.py handled it.
>  > >
>  > >
>  > > I'm on the fence.
>  > > "buildship" of multiple packages is in general problematic.
>  > > I kinda think the whole mechanism needs to be revampled.
>  > >
>  > >
>  > > Let me draw a close analogy.
>  > >
>  > >
>  > > In the slightly larger software world we have:
>  > >
>  > > make
>  > > make install
>  > > make install DESTDIR=somewhere_new
>  > >
>  > >
>  > > buildship is equivalent make && make install, or a make install that includes the make part,
>  > > as they sometimes do.
>  > > Read up on building glibc and they point out how dangerous this is -- how dangerous make install is w/o DESTDIR.
>  > > We have the same danger on a smaller scale.
>  > > You aren't going to break the "OS", but you can break "cm3".
>  > >
>  > >
>  > > The safe recommendation is make install DESTDIR.
>  > > Basically "install" "everything" "on the side", get it all installed, multiple
>  > > "packages", then copy or tar/untar it all. Or chroot into it and test it out, and
>  > > then do an in-place update. Or swap into the chroot and never return? (tail call install)
>  > >
>  > >
>  > > Don't do a "rolling upgrade". Don't install each package as it is built. No package, no cm3,
>  > > not cm3cg, nothing.
>  > >
>  > >
>  > > I'm basically suggesting that.
>  > >
>  > >
>  > > Now, if you say "don't ever ship, but somehow otherwise manually copy", well
>  > > that does kind of resolve to the same thing.
>  > >
>  > >
>  > > If you say, "almost don't ever ship, but provide a way to", that is almost the same.
>  > >
>  > >
>  > > make-dist.py follows the make install DESTDIR way.
>  > >
>  > > It is "better" I guess because I wrote it later, and because I was following
>  > > the example of the other .sh files that make distributions. So blame/credit Olaf? :)
>  > >
>  > >
>  > > It starts with a working compiler, copies parts into /temp, builds a new compiler into /tmp, and
>  > > uses what it builds to build everything.
>  > >
>  > >
>  > > Again I suggest it be looked into, and maybe be promoted as basically
>  > > the replacement for upgrade.py and upgrade.sh.
>  > >
>  > >
>  > > upgrade.py could wrap it and do the final steps of copy/untar in-place.
>  > > Or make-dist.py could just print the last optional command.
>  > >
>  > >
>  > > Anyway, I have no time lately, just email. :(
>  > >
>  > >
>  > > - Jay
>  > >
>  > >
>  > >
>  > > > Date: Fri, 29 May 2015 11:35:41 -0500
>  > > > From: rodney_bates at lcwb.coop
>  > > > To: adacore at marino.st; wagner at elego.de
>  > > > CC: m3devel at elegosoft.com
>  > > > Subject: Re: [M3devel] m3cgc1: fatal error: *** bad M3CG version stamp (0x100), expected 0x110
>  > > >
>  > > > Setting INSTALL_CM3_IN_BIN=yes is what forces the premature install of cm3cg.
>  > > > I see I had encountered this problem two years ago, but then it was happening
>  > > > regardless of the value of this variable. My fix only addresses the case when
>  > > > the variable is not set. I naively went to extra trouble to preserve the
>  > > > existing behavior when it is set--obviously not right.
>  > > >
>  > > > I don't see an easy way to make the variable work, as there is no
>  > > > constraint on the order that cm3 and cm3cg are built. do-cm3-front.sh
>  > > > happens to do cm3cg first, but that is not necessarily required.
>  > > >
>  > > > So INSTALL_CM3_IN_BIN is broken and hard to fix, if the two need
>  > > > compatible upgrades. For now, it looks like install-cm3-compiler.sh
>  > > > is the only working way.
>  > > >
>  > > > But maybe John and Olaf have found a different way around this.
>  > > >
>  > > > On 05/29/2015 10:55 AM, Rodney M. Bates wrote:
>  > > > > I don't know yet why, or what needs to change, but here is definitely the problem, starting
>  > > > > in line 4575 of your m3c log:
>  > > > >
>  > > > >
>  > > > > g++48 -O2 -pipe -fstack-protector -Wl,-rpath=/usr/local/lib/gcc48 -fno-strict-aliasing -Wl,-rpath=/usr/local/lib/gcc48 -DIN_GCC -Wno-missing-field-initializers -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -DHAVE_CONFIG_H -o m3cgc1 m3cg/parse.o attribs.o main.o tree-browser.o libbackend.a libcommon-target.a libcommon.a ../libcpp/libcpp.a libcommon.a ../libcpp/libcpp.a ../libiberty/libiberty.a
>  > > > > gmake[1]: Leaving directory `/mech/construction/mech/ptrees/default/lang/modula3/work/cm3-8c1b86a/m3-sys/m3cc/AMD64_FREEBSD/gcc'
>  > > > >
>  > > > >
>  > > > > This is copying the newly-built cm3cg (the compiler back end) to your
>  > > > > bin directory:
>  > > > >
>  > > > > Forced ship of cm3cg.
>  > > > > --- shipping from AMD64_FREEBSD ---
>  > > > >
>  > > > > . => /mech/construction/mech/ptrees/default/lang/modula3/work/bootstrap/bin
>  > > > > cm3cg
>  > > > > ==> m3-sys/m3cc done
>  > > > >
>  > > > > === package m3-libs/m3core ===
>  > > > >
>  > > > > And this is using the cm3 (compiler front end) that was already in the bin
>  > > > > directory, and is the release version.
>  > > > >
>  > > > > +++ /mech/construction/mech/ptrees/default/lang/modula3/work/bootstrap/bin/cm3 -build -DROOT='/mech/construction/mech/ptrees/default/lang/modula3/work/cm3-8c1b86a' $RARGS && /mech/construction/mech/ptrees/default/lang/modula3/work/bootstrap/bin/cm3 -ship $RARGS -DROOT='/mech/construction/mech/ptrees/default/lang/modula3/work/cm3-8c1b86a' +++
>  > > > > --- building in AMD64_FREEBSD ---
>  > > > >
>  > > > > ignoring ../src/m3overrides
>  > > > >
>  > > > > new source -> compiling RTHooks.i3
>  > > > >
>  > > > > This is cm3cg (aka m3cgc1) detecting the incompatibility:
>  > > > >
>  > > > > m3cgc1: fatal error: *** bad M3CG version stamp (0x100), expected 0x110
>  > > > > compilation terminated.
>  > > > >
>  > > > > Here is the counterpart section of my log:
>  > > > >
>  > > > > g++ -g -O2 -DIN_GCC -Wno-missing-field-initializers -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -DHAVE_CONFIG_H -o m3cgc1 m3cg/parse.o attribs.o main.o tree-browser.o libbackend.a libcommon-target.a libcommon.a ../libcpp/libcpp.a libcommon.a ../libcpp/libcpp.a ../libiberty/libiberty.a
>  > > > >
>  > > > > On my build, it's not shipping:
>  > > > >
>  > > > > Not shipping cm3cg.
>  > > > > --- shipping from AMD64_LINUX ---
>  > > > >
>  > > > > missing ".M3SHIP" file, build the package first.
>  > > > > ==> /home/rodney/proj/m3/git/cm3/m3-sys/m3cc done
>  > > > >
>  > > > > === package /home/rodney/proj/m3/git/cm3/m3-libs/m3core ===
>  > > > > +++ cm3 -build -DROOT='/home/rodney/proj/m3/git/cm3' $RARGS && cm3 -ship $RARGS -DROOT='/home/rodney/proj/m3/git/cm3' +++
>  > > > > --- building in AMD64_LINUX ---
>  > > > >
>  > > > > ignoring ../src/m3overrides
>  > > > >
>  > > > > new source -> compiling RTHooks.i3
>  > > > > new source -> compiling RT0.i3
>  > > > >
>  > > > > When it works, the copying of both cm3 and cm3cg to the bin directory is delayed until after both have been
>  > > > > built, in the 'install-cm3-compiler.sh upgrade' step.
>  > > > >
>  > > > > I see your build is using a full path to the bin directory and mine is letting
>  > > > > it come from the usual $PATH search, but that should make no difference, as long
>  > > > > as the bin directory is found in a consistent way, which seems to be the case
>  > > > > in both builds.
>  > > > >
>  > > > >
>  > > > >
>  > > > > On 05/29/2015 04:51 AM, John Marino wrote:
>  > > > >> On 5/29/2015 11:39, Olaf Wagner wrote:
>  > > > >>> On Fri, 29 May 2015 11:10:50 +0200
>  > > > >>> John Marino <adacore at marino.st> wrote:
>  > > > >>>
>  > > > >>>> On 5/29/2015 10:54, Olaf Wagner wrote:
>  > > > >>>>> Please try what I suggested in the attached mail: define
>  > > > >>>>> INSTALL_CM3_IN_BIN=yes
>  > > > >>>>> in the environment and see if that helps.
>  > > > >>>>>
>  > > > >>>>
>  > > > >>>> Unfortunately it seems the result is exactly the same, even when
>  > > > >>>> INSTALL_CM3_IN_BIN is defined in the environment.
>  > > > >>>
>  > > > >>> Can I see the new logs? Has the new backend (cm3cg) been installed now?
>  > > > >>>
>  > > > >>
>  > > > >> It's here:
>  > > > >> http://leaf.dragonflybsd.org/~marino/m3c.log
>  > > > >>
>  > > > >> It never completes "do-cm3-front.sh buildship "
>  > > > >>
>  > > > >> Note that I'm using a bootstrap install and set INSTALL_ROOT to another
>  > > > >> location, but it never gets to the point of installing anything.
>  > > > >>
>  > > > >> John
>  > > > >>
>  > > > >
>  > > >
>  > > > --
>  > > > Rodney Bates
>  > > > rodney.m.bates at acm.org
>  >
>  > --
>  > Rodney Bates
>  > rodney.m.bates at acm.org

-- 
Rodney Bates
rodney.m.bates at acm.org