[M3devel] ship vs. where build outputs go, etc.?

Tue Jan 20 19:50:40 CET 2009

The point is it would not always be the "public" repositoy.

Today we have two places, the source tree and the repository.
They have different structures.

Instead, you could just have multiple repositories.
They would all have the same structure.

"Install" or "ship" is just recursive copy from a "private"
repository to the "public" repository, instead of
arbitrary rearrangement of files.

OR merely "blessing" a private repository and making it the new
public repository, such as by setting and an environment variable
or possibly delete/rename.

You could get something closer to the current experience by having
two environment variables, CM3_PUBLIC, CM3_PRIVATE.
cm3 -buildship would output to CM3_PUBLIC.
cm3 -build would output to CM3_PRIVATE.
Ideally it would do no extra writes vs. the current scheme.

"Upgrade" would look like:

 set CM3_PRIVATE= some new temporary place 
 build everything 
 if successful
   rm -rf CM3_PUBLIC
   mv CM3_PUBLIC CM3_PRIVATE 

 See, today "upgrade" writes over the live system and is therefore
  "dangerous". In this new scheme, "ship" would lose much of its
  danger. Rather than making changes as it goes, the final "commit"
  would not occur until the end, and would be easier to keep backups
  and such. Granted, if you look at what make-dist does, this is
  already possible today, by twiddling with CM3_INSTALL and always
  shipping. Though that costs extra I/O and you do have to prepopulate
  CM3_INSTALL..at least with a compiler, eh, just two files.
  Also with m3core/libm3 in some bootstrapping scenarios.
   (make-dist assumes so, though it usually is not the case).

 Along with an alternate root for intermediate outputs, this would
  enable multiple concurrent builds as well.

 A similar proposal is that "CM3_PUBLIC" could be colon or semicolon
 delimited -- a "search path" for reads, write to the first location.
 This would probably be confusing if it contained more than two paths though.
 publlic/private is "limited" in that it only supports two, but you can
  always use recursive copy to merge.

 Fixing clean would not take away realclean as an option if that is necessary
 for compat (I forget if compiler even implements realclean; the scripts
 implement it and I always use that version). I find it hard to believe
 that anyone depends on that clean leaves a few files around.
 Which files does it even leave around?

 I'm not sure what is documented or not. Could be that a lot is? 

 - Jay

________________________________
> Date: Tue, 20 Jan 2009 12:07:01 -0500
> From: rcoleburn at scires.com
> To: m3devel at elegosoft.com
> Subject: Re: [M3devel] ship vs. where build outputs go, etc.?
>
>
>
>
>
>
>
> Jay:
>
>
>
> I think your email illustrates part of what I was trying to say in that there are a lot of things that aren't documented very well at this point that differ from the current documentation and practice that we've followed over the years.
>
>
>
> I'm not sure I agree with some of the proposals you offer. To be fair though, I think we should single out one proposal at a time for discussion.
>
>
>
> As for m3overrides and private vs. public repositories, I think you are going to get a lot of push-back on the idea to always ship everything to the public repository. I for one, don't like this idea.
>
>
>
> I do agree that the compiler's idea of certain options, e.g., clean, may not match what everyone expects. Of course, many of us have written scripts, etc. that invoke the compiler with certain options and that do other "stuff" using some of the same terminology (e.g. clean, ship, build, buildship, realclean, zap, etc.). I am arguing that the terminology should be standardized to have a consistent meaning across all of these. Depending on the definitions, some code will need to change make the implementation match the definition.
>
>
>
> Regards,
>
> Randy
>
>>>> Jay 1/20/2009 1:03 AM>>>
>
>> [was truncated and forward to m3commit instead of m3devel, now more content/drivel..]
>
> ps:
>
>
>>> I also want "us" to give explicit,
>>> documented, definitions of the terms
>>> "build", "ship", "clean", "spotless",
>>> etc. etc. that are used in various scripts et al and make these consistent.
>
>
>
> "Clean" doesn't really work imho..I guess it deletes the output files
> that the compiler knows about? Perhaps it is a bug when the
> config file doesn't tell it about all the outputs?
>
>
> "realclean" deletes the entire output directory, no matter what files it thinks it produced.
> It does assume they are all in the particular directory that they strongly tend to be in.
>
>
> It's dumb imho.
> Clean should probably be fixed.
>
>
> The m3-sys/m3cc directory and probably m3gdb also have their own special
> in-between state of "configure having been one", that they create a marker
> file to indicate. And then there is an environment variable to override that.
> Quake is a fairly general purpose language including the ability to read
> environment variables, so various directories are free to do whatever they want.
>
>
> m3cc and m3gdb are two of the longest to build package and configuration is also a slow step, that "rarely" has different output, so going back only to this "partly built" state perhaps is interesting. I rarely do that, at least on purpose.
>
>
> The Linux kernel and ucLibc (an alternative C runtime) do something similar.
> They store their configuration in file starting with a dot, so that on Unix rm -rf * won't delete it, because it skips "hidden" "dot" files.
>
>
>
> As well you can define arbitrary things on the cm3 command line, to guide
> arbitrary decisions around the tree, such as kernel vs. user threads.
>
>
> Given
> foo\src
> build outputs are generally in
> foo\target
>
>
> where "outputs" include "intermediate" files such as object files and "final" outputs such as executables and .dlls.
>
>
> You know...really I think, it's just that the Modula-3 folks knew/thought they were smarter than everyone else and made up their own terminology.
>
>
> In many other projects you have "make" and "make install".
> "ship" is "make install". I think that is all people need to know: "ship" == "install".
>
>
> It copies some of the outputs from foo\target to the installroot.
> Just as "make install" copies some of the outputs from the "intermediate" tree to the "live" tree. They can both be "dangerous" on a live system, and they can both be pointed at "new" empty location for safety (typical GNU make/automake/autoconf install option is make install DESTDIR=somewhere).
>
>
> I propose a different model.
> Always put the "final" outputs directly in a "live" tree, BUT for the case where you are doing something "risky" and would not immediately "ship" / "install", you would be encouraged to define a new clean output root, or an existing "test" root.
>
>
> "Actual install" would then be just a recursive copy from one root to another.
> Perhaps that is too inefficient? cm3 tries to do an incremental copy for example.
> I'm not sure if it is timestamp based or if they compare entire file contents.
>
>
> The downside to this proposal is that source files also get shipped, and doing that for every compile/link is wasteful. This is optional though and maybe can/should be turned off except when "making a distribution"? I'm not sure how optional it is. I'll try out turning it off locally.
>
>
> An upside to this proposal is it supercedes the "override" mechanism.
> You would never need to "override".
> "override" imho is broken because today, in order to keep "override" working, you are expected to encode the source tree structure all over the place. This seems redundant.
>
>
> Perhaps another idea would be for CM3_INSTALL to be colon/semicolon delimited?
> Search multiple package stores?
> That offers functionality not in the current model.
>
>
> A related proposal I have is that even the intermediate files be relocated.
> As I said, on a package-by-package basis, they are not in the source tree.
> Good. But on a multi-package basis, they are. Not good.
>
>
> I want the source tree to have no outputs, to stay "clean", unless I edit a file.
>
>
> Concretely this all looks like:
> 1) There is already the CM3_INSTALL environment variable. That is where "binaries" should be output, in the same structure that "ship" uses today.
>
>
> 2) There shall be a new environment variable, something like CM3_OBJECT_ROOT.
>
> 3) The existing CM3_ROOT that is the root of the source tree shall come into play.
>
>
> Given building in
> CM3_ROOT/foo/bar
>
>
> output shall go into:
> CM3_OBJECT_ROOT/foo/bar/target
> or maybe CM3_OBJECT_ROOT/target/foo/bar
> or maybe CM3_OBJECT_ROOT/foo/bar
>
>
> instead of CM3_ROOT/foo/bar/target.
>
>
> If you are not building under CM3_ROOT, there are a few viable options.
> 1) Do things as they are today.
> 2) Form up a form of CM3_OBJECT_ROOT/the path you are building that is valid and predictable. For example, in Windows, given a full path with a colon in it, remove the colon and you have a valid relative path you can append to another root.
> 3) Give user a way to specify what their "root" or "relative path from root" is.
>
>
> Because outputs litter the source tree today..unless you engage in changing BUILD_DIR, which maybe is the way in a few ways actually..you can't do multiple concurrent builds of the same target, such as with a different %PATH% to a different cm3.exe.
>
>
> Now, there is already BUILD_DIR.
> Perhaps all this is as simple as a change in the config file, changing BUILD_DIR to something like CM3_ROOT/path_of(foo.m3)/TARGET. I'll try that later.
> Maybe it just works. ?
> I'm not confident it'll work but maybe.
>
>
> Am I crazy? Wrong? Too disruptive?
> Most likely the last.
>
>
> - Jay
>
>
>
> ----------------------------------------
>> From: jay.krell at cornell.edu
>> To: m3commit at elegosoft.com; rcoleburn at scires.com
>> Date: Tue, 20 Jan 2009 05:40:07 +0000
>> Subject: [M3commit] FW: [M3devel] move to version 5.7.1?
>>
>>
>> [was truncated]
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ----------------------------------------
>>> From: jay.krell at cornell.edu
>>> To: rcoleburn at scires.com; m3devel at elegosoft.com
>>> Subject: RE: [M3devel] move to version 5.7.1?
>>> Date: Tue, 20 Jan 2009 02:28:07 +0000
>>>
>>>
>>> I meant, can we update the version stamp in the compiler.
>>> Nothing about releases or what not.
>>> But I'll take some of the bait. :)
>>>
>>>
>>> My archives are already pretty easy to use, but need a short readme.
>>>
>>>
>>> HP-UX is not yet supported, but will be fairly soon, maybe this week.
>>> 64bit first, this week, then 32bit, by end of next week?
>>> (HP-UX also runs on IA64 btw, but I couldn't get my hardware to boot
>>> my OS, maybe too old OS, maybe need to fiddle with EFI console settings.
>>> I will be doing IA64_LINUX before long. IA64_FREEBSD also wouldn't boot
>>> for me.)
>>>
>>>
>>> Where is the line between "free Microsoft tools" and "free Python"?
>>>
>>>
>>> Python is NOT needed to build cm3, just as bash is not needed.
>>>
>>>
>>> There are VERY handy scripts for doing so, in both languages.
>>> They are just slightly elaborate ways to cd around and run cm3.
>>>
>>>
>>> I know, I know..they are each line crossing, everything I have to
>>> install is a line. If it was just "free Python" and no need for "free Microsoft",
>>> that would be similar as just "free Microsoft". And "free Microsoft"
>>> is higher quality and more trusted than "free Python", and, clearly,
>>> simply more necessary, unless you use Cygwin.
>>>
>>>
>>> Cmd is an awful language for nearly any purpose.
>>> Please don't ask me to support any cmd files.
>>>
>>>
>>> Maybe I will rewrite the automation in JScript.
>>> It is a half decent language and works plenty well for command line automation.
>>> It is been "in the OS" for many years, probably at least since Windows 2000, and
>>> with any install of Internet Explorer.
>>>
>>>
>>> But really..you know..all the Python I write...is somewhat of an indictment
>>> of either Modula-3 or myself -- I don't "like" Modula-3 enough to write much
>>> of anything in it.. That is..these "scripts" perhaps should be written in Modula-3.
>>>
>>>
>>> Or maybe actually in Quake?
>>> Quake is kind of limited though.
>>> Maybe with the updates though that Olaf made?
>>> I'll think about it.
>>>
>>>
>>> The system is "easier" to build from a "current" system than from an "older" system.
>>> upgrade.sh and upgrade.py work with either.
>>> If your system is already current, you can just build in dependency order.
>>> If your system is not current, you first build "just the compiler", in dependency
>>> order, but skipping and m3core, libm3. Or you possibly hack them slightly.
>>> (Btw Tony there are few other instances of LONGINT in m3core than the one, but not many.)
>>>
>>>
>>> Most of this confusion is probably only for people that build the system.
>>> As well, maybe it could be reduced by just having an "output root"
>>> like I've said, but I know that'd be disruptive.
>>>
>>>
>>> The archives I upload are not bad.
>>> You just extract them, set $PATH, install Python, get/install a
>>> C compiler and linker and runtime (either Cygwin or Microsoft) and go.
>>>
>>>
>>> I think they are better than the cminstall-based ones.
>>>
>>>
>>> I am not likely to get around to much additional polish.
>>>
>>>
>>>
>>> - Jay
>>>
>>>
>>>
>>> ________________________________
>>>> Date: Mon, 19 Jan 2009 18:50:40 -0500
>>>> From: rcoleburn at scires.com
>>>> To: m3devel at elegosoft.com
>>>> Subject: Re: [M3devel] move to version 5.7.1?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Jay et al:
>>>>
>>>>
>>>>
>>>> I have not updated any of my cm3 code base since I released my last application to my customer in late 3Q2008.
>>>>
>>>>
>>>>
>>>> Based on all the activity, I don't think I want to do an update until we've decided to make a new release. This decision should be based primarily on stability of the code base across platforms.
>>>>
>>>>
>>>>
>>>> I can backup my cm3 tree, update to latest code, and run tests on Windows XP using my current apps if that would be helpful. Let me know. Also, I may be able to do some tests on HP-UX if that platform is supported. In addition, I also now have access to a Solaris box.
>>>>
>>>>
>>>>
>>>> One of the things I want to do for the next release is to provide detailed documentation on how to install cm3 on Windows using just the tools that comes with Windows or that are available free from Microsoft. I've already got a good start on this, but will want to get input on the exact order of package builds and which have to be repeated when.
>>>>
>>>>
>>>>
>>>> My idea is that we should have a minimal install archive for Windows (NT386) available from which the "customer" (end-user) can build all of the supported packages in the correct order, including rebuilding the base/core cm3 to prove his system is fully functional. On Windows, the build would be handled via .CMD files and not require python or anything else not supplied with Windows or free from Microsoft.
>>>>
>>>>
>>>>
>>>> I also want "us" to give explicit, documented, definitions of the terms "build", "ship", "clean", "spotless", etc. etc. that are used in various scripts et al and make these consistent. We also need to have good documentation on all the pragmas, environment variables, options, and command line switches that CM3 and CM3IDE recognize. I believe a number of these currently exist that are not well-known or properly documented.
>>>>
>>>>
>>>>
>>>> IMO, this next release needs to be as professional as possible and provide end users (customers) with the best possible experience in terms of ease of installation and use.
>>>>
>>>>
>>>>
>>>> I'll help as much as I can.
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Randy Coleburn
>>>>
>>>>>>> Jay 1/19/2009 5:39 PM>>>
>>>>
>>>> Should we move to move version 5.7.1?
>>>>
>>>> To mark the new year, and NT386GNU defaulting to pthreads (I'm thinking I should go back and make it a command line option, it's very very easy).
>>>>
>>>> Just edit two lines in sysinfo.sh and everything keeps working, right?
>>>>
>>>> - Jay