[M3devel] package groups question

Fri Jul 31 06:14:54 CEST 2009

Jay:

Thanks for your reply.  I really appreciate your patience in dealing with all my questions, especially during this busy time preparing for the release.

In response, I offer the following and also ask a few more questions:

My use of the term "minimal" is consistent with what has been documented in the working release documentation.  Namely, that "min" is the smallest distribution we plan to make available, versus "std" that contains a "standard" distribution, versus "all" that contains everything.

Now as to the exact composition of "min" and "std" that is part of my question.  I assumed someone (you, Olaf, Tony, ?) has defined what these mean.  Further, if you look at PkgInfo.txt, it appears that these definitions are recorded there, but again back to my question as to whether these definitions are accurate, complete, what the "cm3 community" wants/expects, and whether my understanding of how they are used in the context of cm3 is indeed correct.

I am well-versed in compilers and operating systems principles and design, so I don't need a lecture on these fundamentals.  My line of questioning is concrete with respect to the impending cm3 release.  

My use of the term "build" is in the context of "cm3 -build", and to be useful, subsequently "cm3 -ship".

As a software/systems engineer, I understand what you mean when you say that the code is the documentation when there is no external documentation.  However, I would hope you agree that no external documentation is generally a bad thing.  Hence my questions.  I'm not a python code writer yet, though I'm confident I could be, given some time to learn.  Thus, I can't presently rely on reading a python script as documentation.  I am confident that if you, Olaf, Tony, or someone can confirm or correct my understanding of what I've posed as the way to install the "min" distribution on Windows and "build" the rest, that I can and will be able to document this process externally for others to review.  We can then check scripts to see if they deviate from the documentation and make changes (to either scripts or documentation) as needed to reflect reality.

You state that I have not defined the "goal."  Let me attempt to make this obvious.  I'm not trying to upset the apple cart here or change the way you work to achieve progress/success.  My goal simply is to "understand" wrt the impending release so I can document, and so I can "do the right thing" to make this work correctly on my system.  We can discuss in the future how things should change for the next release.

Thanks for help with my 13-steps.  I have some comments and a few more questions below.

WRT #1, Yes, I know I may not need all of Visual Studio, but most users will just choose the typical installation.  If we have specific minimums that need to be present, we should identify (document) those for users who choose to customize their installation.

WRT #2, you have to start somewhere.  If we have both "min" and "std" distributions, one can choose which to start with, but your discussion of which parts to trim out of "min" should not be left to the end user.  The release team needs to establish what constitutes "min".  My "opinion" is that "min" should contain only what is necessary for a working cm3 compiler, given the dependency on the platform-specific linker.  Using "min" one should be able to build (cm3 -build -ship) all other packages.  Anything added to "min" that isn't necessary to accomplish building the remaining packages would be considered superfluous, in my opinion.  Some amount of superfluous on a given platform may be tolerable in order to reach consistency among all supported platforms regarding the composition of "min".  Thus, if cm3cg is needed for "min" on some platforms, but not all, it is tolerable (IMO) to make it a standard component of "min" on all platforms.

WRT #3, point taken.  However, unless we have an easy way to specify and checkout only those parts of the tree relevant to a particular platform, checking out everything ensures you haven't missed something that is going to cause problems later.

WRT#4, although doc may not be needed to run cm3, it is needed for folks to reference and understand cm3.  This is a long-standing gripe of mine that we don't make the installation of doc part of the typical installation.  The naive user new to cm3 normally expects to get some documentation.  If the default is no documentation, how does the poor sap know how to get the documentation and install it?  Also, cm3ide, does have links to the doc folder.  If you click on these links and they are broken, the average user is going to rate cm3 as a poor product and move on to something else.  Why create a problem for the new user?  Give him the documentation he needs by default and let the expert user remove it if it is unwanted.  I know that Critical Mass's 4.1 installation put the doc folder there.  When did someone decide it wasn't appropriate for the default installation?

WRT#5, examples is needed for cm3ide.  Right now, my bet is that most everyone in the cm3 community doesn't know they should copy the examples folder to the root of their cm3 installation.  That's because Reactor wasn't open-sourced until recently as cm3ide.  Right now the only documentation about this is a README file buried in the repository and even that wouldn't be there if I hadn't put it there.  Again, don't create a problem for the new user.  The "typical" installation choice should install it by default.  Let the expert user choose to remove it as part of a custom install if it is not wanted.

WRT#6, agree.  That's why I wasn't specific on how to do it, but for the newbie, it would probably be best to list at least one way.

WRT#7, agree.  I was just stating my interpretation of what you had communicated to me earlier about setting up config properly.

WRT#8, ok, but you do get an error message if the folder exists already.

WRT#9, ditto #7

WRT#11, now we are finally getting to the meat of my questions.  So, if I understand you correctly, you are saying that building the "min" group of packages does not recreate for me exactly what is being published as the "min" distribution.  In my view, we have a conflict of terminology here that needs to be cleaned up.  So what exactly is "min" as defined in PkgInfo.txt if it doesn't represent the minimal binary install?  Furthermore, if I want to use cm3 to build the sources for the compiler itself, it would seem from your comments that "min" won't do it.  Which packages need to built, and in what order, to recreate cm3.exe ?

Again you ask about the goal here, so let me be more explicit.  Given that cm3 continues to evolve and that minimal distributions are created at unpredictable intervals, if one wants to ensure the cm3 compiler is up-to-date with the latest sources, one would want to start with some existing cm3 installation and build and ship the compiler's source package(s) along with any other packages where a dependency exists.  In these instructions, I started with obtaining the latest "min" distribution, then grabbed the current source tree, so I want to proceed from this point to build a current cm3 compiler and any other requisite components to again create a "minimal" installation on my platform.  From there, one can build some/all of the remaining packages.

WRT#12, the repeat is due to my prior understanding.  Perhaps this is incorrect, or perhaps the recent changes have made this unnecessary.  But, it seems my understanding of how to rebuild cm3.exe is flawed given your response to #11.  Are you saying I should use "front" to build cm3.exe?  Am I the only one to whom all this is unclear?  Again, we need to document for understanding.

WRT#13, not sure what you mean by "sort of".  What are the caveats?

Regarding your comment at the end about scripts, I will respectfully choose to disagree with you and hope that we work together to improve documentation.  After all, if we all get run over by a truck, what we know in our brains isn't available to others unless we've documented it somewhere.

Thanks for your patience in putting up with all my questions!  Thanks also for all you are doing for cm3!

Regards,
Randy Coleburn

>>> Jay K <jay.krell at cornell.edu> 7/30/2009 4:11 PM >>>

Randy, the short answer is:

Find a goal state in a graph, which you haven't clearly done, walk that graph from the bottom up, and if there are circularities (there are, you need to have cm3 first, before you can build cm3), you need to download prebuilt files from somewhere (cm3 and mklib).

Everything else is filling in the details of the goal state, discovering the circularities, and that we have the graph recorded in a redundant fashion in pkginfo.txt. The real data is the m3makefiles.

longer answer:

Randy, what is a "minimal" distribution? You have't defined the goal.
What is "build"? You haven't defined the goal.
I could just say: "Whatever you need, to build it, just download it from here."

I'll define it as some arbitrary very minimal compiler and runtime set, specifically m3core and libm3.
But you can have working Modula-3 programs certainly without libm3.
And some Modula-3 programs will have a GUI so will want more libraries.
What is "minimal"?
And depending on future changes, the steps might change.
  Or maybe just pkginfo.txt will change.
If there is no documentation and there is code, the code is the documentation.

A lot of the code is declarative -- you look at the import statements in m3makefile.
For given package foo, say "cm3", you walk the list of imports transitively.
I'm not sure cm3 actually uses libm3 for example, and maybe it won't in the future.
This is a very real point. pkginfo.txt duplicates dependency information by
nature of being "in the correct order", but that information is already contained
in the m3makefiles. Your questions have made me realize the redundancy we have
and that we should fix it.

Or, really, older cm3 did not use sysutils, but now it does.
Things change.
And you haven't really defined the goal.

Instead of "walk pkginfo.txt in order", it should be "pick a package you want,
say cm3, and build its dependency tree from the bottom up, using the m3makefiles
to discover that tree, plus the PKGS file to locate packages".

I'm going to make up a goal that might resemble yours and correct your instructions.

> 1. Install Visual C++ 2008 Express Edition.

ok. Though you don't need all of it by far.
You don't need the entire IDE.

>
> 2. Download latest minimal binary distribution and unzip to "C:\cm3" so that you have C:\cm3\bin, C:\cm3\pkg, etc..

No. You don't need all of it. You only need two files -- cm3.exe and mklib.exe.
Other platforms need cm3cg and don't need mklib.
  Except you never really need cm3cg, it falls out of the instructions -- cm3 can build it.
  It depends on how much you want to build vs. download.
And optionally the config files. You can get them from either the source tree or download them.
See, two legitimate places for things -- download from one place or another.

>
>
> 3. Checkout complete CVS CM3 tree and place in say "C:\cm3\Sandbox", so now Sandbox has m3-sys, m3-libs, scripts, etc.

No. You don't need all of it.
You need, roughly, only m3-libs\m3core, m3-libs\libm3, m3-libs\sysutils, some of m3-sys.
m3-sys/m3cc is needed for most platforms, but not the gcc-apple directory, that's
only for iphone.

>
> 4. Copy "C:\cm3\Sandbox\doc" folder to "C:\cm3\doc"

Not needed.

>
> 5. Copy "C:\cm3\Sandbox\examples" folder to "C:\cm3\examples"

Not needed.

>
> 6. Open Visual Studio Command prompt window and put "C:\cm3\bin" on your path.

Many ways to do that.

>
>
> 7. For each file in "Sandbox\m3-sys\cm3install\src\config-no-install" and "Sandbox\m3-sys\cm3install\src\config" delete the corresponding file in "C:\cm3\bin" if it exists.

Multiple ways to do this but ok.

>
>
> 8. mkdir "C:\cm3\bin\config" if it is missing

ok. You can just mkdir unconditionally.

>
>
> 9. copy contents of "C:\cm3\Sandbox\m3-sys\cm3install\src\config-no-install" to "C:\cm3\bin\config"

You don't need all of it, but ok, that is what I do.

>
>
> 10. (Re)create "C:\cm3\cm3.cfg" to contain the following 2 lines only:
>
> INSTALL_ROOT = path() & "/.."
>
> include(path() & "/config/" & HOST)

ok

>
>
> 11. Rebuild the compiler and minimal install using the latest sources in "C:\cm3\Sandbox". The only packages that need to be built are those listed in the "min" group in "Sandbox\scripts\PkgInfo.txt". Build them in the order listed using -realclean -build -ship. (Granted, there may be scripts for this, but is what I am saying here what is actually required?)

No. Well, it depends on the goal. Really.
You already have a cm3.
"min" gives you m3core and libm3.
Combine that with the cm3 you already have and great, you have a slightly larger more useful set of libraries.

>
>
> 12. Repeat step #11 to ensure the new compiler is used to rebuild the core stuff.

NOt really. The repitition isn't doing what you say.

Look at the group called "front".
Which isn't completely accurate since it includes m3cc.
Perhaps there should be a group compiler.

But really, again, this is cm3 plus its transitive dependencies.
I think maybe we need to partly ditch "groups".
In paritcular, "groups" should contain useful end results and not their dependencies.
m3middle doesn't belong in any group, it is just implied by cm3.
Seriously.

But the scripts don't currently read m3makefiles.
Redoing things in cm3 would be better?

>
>
> 13. Now, for any or all packages listed in "Sandbox\scripts\PkgInfo.txt", use cm3 to build and ship those that you want. You could build everything, but for example, if you wanted only a "std" installation, you could simply build and ship only those packages tagged as "std".

Ok. Sort of.

>
> I know you've probably got scripts for all of this

Yep.

> Scripts are an expression of the requirements

Sometimes it is the other way around.

- Jay

CONFIDENTIALITY NOTICE:  This email and any attachments are intended solely for the use of the named recipient(s). This e-mail may contain confidential and/or proprietary information of Scientific Research Corporation.  If you are not a named recipient, you are prohibited from making any use of the information in the email and attachments.  If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.

EXPORT COMPLIANCE NOTICE:  This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR).  Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require export authorization by the appropriate U.S. Government agency prior to export or transfer.  In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization.  By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20090731/daaf1a53/attachment-0002.html>