[M3devel] ROOT, $ORIGIN, runpath, LD_LIBRARY_PATH, symlinks, hardlinks, etc.

Thu Jul 2 20:51:22 CEST 2009

darnit I need to be more careful.

program -- not a shared library

binary -- program or shared library in some contexts, but I used it below also to mean program

shared library -- .dll, .so, etc.

"I make no account for binaries in the pkg store" -- I meant programs.

Darn I need to write this better...hopefully some of it will come through...

 - Jay

From: jay.krell at cornell.edu
To: wagner at elegosoft.com; m3devel at elegosoft.com
Date: Thu, 2 Jul 2009 18:41:27 +0000
Subject: Re: [M3devel] ROOT, $ORIGIN, runpath, LD_LIBRARY_PATH, symlinks, hardlinks, etc.

Before I explain, more options:

 copy and commit the source to cm3 is viable

 heck, use a relative instead of absolute from ROOT

 go back to the old way (I'll explain)

 There is also the option to relink or "fixup"
  upon ship and/or install. Some platforms have a "fixup"
  utility for exactly "these reasons".
  MacOS X has one.

The question is:

How does program foo find libbar.so?
 At runtime.

How does libbar.so find libfoo.so?

There are a bewildering array of choices.
They are combinable.
It varies somewhat from platform by platform, but not much really.

The main options are:

1: A full path written into program foo
or libbar pointing to e.g. /usr/lib/libfoo.so.
Actually this is usually separate libfoo.so and
a list of directories that apply to all .so files.
There isn't generally a matching up of directory to file,
I think, but a "search path" aka "run path".

2: A relative path written into program foo
or libbar.so point to e.g. $ORIGIN/../lib/libfoo.so.
Again, the directories are split from the files.

MacOS X doesn't have $ORIGIN by that name, but it
has three similar things, like @executable_path.
@executable_path goes way back and is presumably
relative to the primary main executable.
There are two other similar options.
One is new in 10.5. Presumably one is relative
to the reference. And maybe one is some concatenation
of all directories anything found in so far? I don't know.
The documenation was not clear to me.

NetBSD prior to 5.0 doesn't have $ORIGIN.

NT searches $PATH always.
We put the .dlls in bin.
Cygwin same.

Approximately all other systems seem to support about the same $ORIGIN feature:
  OpenBSD, FreeBSD, Linux, Solaris, NetBSD 5.0, even Interix, and others (Aix? Irix? HP-UX?)
  The MacOS X feature is close enough to lump in.

3: Search $LD_LIBRARY_PATH or similar.
 This has a few different names.
  LD_LIBRARY_PATH32, LD_LIBRARY_PATH64, DYLD_SOMETHING.

Now, our historical implementation was
 - record a full path /usr/local/cm3/pkg/foo/LINUX/libfoo.so,
   a path for every dependency
 The file formats are generally such that there are leaf file names
  separate from an array of directories to look in. A "normal"
  non-Modula-3 program would generally I assume have a prety small
  such array of directories. A "normal" Modula-3 program would have
  a fairly large array.
 As well, doing builds for distribution would often leave you with /tmp/cm3
  which is a possible security problem. This definitely did occur.

Now, again, there are many options, and they can be combined.
Portability is perhaps a concern, but perhaps not so much, given today's landscape.

Now, if we are to use relative paths, well, my understanding is that
the "real" files need to be in relatively few directories, e.g. /usr/local/cm3/lib,
and not /usr/local/cm3/pkg/foo/linux etc.

This way, files in lib and bin can point to $ORIGIN/../lib.

Now, keep in mind, that unshipped binaries sometimes people like to work,
as well, perhaps also programs (not shared libraies, as far as I know)
buried in the pkg store /usr/local/cm3/pkg/foo/linux/foo.
I know cm3 sits in there, though that copy isn't used.

I make no account for binaries buried in the package store.

I do make some account for unshipped binaries.

By default the run paths we are recording in current config files 
are
$ORIGIN/../lib:/usr/local/cm3/lib
Not a bunch of paths into pkg, just two entries.

The "security" issue still is present by default.
But the run path is "prettier" and probably more efficient.

When you build a distribution however, you are expected to omit /usr/local/cm3/lib (it might be /tmp/cm3).
This is controled by an environment variable, which I set in Olaf's scripts, but oops
I think I set it too late. I'll fix that very very soon.

To recap:
 - full paths
 - relative paths from $ORIGIN
 - LD_LIBRARY_PATH

They each have advantages and disadvantages.
$ORIGIN provides for a "movable" install. I can install to /usr/local/cm3,
you can install to /opt/cm3. It'll just work. No need to change LD_LIBRARY_PATH.

However if I want to move just some of the files, not the whole binary tree
together, then $ORIGIN breaks down.

If really want one file here, another there, and another somewhere else, $ORIGIN doesn't do.
You are left with full paths or LD_LIBRARY_PATH.

Oh, and moral equivalent of LD_LIBRARY_PATH, like /etc/ld.so.conf or such.
Look on birch for example, it is /usr/local/cm3/lib.
That tripped me up actually, because, you know, my cm3 is in $HOME/cm3 there.
This demonstrated another problem in this area!
$ORIGIN keeps things more tightly together, which you often want, sometimes might not not.
LD_LIBRARY_PATH lets things be spread around, again, sometimes you want that, sometimes not.
I figure tightly point is the more common, but imagine if you just want to replace one file.
 Is that reasonable? You replace it by putting it elsewhere? (Imagine "patching" an install
 this way.)

So, here now I'm rambling.
I've laid out the options and /some/ of the disadvantages/advantages.

There is no perfect solution, and the various solutions are combinable.
I find combining them though, I start to get confused.
Because you end up in a "probing" situation and you start wondering
which supercedes which, and you just hope there aren't any "duplicates"
so that the probe order is mostly moot (assuming there remain no duplicates).

Somewhat clear?

You run, e.g. elfdump -p foo | grep -i path or somesuch, or ldd foo or ldd -v or -V foo and
investigate the various options and their effects. Solaris has a very nice and verbose
ldd form.

Now, the files in pkg.
I consider it somewhat philosophically significant that "everything is in pkg".
 There is a well structured "repository" not a randomly organized bunch of files.
I think. Maybe this is just a suggestion?
You know, if they are hardlinks and we don't really use the ones in pkg, is
there a point?

Or, on the other hand, is the philosophy a big deal and we should go back to
lib symlinking to pkg to appease the philosophy?

"ship/install" certainly "likes" to put things in pkg.
I don't think the option of using lib and not pkg is doable without builder changes.
But that is doable.

A few other folks here did pipe up some on this in the recent past and
I think there was kinda sorta consensus that the old ways were not great
(particularly 1) the big runpath 2) the runpath into /tmp)
and the new ways maybe better, but I admit there was no clear consensus.
I acted somewhat unilaterally. You know, I'm impatient and arrogant so...
and Tinderbox is there to cause some restraint.

There was definitely pushback on wanting unshipped binaries to keep working,
which they largely do as a result of the discussion/pushback.

As well, any unshipped binaries that we know we use in the build, like PklFonts,
stubgen, m3bundle, we build standalone, to evade/avoid problems here,
since $ORIGIN/../lib won't work and the full paths are meant to be omitted.
I consider this not great.

I think, really, we should be able to dynamically link everything really,
including cm3. I don't think we should static link just to make up for
the need for a backup. There are plenty of backups available and you can
make plenty more. However this will have to wait and probably take
more arguing and maybe demonstration of its viability.
I will require some work to make work anyway.

At one point I was confused about the need for static linking so accepted it strongly.
Now I don't see it. If you just need backups, again, they exist, and you can make more.
Static linking isn't imho a substitute for backups.
There are technical problems to be solved though.

Could be that these issues are related though. If your backup might
load the wrong m3core.so too easily, making it not a very reliable backup,
static linking could mitigate that. But $origin seems like a good mitigation.
I'm strongly considering add just plain $origin, in addition to $origin/../lib to runpath.
That way you can just put everything in one messy but easier to move/copy directory.

 - Jay

> Date: Thu, 2 Jul 2009 20:04:36 +0200
> From: wagner at elegosoft.com
> To: m3devel at elegosoft.com
> Subject: Re: [M3devel] ROOT
> 
> Quoting Tony Hosking <hosking at cs.purdue.edu>:
> 
> > That sounds plausible. Is there anything in the current setup that
> > depends on the .so files being in pkg instead of lib?
> 
> I don't think so offhand. But I must admit that I've lost the overview
> in this area, too :-/
> 
> Olaf
> 
> > On 2 Jul 2009, at 13:34, jay.krell at cornell.edu wrote:
> >
> >> How about we put .so files only in lib and not in pkg? Or symlink 
> >> from pkg to lib instead of how it used to be the other way around? 
> >> Those would also fix all this.
> >>
> >> - Jay (phone)
> >>
> >> On Jul 2, 2009, at 9:21 AM, Jay <jay.krell at cornell.edu> wrote:
> >>
> >>>
> >>> Are you up to date in m3-libs/m3core?
> >>> Lots of builds have succeeded with this..
> >>> It isn't great, but that requirement to build from previous 5.4 release..
> >>>
> >>> - Jay
> >>>
> >>> ________________________________
> >>>> From: hosking at cs.purdue.edu
> >>>> To: hosking at cs.purdue.edu
> >>>> Date: Thu, 2 Jul 2009 11:54:51 -0400
> >>>> CC: m3devel at elegosoft.com
> >>>> Subject: Re: [M3devel] ROOT
> >>>>
> >>>> If I define ROOT then I get the following error. I think this all 
> >>>> needs reverting!
> >>>>
> >>>> --- building in SOLgnu ---
> >>>>
> >>>> ignoring ../src/m3overrides
> >>>>
> >>>> "/.gollum/u110/u86/hosking/cm3/m3-sys/cm3/src/m3makefile", line 
> >>>> 12: quake runtime error: unable to copy 
> >>>> "~/cm3/m3-libs/m3core/src/ unix/Common/UnixLink.c" to 
> >>>> "./UnixLink.c": errno=2
> >>>>
> >>>> --procedure-- -line- -file---
> >>>> cp_if --
> >>>> include_dir 12 /.gollum/u110/u86/hosking/cm3/m3-sys/cm3/src/ m3makefile
> >>>> 6 /.gollum/u110/u86/hosking/cm3/m3-sys/cm3/SOLgnu/m3make.args
> >>>>
> >>>> Fatal Error: package build failed
> >>>> n
> >>>>
> >>>> On 2 Jul 2009, at 11:49, Tony Hosking wrote:
> >>>>
> >>>> Where did variable ROOT come from? Do I really need to define it? 
> >>>> Seems broken to me to depend on the source directory structure 
> >>>> as it sets that structure in stone.
> >>>>
> >>>>
> 
> 
> 
> -- 
> Olaf Wagner -- elego Software Solutions GmbH
> Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
> phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23 45 86 95
> http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin
> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20090702/366d144d/attachment-0002.html>