[M3devel] ROOT, $ORIGIN, runpath, LD_LIBRARY_PATH, symlinks, hardlinks, etc.

[Tony Hosking]

On 2 Jul 2009, at 14:41, Jay wrote:

> Before I explain, more options:
>
>  copy and commit the source to cm3 is viable

I don't know what this sentence means.

>   heck, use a relative instead of absolute from ROOT

ditto

>   go back to the old way (I'll explain)

I think I understand the shortcomings here but Jay should explain.

>   There is also the option to relink or "fixup"
>   upon ship and/or install. Some platforms have a "fixup"
>   utility for exactly "these reasons".
>   MacOS X has one.

Sounds feasible, but I worry about portability of this approach.

>  The question is:
>
> How does program foo find libbar.so?
>  At runtime.

Right.  rpaths in the executable should do the trick.

> How does libbar.so find libfoo.so?

Same again.

>  There are a bewildering array of choices.
> They are combinable.
> It varies somewhat from platform by platform, but not much really.

Ideally we use the same approach for *all* platforms.  So, what do all  
platforms have in common.

> The main options are:
>
>
> 1: A full path written into program foo
> or libbar pointing to e.g. /usr/lib/libfoo.so.
> Actually this is usually separate libfoo.so and
> a list of directories that apply to all .so files.
> There isn't generally a matching up of directory to file,
> I think, but a "search path" aka "run path".

i.e., rpath?

>  2: A relative path written into program foo
> or libbar.so point to e.g. $ORIGIN/../lib/libfoo.so.
> Again, the directories are split from the files.

Please explain the implications of this w.r. to hard links!  I don't  
want hard links.

>  MacOS X doesn't have $ORIGIN by that name, but it
> has three similar things, like @executable_path.
> @executable_path goes way back and is presumably
> relative to the primary main executable.

Then we should avoid $ORIGIN.  Use the linker options to specify  
rpaths (and if there is need for override use $LD_LIBRARY_PATH.

> There are two other similar options.
> One is new in 10.5. Presumably one is relative
> to the reference. And maybe one is some concatenation
> of all directories anything found in so far? I don't know.
> The documenation was not clear to me.

Again, I'd like to use the same approach on all platforms (except  
perhaps Windows).

>  NetBSD prior to 5.0 doesn't have $ORIGIN.

A good reason not to use it.

> NT searches $PATH always.
> We put the .dlls in bin.
> Cygwin same.

OK, so things work fine there.

>  Approximately all other systems seem to support about the same  
> $ORIGIN feature:
>   OpenBSD, FreeBSD, Linux, Solaris, NetBSD 5.0, even Interix, and  
> others (Aix? Irix? HP-UX?)
>   The MacOS X feature is close enough to lump in.
>
> 3: Search $LD_LIBRARY_PATH or similar.
>  This has a few different names.
>   LD_LIBRARY_PATH32, LD_LIBRARY_PATH64, DYLD_SOMETHING.

These are fallbacks for the situation when the default installed  
configuration needs overriding.

> Now, our historical implementation was
>  - record a full path /usr/local/cm3/pkg/foo/LINUX/libfoo.so,
>    a path for every dependency
>  The file formats are generally such that there are leaf file names
>   separate from an array of directories to look in. A "normal"
>   non-Modula-3 program would generally I assume have a prety small
>   such array of directories. A "normal" Modula-3 program would have
>   a fairly large array.

But if everything goes in INSTALL_ROOT/lib we have only one directory,  
right?

>  As well, doing builds for distribution would often leave you with / 
> tmp/cm3
>   which is a possible security problem. This definitely did occur.

Surely, building for a distro can build in the right path.  Where did / 
tmp/cm3 come from?  If you need to be able to override while building  
then use $LD_LIBRARY_PATH.

> Now, again, there are many options, and they can be combined.
> Portability is perhaps a concern, but perhaps not so much, given  
> today's landscape.

Elegance is also a concern.

>  Now, if we are to use relative paths, well, my understanding is that
> the "real" files need to be in relatively few directories, e.g. /usr/ 
> local/cm3/lib,
> and not /usr/local/cm3/pkg/foo/linux etc.

I prefer this approach.

> This way, files in lib and bin can point to $ORIGIN/../lib.

Fair enough.

> Now, keep in mind, that unshipped binaries sometimes people like to  
> work,
> as well, perhaps also programs (not shared libraies, as far as I know)
> buried in the pkg store /usr/local/cm3/pkg/foo/linux/foo.
> I know cm3 sits in there, though that copy isn't used.

I don't know understand what you are trying to say here.

>  I make no account for binaries buried in the package store.
>
> I do make some account for unshipped binaries.
>
> By default the run paths we are recording in current config files
> are
> $ORIGIN/../lib:/usr/local/cm3/lib
> Not a bunch of paths into pkg, just two entries.
>
>
> The "security" issue still is present by default.
> But the run path is "prettier" and probably more efficient.
>
>
> When you build a distribution however, you are expected to omit /usr/ 
> local/cm3/lib (it might be /tmp/cm3).
> This is controled by an environment variable, which I set in Olaf's  
> scripts, but oops
> I think I set it too late. I'll fix that very very soon.

The following is a bit rambling for me to respond to in detail.   
Perhaps someone else should weigh in.

>
>
> To recap:
>  - full paths
>  - relative paths from $ORIGIN
>  - LD_LIBRARY_PATH
>
> They each have advantages and disadvantages.
> $ORIGIN provides for a "movable" install. I can install to /usr/ 
> local/cm3,
> you can install to /opt/cm3. It'll just work. No need to change  
> LD_LIBRARY_PATH.
>
>
> However if I want to move just some of the files, not the whole  
> binary tree
> together, then $ORIGIN breaks down.
>
>
> If really want one file here, another there, and another somewhere  
> else, $ORIGIN doesn't do.
> You are left with full paths or LD_LIBRARY_PATH.
>
>
> Oh, and moral equivalent of LD_LIBRARY_PATH, like /etc/ld.so.conf or  
> such.
> Look on birch for example, it is /usr/local/cm3/lib.
> That tripped me up actually, because, you know, my cm3 is in $HOME/ 
> cm3 there.
> This demonstrated another problem in this area!
> $ORIGIN keeps things more tightly together, which you often want,  
> sometimes might not not.
> LD_LIBRARY_PATH lets things be spread around, again, sometimes you  
> want that, sometimes not.
> I figure tightly point is the more common, but imagine if you just  
> want to replace one file.
>  Is that reasonable? You replace it by putting it elsewhere?  
> (Imagine "patching" an install
>  this way.)
>
>
> So, here now I'm rambling.
> I've laid out the options and /some/ of the disadvantages/advantages.
>
>
> There is no perfect solution, and the various solutions are  
> combinable.
> I find combining them though, I start to get confused.
> Because you end up in a "probing" situation and you start wondering
> which supercedes which, and you just hope there aren't any  
> "duplicates"
> so that the probe order is mostly moot (assuming there remain no  
> duplicates).
>
>
> Somewhat clear?
>
>
> You run, e.g. elfdump -p foo | grep -i path or somesuch, or ldd foo  
> or ldd -v or -V foo and
> investigate the various options and their effects. Solaris has a  
> very nice and verbose
> ldd form.
>
>
> Now, the files in pkg.
> I consider it somewhat philosophically significant that "everything  
> is in pkg".
>  There is a well structured "repository" not a randomly organized  
> bunch of files.
> I think. Maybe this is just a suggestion?
> You know, if they are hardlinks and we don't really use the ones in  
> pkg, is
> there a point?
>
>
> Or, on the other hand, is the philosophy a big deal and we should go  
> back to
> lib symlinking to pkg to appease the philosophy?
>
>
> "ship/install" certainly "likes" to put things in pkg.
> I don't think the option of using lib and not pkg is doable without  
> builder changes.
> But that is doable.
>
>
> A few other folks here did pipe up some on this in the recent past and
> I think there was kinda sorta consensus that the old ways were not  
> great
> (particularly 1) the big runpath 2) the runpath into /tmp)
> and the new ways maybe better, but I admit there was no clear  
> consensus.
> I acted somewhat unilaterally. You know, I'm impatient and arrogant  
> so...
> and Tinderbox is there to cause some restraint.
>
>
> There was definitely pushback on wanting unshipped binaries to keep  
> working,
> which they largely do as a result of the discussion/pushback.
>
>
> As well, any unshipped binaries that we know we use in the build,  
> like PklFonts,
> stubgen, m3bundle, we build standalone, to evade/avoid problems here,
> since $ORIGIN/../lib won't work and the full paths are meant to be  
> omitted.
> I consider this not great.
>
> I think, really, we should be able to dynamically link everything  
> really,
> including cm3. I don't think we should static link just to make up for
> the need for a backup. There are plenty of backups available and you  
> can
> make plenty more. However this will have to wait and probably take
> more arguing and maybe demonstration of its viability.
> I will require some work to make work anyway.
>
>
> At one point I was confused about the need for static linking so  
> accepted it strongly.
> Now I don't see it. If you just need backups, again, they exist, and  
> you can make more.
> Static linking isn't imho a substitute for backups.
> There are technical problems to be solved though.
>
> Could be that these issues are related though. If your backup might
> load the wrong m3core.so too easily, making it not a very reliable  
> backup,
> static linking could mitigate that. But $origin seems like a good  
> mitigation.
> I'm strongly considering add just plain $origin, in addition to  
> $origin/../lib to runpath.
> That way you can just put everything in one messy but easier to move/ 
> copy directory.
>
>
>  - Jay
>
>
>
>
>
> > Date: Thu, 2 Jul 2009 20:04:36 +0200
> > From: wagner at elegosoft.com
> > To: m3devel at elegosoft.com
> > Subject: Re: [M3devel] ROOT
> >
> > Quoting Tony Hosking <hosking at cs.purdue.edu>:
> >
> > > That sounds plausible. Is there anything in the current setup that
> > > depends on the .so files being in pkg instead of lib?
> >
> > I don't think so offhand. But I must admit that I've lost the  
> overview
> > in this area, too :-/
> >
> > Olaf
> >
> > > On 2 Jul 2009, at 13:34, jay.krell at cornell.edu wrote:
> > >
> > >> How about we put .so files only in lib and not in pkg? Or symlink
> > >> from pkg to lib instead of how it used to be the other way  
> around?
> > >> Those would also fix all this.
> > >>
> > >> - Jay (phone)
> > >>
> > >> On Jul 2, 2009, at 9:21 AM, Jay <jay.krell at cornell.edu> wrote:
> > >>
> > >>>
> > >>> Are you up to date in m3-libs/m3core?
> > >>> Lots of builds have succeeded with this..
> > >>> It isn't great, but that requirement to build from previous  
> 5.4 release..
> > >>>
> > >>> - Jay
> > >>>
> > >>> ________________________________
> > >>>> From: hosking at cs.purdue.edu
> > >>>> To: hosking at cs.purdue.edu
> > >>>> Date: Thu, 2 Jul 2009 11:54:51 -0400
> > >>>> CC: m3devel at elegosoft.com
> > >>>> Subject: Re: [M3devel] ROOT
> > >>>>
> > >>>> If I define ROOT then I get the following error. I think this  
> all
> > >>>> needs reverting!
> > >>>>
> > >>>> --- building in SOLgnu ---
> > >>>>
> > >>>> ignoring ../src/m3overrides
> > >>>>
> > >>>> "/.gollum/u110/u86/hosking/cm3/m3-sys/cm3/src/m3makefile", line
> > >>>> 12: quake runtime error: unable to copy
> > >>>> "~/cm3/m3-libs/m3core/src/ unix/Common/UnixLink.c" to
> > >>>> "./UnixLink.c": errno=2
> > >>>>
> > >>>> --procedure-- -line- -file---
> > >>>> cp_if --
> > >>>> include_dir 12 /.gollum/u110/u86/hosking/cm3/m3-sys/cm3/src/  
> m3makefile
> > >>>> 6 /.gollum/u110/u86/hosking/cm3/m3-sys/cm3/SOLgnu/m3make.args
> > >>>>
> > >>>> Fatal Error: package build failed
> > >>>> n
> > >>>>
> > >>>> On 2 Jul 2009, at 11:49, Tony Hosking wrote:
> > >>>>
> > >>>> Where did variable ROOT come from? Do I really need to define  
> it?
> > >>>> Seems broken to me to depend on the source directory structure
> > >>>> as it sets that structure in stone.
> > >>>>
> > >>>>
> >
> >
> >
> > --
> > Olaf Wagner -- elego Software Solutions GmbH
> > Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
> > phone: +49 30 23 45 86 96 mobile: +49 177 2345 869 fax: +49 30 23  
> 45 86 95
> > http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz:  
> Berlin
> > Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr:  
> DE163214194
> >

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20090702/8e612487/attachment-0002.html>