ld.so behavior with $ORIGIN

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

ld.so behavior with $ORIGIN

Aurélien Vallée
Hello,

I have troubles understanding the interpretation of $ORIGIN on
OpenBSD. I'm switching to OpenBSD from Linux, so I may be biased in my
assumptions.

I built a program (python in this example) with the following ld parameters:

-Wl,origin,z
-Wl,rpath,'$ORIGIN/../lib'

I can then check that the proper attributes were set on the binary:

$ readelf -d python
...
0x0000000000000001 (NEEDED) Shared library: [libpython2.7.so.1.0]
...
0x000000000000000f (RPATH) Library rpath: [$ORIGIN/../lib]
...
0x000000006ffffffb (FLAGS_1) Flags: ORIGIN

On the filesystem, everything is also correct, say I have:

- The python binary located in ~/bin/python
- The python library located in ~/lib/libpython2.7.so.1.0

Now if I run python using its absolute or relative path, everything
works fine. However when I add python to my $PATH, and call it by name
(just "python"), the loader complains it cannot find
libpython2.7.so.1.0.

e.g. with python in my $PATH:

$ python
python: can't load library 'libpython2.7.so.1.0'

That won't work, while:

$ $(which python)

or

$ ~/bin/python

or

$ /home/.../bin/python

These will work.

Strangely, if I cd to the directory where python is located, then I
can issue just "python" and it will work. e.g:

$ cd ~/bin
$ python

It seems to me that $ORIGIN is interpreted as a relative path from the
elf file when calling it with an absolute path, or relative to the
path of the caller when using $PATH.

I did not expect this behavior, as "man ld.so" states:

> When resolving dependencies for the loaded objects, ld-elf.so.1 may be
>     allowed to translate dynamic token strings in rpath and soname by setting
>     -z origin option of the static linker ld(1).  The following strings are
>     recognized now:
>     $ORIGIN Translated to the full path of the loaded object.

Am I doing something wrong here, or is expected behavior?

I've spent some time digging libexec/ld.so/resolve.c for an
explanation (I'm on -current snapshots from Dec 3 2015, last week).
It's not an easy lecture for a newcomer, so please forgive me if what
I say below is complete garbage.

Here is what I found:

1) My elf binary is loaded and ends up in _dl_finalize_object()

2) My binary has RPATH and FLAGS_1 to ORIGIN, so _dl_origin_subst() is called
if (object->dyn.rpath) {
  object->rpath = _dl_split_path(object->dyn.rpath);
if ((object->obj_flags & DF_1_ORIGIN) && _dl_trust)
  _dl_origin_subst(object);
}

3) _dl_origin_subst() retrieves the original path of the elf object,
and applies $ORIGIN interpolation if necessary
if (_dl_origin_path(object, origin_path) != 0)
  return;

/* perform path substitutions on each segment of rpath */
for (pp = object->rpath; *pp != NULL; pp++) {
  _dl_origin_subst_path(object, origin_path, pp);
}

4) _dl_origin_subst_path() will use origin_path in case of $ORIGIN, so
back to _dl_origin_path() to understand what is used.
case SUBST_ORIGIN:
  value = origin_path;
  break;

5) _dl_origin_path() just uses _dl_realpath(_dl_dirname()) on the elf object.

6) _dl_dirname() will return "." if the path provided does not contain a '/'

7) _dl_realpath() will prepend the CWD to this.

These last steps seems odd to me. Why is the CWD used here, when what
we really want is the directory containing the binary of the currently
running process (i.e. /proc/self/exe on linux).
Just my guess, I don't really know what I'm talking about.

Anyway, explanations  greatly appreciated! And again, please forgive
me if all that is just me doing stupid things.

Reply | Threaded
Open this post in threaded view
|

Re: ld.so behavior with $ORIGIN

Aurélien Vallée
Just found I can set LD_DEBUG to see the full translation process of ld.so.
This seems to confirm what I've seen in the source: ld.so uses cwd
instead of process file location for $ORIGIN interpolation.

$ mkdir -p /tmp/dummy/working/directory

$ cd /tmp/dummy/working/directory

$ which python
/home/avallee/bin/python

$ readelf -d $(which python)
0x0000000000000001 (NEEDED) Shared library: [libpython2.7.so.1.0]
0x000000000000000f (RPATH) Library rpath: [$ORIGIN/../lib]
0x000000006ffffffb (FLAGS_1) Flags: ORIGIN

$ LD_DEBUG=1 python
rtld loading: 'python'
...
orig_path $ORIGIN/../lib
new_path /tmp/dummy/working/directory/../lib
<failure here>

$ LD_DEBUG=1 $(which python)
...
orig_path $ORIGIN/../lib
new_path /home/avallee/bin/../lib
<success here>

On Fri, Dec 11, 2015 at 5:04 PM, Aurélien Vallée
<[hidden email]> wrote:
> Hello,
>
> I have troubles understanding the interpretation of $ORIGIN on
> OpenBSD. I'm switching to OpenBSD from Linux, so I may be biased in my
> assumptions.
>
> I built a program (python in this example) with the following ld
parameters:

>
> -Wl,origin,z
> -Wl,rpath,'$ORIGIN/../lib'
>
> I can then check that the proper attributes were set on the binary:
>
> $ readelf -d python
> ...
> 0x0000000000000001 (NEEDED) Shared library: [libpython2.7.so.1.0]
> ...
> 0x000000000000000f (RPATH) Library rpath: [$ORIGIN/../lib]
> ...
> 0x000000006ffffffb (FLAGS_1) Flags: ORIGIN
>
> On the filesystem, everything is also correct, say I have:
>
> - The python binary located in ~/bin/python
> - The python library located in ~/lib/libpython2.7.so.1.0
>
> Now if I run python using its absolute or relative path, everything
> works fine. However when I add python to my $PATH, and call it by name
> (just "python"), the loader complains it cannot find
> libpython2.7.so.1.0.
>
> e.g. with python in my $PATH:
>
> $ python
> python: can't load library 'libpython2.7.so.1.0'
>
> That won't work, while:
>
> $ $(which python)
>
> or
>
> $ ~/bin/python
>
> or
>
> $ /home/.../bin/python
>
> These will work.
>
> Strangely, if I cd to the directory where python is located, then I
> can issue just "python" and it will work. e.g:
>
> $ cd ~/bin
> $ python
>
> It seems to me that $ORIGIN is interpreted as a relative path from the
> elf file when calling it with an absolute path, or relative to the
> path of the caller when using $PATH.
>
> I did not expect this behavior, as "man ld.so" states:
>
>> When resolving dependencies for the loaded objects, ld-elf.so.1 may be
>>     allowed to translate dynamic token strings in rpath and soname by
setting
>>     -z origin option of the static linker ld(1).  The following strings
are

>>     recognized now:
>>     $ORIGIN Translated to the full path of the loaded object.
>
> Am I doing something wrong here, or is expected behavior?
>
> I've spent some time digging libexec/ld.so/resolve.c for an
> explanation (I'm on -current snapshots from Dec 3 2015, last week).
> It's not an easy lecture for a newcomer, so please forgive me if what
> I say below is complete garbage.
>
> Here is what I found:
>
> 1) My elf binary is loaded and ends up in _dl_finalize_object()
>
> 2) My binary has RPATH and FLAGS_1 to ORIGIN, so _dl_origin_subst() is
called

> if (object->dyn.rpath) {
>   object->rpath = _dl_split_path(object->dyn.rpath);
> if ((object->obj_flags & DF_1_ORIGIN) && _dl_trust)
>   _dl_origin_subst(object);
> }
>
> 3) _dl_origin_subst() retrieves the original path of the elf object,
> and applies $ORIGIN interpolation if necessary
> if (_dl_origin_path(object, origin_path) != 0)
>   return;
>
> /* perform path substitutions on each segment of rpath */
> for (pp = object->rpath; *pp != NULL; pp++) {
>   _dl_origin_subst_path(object, origin_path, pp);
> }
>
> 4) _dl_origin_subst_path() will use origin_path in case of $ORIGIN, so
> back to _dl_origin_path() to understand what is used.
> case SUBST_ORIGIN:
>   value = origin_path;
>   break;
>
> 5) _dl_origin_path() just uses _dl_realpath(_dl_dirname()) on the elf
object.
>
> 6) _dl_dirname() will return "." if the path provided does not contain a
'/'

>
> 7) _dl_realpath() will prepend the CWD to this.
>
> These last steps seems odd to me. Why is the CWD used here, when what
> we really want is the directory containing the binary of the currently
> running process (i.e. /proc/self/exe on linux).
> Just my guess, I don't really know what I'm talking about.
>
> Anyway, explanations  greatly appreciated! And again, please forgive
> me if all that is just me doing stupid things.



--
Aurélien Vallée
Phone +33 9 77 19 85 61

Reply | Threaded
Open this post in threaded view
|

Re: ld.so behavior with $ORIGIN

Theo de Raadt
> Just found I can set LD_DEBUG to see the full translation process of ld.so.
> This seems to confirm what I've seen in the source: ld.so uses cwd
> instead of process file location for $ORIGIN interpolation.
             ^^^^^^^^^^^^^^^^^^^^^

What is that?  Generally Unix has no way of doing this.  You can inspect
your executable's __progname, which is cloned from argv at crt startup,
but it may not contain an absolute path.  So what do you really mean here?

Reply | Threaded
Open this post in threaded view
|

Re: ld.so behavior with $ORIGIN

Philip Guenther-2
On Fri, Dec 11, 2015 at 10:18 AM, Theo de Raadt <[hidden email]> wrote:
>> Just found I can set LD_DEBUG to see the full translation process of ld.so.
>> This seems to confirm what I've seen in the source: ld.so uses cwd
>> instead of process file location for $ORIGIN interpolation.
>              ^^^^^^^^^^^^^^^^^^^^^
>
> What is that?  Generally Unix has no way of doing this.  You can inspect
> your executable's __progname, which is cloned from argv at crt startup,
> but it may not contain an absolute path.  So what do you really mean here?

I'm not sure what Solaris does for this, but it looks like glibc on
Linux does readlink("/proc/self/exe") and if that fails and the
process trusts its environment** then it falls back to the
LD_ORIGIN_PATH environment variable.  $ORIGIN then expands to
dirname() of that.

What's desired, I think, is to base it on the resolved (ala
realpath()) value of the 'path' argument to execve().  At least that's
what the Solaris docs seem to imply.  Doesn't matter if that path
resolves to a different underlying file later: it's just going to be
used for the directory part anyway and it's ignored for privileged
processes.

I guess we could stick 'path' into an auxiliary vector value and have
ld.so do the realpath() call if $ORIGIN is used?  It would be that or
have the kernel store the whole path for the life of the process for
obtaining with sysctl().  Right now it only stores the last component
of the (resolved) original path in p_comm.


Philip Guenther


** i.e, not marked as 'secure', their sorta-similar-to-issetugid-but-not-really

Reply | Threaded
Open this post in threaded view
|

Re: ld.so behavior with $ORIGIN

Aurélien Vallée
> glibc on
> Linux does readlink("/proc/self/exe") and if that fails and the
> process trusts its environment** then it falls back to the
> LD_ORIGIN_PATH environment variable.  $ORIGIN then expands to
> dirname() of that.

This is also what musl-libc do, except that it does not bother trying
something
else if that fails.

> It would be that or
> have the kernel store the whole path for the life of the process for
> obtaining with sysctl()

That would be great. ps and top would be able to display the path too,
pretty handy.

On Fri, Dec 11, 2015 at 9:04 PM, Philip Guenther <[hidden email]> wrote:
> On Fri, Dec 11, 2015 at 10:18 AM, Theo de Raadt <[hidden email]>
wrote:
>>> Just found I can set LD_DEBUG to see the full translation process of
ld.so.

>>> This seems to confirm what I've seen in the source: ld.so uses cwd
>>> instead of process file location for $ORIGIN interpolation.
>>              ^^^^^^^^^^^^^^^^^^^^^
>>
>> What is that?  Generally Unix has no way of doing this.  You can inspect
>> your executable's __progname, which is cloned from argv at crt startup,
>> but it may not contain an absolute path.  So what do you really mean here?
>
> I'm not sure what Solaris does for this, but it looks like glibc on
> Linux does readlink("/proc/self/exe") and if that fails and the
> process trusts its environment** then it falls back to the
> LD_ORIGIN_PATH environment variable.  $ORIGIN then expands to
> dirname() of that.
>
> What's desired, I think, is to base it on the resolved (ala
> realpath()) value of the 'path' argument to execve().  At least that's
> what the Solaris docs seem to imply.  Doesn't matter if that path
> resolves to a different underlying file later: it's just going to be
> used for the directory part anyway and it's ignored for privileged
> processes.
>
> I guess we could stick 'path' into an auxiliary vector value and have
> ld.so do the realpath() call if $ORIGIN is used?  It would be that or
> have the kernel store the whole path for the life of the process for
> obtaining with sysctl().  Right now it only stores the last component
> of the (resolved) original path in p_comm.
>
>
> Philip Guenther
>
>
> ** i.e, not marked as 'secure', their
sorta-similar-to-issetugid-but-not-really



--
Aurélien Vallée
Phone +33 9 77 19 85 61

Reply | Threaded
Open this post in threaded view
|

Re: ld.so behavior with $ORIGIN

Theo de Raadt
In reply to this post by Philip Guenther-2
> On Fri, Dec 11, 2015 at 10:18 AM, Theo de Raadt <[hidden email]> wrote:
> >> Just found I can set LD_DEBUG to see the full translation process of ld.so.
> >> This seems to confirm what I've seen in the source: ld.so uses cwd
> >> instead of process file location for $ORIGIN interpolation.
> >              ^^^^^^^^^^^^^^^^^^^^^
> >
> > What is that?  Generally Unix has no way of doing this.  You can inspect
> > your executable's __progname, which is cloned from argv at crt startup,
> > but it may not contain an absolute path.  So what do you really mean here?
>
> I'm not sure what Solaris does for this, but it looks like glibc on
> Linux does readlink("/proc/self/exe") and if that fails and the
> process trusts its environment** then it falls back to the
> LD_ORIGIN_PATH environment variable.  $ORIGIN then expands to
> dirname() of that.

Oh, I was certain that is what this would do.

That is why I said "Generally Unix has no way of doing this".

/proc/self/exe, or anything else like it, is DEFINATELY NOT
in POSIX or any other cross-vendor Unix standard.  

> I guess we could stick 'path' into an auxiliary vector value and have
> ld.so do the realpath() call if $ORIGIN is used?  It would be that or
> have the kernel store the whole path for the life of the process for
> obtaining with sysctl().  Right now it only stores the last component
> of the (resolved) original path in p_comm.

Quite expensive, for such a small need.

Since Unix operating systems don't have a way to do this in general,
how did this become part of the ld.so spec?

Reply | Threaded
Open this post in threaded view
|

Re: ld.so behavior with $ORIGIN

Theo de Raadt
In reply to this post by Aurélien Vallée
> > It would be that or
> > have the kernel store the whole path for the life of the process for
> > obtaining with sysctl()
>
> That would be great. ps and top would be able to display the path too,
> pretty handy.

How did people get by without needing this in the last three decades?

Reply | Threaded
Open this post in threaded view
|

Re: ld.so behavior with $ORIGIN

Aurélien Vallée
> How did people get by without needing this in the last three decades?

Just trying to be positive on a feature that would have interest for me.
Nevermind.

It seems like most other unixes out there do have a way to retrieve the full
path of a running program, mainly through /proc (be it /proc/pid/exe on
Linux,
or /proc/pid/path/a.out on Solaris (TBV).

From what I have understood there is no /proc anymore of OpenBSD, and these
info are now accessible through sysctl, so I thought that would fit
nicely for this
feature too.

> Since Unix operating systems don't have a way to do this in general,
> how did this become part of the ld.so spec?

It is part of  the System V ELF gABI.
Current description:
http://www.sco.com/developers/gabi/latest/ch5.dynamic.html#substitution

Revision history available at:
http://www.sco.com/developers/gabi/latest/revision.html
It was added in 2nd draft from 1999 may 3:

> New dynamic section tags DT_RUNPATH and DT_FLAGS added. Dynamic section tag
DT_RPATH moved to level 2.
> New semantics for shared object path searching, including new ``Substitution
Sequences''.

On Fri, Dec 11, 2015 at 11:32 PM, Theo de Raadt <[hidden email]>
wrote:
>> > It would be that or
>> > have the kernel store the whole path for the life of the process for
>> > obtaining with sysctl()
>>
>> That would be great. ps and top would be able to display the path too,
>> pretty handy.
>
> How did people get by without needing this in the last three decades?



--
Aurélien Vallée
Phone +33 9 77 19 85 61

Reply | Threaded
Open this post in threaded view
|

Re: ld.so behavior with $ORIGIN

Raul Miller
On Sat, Dec 12, 2015 at 6:34 AM, Aurélien Vallée
<[hidden email]> wrote:
> It seems like most other unixes out there do have a way to retrieve the
full
> path of a running program, mainly through /proc (be it /proc/pid/exe on
> Linux, or /proc/pid/path/a.out on Solaris (TBV).

An issue, though, is that this path does not have to refer to a file
which has anything to do with what the process is doing.

Here's an illustration:

#!/bin/sh
set -e
mkdir -p $HOME/bin
rm -f $HOME/bin/example[12]
cp /bin/sleep $HOME/bin/example1
ln $HOME/bin/example1 $HOME/bin/example2
$HOME/bin/example2 100 &
rm -f $HOME/bin/example2
cp /usr/bin/false $HOME/bin/example2
echo "what is the full path for process $!?"

Seriously, though: understanding the unix process/file model is
critically important if you are going to make any useful contributions
to the implementation. But I am not sure where to refer you to get the
basic idea. (I got mine from reading source code comments for a very
early version of unix.)

All the docs I find through searching go into a lot of detail about
steps and procedures, but what I think you really need is something
cruder. I don't know where to point you for that, though - perhaps
such things have been lost to the mists of history?

Good luck,

--
Raul

Reply | Threaded
Open this post in threaded view
|

Re: ld.so behavior with $ORIGIN

Aurélien Vallée
>> It seems like most other unixes out there do have a way to retrieve the
full
>> path of a running program, mainly through /proc (be it /proc/pid/exe on
>> Linux, or /proc/pid/path/a.out on Solaris (TBV).

> An issue, though, is that this path does not have to refer to a file
> which has anything to do with what the process is doing.

You are right, apologizes for being very unclear on my wordings here.

I totally understand that there is no way to literally retrieve "the
path of the executable
currently running". That doesn't make sense, as you stated and
demonstrated. The file
could have been removed, replaced, the partition unmounted, or
whatever other million
of reasons. Maintaining a relationship between a path and a running
process on unix
doesn't make sense.

Allow me to rephrase it with a clearer wording then:
We need the path that was provided to execve(), and thus contained -
at that time - the
ELF that was loaded".

We don't need to - and can't - make any sort of guarantee on this
path, it is just informational, and
can be used by ld.so to perform $ORIGIN substitution.

On Sat, Dec 12, 2015 at 2:19 PM, Raul Miller <[hidden email]> wrote:
> On Sat, Dec 12, 2015 at 6:34 AM, Aurélien Vallée
> <[hidden email]> wrote:
>> It seems like most other unixes out there do have a way to retrieve the
full

>> path of a running program, mainly through /proc (be it /proc/pid/exe on
>> Linux, or /proc/pid/path/a.out on Solaris (TBV).
>
> An issue, though, is that this path does not have to refer to a file
> which has anything to do with what the process is doing.
>
> Here's an illustration:
>
> #!/bin/sh
> set -e
> mkdir -p $HOME/bin
> rm -f $HOME/bin/example[12]
> cp /bin/sleep $HOME/bin/example1
> ln $HOME/bin/example1 $HOME/bin/example2
> $HOME/bin/example2 100 &
> rm -f $HOME/bin/example2
> cp /usr/bin/false $HOME/bin/example2
> echo "what is the full path for process $!?"
>
> Seriously, though: understanding the unix process/file model is
> critically important if you are going to make any useful contributions
> to the implementation. But I am not sure where to refer you to get the
> basic idea. (I got mine from reading source code comments for a very
> early version of unix.)
>
> All the docs I find through searching go into a lot of detail about
> steps and procedures, but what I think you really need is something
> cruder. I don't know where to point you for that, though - perhaps
> such things have been lost to the mists of history?
>
> Good luck,
>
> --
> Raul



--
Aurélien Vallée
Phone +33 9 77 19 85 61

Reply | Threaded
Open this post in threaded view
|

Re: ld.so behavior with $ORIGIN

Aurélien Vallée
> I asked a question, and you didn't answer it.

I did my best to try to find this out, but all I was able to dig are these
release notes of ELF gABI.
I will try to find how that was supposed to be solved at the time, but
I fear these details are lost in time, and not available on much online
documents I could get a hand on.

> We could come up with a hack.

If you don't mind detailing the hack you're thinking of, I could try to make
a patch implementing it.

On Sat, Dec 12, 2015 at 3:23 PM, Theo de Raadt <[hidden email]>
wrote:
>> Allow me to rephrase it with a clearer wording then:
>> We need the path that was provided to execve(), and thus contained -
>> at that time - the
>> ELF that was loaded".
>
> And once again, I am saying Unix doesn't have a standardized way of
> doing this.
>
> We could come up with a hack.



--
Aurélien Vallée
Phone +33 9 77 19 85 61

Reply | Threaded
Open this post in threaded view
|

Re: ld.so behavior with $ORIGIN

Aurélien Vallée
Just as an update:
I just spent some time reading NetBSD source code to understand how
they are doing it. They seem to do exactly what Philip suggested: pack the
execname from kern_exec in the elf auxiliary vector, and dirname() that
on ld.so.

On Sat, Dec 12, 2015 at 4:08 PM, Aurélien Vallée
<[hidden email]> wrote:

>> I asked a question, and you didn't answer it.
>
> I did my best to try to find this out, but all I was able to dig are these
> release notes of ELF gABI.
> I will try to find how that was supposed to be solved at the time, but
> I fear these details are lost in time, and not available on much online
> documents I could get a hand on.
>
>> We could come up with a hack.
>
> If you don't mind detailing the hack you're thinking of, I could try to
make
> a patch implementing it.
>
> On Sat, Dec 12, 2015 at 3:23 PM, Theo de Raadt <[hidden email]>
wrote:

>>> Allow me to rephrase it with a clearer wording then:
>>> We need the path that was provided to execve(), and thus contained -
>>> at that time - the
>>> ELF that was loaded".
>>
>> And once again, I am saying Unix doesn't have a standardized way of
>> doing this.
>>
>> We could come up with a hack.
>
>
>
> --
> Aurélien Vallée
> Phone +33 9 77 19 85 61



--
Aurélien Vallée
Phone +33 9 77 19 85 61