Python modules installed by pip in virtualenv can't find shared objects.

  • Done
  • quality assurance status badge
Details
4 participants
  • Ludovic Courtès
  • Ricardo Wurmus
  • Fis Trivial
  • zimoun
Owner
unassigned
Submitted by
Fis Trivial
Severity
normal
F
F
Fis Trivial wrote on 23 Jan 2018 12:49
(name . bug-guix@gnu.org)(address . bug-guix@gnu.org)
MWHPR16MB0063656F9561DCD3EF12634592E30@MWHPR16MB0063.namprd16.prod.outlook.com
When a python module needs to load a dynamic shared object, it looks in the
path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this environment
variable to export the needed path for python.
* Backtrace
In my case, it's lightgbm (installed by pip), needs libgomp.so from gcc:lib for
openmp support. Here is the backtrace:
Toggle snippet (15 lines)
Traceback (most recent call last):
File "ex1.py", line 5, in <module>
import lightgbm as lgb
File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/__init__.py", line 8, in <module>
from .basic import Booster, Dataset
File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 32, in <module>
_LIB = _load_lib()
File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 27, in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 425, in LoadLibrary
return self._dlltype(name)
File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 347, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libgomp.so.1: cannot open shared object file: No such file or directory
* Reproduce
To install lightgbm, simply use `pip install lightgbm`(in virtualenv).
Then in python shell:
import lightgbm as lgb
* Ad-hoc
export LD_LIBRARY_PATH=~/.guix-profile/lib:$LD_LIBRARY_PATH
F
F
Fis Trivial wrote on 23 Jan 2018 14:21
(name . Ricardo Wurmus)(address . rekado@elephly.net)(name . 30229@debbugs.gnu.org)(address . 30229@debbugs.gnu.org)
MWHPR16MB0063FE6E9AB2B17F44A8B9A692E30@MWHPR16MB0063.namprd16.prod.outlook.com
Toggle quote (4 lines)
>
> In general, though, I recommend using Guix for package management and
> development instead of virtualenv and pip.
>
I understand the importance for having a self-contained dependency graph.
After trying guix a few months, I came to realize that, at the a theoretical
level, the environment variables are not treated as part of the dependency
graph in the Guix world. Dangling environment variables happens, and can not
be solved by installing everything with guix(1).
I really hope that we can come up with some methods to solve these problems
in a systematical way. I will keep reporting bugs caused by environment
variables, hoping that it might be helpful in future development.
L
L
Ludovic Courtès wrote on 24 Jan 2018 16:01
(name . Fis Trivial)(address . ybbs.daans@hotmail.com)(address . 30229@debbugs.gnu.org)
87k1w7xpux.fsf@gnu.org
Hi,

Fis Trivial <ybbs.daans@hotmail.com> skribis:

Toggle quote (18 lines)
> In my case, it's lightgbm (installed by pip), needs libgomp.so from gcc:lib for
> openmp support. Here is the backtrace:
>
> Traceback (most recent call last):
> File "ex1.py", line 5, in <module>
> import lightgbm as lgb
> File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/__init__.py", line 8, in <module>
> from .basic import Booster, Dataset
> File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 32, in <module>
> _LIB = _load_lib()
> File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 27, in _load_lib
> lib = ctypes.cdll.LoadLibrary(lib_path[0])
> File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 425, in LoadLibrary
> return self._dlltype(name)
> File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 347, in __init__
> self._handle = _dlopen(self._name, mode)
> OSError: libgomp.so.1: cannot open shared object file: No such file or directory

I’d suggest hard-coding the absolute file name of libgomp.so.1 in
lightgbm/basic.py.

This is what we do, for instance, for Guile packages that use the
foreign function interface (FFI) and dynamically load shared libraries
(see ‘guile-gdbm-ffi’ as an example.)

HTH,
Ludo’.
R
R
Ricardo Wurmus wrote on 23 Jan 2018 13:00
(name . Fis Trivial)(address . ybbs.daans@hotmail.com)(address . 30229@debbugs.gnu.org)
87h8rczsx3.fsf@elephly.net
Hi,

Toggle quote (4 lines)
> When a python module needs to load a dynamic shared object, it looks in the
> path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this environment
> variable to export the needed path for python.

We cannot set this environment variable by default lest we break other
packages that may be installed. LD_LIBRARY_PATH is dangerous as it
tells the runtime linker to prefer libraries in the specified
directories.

For Guix packages we use different means to embed store paths in
binaries, which are looked up at runtime. For binaries that’s achieved
with RUNPATH; for others we patch the sources to ensure that libraries
are not looked up merely by name but by absolute path.

In your particular case (installing packages without Guix) I think the
best way is to manually set LD_LIBRARY_PATH on demand, or to set
LD_PRELOAD to the specific libraries that are required.

In general, though, I recommend using Guix for package management and
development instead of virtualenv and pip.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
Z
Z
zimoun wrote on 25 Nov 2021 00:39
(name . Ricardo Wurmus)(address . rekado@elephly.net)
867dcxt97a.fsf@gmail.com
Hi,

On Tue, 23 Jan 2018 at 13:00, Ricardo Wurmus <rekado@elephly.net> wrote:

Toggle quote (21 lines)
>> When a python module needs to load a dynamic shared object, it looks in the
>> path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this environment
>> variable to export the needed path for python.
>
> We cannot set this environment variable by default lest we break other
> packages that may be installed. LD_LIBRARY_PATH is dangerous as it
> tells the runtime linker to prefer libraries in the specified
> directories.
>
> For Guix packages we use different means to embed store paths in
> binaries, which are looked up at runtime. For binaries that’s achieved
> with RUNPATH; for others we patch the sources to ensure that libraries
> are not looked up merely by name but by absolute path.
>
> In your particular case (installing packages without Guix) I think the
> best way is to manually set LD_LIBRARY_PATH on demand, or to set
> LD_PRELOAD to the specific libraries that are required.
>
> In general, though, I recommend using Guix for package management and
> development instead of virtualenv and pip.

Regarding the improvements of ’guix import pypi’ since 2018, and because
tweaking LD_LIBRARY_PATH is dangerous, I do not see what could be the
next action to solve this.

Therefore, I propose to close it. WDYT?


Cheers,
simon
Z
Z
zimoun wrote on 13 Jan 2022 15:59
(name . Ricardo Wurmus)(address . rekado@elephly.net)
864k67k7fl.fsf@gmail.com
Hi,

On Thu, 25 Nov 2021 at 00:39, zimoun <zimon.toutoune@gmail.com> wrote:
Toggle quote (30 lines)
> On Tue, 23 Jan 2018 at 13:00, Ricardo Wurmus <rekado@elephly.net> wrote:
>
>>> When a python module needs to load a dynamic shared object, it looks in the
>>> path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this
>>> environment
>>> variable to export the needed path for python.
>>
>> We cannot set this environment variable by default lest we break other
>> packages that may be installed. LD_LIBRARY_PATH is dangerous as it
>> tells the runtime linker to prefer libraries in the specified
>> directories.
>>
>> For Guix packages we use different means to embed store paths in
>> binaries, which are looked up at runtime. For binaries that’s achieved
>> with RUNPATH; for others we patch the sources to ensure that libraries
>> are not looked up merely by name but by absolute path.
>>
>> In your particular case (installing packages without Guix) I think the
>> best way is to manually set LD_LIBRARY_PATH on demand, or to set
>> LD_PRELOAD to the specific libraries that are required.
>>
>> In general, though, I recommend using Guix for package management and
>> development instead of virtualenv and pip.
>
> Regarding the improvements of ’guix import pypi’ since 2018, and because
> tweaking LD_LIBRARY_PATH is dangerous, I do not see what could be the
> next action to solve this.
>
> Therefore, I propose to close it. WDYT?

After 7 weeks of delay, I am closing.


Cheers,
simon
Closed
?