R会自动配置LD_LIBRARY_PATH

事情是这样子的,单细胞课程中有一个构建图谱的代码,其中有一个函数FindOptimalK(Jarning写的)在我负责的服务器运行的时候出现了如下问题。

然后这个问题困扰了学员半个月了,还是没解决,他已经将scipy库降级到很低的版本了,还是无法运行,查的这个方法需要sudo权限,尝试了拷贝别人的相关文件,但是没有权限,于是求助我。

如果你向GPT求助,它会建议你升级,但是作为系统管理员,我知道这玩意可不能随便拷贝,而且也不能随便升级,可能会牵一发动全身,导致其他软件无法正常运行(我之前升级过glibc,导致服务器连ls都用不了)

我尝试去定位了问题,发现在导入cnmf的时候,会出现如下报错,实际上就是scipy依赖于的libstdc的版本比较高。

> reticulate::import("cnmf")
Error in py_module_import(module, convert = convert) : 
  ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found 
  (required by /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/_ckdtree.cpython-39-x86_64-linux-gnu.so)
Run `reticulate::py_last_error()` for details.

然而,启动对应环境,然后导入的时候,并没有问题

conda activate ~/.local/share/r-miniconda/envs/r-reticulate
python -c "import cnmf"

因为我检查了对应了动态库的依赖情况,确实能够对应到conda中的libstdc

ldd /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/_ckdtree.cpython-39-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffd4d5d4000)
        libstdc++.so.6 => /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/../../../../libstdc++.so.6 (0x00007fc050e29000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc050cb9000)
        libgcc_s.so.1 => /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/../../../../libgcc_s.so.1 (0x00007fc050c9e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc050aac000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc050aa2000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc0510b6000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc050a7d000)

这就奇怪了,且用户没有设置LD_LIBRARY_PATH

echo $LD_LIBRARY_PATH
# 无显示

但是,我打开R之后,却发现里面有LD_LIBRARY_PATH,并且出现了/usr/lib/x86_64-linux-gnu ,根据我多年的经验,肯定是R优先用了/usr/lib/x86_64-linux-gnu的libstd,而不是conda的

> Sys.getenv("LD_LIBRARY_PATH")
[1] "/opt/R/4.3.2/lib/R/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server"

可我已经检查过了,用户没有设置LD_LIBRARY_PATH啊,所以,我就手动配置环境

echo 'LD_LIBRARY_PATH=/opt/R/4.3.2/lib/R/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib' >> ~/.Renviron

路径对了,但是结果还是出错了(我甚至都在R里面设置环境变量了)

> Sys.getenv("LD_LIBRARY_PATH")
[1] "/opt/R/4.3.2/lib/R/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib"
> reticulate::import("cnmf")
Error in py_module_import(module, convert = convert) : 
  ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/_ckdtree.cpython-39-x86_64-linux-gnu.so)
Run `reticulate::py_last_error()` for details.

> Sys.setenv("LD_LIBRARY_PATH"="/opt/R/4.3.2/lib/R/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib")
> reticulate::import("cnmf")
Error in py_module_import(module, convert = convert) : 
  ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/_ckdtree.cpython-39-x86_64-linux-gnu.so)
Run `reticulate::py_last_error()` for details.

我就非常纳闷了,到底咋回事啊,通过R调用python,然后查看LD_LIBRARY_PATH,里面也没有发现/usr/lib/x86_64-linux-gnu啊,为啥还是出错啊!

# 加载reticulate包
library(reticulate)

# 使用Python的os模块获取环境变量
os <- import("os")
ld_library_path <- os$getenv('LD_LIBRARY_PATH')
print(ld_library_path)
[1] "/opt/R/4.3.2/lib/R/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib"

GPT4给了我一些建议,让我检查如下的文件,是不是里面设置了LD_LIBRARY_PATH,可是我发了个遍,都没有找到

ls /etc/profile.d/
cat /etc/profile.d/*
cat /etc/environment
cat ~/.bashrc
cat ~/.bash_profile
cat ~/.profile
cat ~/.zshrc
cat ~/.Rprofile
cat /opt/R/4.3.2/lib/R/etc/Rprofile.site

正当我一筹莫展的时候,我在/opt/R/4.3.2/lib/R/etc里ls了一下,发现一个ldpaths,

: ${JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre}
: ${R_JAVA_LD_LIBRARY_PATH=${JAVA_HOME}/lib/amd64/server}
if test -n "/usr/local/lib:/usr/lib/x86_64-linux-gnu"; then
: ${R_LD_LIBRARY_PATH=${R_HOME}/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu}
else
: ${R_LD_LIBRARY_PATH=${R_HOME}/lib}
fi
if test -n "${R_JAVA_LD_LIBRARY_PATH}"; then
  R_LD_LIBRARY_PATH="${R_LD_LIBRARY_PATH}:${R_JAVA_LD_LIBRARY_PATH}"
fi
## This is DYLD_FALLBACK_LIBRARY_PATH on Darwin (macOS) and
## LD_LIBRARY_PATH elsewhere.
## However, on macOS >=10.11 (if SIP is enabled, the default), the
## environment value will not be passed to a script such as R.sh, so
## would not seen here.
if test -z "${LD_LIBRARY_PATH}"; then
  LD_LIBRARY_PATH="${R_LD_LIBRARY_PATH}"
else
  LD_LIBRARY_PATH="${R_LD_LIBRARY_PATH}:${LD_LIBRARY_PATH}"
fi
export LD_LIBRARY_PATH

好家伙,原来是你啊,于是我注释了其中三行,确保${R_LD_LIBRARY_PATH=${R_HOME}/lib},避免引入/usr/lib/x86_64-linux-gnu这个害人精

: ${JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre}
: ${R_JAVA_LD_LIBRARY_PATH=${JAVA_HOME}/lib/amd64/server}
#if test -n "/usr/local/lib:/usr/lib/x86_64-linux-gnu"; then
#: ${R_LD_LIBRARY_PATH=${R_HOME}/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu}
#else
: ${R_LD_LIBRARY_PATH=${R_HOME}/lib}
#fi
if test -n "${R_JAVA_LD_LIBRARY_PATH}"; then
  R_LD_LIBRARY_PATH="${R_LD_LIBRARY_PATH}:${R_JAVA_LD_LIBRARY_PATH}"
fi
## This is DYLD_FALLBACK_LIBRARY_PATH on Darwin (macOS) and
## LD_LIBRARY_PATH elsewhere.
## However, on macOS >=10.11 (if SIP is enabled, the default), the
## environment value will not be passed to a script such as R.sh, so
## would not seen here.
if test -z "${LD_LIBRARY_PATH}"; then
  LD_LIBRARY_PATH="${R_LD_LIBRARY_PATH}"
else
  LD_LIBRARY_PATH="${R_LD_LIBRARY_PATH}:${LD_LIBRARY_PATH}"
fi
export LD_LIBRARY_PATH

我原以为没问题了,但是还是出错了!

> Sys.getenv("LD_LIBRARY_PATH")
[1] "/opt/R/4.3.2/lib/R/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server"
> reticulate::import("cnmf")
Error in py_module_import(module, convert = convert) : 
  ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/_ckdtree.cpython-39-x86_64-linux-gnu.so)
Run `reticulate::py_last_error()` for details.

这为啥啊!不应该啊,我不是已经删掉了吗?怎么还是阴魂不散啊!

但是,突然间,我灵机一动,会不会是需要自己加一个LD_LIBRARY_PATH,加上r-reticulate的动态库,如下

export LD_LIBRARY_PATH=/home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib

然后,这个问题就被我解决了。

> reticulate::import("cnmf")
Module(cnmf)

这个问题折磨了我快4个小时,好在最后解决了,我是真的没想到,R里面居然会偷偷摸摸给我加LD_LIBRARY_PATH,就算你加了,你好歹把用户的LD_LIBRARY_PATH 放在前面啊!

PS:除了上面我写的一些尝试外,我还做了如下的尝试

例如,设置其他conda路径,但还是出错

library(reticulate)
#use_virtualenv("myenv")
use_condaenv("/home/data/gz0310/mambaforge/envs/cnmf_env")
reticulate::import("cnmf")

我本来是打算删除libapths,但是删了就会导致R无法正常运行。

mv /opt/R/4.3.2/lib/R/etc/ldpaths /opt/R/4.3.2/lib/R/etc/ldpaths.bak

评论

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×