事情是这样子的,单细胞课程中有一个构建图谱的代码,其中有一个函数FindOptimalK(Jarning写的)在我负责的服务器运行的时候出现了如下问题。
然后这个问题困扰了学员半个月了,还是没解决,他已经将scipy库降级到很低的版本了,还是无法运行,查的这个方法需要sudo权限,尝试了拷贝别人的相关文件,但是没有权限,于是求助我。
如果你向GPT求助,它会建议你升级,但是作为系统管理员,我知道这玩意可不能随便拷贝,而且也不能随便升级,可能会牵一发动全身,导致其他软件无法正常运行(我之前升级过glibc,导致服务器连ls都用不了)
我尝试去定位了问题,发现在导入cnmf的时候,会出现如下报错,实际上就是scipy依赖于的libstdc的版本比较高。
> reticulate::import("cnmf")
Error in py_module_import(module, convert = convert) :
ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found
(required by /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/_ckdtree.cpython-39-x86_64-linux-gnu.so)
Run `reticulate::py_last_error()` for details.
然而,启动对应环境,然后导入的时候,并没有问题
conda activate ~/.local/share/r-miniconda/envs/r-reticulate
python -c "import cnmf"
因为我检查了对应了动态库的依赖情况,确实能够对应到conda中的libstdc
ldd /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/_ckdtree.cpython-39-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffd4d5d4000)
libstdc++.so.6 => /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/../../../../libstdc++.so.6 (0x00007fc050e29000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc050cb9000)
libgcc_s.so.1 => /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/../../../../libgcc_s.so.1 (0x00007fc050c9e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc050aac000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc050aa2000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc0510b6000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc050a7d000)
这就奇怪了,且用户没有设置LD_LIBRARY_PATH
echo $LD_LIBRARY_PATH
# 无显示
但是,我打开R之后,却发现里面有LD_LIBRARY_PATH,并且出现了/usr/lib/x86_64-linux-gnu
,根据我多年的经验,肯定是R优先用了/usr/lib/x86_64-linux-gnu
的libstd,而不是conda的
> Sys.getenv("LD_LIBRARY_PATH")
[1] "/opt/R/4.3.2/lib/R/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server"
可我已经检查过了,用户没有设置LD_LIBRARY_PATH啊,所以,我就手动配置环境
echo 'LD_LIBRARY_PATH=/opt/R/4.3.2/lib/R/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib' >> ~/.Renviron
路径对了,但是结果还是出错了(我甚至都在R里面设置环境变量了)
> Sys.getenv("LD_LIBRARY_PATH")
[1] "/opt/R/4.3.2/lib/R/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib"
> reticulate::import("cnmf")
Error in py_module_import(module, convert = convert) :
ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/_ckdtree.cpython-39-x86_64-linux-gnu.so)
Run `reticulate::py_last_error()` for details.
> Sys.setenv("LD_LIBRARY_PATH"="/opt/R/4.3.2/lib/R/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib")
> reticulate::import("cnmf")
Error in py_module_import(module, convert = convert) :
ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/_ckdtree.cpython-39-x86_64-linux-gnu.so)
Run `reticulate::py_last_error()` for details.
我就非常纳闷了,到底咋回事啊,通过R调用python,然后查看LD_LIBRARY_PATH,里面也没有发现/usr/lib/x86_64-linux-gnu
啊,为啥还是出错啊!
# 加载reticulate包
library(reticulate)
# 使用Python的os模块获取环境变量
os <- import("os")
ld_library_path <- os$getenv('LD_LIBRARY_PATH')
print(ld_library_path)
[1] "/opt/R/4.3.2/lib/R/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib"
GPT4给了我一些建议,让我检查如下的文件,是不是里面设置了LD_LIBRARY_PATH,可是我发了个遍,都没有找到
ls /etc/profile.d/
cat /etc/profile.d/*
cat /etc/environment
cat ~/.bashrc
cat ~/.bash_profile
cat ~/.profile
cat ~/.zshrc
cat ~/.Rprofile
cat /opt/R/4.3.2/lib/R/etc/Rprofile.site
正当我一筹莫展的时候,我在/opt/R/4.3.2/lib/R/etc
里ls了一下,发现一个ldpaths,
: ${JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre}
: ${R_JAVA_LD_LIBRARY_PATH=${JAVA_HOME}/lib/amd64/server}
if test -n "/usr/local/lib:/usr/lib/x86_64-linux-gnu"; then
: ${R_LD_LIBRARY_PATH=${R_HOME}/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu}
else
: ${R_LD_LIBRARY_PATH=${R_HOME}/lib}
fi
if test -n "${R_JAVA_LD_LIBRARY_PATH}"; then
R_LD_LIBRARY_PATH="${R_LD_LIBRARY_PATH}:${R_JAVA_LD_LIBRARY_PATH}"
fi
## This is DYLD_FALLBACK_LIBRARY_PATH on Darwin (macOS) and
## LD_LIBRARY_PATH elsewhere.
## However, on macOS >=10.11 (if SIP is enabled, the default), the
## environment value will not be passed to a script such as R.sh, so
## would not seen here.
if test -z "${LD_LIBRARY_PATH}"; then
LD_LIBRARY_PATH="${R_LD_LIBRARY_PATH}"
else
LD_LIBRARY_PATH="${R_LD_LIBRARY_PATH}:${LD_LIBRARY_PATH}"
fi
export LD_LIBRARY_PATH
好家伙,原来是你啊,于是我注释了其中三行,确保${R_LD_LIBRARY_PATH=${R_HOME}/lib}
,避免引入/usr/lib/x86_64-linux-gnu
这个害人精
: ${JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre}
: ${R_JAVA_LD_LIBRARY_PATH=${JAVA_HOME}/lib/amd64/server}
#if test -n "/usr/local/lib:/usr/lib/x86_64-linux-gnu"; then
#: ${R_LD_LIBRARY_PATH=${R_HOME}/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu}
#else
: ${R_LD_LIBRARY_PATH=${R_HOME}/lib}
#fi
if test -n "${R_JAVA_LD_LIBRARY_PATH}"; then
R_LD_LIBRARY_PATH="${R_LD_LIBRARY_PATH}:${R_JAVA_LD_LIBRARY_PATH}"
fi
## This is DYLD_FALLBACK_LIBRARY_PATH on Darwin (macOS) and
## LD_LIBRARY_PATH elsewhere.
## However, on macOS >=10.11 (if SIP is enabled, the default), the
## environment value will not be passed to a script such as R.sh, so
## would not seen here.
if test -z "${LD_LIBRARY_PATH}"; then
LD_LIBRARY_PATH="${R_LD_LIBRARY_PATH}"
else
LD_LIBRARY_PATH="${R_LD_LIBRARY_PATH}:${LD_LIBRARY_PATH}"
fi
export LD_LIBRARY_PATH
我原以为没问题了,但是还是出错了!
> Sys.getenv("LD_LIBRARY_PATH")
[1] "/opt/R/4.3.2/lib/R/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server"
> reticulate::import("cnmf")
Error in py_module_import(module, convert = convert) :
ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib/python3.9/site-packages/scipy/spatial/_ckdtree.cpython-39-x86_64-linux-gnu.so)
Run `reticulate::py_last_error()` for details.
这为啥啊!不应该啊,我不是已经删掉了吗?怎么还是阴魂不散啊!
但是,突然间,我灵机一动,会不会是需要自己加一个LD_LIBRARY_PATH,加上r-reticulate的动态库,如下
export LD_LIBRARY_PATH=/home/data/gz0310/.local/share/r-miniconda/envs/r-reticulate/lib
然后,这个问题就被我解决了。
> reticulate::import("cnmf")
Module(cnmf)
这个问题折磨了我快4个小时,好在最后解决了,我是真的没想到,R里面居然会偷偷摸摸给我加LD_LIBRARY_PATH,就算你加了,你好歹把用户的LD_LIBRARY_PATH
放在前面啊!
PS:除了上面我写的一些尝试外,我还做了如下的尝试
例如,设置其他conda路径,但还是出错
library(reticulate)
#use_virtualenv("myenv")
use_condaenv("/home/data/gz0310/mambaforge/envs/cnmf_env")
reticulate::import("cnmf")
我本来是打算删除libapths,但是删了就会导致R无法正常运行。
mv /opt/R/4.3.2/lib/R/etc/ldpaths /opt/R/4.3.2/lib/R/etc/ldpaths.bak