超算服务器上CUDA无法正常使用

AquaIndigo · 2021 年5 月 13 日 01:10

报错信息


  Activating environment at `~/codes/KMarkovGenSeq.jl/Project.toml`
     Project KMarkovGenSeq v0.1.0
      Status `~/codes/KMarkovGenSeq.jl/Project.toml`
  [052768ef] CUDA v2.6.3
  [e30172f5] Documenter v0.26.3
  [c2308a5c] FASTX v1.1.3
  [0337cf30] GRUtils v0.6.1
  [295af30f] Revise v3.1.16
  [37e2e46d] LinearAlgebra
  Activating environment at `~/codes/KMarkovGenSeq.jl/Project.toml`
┌ Debug: Initializing CUDA driver
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:88
┌ Debug: Trying to use artifacts...
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:137
┌ Debug: Selecting artifacts based on driver compatibility 10.2.0
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:149
 Downloading artifact: CUDA102
 Downloading artifact: CUDA102
┌ Debug: Could not load the CUDA 10.2.0 artifact
│   exception =
│    Unable to automatically install 'CUDA102' from '/home/ug2018/ug518111910129/.julia/packages/CUDA/M4jkK/Artifacts.toml'
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:33
│      [2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::IOStream)
│        @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:443
│      [3] ensure_artifact_installed(name::String, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, pkg_uuid::Nothing, verbose::Bool, quiet_download::Bool, io::IOStream)
│        @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:406
│      [4] _artifact_str(__module__::Module, artifacts_toml::String, name::SubString{String}, path_tail::String, artifact_dict::Dict{String, Any}, hash::Base.SHA1, platform::Base.BinaryPlatforms.Platform, lazyartifacts::Any)
│        @ Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Artifacts/src/Artifacts.jl:547
│      [5] #invokelatest#2
│        @ ./essentials.jl:708 [inlined]
│      [6] invokelatest
│        @ ./essentials.jl:706 [inlined]
│      [7] macro expansion
│        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Artifacts/src/Artifacts.jl:689 [inlined]
│      [8] lazy_artifact(x::String)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:124
│      [9] (::CUDA.var"#83#88")()
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:132
│     [10] use_artifact_cuda()
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:160
│     [11] __init_dependencies__()
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:382
│     [12] __runtime_init__()
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:103
│     [13] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:32 [inlined]
│     [14] macro expansion
│        @ ./lock.jl:209 [inlined]
│     [15] _functional(show_reason::Bool)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:26
│     [16] functional(show_reason::Bool)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:19
│     [17] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:47 [inlined]
│     [18] toolkit_version
│        @ ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:26 [inlined]
│     [19] versioninfo(io::IOStream) (repeats 2 times)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/utilities.jl:43
│     [20] top-level scope
│        @ none:1
│     [21] eval
│        @ ./boot.jl:360 [inlined]
│     [22] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:261
│     [23] _start()
│        @ Base ./client.jl:485
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:164
 Downloading artifact: CUDA101
 Downloading artifact: CUDA101
┌ Debug: Could not load the CUDA 10.1.0 artifact
│   exception =
│    Unable to automatically install 'CUDA101' from '/home/ug2018/ug518111910129/.julia/packages/CUDA/M4jkK/Artifacts.toml'
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:33
│      [2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::IOStream)
│        @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:443
│      [3] ensure_artifact_installed(name::String, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, pkg_uuid::Nothing, verbose::Bool, quiet_download::Bool, io::IOStream)
│        @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:406
│      [4] _artifact_str(__module__::Module, artifacts_toml::String, name::SubString{String}, path_tail::String, artifact_dict::Dict{String, Any}, hash::Base.SHA1, platform::Base.BinaryPlatforms.Platform, lazyartifacts::Any)
│        @ Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Artifacts/src/Artifacts.jl:547
│      [5] #invokelatest#2
│        @ ./essentials.jl:708 [inlined]
│      [6] invokelatest
│        @ ./essentials.jl:706 [inlined]
│      [7] macro expansion
│        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Artifacts/src/Artifacts.jl:689 [inlined]
│      [8] lazy_artifact(x::String)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:124
│      [9] (::CUDA.var"#84#89")()
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:133
│     [10] use_artifact_cuda()
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:160
│     [11] __init_dependencies__()
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:382
│     [12] __runtime_init__()
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:103
│     [13] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:32 [inlined]
│     [14] macro expansion
│        @ ./lock.jl:209 [inlined]
│     [15] _functional(show_reason::Bool)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:26
│     [16] functional(show_reason::Bool)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:19
│     [17] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:47 [inlined]
│     [18] toolkit_version
│        @ ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:26 [inlined]
│     [19] versioninfo(io::IOStream) (repeats 2 times)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/utilities.jl:43
│     [20] top-level scope
│        @ none:1
│     [21] eval
│        @ ./boot.jl:360 [inlined]
│     [22] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:261
│     [23] _start()
│        @ Base ./client.jl:485
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:164
┌ Debug: Could not find a compatible artifact.
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:168
┌ Debug: Trying to use local installation...
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:204
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_HOME
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:379
┌ Debug: Request to look for binary nvdisasm
│   locations =
│    1-element Vector{String}:
│     "/usr/local/cuda"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:118
┌ Debug: Looking for binary nvdisasm
│   locations =
│    2-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/bin"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:128
┌ Debug: Found binary nvdisasm at /usr/local/cuda-9.0/bin/nvdisasm
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:134
┌ Debug: Request to look for binary cuda-memcheck
│   locations =
│    1-element Vector{String}:
│     "/usr/local/cuda"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:118
┌ Debug: Looking for binary cuda-memcheck
│   locations =
│    2-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/bin"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:128
┌ Debug: Found binary cuda-memcheck at /usr/local/cuda-9.0/bin/cuda-memcheck
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:134
┌ Debug: CUDA toolkit identified as 9.0.176
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:470
┌ Debug: Request to look for library cublas 9.0.176
│   locations =
│    1-element Vector{String}:
│     "/usr/local/cuda"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:79
┌ Debug: Looking for library libcublas.so.9.0, libcublas.so.9
│   locations =
│    4-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/lib"
│     "/usr/local/cuda/lib64"
│     "/usr/local/cuda/libx64"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:99
┌ Debug: Found library libcublas.so.9.0 at /usr/local/cuda/lib64
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:107
┌ Debug: Request to look for library cusparse 9.0.176
│   locations =
│    1-element Vector{String}:
│     "/usr/local/cuda"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:79
┌ Debug: Looking for library libcusparse.so.9.0, libcusparse.so.9
│   locations =
│    4-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/lib"
│     "/usr/local/cuda/lib64"
│     "/usr/local/cuda/libx64"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:99
┌ Debug: Found library libcusparse.so.9.0 at /usr/local/cuda/lib64
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:107
┌ Debug: Request to look for library cusolver 9.0.176
│   locations =
│    1-element Vector{String}:
│     "/usr/local/cuda"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:79
┌ Debug: Looking for library libcusolver.so.9.0, libcusolver.so.9
│   locations =
│    4-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/lib"
│     "/usr/local/cuda/lib64"
│     "/usr/local/cuda/libx64"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:99
┌ Debug: Found library libcusolver.so.9.0 at /usr/local/cuda/lib64
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:107
┌ Debug: Request to look for library cufft 9.0.176
│   locations =
│    1-element Vector{String}:
│     "/usr/local/cuda"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:79
┌ Debug: Looking for library libcufft.so.9.0, libcufft.so.9
│   locations =
│    4-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/lib"
│     "/usr/local/cuda/lib64"
│     "/usr/local/cuda/libx64"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:99
┌ Debug: Found library libcufft.so.9.0 at /usr/local/cuda/lib64
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:107
┌ Debug: Request to look for library curand 9.0.176
│   locations =
│    1-element Vector{String}:
│     "/usr/local/cuda"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:79
┌ Debug: Looking for library libcurand.so.9.0, libcurand.so.9
│   locations =
│    4-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/lib"
│     "/usr/local/cuda/lib64"
│     "/usr/local/cuda/libx64"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:99
┌ Debug: Found library libcurand.so.9.0 at /usr/local/cuda/lib64
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:107
┌ Debug: Request to look for library cupti 9.0.176
│   locations =
│    2-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/extras/CUPTI"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:79
┌ Debug: Looking for library libcupti.so.9.0, libcupti.so.9
│   locations =
│    8-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/lib"
│     "/usr/local/cuda/lib64"
│     "/usr/local/cuda/libx64"
│     "/usr/local/cuda/extras/CUPTI"
│     "/usr/local/cuda/extras/CUPTI/lib"
│     "/usr/local/cuda/extras/CUPTI/lib64"
│     "/usr/local/cuda/extras/CUPTI/libx64"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:99
┌ Debug: Found library libcupti.so.9.0 at /usr/local/cuda/extras/CUPTI/lib64
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:107
┌ Debug: Request to look for library nvToolsExt 1.0.0
│   locations =
│    2-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/extras/CUPTI"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:79
┌ Debug: Looking for library libnvToolsExt.so.1.0, libnvToolsExt.so.1
│   locations =
│    8-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/lib"
│     "/usr/local/cuda/lib64"
│     "/usr/local/cuda/libx64"
│     "/usr/local/cuda/extras/CUPTI"
│     "/usr/local/cuda/extras/CUPTI/lib"
│     "/usr/local/cuda/extras/CUPTI/lib64"
│     "/usr/local/cuda/extras/CUPTI/libx64"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:99
┌ Debug: Found library libnvToolsExt.so.1 at /usr/local/cuda/lib64
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:107
┌ Debug: Request to look for libcudadevrt 
│   locations =
│    2-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/extras/CUPTI"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:517
┌ Debug: Looking for CUDA device runtime library libcudadevrt.a
│   locations =
│    6-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/lib"
│     "/usr/local/cuda/lib64"
│     "/usr/local/cuda/extras/CUPTI"
│     "/usr/local/cuda/extras/CUPTI/lib"
│     "/usr/local/cuda/extras/CUPTI/lib64"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:546
┌ Debug: Found CUDA device runtime library libcudadevrt.a at /usr/local/cuda/lib64
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:553
┌ Debug: Request to look for libdevice
│   locations =
│    2-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/extras/CUPTI"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:484
┌ Debug: Look for libdevice
│   locations =
│    3-element Vector{String}:
│     "/usr/local/cuda-9.0"
│     "/usr/local/cuda/nvvm/libdevice"
│     "/usr/local/cuda/extras/CUPTI"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:496
┌ Debug: Found unified device library at /usr/local/cuda/nvvm/libdevice/libdevice.10.bc
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:501
┌ Debug: Found local CUDA 9.0.176 at /usr/local/cuda, /usr/local/cuda/extras/CUPTI
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:266
┌ Debug: Request to look for library cudnn 8.0.0
│   locations =
│    2-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/extras/CUPTI"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:79
┌ Debug: Looking for library libcudnn.so.8.0, libcudnn.so.8
│   locations =
│    8-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/lib"
│     "/usr/local/cuda/lib64"
│     "/usr/local/cuda/libx64"
│     "/usr/local/cuda/extras/CUPTI"
│     "/usr/local/cuda/extras/CUPTI/lib"
│     "/usr/local/cuda/extras/CUPTI/lib64"
│     "/usr/local/cuda/extras/CUPTI/libx64"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:99
┌ Debug: Request to look for library cutensor 1.0.0
│   locations =
│    2-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/extras/CUPTI"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:79
┌ Debug: Looking for library libcutensor.so.1.0, libcutensor.so.1
│   locations =
│    8-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/lib"
│     "/usr/local/cuda/lib64"
│     "/usr/local/cuda/libx64"
│     "/usr/local/cuda/extras/CUPTI"
│     "/usr/local/cuda/extras/CUPTI/lib"
│     "/usr/local/cuda/extras/CUPTI/lib64"
│     "/usr/local/cuda/extras/CUPTI/libx64"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:99
┌ Debug: Request to look for library cutensor nothing
│   locations =
│    2-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/extras/CUPTI"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:79
┌ Debug: Looking for library libcutensor.so
│   locations =
│    8-element Vector{String}:
│     "/usr/local/cuda"
│     "/usr/local/cuda/lib"
│     "/usr/local/cuda/lib64"
│     "/usr/local/cuda/libx64"
│     "/usr/local/cuda/extras/CUPTI"
│     "/usr/local/cuda/extras/CUPTI/lib"
│     "/usr/local/cuda/extras/CUPTI/lib64"
│     "/usr/local/cuda/extras/CUPTI/libx64"
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/discovery.jl:99
┌ Warning: This version of CUDA.jl only supports CUDA 10.1 or higher (your toolkit provides CUDA 9.0.0)
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:106
┌ Debug: Toolchain with LLVM 11.0.1, CUDA driver 10.2.0 and toolkit 9.0.176 supports devices 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2 and 7.0; PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0 and 6.0
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/compatibility.jl:235
┌ Error: Error during initialization of CUDA.jl
│   exception =
│    could not load symbol "cublasSetLoggerCallback":
│    /usr/local/cuda/lib64/libcublas.so.9.0: undefined symbol: cublasSetLoggerCallback
│    Stacktrace:
│      [1] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/lib/cublas/libcublas.jl:99 [inlined]
│      [2] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/src/pool.jl:408 [inlined]
│      [3] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/lib/cublas/error.jl:56 [inlined]
│      [4] cublasSetLoggerCallback(userCallback::Ptr{Nothing})
│        @ CUDA.CUBLAS ~/.julia/packages/CUDA/M4jkK/lib/utils/call.jl:26
│      [5] __runtime_init__()
│        @ CUDA.CUBLAS ~/.julia/packages/CUDA/M4jkK/lib/cublas/CUBLAS.jl:162
│      [6] __runtime_init__()
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:134
│      [7] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:32 [inlined]
│      [8] macro expansion
│        @ ./lock.jl:209 [inlined]
│      [9] _functional(show_reason::Bool)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:26
│     [10] functional(show_reason::Bool)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:19
│     [11] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:47 [inlined]
│     [12] toolkit_version
│        @ ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:26 [inlined]
│     [13] versioninfo(io::IOStream) (repeats 2 times)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/utilities.jl:43
│     [14] top-level scope
│        @ none:1
│     [15] eval
│        @ ./boot.jl:360 [inlined]
│     [16] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:261
│     [17] _start()
│        @ Base ./client.jl:485
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:34
ERROR: CUDA.jl did not successfully initialize, and is not usable.
If you did not see any other error message, try again in a new session
with the JULIA_DEBUG environment variable set to 'CUDA'.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] macro expansion
   @ ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:48 [inlined]
 [3] toolkit_version
   @ ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:26 [inlined]
 [4] versioninfo(io::IOStream) (repeats 2 times)
   @ CUDA ~/.julia/packages/CUDA/M4jkK/src/utilities.jl:43
 [5] top-level scope
   @ none:1
Host key verification failed.

环境信息


Julia Version 1.6.1
Commit 6aaedecc44 (2021-04-23 05:59 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake-avx512)
Environment:
  JULIA_NUM_THREAD = 4
  JULIA_PKG_SERVER = https://mirrors.sjtug.sjtu.edu.cn/julia
  JULIA_GR_PROVIDER = BinaryBuilder
nothing
nothing
Thu May 13 09:12:46 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.14       Driver Version: 430.14       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  On   | 00000000:3D:00.0 Off |                  N/A |
| 23%   38C    P2   107W / 250W |    229MiB / 11178MiB |     80%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  On   | 00000000:3E:00.0 Off |                  N/A |
| 23%   13C    P8     7W / 250W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 108...  On   | 00000000:3F:00.0 Off |                  N/A |
| 23%   16C    P8     8W / 250W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 108...  On   | 00000000:40:00.0 Off |                  N/A |
| 23%   22C    P8     8W / 250W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     82081      C   pmemd.cuda                                   219MiB |
+-----------------------------------------------------------------------------+
/usr/local/cuda/bin/nvcc

提交脚本：

julia -e "using InteractiveUtils; versioninfo() |> println"
julia -e "using Pkg; Pkg.status() |> println"
nvidia-smi
which nvcc
JULIA_DEBUG=CUDA julia -e "using CUDA; println(CUDA.versioninfo())"

Jun · 2021 年5 月 13 日 04:10

你服务器上装的是cuda9 有点老了，估计驱动啥的不匹配

xgdgsc · 2021 年5 月 13 日 06:20

没挂梯子吧？ zzzz

johnnychen94 · 2021 年5 月 13 日 06:22

这里设置了 Pkg 镜像应该不需要梯子。话说 CUDA.jl 能否设置使用系统自带的 CUDA呀？

AquaIndigo · 2021 年5 月 13 日 06:24

nvidia-smi显示的是

NVIDIA-SMI 430.14 Driver Version: 430.14 CUDA Version: 10.2

xgdgsc · 2021 年5 月 13 日 06:28

https://juliagpu.gitlab.io/CUDA.jl/installation/overview/#Local-installation

AquaIndigo · 2021 年5 月 13 日 06:32

设置

ENV["JULIA_CUDA_USE_BINARYBUILDER"] = false

后，再提交会显示

┌ Warning: This version of CUDA.jl only supports CUDA 10.1 or higher (your toolkit provides CUDA 9.0.0)
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:106
┌ Debug: Toolchain with LLVM 11.0.1, CUDA driver 10.2.0 and toolkit 9.0.176 supports devices 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2 and 7.0; PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0 and 6.0
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/deps/compatibility.jl:235
┌ Error: Error during initialization of CUDA.jl
│   exception =
│    could not load symbol "cublasSetLoggerCallback":
│    /usr/local/cuda/lib64/libcublas.so.9.0: undefined symbol: cublasSetLoggerCallback
│    Stacktrace:
│      [1] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/lib/cublas/libcublas.jl:99 [inlined]
│      [2] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/src/pool.jl:408 [inlined]
│      [3] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/lib/cublas/error.jl:56 [inlined]
│      [4] cublasSetLoggerCallback(userCallback::Ptr{Nothing})
│        @ CUDA.CUBLAS ~/.julia/packages/CUDA/M4jkK/lib/utils/call.jl:26
│      [5] __runtime_init__()
│        @ CUDA.CUBLAS ~/.julia/packages/CUDA/M4jkK/lib/cublas/CUBLAS.jl:162
│      [6] __runtime_init__()
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:134
│      [7] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:32 [inlined]
│      [8] macro expansion
│        @ ./lock.jl:209 [inlined]
│      [9] _functional(show_reason::Bool)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:26
│     [10] functional(show_reason::Bool)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:19
│     [11] macro expansion
│        @ ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:47 [inlined]
│     [12] toolkit_version
│        @ ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:26 [inlined]
│     [13] versioninfo(io::IOStream) (repeats 2 times)
│        @ CUDA ~/.julia/packages/CUDA/M4jkK/src/utilities.jl:43
│     [14] top-level scope
│        @ none:1
│     [15] eval
│        @ ./boot.jl:360 [inlined]
│     [16] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:261
│     [17] _start()
│        @ Base ./client.jl:485
└ @ CUDA ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:34
ERROR: CUDA.jl did not successfully initialize, and is not usable.
If you did not see any other error message, try again in a new session
with the JULIA_DEBUG environment variable set to 'CUDA'.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] macro expansion
   @ ~/.julia/packages/CUDA/M4jkK/src/initialization.jl:48 [inlined]
 [3] toolkit_version
   @ ~/.julia/packages/CUDA/M4jkK/deps/bindeps.jl:26 [inlined]
 [4] versioninfo(io::IOStream) (repeats 2 times)
   @ CUDA ~/.julia/packages/CUDA/M4jkK/src/utilities.jl:43
 [5] top-level scope
   @ none:1
Host key verification failed.

AquaIndigo · 2021 年5 月 13 日 06:36

似乎确实是运行库版本的问题，nvcc -V也会显示9.0的版本

xgdgsc · 2021 年5 月 13 日 06:37

你看看有没有 /usr/local/cuda-10.2/ 然后设置变量 CUDA_HOME, CUDA_ROOT or CUDA_PATH

AquaIndigo · 2021 年5 月 13 日 07:52

折腾了一下在.local底下装了个CUDA-10.1，但是为什么安装的库里没有libcublas.so，CUDA.jl依旧没法运行

xgdgsc · 2021 年5 月 13 日 09:39

tensorflow - ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory - Stack Overflow 10.1不行，要10.2

AquaIndigo · 2021 年5 月 13 日 10:06

CUDA.jl说支持CUDA 10.1 or newer，用的时候却缺少cublas，服务器的driver只能支持到10.1的运行库，所以我自闭了

Eggiverse · 2021 年5 月 13 日 12:48

我试过，10.1可以，不过得用CUDA.jl 3.1及以下