ProtoBuf.jl的使用问题

我计划为 milvus 做个julia api/sdk.
milvus 是一个向量搜索库, 已经有python, go, java, c++ sdk, 都是通过proto制作的sdk.
我也在尝试用julia的ProtoBuf.jl包来做下这个julia的sdk.
我用ProtoBuf 来把 milvus-proto文件 milvus-proto 生成为 *_pb.jl 文件. 但是过程中遇到了一些问题.

环境:

ulia> versioninfo()
Julia Version 1.8.0-rc1
Commit 6368fdc656 (2022-05-27 18:33 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: 96 × Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, cascadelake)
  Threads: 1 on 96 virtual cores
Environment:
  JULIA_PKG_SERVER = https://mirrors.bfsu.edu.cn/julia/static

(Milvus) pkg> st
Project Milvus v0.1.0
Status `~/codes/Milvus.jl/Project.toml`
  [c27321d9] Glob v1.3.0
  [3349acd9] ProtoBuf v0.11.5
  [aaca4a50] gRPCClient v0.1.2
  [c7845625] protoc_jll v3.19.4+0

代码:

using ProtoBuf
using gRPCClient
using Glob

function gen_jl_1()
    # used ProtoBuf.jl
    dir = "/home/zhangyong/codes/Milvus.jl/src/grpc_gen/"
    proto_dir = joinpath(dir, "milvus-proto/proto/")

    # Output folder
    module_out_dir = dir   # joinpath(dir, "")

    # Input files
    infiles = glob("*.proto", proto_dir)
    input_file = joinpath(proto_dir, "common.proto")

    includes_str = proto_dir

    # run(ProtoBuf.protoc(`-I=$includes_str --plugin=protoc-gen-julia=/home/zhangyong/plugin/protoc-gen-julia --julia_out=$module_out_dir $infiles `))
    ProtoBuf.protoc(`-I=$includes_str --julia_out=$module_out_dir $input_file`)

end

gen_jl_1()

报错:

(base) zhangyong@VM-1-8-ubuntu:~/codes/Milvus.jl$ julia --project=/home/zhangyong/codes/Milvus.jl/Project.toml /home/zhangyong/codes/Milvus.jl/src/grpc_gen/julia_gen.jl
ERROR: LoadError: ArgumentError: Package protoc_jll [c7845625-083e-5bbe-8504-b32d602b7110] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
  [1] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1172
  [2] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1091
  [3] macro expansion
    @ ./loading.jl:1071 [inlined]
  [4] macro expansion
    @ ./lock.jl:223 [inlined]
  [5] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1035
  [6] include(mod::Module, _path::String)
    @ Base ./Base.jl:422
  [7] include(x::String)
    @ ProtoBuf ~/.julia/packages/ProtoBuf/A71jO/src/ProtoBuf.jl:1
  [8] top-level scope
    @ ~/.julia/packages/ProtoBuf/A71jO/src/ProtoBuf.jl:20
  [9] include
    @ ./Base.jl:422 [inlined]
 [10] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1407
 [11] top-level scope
    @ stdin:1
in expression starting at /home/zhangyong/.julia/packages/ProtoBuf/A71jO/src/utils.jl:59
in expression starting at /home/zhangyong/.julia/packages/ProtoBuf/A71jO/src/ProtoBuf.jl:1
in expression starting at stdin:1
ERROR: Failed to precompile ProtoBuf [3349acd9-ac6a-5e09-bcdb-63829b23a429] to /home/zhangyong/.julia/compiled/v1.8/ProtoBuf/jl_1vCllE.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
   @ Base ./loading.jl:1558
 [3] compilecache
   @ ./loading.jl:1502 [inlined]
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1203
 [5] _require_prelocked(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1091
 [6] macro expansion
   @ ./loading.jl:1071 [inlined]
 [7] macro expansion
   @ ./lock.jl:223 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1035
--julia_out: protoc-gen-julia: Plugin failed with status code 1.
ERROR: LoadError: failed process: Process(setenv(`/home/zhangyong/.julia/artifacts/3c4b745a8e6f9478ffc074bb9ead269b343495c4/bin/protoc --plugin=protoc-gen-julia=/home/zhangyong/.julia/packages/ProtoBuf/A71jO/plugin/protoc-gen-julia -I=/home/zhangyong/codes/Milvus.jl/src/grpc_gen/milvus-proto/proto/ --julia_out=/home/zhangyong/codes/Milvus.jl/src/grpc_gen/ /home/zhangyong/codes/Milvus.jl/src/grpc_gen/milvus-proto/proto/common.proto`,["_CE_M=", "PATH=/home/zhangyong/.julia/packages/ProtoBuf/A71jO/plugin:/home/zhangyong/.julia/artifacts/3c4b745a8e6f9478ffc074bb9ead269b343495c4/bin:/home/zhangyong/.julia/bin:/home/zhangyong/.juliaup/bin:/home/zhangyong/miniconda3/bin:/home/zhangyong/miniconda3/condabin:/home/zhangyong/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/bin/remote-cli:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin", "LD_LIBRARY_PATH=/home/zhangyong/.julia/artifacts/3c4b745a8e6f9478ffc074bb9ead269b343495c4/lib:/home/zhangyong/.julia/juliaup/julia-1.8.0-rc1+0~x64/bin/../lib/julia:/home/zhangyong/.julia/juliaup/julia-1.8.0-rc1+0~x64/bin/../lib", "JULIA=/home/zhangyong/.julia/juliaup/julia-1.8.0-rc1+0~x64/bin/julia", "USER=zhangyong", "LESSCLOSE=/usr/bin/lesspipe %s %s", "CONDA_PROMPT_MODIFIER=(base) ", "VSCODE_GIT_ASKPASS_NODE=/home/zhangyong/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/node", "LESSOPEN=| /usr/bin/lesspipe %s", "SHELL=/bin/bash"  …  "TERM_PROGRAM_VERSION=1.68.1", "PWD=/home/zhangyong/codes/Milvus.jl", "TERM_PROGRAM=vscode", "VSCODE_GIT_ASKPASS_MAIN=/home/zhangyong/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/extensions/git/dist/askpass-main.js", "SSH_CONNECTION=192.168.100.254 12341 10.9.1.8 22", "MAIL=/var/mail/zhangyong", "CONDA_PREFIX=/home/zhangyong/miniconda3", "XDG_SESSION_ID=773874", "PROMPT_COMMAND=history -a; history -a; history -a; ", "LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:"]), ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:562 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:477
  [3] run
    @ ./process.jl:474 [inlined]
  [4] (::ProtoBuf.var"#11#12"{Cmd, String, String})(protoc_path::String)
    @ ProtoBuf ~/.julia/packages/ProtoBuf/A71jO/src/utils.jl:68
  [5] (::JLLWrappers.var"#2#3"{ProtoBuf.var"#11#12"{Cmd, String, String}, String})()
    @ JLLWrappers ~/.julia/packages/JLLWrappers/QpMQW/src/runtime.jl:49
  [6] withenv(::JLLWrappers.var"#2#3"{ProtoBuf.var"#11#12"{Cmd, String, String}, String}, ::Pair{String, String}, ::Vararg{Pair{String, String}})
    @ Base ./env.jl:172
  [7] withenv_executable_wrapper(f::Function, executable_path::String, PATH::String, LIBPATH::String, adjust_PATH::Bool, adjust_LIBPATH::Bool)
    @ JLLWrappers ~/.julia/packages/JLLWrappers/QpMQW/src/runtime.jl:48
  [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./essentials.jl:729
  [9] invokelatest(::Any, ::Any, ::Vararg{Any})
    @ Base ./essentials.jl:726
 [10] protoc(f::Function; adjust_PATH::Bool, adjust_LIBPATH::Bool)
    @ protoc_jll ~/.julia/packages/JLLWrappers/QpMQW/src/products/executable_generators.jl:21
 [11] protoc(f::Function)
    @ protoc_jll ~/.julia/packages/JLLWrappers/QpMQW/src/products/executable_generators.jl:19
 [12] protoc(args::Cmd)
    @ ProtoBuf ~/.julia/packages/ProtoBuf/A71jO/src/utils.jl:64
 [13] gen_jl_1()
    @ Main ~/codes/Milvus.jl/src/grpc_gen/julia_gen.jl:21
 [14] top-level scope
    @ ~/codes/Milvus.jl/src/grpc_gen/julia_gen.jl:40
in expression starting at /home/zhangyong/codes/Milvus.jl/src/grpc_gen/julia_gen.jl:40

希望有人能帮我看下是怎么原因…

信息让你调用Pkg.instantiate(),就做下试试
或者Pkg.add("protoc_jll")
可能是某个包内部调用Pkg没弄好,配置也没写

Pkg.instantiate() 和 Pkg.add(“protoc_jll”) 我都做了, 但是还是这样
(Milvus) pkg> st
Project Milvus v0.1.0
Status ~/codes/Milvus.jl/Project.toml
[c27321d9] Glob v1.3.0
[3349acd9] ProtoBuf v0.11.5
[aaca4a50] gRPCClient v0.1.2
[c7845625] protoc_jll v3.19.4+0

抱歉,看了半天没有找出问题,卸载重装试试吧,不行就直接去仓库提issue

手动点赞,这个还是蛮有意义的~

这个是 ProtoBuf.jl 的一个小bug,不过似乎一致没有人修。。。

你先直接在global 环境下(别切到 --project)install ProtoBufprotoc_jll,然后再试试。

另外,有遇到ProtoBuf相关的其它问题的话,也可以在这里交流,之前趟过一些坑…

我按照你说的做了, 可以正常生成 *_pb.jl

但我生成所有 *.proto 文件时, 有个报错:

Exception while generating Julia code
ERROR: unresolved name milvus.proto.common.Status at scope 
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] qualify_in_hierarchy(name::String, scope::ProtoBuf.Gen.Scope)
    @ ProtoBuf.Gen ~/.julia/packages/ProtoBuf/A71jO/src/gen.jl:154
  [3] qualify_in_hierarchy(name::String, scope::ProtoBuf.Gen.Scope) (repeats 3 times)
    @ ProtoBuf.Gen ~/.julia/packages/ProtoBuf/A71jO/src/gen.jl:152
  [4] scoped_svc_type_name(scope::ProtoBuf.Gen.Scope, typ_name::String)
    @ ProtoBuf.Gen ~/.julia/packages/ProtoBuf/A71jO/src/gen.jl:609
  [5] generate_svc(io::IOBuffer, errio::IOBuffer, stype::ProtoBuf.GoogleProtoBuf.ServiceDescriptorProto, scope::ProtoBuf.Gen.Scope, svcidx::Int64, exports::Vector{String})
    @ ProtoBuf.Gen ~/.julia/packages/ProtoBuf/A71jO/src/gen.jl:626
  [6] generate_file(io::IOBuffer, errio::IOBuffer, protofile::ProtoBuf.GoogleProtoBuf.FileDescriptorProto)
    @ ProtoBuf.Gen ~/.julia/packages/ProtoBuf/A71jO/src/gen.jl:771
  [7] codegen(srcio::Base.PipeEndpoint)
    @ ProtoBuf.Gen ~/.julia/packages/ProtoBuf/A71jO/src/gen.jl:859
  [8] (::ProtoBuf.Gen.var"#21#22")()
    @ ProtoBuf.Gen ~/.julia/packages/ProtoBuf/A71jO/src/gen.jl:912
  [9] with_debug
    @ ~/.julia/packages/ProtoBuf/A71jO/src/gen.jl:895 [inlined]
 [10] gen()
    @ ProtoBuf.Gen ~/.julia/packages/ProtoBuf/A71jO/src/gen.jl:910
 [11] top-level scope
    @ none:1
--julia_out: protoc-gen-julia: Plugin failed with status code 1.

不知道为什么?

The Julia code generated can be improper if the package name declared in the proto specification has . . Set a suitable package name without . .

没办法,现在不支持带 . 的名字,得自己手动改一个名字,这个很烦,但是目前并没有很好的解决办法。改完了之后,可能会有些依赖问题,你可以看下生成后得到的文件是什么样子,手动处理下