Julia 1.4-Dev Pkg服务器使用测试

[johnnychen94]

Julia 1.4 下可以通过设置JULIA_PKG_SERVER来调整registry和package的下载来源，对于国内用户来说应该是一个好消息，目前还在测试阶段，因此只有官方源pkg.julialang.org，未来1.4正式发布之后应该会有国内镜像站。这里简单测试一下设置与不设置时对于装包时的时间影响：

设置的方式: shell下直接运行JULIA_PKG_SERVER=pkg.julialang.org ./julia，注意这里需要从master手动编译julia

• pre-test: rm ~/.julia/packages ~/.julia/artifacts （也可以设置DEPOT_PATH）
• test: time ./julia -e 'using Pkg; Pkg.add(["Images", "Plots", "Flux"])' 取系统实际时间(最后一项) 然后利用ifstat监测网络流量

测试网络：上海电信100M
时间：2019.12.20 01:00 UTC+8

• 挂美国代理不设置JULIA_PKG_SERVER： 6分33秒 （大概下载速度在500KB/s，最高下载速度在4MB/s)
• 直接连接，设置JULIA_PKG_SERVER: 21分11秒（大概下载速度在300KB/s，最高速度在2MB/s)
• 直接连接，不设置JULIA_PKG_SERVER: 完全下载不动… 长期处于0KB/s，最高100KB/s

Julia Version 1.4.0-DEV.639
Commit 320ee5acba (2019-12-19 14:33 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)

有兴趣可以跟踪一下这个issue: https://github.com/JuliaLang/Pkg.jl/issues/1377

[azurefx]

AS133119 (China Unicom IP network)

直接连接 704.498471 seconds
Pkg server 414.403850 seconds

Julia Version 1.4.0-DEV.639
Commit 320ee5acba (2019-12-19 14:33 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.6.0)
  CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
Environment:
  JULIA_PKG_SERVER = pkg.julialang.org

另外一楼环境变量那块和Pkg.add写错了，改一下？

[daxinnan]

我关掉ssr，在虚拟机上测试的，学校实验室的网测了一下实时网速在5.5M/s.
直接连接 完成时间在7min36.5s
export JULIA_PKG_SERVER=pkg.julialang.org 有问题，试了三次都是装到某个包就会卡住不动了。
我在科大镜像上下载的latest版本，难道是这个的问题？

julia> versioninfo()
Julia Version 1.4.0-DEV.648
Commit 608567f856 (2019-12-20 08:29 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, haswell)

[johnnychen94]

利用 GitHub - JuliaPackaging/PkgServer.jl 自建了pkgserver然后在同网段的另服务器内下做测试（千兆网线）

time env JULIA_DEPOT_PATH=~/local/julia JULIA_PKG_SERVER=http://juliapkg.lflab.cn ./julia -e 'using Pkg; Pkg.add(["Images", "Plots", "Flux"])'

大概30s搞定… 其中峰值速度可以达到100Mb/s

服务器的使用量

Client:

jc@mathlf1:~/l/julia
» du -sh *
757M	artifacts
788K	compiled
36K	environments
12K	logs
237M	packages
8.0K	prefs
58M	registries
12K	servers

PkgServer:

jc@storage:/m/julia_pkg
» du -sh *
259M	artifact
29M	package
512	registries
36M	registry

[johnnychen94]

Julia Computing 新增了一个测试用的Pkg server，位于首尔，使用方式如下：

在bash下：JULIA_PKG_SERVER=kr.pkg.julialang.org julia

julia> versioninfo()
Julia Version 1.4.0-rc1.0
Commit b0c33b0cf5 (2020-01-23 17:23 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.0.0)
  CPU: Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, broadwell)
Environment:
  JULIA_PKG_SERVER = kr.pkg.julialang.org

总之设置环境变量JULIA_PKG_SERVER就好了

[shihaoyan2018]

不设置JULIA_PKG_SERVER：

无法连接github下载registry，

Cloning registry from "https://github.com/JuliaRegistries/General.git" ERROR: LoadError: failed to clone from https://github.com/JuliaRegistries/General.git, error: GitError(Code:ERROR, Class:OS, failed to send request: 操作超时 )

设置JULIA_PKG_SERVER=kr.pkg.julialang.org：

registry都无法下载，

设置JULIA_PKG_SERVER=pkg.julialang.org：

registry也无法下载，

ASN： AS4837 CHINA UNICOM China169 Backbone

[staticfloat]

Can you show me what happens when you do this:

julia> import Pkg
       Pkg.probe_platform_engines!()
       mktempdir() do dir
         registries_file = joinpath(dir, "registries.txt")
         Pkg.PlatformEngines.download("https://kr.pkg.julialang.org/registries", registries_file; verbose=true)
       end
[ Info: Downloading https://kr.pkg.julialang.org/registries to /var/folders/xz/tg6x2wt15y70nmr5cb0bq2k00000gn/T/jl_mezZUZ/registries.txt...
######################################################################## 100.0%
Process(`curl -C - '-#' -f -o /var/folders/xz/tg6x2wt15y70nmr5cb0bq2k00000gn/T/jl_mezZUZ/registries.txt -L https://kr.pkg.julialang.org/registries`, ProcessExited(0))

If you have HTTP.jl installed, I would also be interested in seeing this output:

julia> using HTTP
       HTTP.get("https://kr.pkg.julialang.org/registries")
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Server: nginx/1.17.7
Date: Mon, 03 Feb 2020 19:24:15 GMT
Content-Length: 88
Connection: keep-alive
Content-Encoding: gzip

/registry/23338594-aafe-5451-b93e-139f81909106/86db14e6938f74dc2c6b6fbdb8242a8ac3ada08c
"""

[dong2221]

when run the code：

import Pkg
      Pkg.probe_platform_engines!()
      mktempdir() do dir
        registries_file = joinpath(dir, "registries.txt")
        Pkg.PlatformEngines.download("https://kr.pkg.julialang.org/registries", registries_file; verbose=true)
      end

The following errors appear

设置“SecurityProtocol”时发生异常:“由于枚举值无效，无法将空值转换为类型“System.Net.SecurityProtocol
Type”。请指定以下枚举值之一，然后重试。可能的枚举值为“Ssl3、Tls”。”
所在位置 行:1 字符: 35
+ [System.Net.ServicePointManager]:: <<<< SecurityProtocol =
   + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
   + FullyQualifiedErrorId : PropertyAssignmentException

Process(`powershell -NoProfile -Command '[System.Net.ServicePointManager]::SecurityProtocol =
   [System.Net.SecurityProtocolType]::Tls12;
$webclient = (New-Object System.Net.Webclient);
$webclient.UseDefaultCredentials = $true;
$webclient.Proxy.Credentials = $webclient.Credentials;
$webclient.Headers.Add("user-agent", "Pkg.jl (https://github.com/JuliaLang/Pkg.jl)");
$webclient.Headers.Add("Julia-Pkg-Protocol", "1.0");
$webclient.Headers.Add("Julia-Version", "1.4.0-rc1.0");
$webclient.Headers.Add("Julia-System", "x86_64-w64-mingw32-libgfortran4-cxx11");
$webclient.Headers.Add("Julia-Client-UUID", "768bf803-7b88-44a1-bb08-7fa6108ceb94");
$webclient.Headers.Add("Julia-Salt-Hash", "0d03c6770b5286d37468074cc5c138a890a0337b");
$webclient.Headers.Add("Julia-Project-Hash", "723cb3c250adbbf8bc26376b7f1f5482a5007266");
$webclient.Headers.Add("Julia-CI-Variables", "APPVEYOR=n;CI=n;CIRCLECI=n;CONTINUOUS_INTEGRATION=n;GITHUB_ACTION=n;GITLAB_CI=n;JULIA_CI=n;TF_BUILD=n;TRAVIS=n");
$webclient.Headers.Add("Julia-Interactive", "true");
$webclient.DownloadFile("https://kr.pkg.julialang.org/registries", "C:\Users\XUNG~1\AppData\Local\Temp\jl_sFEvLb\registries.txt")
'`, ProcessExited(0))

How to solve this problem ? Thank you very much！