Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The PkgServer is an open source caching server, available here: https://github.com/JuliaPackaging/PkgServer.jl

The PkgServer protocol serves content-addressed chunks of data to Julia Pkg clients, so to download a certain version of a package, the Pkg client will request things like `GET /package/${pkg_uuid}/${content_hash}`. If you can pre-fill a PkgServer with all of the packages that you want, then it can serve a client just fine.

The full design is that there are a small number of "Storage Servers" that continually explore the global registry of packages (called `General`, located here: https://github.com/JuliaRegistries/General), downloading and storing tarballs for every version of every package that is available. These storage servers store everything forever, while Pkg servers contain an LRU cache to allow them to be deployed close to whatever compute resources will be requesting packages. For an airgapped solution, you could generate a static snapshot of some selection of the packages you want to serve, then serve them with nginx and point to that "static storage server" with the opensource Pkg Server, and it would all "just work".

An example of how to generate a static storage server is here: https://github.com/JuliaPackaging/PkgServer.jl/blob/master/b..., you would serve the resultant directory structure with something like nginx, then point the Pkg Server to that server as the upstream storage server.

I will note that Julia Computing offers an enterprise solution for dealing with secure/restrictive environments called JuliaTeam which provides this in a convenient, managed bundle, along with many other useful features.

EDIT: Ah, I forgot to mention, Stefan and I gave a talk at JuliaCon 2020 that touches on some of this, here's a link to the timestamp of the relevant section: https://youtu.be/xPhnJCAkI4k?t=350

And for more info, here's the original planning issue (note some things have changed as we've implemented it over the last year, but the bones are the same): https://github.com/JuliaLang/Pkg.jl/issues/1377#issue-492482...



Thanks! I will give this a try.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: