Hacker News new | past | comments | ask | show | jobs | submit login

Alpine Linux uses musl libc, and SourceHut runs Alpine on all hosts and VMs. It's lean and mean and fast, and together with Alpine makes for a wonderfully simple and auditable system. I wouldn't dream of using anything else in production.



Dependency management can be a pain. Want to install numpy? You have to use alpine's version because compiling it from pip requires glibc.


Also, annoyingly, there is still no platform tag for python wheels for alpine/musl, so you cannot get binary wheels for it from pypi.


Using anything other than distro packages is a pain. pip is an anti-pattern and one giant security risk.


Disagree, although pip should assume --user option by default which installs the packages in user directory instead of system directory (some distros (was it Debian?) modify it to work that way).

The best way of using Python though, is to create a virtualenv (since Python 3, venv module is built in which makes it straight forward, python -m venv <directory>) and install packages inside of it. This gives you control to use exact versions of packages you need in your applications and makes your application not tied to the OS, so system upgrades and Python version updates are much easier to perform.


> This gives you control to use exact versions of packages you need in your applications and makes your application not tied to the OS

Is this true for drivers, too? like the coupling of cuda versions to tensorflow versions?


Unfortunately that's one weakness, but unless you package your application using system packager (which comes with its own set of issues[1])

If you have any dependencies that depend on system libraries there are two options:

- the libraries can be compiled statically, they call them manylinux wheels, this generally works well, except if your dependencies are overlapping with python or other packages' dependencies. Most commonly this happen if package depends on openssl. If the compiled-in openssl is different than the version on the system in certain circumstances python might crash. This is for example why psycopg2 was initially distributed as manylinux but now they opt for the second method (they still provide psycopg2-binary but they discourage its use; most time it works fine but it is a problem on certain distros)

- make python create bindings on installation, this makes installation process do small compilation to create bindings between system library and python, this makes the packages robust, but tied to the OS. In my experience this is not an issue, but it can be annoying, because you still don't have 100% control of dependencies.

This issue is what made me investigate Nix[2] package manager, because it gives you full control of all dependencies down to libc making everything fully reproducible, so you can control the exact version of python to use and all system and python dependencies.

[1] in one of my previous jobs on a team that I joined they were running their applications on CentOS 5 that was already EOL, because rebuilding RPMs to CentOS 7 (at the time the newest version) was a lot of work, another issue was that they were bound to python package version that came with the system, they could create their own RPMs, but no one did it because maintaining that was adding more work. I spent time converting the python code to be packaged using setuptools. Once that was done switching the OS was trivial. We finally also could use the latest versions of many of our dependencies.

[2] This is IMO good article which describes tooling that makes Nix much more enjoyable to use: https://christine.website/blog/how-i-start-nix-2020-03-08


by doing this, do you suggest also tying your dependencies versions to those packages with the distro version you're running?


Yes, or just shadow them with newer versions and run your own repo.


Can you do something like `pip install --no-binary :all:` (or equivalent pip.conf entry) ?


You normally will not get wheels with compiled binaries on alpine anyway because it is not compatible with any of the platform tags (manylinux* is glibc only). I think they're saying numpy won't compile with musl off the shelf, you need patches for alpine. Pip just won't help you there.


It might be lean and mean - but isn't generally glibc faster? I'd be curious to hear if you benchmarked cases where musl is faster?




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: