All the Ways to Compress and Archive Files in Python

[dazzling_nash]

https://towardsdatascience.com/all-the-ways-to-compress-and-archive-files-in-python-e8076ccedb4b

[BiteCode_dev]

Only tangential, but it's a very little know fact that python can import and even run code from a zip archive. There is actually a module for that in the stdlib: zipapp.

This means that you can zip your entire project, including deps, into a pyz file, and ship that to your serveur as one big archive and run it like you would a .py file. It makes deployments very easy.

It also means you don't have to worry about using dependancies in quick scripts: use as many as you want and bundle them!

A tool exists to create the bundle easily and make it extract itself on first run for better perfs:

https://shiv.readthedocs.io

Is that instead of the zipapp module.

[jjice → BiteCode_dev]

Does zipapp automatically bundle dependencies, or do you need to specify that somehow? Also, how does this handle native extensions?

Edit: Answered by first question: https://docs.python.org/3/library/zipapp.html#creating-stand...

[hprotagonist → jjice]

for bundled deps, see shiv or pex.

https://shiv.readthedocs.io/en/latest/

https://pex.readthedocs.io/en/v2.1.49/

You still need the host to have a base python interpreter.

[BiteCode_dev → hprotagonist]

Carefull though, pex doesn't work on windows

[BiteCode_dev → jjice]

pip install --target to download your deps then use shiv

[seasily]

The only viable compression library, for internal use, is Zstandard

https://python-zstandard.readthedocs.io/en/latest/

https://engineering.fb.com/2016/08/31/core-data/smaller-and-...

The compression ratio and decompression speed just blow everything else away.