
Building Standalone Python Applications with PyOxidizer - sametmax
https://gregoryszorc.com/blog/2019/06/24/building-standalone-python-applications-with-pyoxidizer/
======
ForHackernews
Obviously this has real advantages for making it easier for non-technical
users to run Python applications, but I despair for the future of computing.

A desktop computer in 2025 is going to have 47 redundant installations of
Chromium for various electron apps, 78 separate-but-basically-identical Python
runtimes as every small utility packages the entire Python interpreter +
stdlib, 200,000 redundant npm packages...

~~~
jedberg
Disk is cheap, internet is pretty fast. I'd rather have the consistency of
packaged software than deal with getting a bunch of libraries just right to
save a little space.

~~~
squaresmile
> Disk is cheap, internet is pretty fast

Until it isn't [0]. I think it's fine to only develop/care for faster hardware
but I'm not a fan of broad statement like this when it doesn't apply for so
many people.

[0] [https://www.speedtest.net/global-
index#mobile](https://www.speedtest.net/global-index#mobile)

~~~
jedberg
There is a difference between catering to fast runtime internet vs fast
install internet. I agree with you that we should optimize apps to be quick on
slow internet, especially with so many mobile only users. But making an app
that is large is a one time hit that isn't so bad, even on slow internet.

------
mehrdadn
Update 2: Gave up on MSYS2 and just used rustup. Managed to compile and run,
but exit() doesn't work. Also I just realized the executable contains the
compilation path, which leaks private information? Makes it hard to compile
and redistribute stuff, but otherwise I'd be very excited about the project.

__________________________________________________

Update: Still can't get it to work. Also, the tool seems to download random
third-party packages that aren't documented?

    
    
      > pyoxidizer run
      no existing PyOxidizer artifacts found
      processing config file pyoxidizer.toml
      resolving Python distribution...
      downloading https://github.com/indygreg/python-build-standalone/releases/download/{...}/cpython-{...}.tar.zst
      ...
      error[E0463]: can't find crate for `std`
        |
        = note: the `x86_64-pc-windows-msvc` target may not be installed
      
      error: aborting due to previous error
      
      For more information about this error, try `rustc --explain E0463`.
      error: Could not compile `rand_core`.
      error: cargo build failed
    

__________________________________________________

I ran 'cargo install pyoxidizer' and got

    
    
      cargo install pyoxidizer
          Updating registry `https://github.com/rust-lang/crates.io-index`
      error: failed to parse manifest at `%USERPROFILE%\.cargo\registry\src\github.com-1ecc6299db9ec823\pyoxidizer-0.1.1\Cargo.toml`
      
      Caused by:
        editions are unstable
      
      Caused by:
        feature `edition` is required
      
      this Cargo does not support nightly features, but if you switch to nightly channel you can add `cargo-features = ["edition"]` to enable this feature
    

Not sure where to go next... I'm not a Rust programmer and frustratingly I
never seem to get Rust programs working on the first try. Do I have to
download another version of Rust now? Why can't Rust just install the package?

Edit: Would be nice if someone could also explain how this is a manifest
_parsing_ error?

~~~
codetrotter
Sounds like you might be using an old version of the Rust toolchain, from
before the edition feature was in the stable channel.

Try this:

    
    
      rustup update
    

And then try again

    
    
      cargo install pyoxidizer
    

Edit: That is, assuming you installed the Rust toolchain via
[https://rustup.rs/](https://rustup.rs/), which you should -- the official
package repos of most distros are sure to contain really old outdated versions
of the Rust toolchain.

~~~
mehrdadn
Huh I see. There's no rustup anywhere to be found. Rust just from MSYS2 in
Oct. 2018, just 8 months ago. Updated it now and now it's installing fine,
thanks. Here's what I had:

    
    
      $ pacboy -Qi rust:x
      Name            : mingw-w64-x86_64-rust
      Version         : 1.29.2-1
      Build Date      : Sun, Oct 28, 2018 9:43:06 PM
      ...
    

Now I have 1.35.0-1 which seems to be the most recent version (with rustup
still nowhere to be found).

~~~
codetrotter
Yeah I realized after commenting that that you might not have installed it via
rustup.rs. Glad to hear that you were able to get a new enough version after
updating.

8 months might not seem like a long time but quite a few things have happened
in that time, including some features having become stabilized.
[https://github.com/rust-
lang/rust/blob/master/RELEASES.md](https://github.com/rust-
lang/rust/blob/master/RELEASES.md)

1.35.0 is the current stable Rust toolchain indeed.

~~~
mehrdadn
Yup, updated my post. Would you know how to prevent Rust from embedding
absolute file paths into the binary?

------
mherrmann
I'd be interested to learn what the motivation to create this was, when
PyInstaller [1] already solves this (hard) problem in a fair way.

1: [https://www.pyinstaller.org](https://www.pyinstaller.org)

~~~
mehrdadn
It probably requires unpacking files onto the disk every single time? At least
that's what I remember of every tool I checked out a while ago.

~~~
mherrmann
I think PyInstaller does, yes, _if_ you use single file mode. The default mode
simply uses an exploded directory. Not pretty, but really not a problem in
practice.

~~~
mehrdadn
How is this not a problem in practice? So much write I/O into a temp directory
is slow and error-prone, and also you can't guarantee cleanup in case the
process quits abnormally. And it's not like the average user will be fond of
downloading and unzipping things manually. I know I absolutely loathe programs
that self-extract files on a regular basis, and judging from the pains people
went through to avoid it here I don't think I'm alone.

~~~
mherrmann
I meant the exploded directory is not a problem. And the answer to unzipping
is an installer.

------
metroholografix
Cool project!

The parts I don't like:

\+ Static linking

\+ No extension (shared library) support

\+ The module importers look over-engineered / too complicated

\+ Custom Python distributions

\+ Not sure what Rust gets you here, besides additional complexity

\+ No support for Python 2.7 (Python 3 drops support for Windows XP which is
still a legit platform for many scenarios)

Many years ago, I solved similar problems (Windows-only at the time) with an
approach that:

\+ Requires no static linking, absolutely no compilation of Python

\+ Supports packaging any pre-installed stock Python 2 distribution, no
modifications except to fix bugs in the standard library

\+ Uses in-memory loading for _everything_ including Python extensions (shared
libraries)

\+ Is based on top of a minimal C-bootstrapping layer that can be compiled
once and reused

\+ Includes a better loader than the one in py2exe :-]

\+ Delivers an injectable DLL or executable with absolutely no dependencies
except OS built-ins

\+ Works on every version of Windows from XP SP0 and beyond

The product is commercial (platform for modelling APT attacks and performing
automated exfiltration pentests) but I did a presentation [1] which lays out
the entire loader and the benefits of my approach in a way that is easy to
independently re-engineer. I can see some benefits of static linking on Unix
(even though I'd still avoid it there), but I'm convinced that it's absolutely
the wrong thing to do for Windows.

[1] [https://downloads.immunityinc.com/infiltrate-
archives/python...](https://downloads.immunityinc.com/infiltrate-
archives/python_deflowered.pdf)

~~~
gravypod
Why is "Static linking" bad? A majority of software engineers are shifting to
statically linked to reduce runtime complexity.

Python 2.7 is also deprecated. New projects shouldn't really be supporting it
going forward in my opinion. We're coming up on the 2020 complete end of life
cycle.

~~~
metroholografix
It depends on the problem. Someone could say that static linking optimizes for
runtime costs but for the problems that OP (and me) are trying to solve, I
find that it increases runtime complexity since:

\+ It tightly couples independent subsystems together, you can no longer see
them (e.g. update) in isolation. Some of those subsystems explode when
statically linked due to second-order effects (e.g. memory management).

\+ Requires compilation.

\+ Licensing woes that do not apply with dynamic linking.

\+ Most important: Static linking reduces your runtime flexibility by being
the opposite of late-binding and forces you to commit upfront. Many problems
can be elegantly solved when one can take additional context into account
(e.g. delay solving them until the 'how' becomes clear).

On top of that, static linking does not optimize for deployment costs, which
can be substantial depending on the application and frequency of component
changes.

For all these reasons, it should be avoided when possible. The counterpoint is
that in practice, it takes a lot more effort to design a system that does not
deploy static linking in order to solve the problems described by OP and me.
Static linking is thus an easy and quick solution but it comes with many
strings attached.

------
jchw
Wow, very cool. This sounds like a lot of existing projects but done in a much
cleaner fashion.

If it was mentioned, I missed it, but I’m really curious to hear how things
like cffi, etc. end up working. It may, with work, be possible to statically
link CPython extensions, but cffi seems like a whole new packaging problem.

To avoid the X-Y problem, I’ll state a high level goal. Let’s say I want to
release an application using PyQt. Can PyOxidizer handle this in any way
today? In the future, does PyOxidizer plan on handling any more of the
problem? Since existing Python packaging infrastructure doesn’t attempt to
answer problems like this, it’d be awesome to see new initiatives that do.

~~~
mherrmann
Re deploying PyQt apps: check out my [https://build-
system.fman.io](https://build-system.fman.io). It solves the bejeezus out of
this problem.

~~~
jchw
Ah, that I _have_ seen, thanks. I suppose that would be the end to end
solution for this specific problem.

------
zerogvt
""" Rust is the first programming language I've used where I feel like the
language itself, the compiler, the tools around it (cargo, rustfmt, clippy,
rustup, etc), and the community surrounding it all actually care about and
assist me with writing high quality software. Nothing else I've used comes
even close. """

I've been hearing nice words about Rust for a while now but it doesn't seem to
catch up in popularity which I find weird.

~~~
sametmax
> it doesn't seem to catch up in popularity which I find weird.

What's your metric to say that ? After all, it's a system language, not a web
one or scripting one, so it's not likely to go to the top trending stats very
fast. Not to mention we had a stable version for only 4 years.

~~~
zerogvt
[https://www.tiobe.com/tiobe-index/](https://www.tiobe.com/tiobe-index/)
currently Rust is 38th

~~~
sametmax
Seems fair to me, giving the nature of the language, the inertia of c/c++ and
low level stuff, the learning curve of this domain, the popularity of
web/scripting and the 4 years I mentioned.

------
science404
Is PyOxidizer going to make it any harder to 'decompile' one of these Python
executables? Of course, once you have the bytecode it's easy to recover the
source code, but I wonder if something could be built into this tool to make
it ever-so-slightly more difficult.

Nuitka has the potential to solve this issue, but it seems not ready yet.

Anyone know if/how Dropbox does it?

~~~
c8g
Dropbox uses a modified python interpreter

[https://github.com/anvilventures/lookinsidethebox](https://github.com/anvilventures/lookinsidethebox)

[https://anvilventures.com/blog/looking-inside-the-
box.html](https://anvilventures.com/blog/looking-inside-the-box.html)

------
hprotagonist
Can this crossbuild? I cannot produce windows exes with pyinstaller on my
linux box; can i with pyoxidizer?

~~~
montecarl
I've built windows exe files using pyinstaller on Linux using docker and wine.
Someone else has helped automate this. Check out
[https://github.com/cdrx/docker-pyinstaller](https://github.com/cdrx/docker-
pyinstaller)

------
lawik
Oooh, this seems like exactly what I need. I hope it gains some traction.
Trying it out right now for small util my wife needs that has to run on a
machine she doesn't fully control.

------
luord
Was liking the project until I got to the gotcha: the incongruous decision of
using rust (or any other programming language, for that matter).

Not only that will stifle any kind of community forming around the project
(not impossible, but more difficult), but also because after that point the
article became yet another mostly inaccurate proselytizing of "[x] language I
use because [I like it, but I'll pretend there are sound technical reasons why
it's better, even though every single one is contested]".

And I ended up closing the tab entirely after this sentence "in the same way
that python is technically a C application", which pretty much encapsulates my
previous paragraph.

I hope I'm wrong and the project takes off; python does need better packaging
and distribution. But in the meantime, I'll keep looking into python-based
solutions to the problem.

~~~
dual_basis
So, to be clear, you don't like it when projects choose to use their preferred
language, and so you will search for a project which uses your preferred
language?

~~~
luord
If I'm solving a problem with Go, I will first look into solutions written in
Go. If I'm solving a problem with Javascript, I'll first look into solutions
written in JavaScript. If I'm solving a problem in Python, I'll first look
into solutions written in Python. Etc, etc, etc.

If this project does prove to be better than something like PyInstaller, I
will welcome it; I wouldn't go against the grain and would rather use a
standard, de facto or official. In the meantime, I'd rather not leave an
ecosystem just so I can develop _for_ that ecosystem.

------
zarmin
I love this project, it is much needed, and your documentation/writeups are
outstanding. Thank you for your work.

------
antoox
Very cool project! As a python developer, I completely share the concerns
mentioned in this article. From my point of view, building a new project from
scratch requires a certain "marketing approach". The idea of having "The
PyOxidizer's functionality rolling up into official packaging tools like pip"
seems to be key to massive adoption.

------
goodside
Also see Facebook’s XAR:
[https://github.com/facebookincubator/xar](https://github.com/facebookincubator/xar)

~~~
sametmax
XAR is only for "Linux or macOS", with FUSE installed.

PyOxidizer offers a solution that:

\- works on Windows

\- can produce executable for Windows, Mac and Linux, but also potentially
cross compile from any of those in the future

\- results in a stand alone executable, not some filesystem abstraction

\- as such, the target machine doesn't need to have anything installed

\- it also offers to embed Python into Rust projects, and Rust into Python
projects

A better alternative would be the excellent nuitka (nuitka.net):

\- it compiles Python to C

\- it's very compatible, very reliable (I battle tested it)

\- works better than the alternatives like py2exe, cx_freeze, etc

\- does provide a standalone executable

\- more mature that both of these projects

But:

\- no cross compilation

\- no rust / python interoperability

\- more complicated to get working on windows

So I'm really excited for PyOxidizer. It may make the task of providing a
stand alone executable even easier than with nuitka, and as a bonus, may build
a bridge between the Python and Rust community, which really are meant to play
together.

~~~
asicsp
>cross compile from any of those in the future

that would be very helpful, I use Linux and to be able to provide executables
for Windows/macOS would be nice

