A desktop computer in 2025 is going to have 47 redundant installations of Chromium for various electron apps, 78 separate-but-basically-identical Python runtimes as every small utility packages the entire Python interpreter + stdlib, 200,000 redundant npm packages...
I definitely share those concerns. But being able to package things in an independent way is incredibly useful.
I hope we can get away from the reliance on incredibly disproportionately sized runtimes (such as Chromium) for making GUI apps in most cases. I think it trades too much of the user's resources for developer convenience. Unless you are trying to do audio/video conferencing in which case I guess I get it.
I've found the work on the Scenic UI framework in Elixir quite cool (only depends on OpenGL + glfw on desktop, nanovg on RPi).
I also had reason to build a simple bit with wxWidgets in Elixir through Erlang. Didn't love it. But that should be able to make reasonable-sized native but cross-platform UIs.
Elixir and Erlang tend to package releases in a nice independent way.
But what I need this for is because every library under the sun is available for Python. And sometimes you need to just get a small utility built and distributed.
Maybe we'll eventually hit hard physical limits on hardware, and then we software developers will be forced to learn how to do our jobs correctly for the first time.
Until it isn't . I think it's fine to only develop/care for faster hardware but I'm not a fan of broad statement like this when it doesn't apply for so many people.
If anything, the technology underlying the abstraction of data storage should be taking care of optimal use of resources.
Update: Still can't get it to work. Also, the tool seems to download random third-party packages that aren't documented?
> pyoxidizer run
no existing PyOxidizer artifacts found
processing config file pyoxidizer.toml
resolving Python distribution...
error[E0463]: can't find crate for `std`
= note: the `x86_64-pc-windows-msvc` target may not be installed
error: aborting due to previous error
For more information about this error, try `rustc --explain E0463`.
error: Could not compile `rand_core`.
error: cargo build failed
I ran 'cargo install pyoxidizer' and got
cargo install pyoxidizer
Updating registry `https://github.com/rust-lang/crates.io-index`
error: failed to parse manifest at `%USERPROFILE%\.cargo\registry\src\github.com-1ecc6299db9ec823\pyoxidizer-0.1.1\Cargo.toml`
editions are unstable
feature `edition` is required
this Cargo does not support nightly features, but if you switch to nightly channel you can add `cargo-features = ["edition"]` to enable this feature
Edit: Would be nice if someone could also explain how this is a manifest parsing error?
cargo install pyoxidizer
$ pacboy -Qi rust:x
Name : mingw-w64-x86_64-rust
Version : 1.29.2-1
Build Date : Sun, Oct 28, 2018 9:43:06 PM
8 months might not seem like a long time but quite a few things have happened in that time, including some features having become stabilized. https://github.com/rust-lang/rust/blob/master/RELEASES.md
1.35.0 is the current stable Rust toolchain indeed.
Nuitka worked immediatly, but still dynamamically links libc and doesn't allow this nice rust/python interraction nor the possibility of future cross compilation.
The parts I don't like:
+ Static linking
+ No extension (shared library) support
+ The module importers look over-engineered / too complicated
+ Custom Python distributions
+ Not sure what Rust gets you here, besides additional complexity
+ No support for Python 2.7 (Python 3 drops support for Windows XP which is still a legit platform for many scenarios)
Many years ago, I solved similar problems (Windows-only at the time) with an approach that:
+ Requires no static linking, absolutely no compilation of Python
+ Supports packaging any pre-installed stock Python 2
distribution, no modifications except to fix bugs in the standard library
+ Uses in-memory loading for _everything_ including Python extensions (shared libraries)
+ Is based on top of a minimal C-bootstrapping layer that can be compiled once and reused
+ Includes a better loader than the one in py2exe :-]
+ Delivers an injectable DLL or executable with absolutely no dependencies except OS built-ins
+ Works on every version of Windows from XP SP0 and beyond
The product is commercial (platform for modelling APT attacks and performing automated exfiltration pentests) but I did a presentation  which lays out the entire loader and the benefits of my approach in a way that is easy to independently re-engineer.
I can see some benefits of static linking on Unix (even though I'd still avoid it there), but I'm convinced that it's absolutely the wrong thing to do for Windows.
Python 2.7 is also deprecated. New projects shouldn't really be supporting it going forward in my opinion. We're coming up on the 2020 complete end of life cycle.
+ It tightly couples independent subsystems together, you can no longer see them (e.g. update) in isolation. Some of those subsystems explode when statically linked due to second-order effects (e.g. memory management).
+ Requires compilation.
+ Licensing woes that do not apply with dynamic linking.
+ Most important: Static linking reduces your runtime flexibility by being the opposite of late-binding and forces you to commit upfront. Many problems can be elegantly solved when one can take additional context into account (e.g. delay solving them until the 'how' becomes clear).
On top of that, static linking does not optimize for deployment costs, which can be substantial depending on the application and frequency of component changes.
For all these reasons, it should be avoided when possible. The counterpoint is that in practice, it takes a lot more effort to design a system that does not deploy static linking in order to solve the problems described by OP and me. Static linking is thus an easy and quick solution but it comes with many strings attached.
> + Static linking
It's actually what I like in this project. Dynamic linking causes many issues, especially on Linux, where you never have quite the same libs depending of the distro. If I want a standalone exe, I want static linking. If I want dynamic linking, I'll make a deb or a rpm.
> + No extension (shared library) support
> + The module importers look over-engineered / too complicated
Man, the project just came out, give it time. You're very harsh on this.
> + Custom Python distributions
Well, yes. That's why it works. Otherwise you are back to require a C compiler on the machine or use dynamic linking.
> + Not sure what Rust gets you here, besides additional complexity
- link to the custom python distrib, allowing you to use the nice rust toolchain instead of the mess that is the c toolchain or dynamic linking
- free rust / python interrop. This is very interesting.
- potentially cross compilation and even make the tools stand alone itself
> + No support for Python 2.7 (Python 3 drops support for Windows XP which is still a legit platform for many scenarios)
Python 2.7 is reaching EOL in a few months. It's perfectly reasonable to not support it. No poney.
I don't remember the details, but ... I do remember failing to find a working solution (myself or in any of the projects I checked at the time).
But my problem was that I needed it to happen (e.g. if using msvcrt) and having no way to do so for DLLs “manually” loaded from memory.
If you use a method that completely bypasses the OS loader, then you have to do a lot more work to have your DLL behave like a DLL loaded by the OS would.
I was under the impression you needed to have some non-trivial tokens to patch NtDll in a way that kernel32 would use (which would make it unusable as a general technique), but it's been ages and I'm probably misremembering.
If it was mentioned, I missed it, but I’m really curious to hear how things like cffi, etc. end up working. It may, with work, be possible to statically link CPython extensions, but cffi seems like a whole new packaging problem.
To avoid the X-Y problem, I’ll state a high level goal. Let’s say I want to release an application using PyQt. Can PyOxidizer handle this in any way today? In the future, does PyOxidizer plan on handling any more of the problem? Since existing Python packaging infrastructure doesn’t attempt to answer problems like this, it’d be awesome to see new initiatives that do.
This can be used with statically or dynamically linking Python easily.
I've been hearing nice words about Rust for a while now but it doesn't seem to catch up in popularity which I find weird.
What's your metric to say that ? After all, it's a system language, not a web one or scripting one, so it's not likely to go to the top trending stats very fast. Not to mention we had a stable version for only 4 years.
Nuitka has the potential to solve this issue, but it seems not ready yet.
Anyone know if/how Dropbox does it?
PyOxider packages and accesses the python files I think, so they are here and available.
Not only that will stifle any kind of community forming around the project (not impossible, but more difficult), but also because after that point the article became yet another mostly inaccurate proselytizing of "[x] language I use because [I like it, but I'll pretend there are sound technical reasons why it's better, even though every single one is contested]".
And I ended up closing the tab entirely after this sentence "in the same way that python is technically a C application", which pretty much encapsulates my previous paragraph.
I hope I'm wrong and the project takes off; python does need better packaging and distribution. But in the meantime, I'll keep looking into python-based solutions to the problem.
If this project does prove to be better than something like PyInstaller, I will welcome it; I wouldn't go against the grain and would rather use a standard, de facto or official. In the meantime, I'd rather not leave an ecosystem just so I can develop for that ecosystem.
PyOxidizer offers a solution that:
- works on Windows
- can produce executable for Windows, Mac and Linux, but also potentially cross compile from any of those in the future
- results in a stand alone executable, not some filesystem abstraction
- as such, the target machine doesn't need to have anything installed
- it also offers to embed Python into Rust projects, and Rust into Python projects
A better alternative would be the excellent nuitka (nuitka.net):
- it compiles Python to C
- it's very compatible, very reliable (I battle tested it)
- works better than the alternatives like py2exe, cx_freeze, etc
- does provide a standalone executable
- more mature that both of these projects
- no cross compilation
- no rust / python interoperability
- more complicated to get working on windows
So I'm really excited for PyOxidizer. It may make the task of providing a stand alone executable even easier than with nuitka, and as a bonus, may build a bridge between the Python and Rust community, which really are meant to play together.
that would be very helpful, I use Linux and to be able to provide executables for Windows/macOS would be nice