I just spent 30 minutes reading about this. I'm so shocked that I'm logging in to comment after 5 years of lurking.
Justine has built a c library that allows you compile a binary once and have it run it on any os or baremetal. The SAME binary. Quite frankly, that sentence doesn't even make sense to me.
Agreed, this is the best programming-related thing I've seen on the internet in a long time. It reminds me of when I was a teenager, excited to become a computer programmer. Then I grew up and joined the real world workforce and it was all far less exciting than I imagined.
The coolest thing about this Actually Portable Executable is that once there's a compiler and linker built with it, I can play around with writing C on Windows without having to faff about with WSL or MinGW or learning what makes MSVC different from the C I learned in university and coded in my first job before I became a Java/JavaScript/Python/etc programmer. When I discovered Go I thought I had discovered the better C that just worked everywhere, but the idea of having plain old C that also just works everywhere is very appealing.
I like Go for the same reason, but I have an irrational fear that someday I might be in a life or death situation and should program my way out using a bare bones computer without Internet; So I keep tabs on C occasionally.
αcτµαlly pδrταblε εxεcµταblε has now piqued my interest, So I might actually indulge in C programming for fun again after all. Thanks Justine.
I think it's really cool project; but could you explain why it's important that a binary works everywhere for you? I don't really get the enthusiasm for this sort of thing.
Maybe I have a blind spot or something or I've just been in Unix-land for too long.
I think the UNIX thing is the blind spot. People who run Linux or BSD don't seem so bothered to recompile for their own platform, safe in the knowledge that it will almost certainly just work.
But on Windows, trying to get open source software written in C to compile has been a pain for decades. There have been some attempts to make it work (notably Cygwin, and now MSYS2), but it can still be challenging, especially with programs that were never designed to be compiled for non-UNIX (or non-Linux!) targets.
Having a C library that is designed from the ground up to target multiple operating systems, and an executable format that means the first compile is the only compile, that takes away a lot of toil that programmers normally have when trying to build their own abstractions around each platform's C library. I guess from UNIX land you could see the benefit as removing the need for autoconf or platform ifdefs.
Agreed. This single sentence at the end of αcτµαlly pδrταblε εxεcµταblε page:
> I believe the best chance we have of doing that [writing software that stands the test of time with minimal toil], is by gluing together the binary interfaces that've already achieved a decades-long consensus, and ignoring the APIs.
...it's a kind of thought I don't think I could ever come up with. Mind blown.
I agree, I got to the page on actually portable executable, and had to read it a few times to make sure I was grokking it properly. This is just sheer cleverness, THIS is the stuff that should be on hacker news!
It's hard to grok cause there are no words to describe it.
When people say that something is cross-platform, they usually mean
a) that the software will build on multiple platforms
b) there is some sort of vm which runs the executable (jvm, beam, wasm)
This is the SAME binary. Running everywhere. Could be super-useful as an archival format for mission critical code. sqlite comes to mind.
Maybe it's late at night, but I'm struggling to find a suitable word for the project that is better than portable or cross-platform.
Literally, I can't find a better word than the project name - "actually portable executable."
@author - if you're reading this please set up something for a few bucks a month on your github sponsors. I don't think I have any use for the library, but this is so outrageous it deserves something more than imaginary internet points.
Thanks for the links. The discussion on the first one is interesting, it wasn't clear to me from the original doc that the UNIX version needs to overwrite itself to get to the ELF header. That's a bit unfortunate, because it means you can't share the program again after you ran it the first time on BSD or Linux. Still very clever, though. I imagine there could be a "repackage for distribution" switch added to put the MZ header back again.
I did the research and what we've accomplished here, while imperfect, is the best of all worlds solution. I like to think of it as an installer that takes a few microseconds, because it only needs to change 64 bytes. I've been considering adding a CLI flag where you can say ./foo.com --restore and it'll put the original 64 byte header back in place. Perhaps one day we can change the Linux, BSD, and XNU kernels so they can recognize the APE executable format. But until that happens we've got a great shell script hack that's now required by POSIX which is exceedingly fast and works on pretty much all systems stretching back decades.
Everything she does has this level of jaw-dropping amazingness. Between her and Fabrice Bellard i don't know many people who consistently get my chin to hit the table.
I hope that Cosmopolitan becomes mainstream, so much more software could have that It Just Works quality.
Understood (with my apologies, I mistakenly assumed
it was on topic).
FWIW, I like and agree with the things I was referencing, I didn't mean it in any way as a jab or provocation. I shall endeavor to do a better job staying on the topic of the article in question.
Strong agree; someone needs to give her like a million dollars a year salary and unlimited resources to explore whatever she wants for life, just to see what she comes up with.
It's funny because the high-level introductions to C I would read as a kid before I really knew much about computing would almost always start out by explaining that one of the biggest strengths of C was its portability, and how it allowed you to write the same code that worked on every platform. Of course I came to learn that's true only in a sense that has no connection to practical reality, this looks like an exception.
Could you please explain to someone from the web domain with very little experience with compiled C programs why this is significantly better than, say, distributing a python program? Python runs on all the platforms mentioned. Or alternatively requiring compilation from source?
The python script/program ultimately depends on a Python interpreter which currently has to be distributed as a different binary for every platform.
So in the end, the difference is purely where the platform-differentiated compilation happens for VM & interpreted languages like Java & Python.
This would instead allow you to ship a single Python executable that simply works everywhere. Whether with a bundled-in script/program, or just the interpreter.
That sort of malware already exists. This does simplify making it by a lot, because now it can much more easily ensure its portability without external communication.
I really don't understand internals ibut I've often downloaded cygwin exe file for commands like grep, tail, etc.. Are you telling me that is no longer required?
On linux, I downloaded the printvideo binary and played a friggin crab video on my terminal after just a chmod. I don't have any other OS to confirm, but it sure looks like it if someone compiles/links coreutils against this.
Wow, that cosmopolitan C library is absolutely awesome. The ABI specialization is extremely interesting. Never seen anything like that before. I thought compilers did that. I don't understand why they couldn't properly optimize memcpy.
I wonder how it handles system calls. Let me check the source...
That malware for OS X that does nothing (so far) from the news earlier this week also has one multi-arch binary for all modern Apple notebook architectures (AMD64 and ARM64)
I absolutely love reading Justine's code. It comes up from time to time here and it just makes me happy. It reminds me of the passage from "Programming Sucks" by Still Drinking [1]
Every programmer occasionally, when nobody’s home, turns off the lights, pours a glass of scotch, puts on some light German electronica, and opens up a file on their computer. It’s a different file for every programmer. Sometimes they wrote it, sometimes they found it and knew they had to save it. They read over the lines, and weep at their beauty, then the tears turn bitter as they remember the rest of the files and the inevitable collapse of all that is good and true in the world.
This file is Good Code. It has sensible and consistent names for functions and variables. It’s concise. It doesn’t do anything obviously stupid. It has never had to live in the wild, or answer to a sales team. It does exactly one, mundane, specific thing, and it does it well. It was written by a single person, and never touched by another. It reads like poetry written by someone over thirty.
Her work is always just a little bit trippy in a good way haha.
The short description is a bit too short; this is mind blowing!
Redbean is a portable, single file executable webserver which also acts as a zip-file for the static content it hosts, and runs on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS without any recompilation...
You can manage the static content using standard pkzip tooling!
Now that's what I call thinking out of the box!
Is it that this binary does some tricks to run on Linux, Mac,
Windows, FreeBSD, OpenBSD, NetBSD, BIOS or does it really work on all operating systems including something like Plan 9 or even TempleOS?
And does it run on different architectures like ARMv6?
Native execution on x86_64 only. "Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS", where "BIOS" means that it will boot a machine and run on bare metal. On non x86_64 systems, the driver script will try to run the binary using QEMU (which has to be installed and on the path):
If i have understood how the portability works, it would be pretty straightforward to add support for another OS which uses the same ABI as the unices and has a compatible system call table - just a matter of adding a column to consts.sh (system call numbers in the nr section, constants from system headers in their appropriate sections) and adding any OS-specific system call rituals to systemfive.S.
Note that all system calls will go through the syscall interface. If your OS has some other way to make system calls, like Linux's vDSO or io_uring, that won't be used.
If your OS had a different system call table, like Windows, then you would need to write more bespoke code for it, like all the Windows wrappers:
It's a neat coincidence that this is on at the same time as an article about CP/M, because I think there could be some overlap in the programs that would be a good fit for cosmopolitan.
Sure, there's server appliances like this, and I hope someone makes a neat services wrapper to abstract at least some platform's intricacies so you can do a "cosmo-service up redbean" on BSD/Linux/Windows.
But coming back to CP/M, there you had a lowest common denominator of terminal applications, too, but spread across different architectures. And you could still produce some quite intersting, if a bit business-like applications. These days you probably can even rely on more ANSI colors and maybe even unicode fonts (hopefull as an option, not mandatory).
The ZIP characteristics of the APE format make it even easier to distribute a whole application in a rather simple way. Yes, sure, you can do regular Unix-style servers and pipe-it-together CLI tools, but I wouldn't mind more self-contained "business" applications with a lo-fi aesthetic regarding interface and API usage. The PICO-8 of TUIs…
Author here. We're living in the most exciting time for developing terminal applications. When Microsoft unexpectedly added support for VT100 and XTERM codes to CMD.EXE it totally changed the equation and ANSI became universal for the first time. Blinkenlights is an example of a TUI application I created using Cosmopolitan and it literally works everywhere. https://justine.lol/blinkenlights/index.html You don't need curses. All that's needed is an ioctl() call which flips a bit in termios. Cosmopolitan polyfills that across operating systems. Another cool example of a demo app is this conway's game of life tui gui: https://justine.lol/apelife/index.html
Cosmopolitan supports Windows 7. The WinMain() polyfill is programmed to gracefully fall back if ANSI isn't available. So the binary still loads fine. You just need to run it in something like MinTTY instead if your TUI program is run on an old version of Windows.
I really like your view of the world, that programs should be portable, tiny, and just work. The slamming of so much functionality into a zip file is inspiring.
Would it be possible to do something similar with Free Pascal, allowing the recreation of something like Turbo Pascal, except really, really portable .com output?
Author here. Absolutely. I used to love Turbo Pascal and Delphi when I was younger. If Free Pascal uses GNU LD.BFD or LLVM LLD when it links programs, then all you'd need to do is is configure it to use cosmopolitan.a when linking system call functions like read(), write(), etc. See https://github.com/jart/cosmopolitan Another option is if Free Pascal wants to write all the system call support from scratch, then doing that now is going to be a whole lot easier since the Cosmopolitan codebase does a really good job documenting all the magic numbers you'll need. See files like https://github.com/jart/cosmopolitan/blob/master/libc/sysv/s... and https://github.com/jart/cosmopolitan/blob/master/libc/sysv/c... I've been working on a tiny C11 compiler called chibicc which has most GNU extensions and I managed to get it working as an actually portable executable with an integrated assembler: https://github.com/jart/cosmopolitan/blob/master/third_party... I also got Antirez's KILO text editor working as an APE binary. https://github.com/jart/cosmopolitan/blob/master/examples/ki... If we can build a linker too then we can get a freestanding single file toolchain + ide that's able to be a modern version of Turbo C.
Since I can't upvote your Show HN twice, I'll upvote this comment. I learned something interesting tonight from your websites. Thanks!
chibicc is a really fun idea! If I was in college, I would take a compiler course from you.
Taken from your README:
"chibicc is developed as the reference implementation for a book I'm currently writing about the C compiler and the low-level programming. The book covers the vast topic with an incremental approach; in the first chapter, readers will implement a "compiler" that accepts just a single number as a "language", which will then gain one feature at a time in each section of the book until the language that the compiler accepts matches what the C11 spec specifies. I took this incremental approach from the paper by Abdulaziz Ghuloum."
I love what you've produced here. I noticed that pthreads seems to be fail-stubbed for the time being, i guess it's a todo with tricky semantics?
As for holding off on UI for the time being is probably the right thing to do, keeping with the minimal style planning on going for raw-x11,etc.
However, as much as I personally love plain framebuffers (late 90s democoder myself), it'd be totally non-accessible for "plain" applications and totally lacking performance compared to anything accelerated when it comes to rendering. Graphics cards are just so many times faster that it's not really even the same kind of usage (only sad part is Apple punting on OpenGL driver updates even if Intel on the low end has become so much better).
That said, having looked for a minimal UI framework and sadly they seem to be far inbetween these days (libUI is fairly compact on win/osX but required GTK on *nix but they've lost a bit of steam since 2018)
Best of luck on it all (hopefully I can snatch out some time to contribute)
So far the only platform where I've managed to get threads to work has been Windows. I came pretty close to getting clone() on Linux working a few days ago. Threads are so hard because no API was ever defined for them in original UNIX and BSD, so therefore each modern system today does them in a completely different way. Another blocker is that not all operating systems let us modify segment registers which is something that the Linux compiler assumes is available for TLS. So it'd be a miracle if it ended up being possible to make threads work portably. fork() on the other hand was easy to get working on all platforms. I love fork() and vfork().
As for GUIs I'm still open to merging raw x11 support. One tricky issue is that we'd need to define a new API that veneers framebuffers, and so far Cosmopolitan has mostly refrained from defining novel interfaces, since the focus has been twiddling with low level magic numbers in order to make the textbook interfaces we already have work everywhere!
Ah segregs is gonna be a pita indeed if it's all hard-wired to the compiler.
Considered making some way to add "portable" modules yet(generic APE code with plat-specific impls) to avoid adding stuff to the core? I know the default calling-conv is slightly different but maybe by smth like defining virtual ports? (ie "send" structs since alignment rules should be mostly the same?)
Threads are a PITA everywhere. On Linux the SYSCALL stuff complicates things because of all the register moves, the FUTEX stuff, and potential races with how threads are exited.
Author here. I've been chatting with the Zig BDFL on Twitter recently about contributing APE support and he's been super supportive so far. It's going to be a nontrivial undertaking but would certainly be rewarding for everyone if we can make it happen.
Speaking of other languages, given that the APE is a zip container, it shouldn't be too hard to have a "starkit" like setup where the binary part is basically node/python/perl/tcl and the main script and all required libraries are in the executable.
Am I the only one having issues trying to get this working?
The webserver starts just fine, but once I add the index.html with zip as in the example, it stops working.
This is on Mac 10.15, CentOS 8 and Ubuntu 18.04 LTS.
Centos:
[centos@test ~]$ ./redbean.com -vv
error: Uncaught SIGSEGV on test.novalocal
./redbean.com
EINVAL/err=22/errno:2/GetLastError:0
Linux test.novalocal 4.18.0-240.10.1.el8_3.x86_64 #1 SMP
Mon Jan 18 17:05:51 UTC 2021
On the mac:
~ my-Maccie$./redbean.com -vv
Killed: 9
On Ubuntu:
ubuntu@localhost:~$ ./redbean.com -vv
error: Uncaught SIGSEGV on localhost
./redbean.com
EINVAL/err=22/errno:2/GetLastError:0
Linux localhost 4.15.0-136-generic #140-Ubuntu SMP Thu Jan
28 05:20:47 UTC 2021
IIRC, from vague memories of things flying past, Big Sur won't let a modified binary run once it's been checked by Gatekeeper. Which means the first run gets checked and notarised but when you add the `index.html`, the on-disk binary has changed and Gatekeeper won't allow it to run - I guess it's to prevent malicious code modifications, etc.
It looks like you can get around this by adding Terminal.app as a "developer tool" (i.e. it can create processes without hitting Gatekeeper), maybe that works? (once I did that and relaunched terminal, I can run cosmopolitan binaries).
> This will also prevent distribution from outside of their appstore giving it will prevent updates.
I think that if an app is correctly codesigned, Gatekeeper has no issue with it. Also if it's not correctly codesigned but you've ticked "allow unsigned binaries", you can still run them.
> the mothership knows everything you run on your machine
I can't remember the details from the last time this came up but IIRC it only sends a hash and possibly even then only the first time you launch it. Either way, they're not going to block out-of-appstore distribution except if you're a known malicious actor.
Author here. It works fine on RHEL5 / CentOs5 for me. I can also confirm RHEL7. Please file an issue. What will help in particular is if you can give me the faulting RIP address. That should be in the crash report or in your dmesg log.
$ zip -v
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.
Addition of index.html shrinks(!) redbean.com from size 204800 to size 204348, with changes starting from char 172938. That clearly corrupts the portable executable format.
EDIT: I am an idiot: it's working just fine... the messages looked like an error to me, but visiting https://localhost:8080/ does show a nice HTML page!
No luck for me on win10. I can open the archive but get errors trying to add to it. 7zip and win explorer zip both fail with either unsupported or corrupted archive errors.
Since it's a .com file, after downloading, win 10 also claims 'this dangerous file comes from the internet and is blocked' which you have to check a box to unblock it. I did, but still get the errors trying to add to it.
Yeah another killed: 9 on Mac BigSur.
Could you export the tool in a way that we don't have to zip new files into it? Instead we would have to gcc the whole thing once and be done with it.
Author here. The zip shouldn't need to be saved in any special format. If WinRAR is posing problems then please file an issue because I'd like to fix that. That error doesn't seem like something that would indicate executable corruption. I want to learn more.
If you want to learn more how these things work I'd highly suggest going through the PoC||GTFO archive (https://github.com/angea/pocorgtfo/blob/master/README.md) and check out entries by Ange Albertini or entries named like "This ZIP is also a PDF".
"This file, pocorgtfo19.pdf, is valid as a PDF document, a ZIP archive, and a HTML
page. It is also available as a Windows PE executable, a PNG image and an MP4 video, all of which have
the same MD5 as this PDF."
Zero/near-zero dependencies, minimal configuration, portable, and trivial to run. Awesome.
Even if Redbean here doesn't end up taking over the world, I hope that more ecosystems and toolchains sit up and take notice. We need more tools that look like this.
OK I see we are going crazy, me too. Thanks @jart.
I have been thinking about portable web apps with embedded SQLite for some time. I do not have the technical chops at C level to pull this off. I am really inspired by this project. I hope Redbean leads a way to distribute self-hosted apps in today's era of the cloud.
I still use the classic version of Tiddlywiki that is a single self-contained self-editing html file, and I "app-ify" it on Windows by changing the extension to *.hta.
But having the similar solution with the SQLite, would be a perfect combination.
I feel like I’ve learned more from your website and your source code in a few hours than I otherwise would have from years of study.
You have a real talent for distilling the complex into something simple. It’s apparent in the purpose of your programs, in your writing, and in your source code. Thank you so much!
Basics of computing architecture that I didn’t especially think were too complex for me to learn, but we’re too abstracted for me to become interested in, are laid bare in each one of these programs and their source. This is all incredibly exciting.
Just make sure you don't have another server running on :8080.
I had syncthing running and kept getting some error whenever I tried to run redbean. The error didn't make much sense, but eventually I realised that this was the casue of the error.
I'm really impressed how portable this is. The only improvement I can see is if it auto-opened your browser to :8080. That way it would be easy to distribute the binary and have people run your application and just interact with it through their browser. No need to ship electron then!
There is the possibility of other programs making HTTP requests so you shouldn't assume trust on any request automatically. What you can do is give the browser a cookie once when your app starts which you then check on every request.
Oh and listen on 127.0.0.1, not 0.0.0.0. That way your app is not needlessly exposed to the network.
I think you could trivially issue a system() call, you'd just have to use the corresponding command to open the default browser for each particular system. (Linux: xdg-open, MacOs: open, Windows: start)
I am floored. Not only is this just breathtaking sorcery, I also needed this exact thing for a project. I don’t understand probably 70% of the description of this software, but I understand what it does, and I’m more excited to carefully study and use this than I have been about anything in years.
TLDR: This is a packed file format that can look to the OS as being executable (for Windows/Linux/OSX). The code is x86 and so offers native performance on those processors. On other platforms, an x86 emulator is built in so can't offer native execution speeds.
While neat, its not the "best of all worlds" due to the lack of native code on anything other than x86/x64. Also, claiming "bare metal" is supported is a stretch as you are limited to having no I/O of any form (as there is no platform code).
I'm just one woman. What Cosmopolitan does so far, it does really well. It was only as recently as a few days ago that I got mmap() and malloc() polyfilled on bare metal. It has serial uart i/o. It's going to have e1000/virtio sockets soon. You can help me will that future into existence. I need people who know o/s dev and can help me write code that does things like correctly set up pic controller.
Sorry, wasn't meant as a criticism, just as an explanation because people were assuming magic...
Taking on bare-metal is a significant job much greater than your original task and you may be better targeting something like buildroot instead, after all what is the Linux kernel but a hardware abstraction layer.
How hard could stdio/sockets on metal possibly be? So far bare metal has been a walk in the park. I love the fact that PML4T lets me do things like move memory without copying it. I want to be able to have that power without schlepping in the entire Linux world. I believe it should be possible for programs to be able to boot on metal as just programs which are tiny and auditable. Especially considering everything runs on hypervisors these days, which are in some respects the true operating system. Ring 0 is the new Ring 3. Who among you is willing to accept being pushed to the outer bounds of computing? If MINIX gets to run in Ring -3 then it should at least be easy for our programs to run in Ring0.
Working on one bare metal system is relatively easy as long as you're willing to dive into drivers. Getting 1001 I2C drivers or maybe writing a drivers for tens of different types of flash devices is a different matter.
Your project will quickly become overrun with drivers for literally tens of thousands of devices... then comes maintenance.
Look at the Linux source tree... Good drivers are not trivial either. Simple ones are of course.
Basically, its a time-sink and it doesn't become useful for claiming "portability" until you're approaching the scale of something like Linux.
You're confusing portability with requirements. I'm aware that a long tail exists. Linux is a big tent for that sort of thing. I think that's great. I also want the freedom to live outside that tent during the times when all I need are the other 85% of use cases, which as I've shown, can be easily achieved by just one person. I don't believe that just because I ship actually portable executables that are able to do e1000/virtio that every hardware manufacturer is going to be kicking down my door eager to get support for their silicon merged. Linux is already doing a great job at that and I think Cosmopolitan is complementary.
As someone who knows programming in higher level languages well but has no experience with low level, bare metal, how can I get started with this stuff you're working with?
> So far bare metal has been a walk in the park. I love the fact that PML4T lets me do things like move memory without copying it.
> I believe it should be possible for programs to be able to boot on metal as just programs which are tiny and auditable.
I would love to be able to learn more about this. Maybe even contribute to your project :) perhaps a good way to start would be to address some small tasks you might have laying around?!
That file runs a few hundred lines of old school assembly in order to bring us into the modern era. Towards the end of the file you'll notice it starts calling functions that are written in C which configure memory: https://github.com/jart/cosmopolitan/blob/master/libc/runtim...
After it configures memory, the ape.S file is able to call _start(). It also sets a bit in __hostos so that functions like write() know to use the serial port instead of issuing system calls: https://github.com/jart/cosmopolitan/blob/master/libc/calls/... That's the simplest possible example of a driver with spin locks which uses the x86 IN/OUT instructions.
If you want to take a dive off the deep end right now with Cosmopolitan on metal -- simulated -- then try the following build commands on your Linux terminal:
git clone https://github.com/jart/cosmopolitan
cd cosmopolitan
make -j8 o//examples/hello2.com
qemu-system-x86_64 -m 16 -nographic -fda o//examples/hello2.com
sed -i -e 's/USE_SYMBOL_HACK 0/USE_SYMBOL_HACK 1/' ape/ape.S
make -j8 o//examples/hello2.com o//tool/build/blinkenlights.com
o//tool/build/blinkenlights.com -rt o//examples/hello2.com
On most computer systems these days can select a set of files and folders and tell the computer to "Send to compressed file", the files that come out of that process usually end in ".zip", and are thus known as "zip files"
This is a zip file, like any other... you can add and remove things from it. However, this ZIP file has a superpower, it can display the contents on the web... it has a web server built into it.
You can take this file, run it on a Windows, Linux, Macintosh, and it will work. You don't need separate versions for each different system... this just works on all three systems, unlike almost every program ever written.
This is one of the most impressive feats of programming things I've seen in my 40+ years of programming. The web serving is clever, but the superpower that it can run on anything really took a huge amount of work, which the author built into a tool called "ape", and has shared with the world, so other people can use it, and help her make it better.
So... will a new browser need to be created in order for this tech to go mainstream? In the case of building a web app does this eliminate the need for a server?
I've been programming since before you were born... and this is the most impressive hacking feat I've ever seen.
A single executable, that doubles as a valid ZIP file, that you can put a complete website into, also doubles as a valid executable on Windows, Mac, Linux.
The build infrastructure is a complete reworking of C to make it far more portable than even K&R thought possible.
I probably wouldn't have believed it possible, but here it is!
I will say, I WOULD absolutely love it, if it weren't for the little problem...that...I can't "just add to the zip"...because for whatever reason, it ruins the special setup...
on windows, 7zip and windows explorer's builtin zip stuff refuse to modify it, as if it was corrupted (understandable), but explorer can view it, on linux, zip (apparently info-zip 3.0 according to the version) can add to it just fine, but ruins the special properties, on linux it just spawns the debug gdb with "no symbol table loaded" and no register or assembly data, and on windows it's either wrong format, or `check failed: 0x1 == 0x0 (32)` so also broken...
I definitely love the idea of this, and the APE in general, and it works as-is, but modification seems impossible to me...changing to .zip doesn't change the breakage
anyone else had luck with adding stuff to it? is it just me? or is it like "it works with openbsd zip command only"?
I looked into building SBCL using chibicc last weekend in the hope that one could build a truly portable lisp runtime using the ape toolchain (Yes, yes, I know you couldn't possibly pick two more different approaches to software portability). Turns out the GNUC extensions for __asm__ are the primary stumbling block (though I'm sure there are others). tcc supports the GNUC extensions needed, and chibicc supports the thread local storage needed for dynamic variables. Looking forward to future developments!
You need __asm__ in chibicc for sbcl? I have good news for you. I did exactly that in cosmopolitan's vendored fork of chibicc here: https://github.com/jart/cosmopolitan/blob/master/third_party... I went on a coding rampage back in December, adding to chibicc pretty much every GNU extension under the moon: https://github.com/jart/cosmopolitan/blob/master/third_party... It should be a perfect fit for use cases needed by Cosmopolitan and SBCL. Right now it's unlikely to get merged back upstream because Rui is still focusing on using Chibicc for his book, and that requires being more conservative about feature inclusion right now.
I would highly recommend using chibicc. Bellard's work on tcc was fantastic when it came out. However it didn't age well. Its GNU extension support is roughly equivalent to GCC 2.x. The x86_64 support that got bolted on later isn't very good. It was much more elegant back when it was only doing i386. Hackability was also laid low by merging a lot of external contributions. The TCC design, while amazingly fast, also carries the tradeoff of making things like inline assembly hacks really hard because it generates the x86 binary content directly, rather than going through the intermediate step of generating an assembly file and running it through a proper assembler -- which chibicc now has!
I'm actually happy to merge your GNU-style inline assembly support and other GNU C extensions to chibicc. But for the purpose of the project, I need to rewrite your patches in the incremental manner, so it's not something I can do right now. I'm currently busy working on my linker project. Once it settles down, I'll take a closer look at your fork.
Thanks for clarifying Rui. I'm happy to do the bulk of that effort for you. As soon as you're ready, just tell me which things you want to see upstreamed, and how fine/coarse you want the granularity to be, and I'll send you pull requests!
That really depends on how I organize chapters of the book, and it will need trial and error, just like I'm continue rewriting book chapters as well as the compiler's commit history. So I think I have to do this myself.
This is something like a "fat binary" with some magic in the header that allows it to run out of the box on Windows and several different UNIXes, a bit like those sneaky shell scripts that run on several platforms[0]. It still needs to include some native code for each of the platforms inside the compiled binary, so it's perhaps better to see it as a slick packaging/distribution solution for C programmers than a total game changer for developers everywhere.
The more interesting thing I think is the mapping of C library functionality between Linux, BSD and Windows. There have been other attempts to do this, but they never seem to get much traction. I think it's because C programmers tend to get wedded to their OS of choice and then invest time in improving that platform's C libraries. The problem with trying to remain portable with everything is that you tend to be stuck working with the lowest common denominator functionality, which can be frustrating when doing OS-specific stuff would open up so many more options.
All that said, the cleverness of this project is inspirational in a way that might encourage other programmers to also focus more on building against simple functionality that is truly portable. It shows that you don't need all the bells and whistles to build cool stuff. That feels aligned with early UNIX philosophy and (more broadly) the hacker ethic, which is why it's neat to see, even if never gets adopted as a mainstream thing.
Appreciate the response. So it's a bigger win for C devs specifically. I am a C# dev so I am trying to understand what opportunities this project unlocks.
Why you have pigeonholed yourself? "X Dev" is so cliche. You can compile C# to x86 machine code - this project has less to do with C and more to do with PE format and syscall tables. It means that it's language agnostic but not standard library agnostic.
Author here. Thanks! I was deeply inspired when I read the source code to early versions of Linux and UNIX. The dream of that simple elegant power is something I'm hoping to restore for a new generation. All I'm doing to achieve that is simply standing on the shoulders of the giants who came before me.
Can you blog a bit of the career/cognitive/skill evolutionary history behind this? Did the cosmo stuff come out of this or something else?
Learned this in X level school, found out about Y, read about Y, learned this, got this job doing this, learned this from Z, got experience of AA... etc.
Of course, assuming this exists, and you exist. And blogs "exist".
Author here. When I first discovered the idea for Actually Portable Executable I thought I was going mad like Terry Davis. I couldn't even believe that it was possible until I actually did it. It cost me my job but I couldn't be happier with the result. This project is going to make problems disappear for so many developers. It's the sort of tool people can get excited about. I'm so happy that my prior career success and top-notch training have granted me the privilege of bringing something this cool to the world. Enjoy!
In all seriousness: whether it's exciting or not, this is something that should be getting federal funding. It really shouldn't be difficult for large, government-backed organizations to justify their interest. I don't have a lead for you (otherwise, I'd have invoked it by now to work on my own project[1], which has something in common through the use of polyglot files to achieve a similar effect[2], while mostly otherwise taking a different approach), but since you have a high profile, casting a wide net to attract the attention of someone able to secure the funding is something to consider.
Have you considered trying to make system calls inline to avoid the call overhead? Or already thought about it?
I've been banging my head against the wall of trying to make a micro KVM guest and the glibc startup code uses inline SYSCALL everywhere, I assume to make things faster. It does seem to call CPUID a lot too, but I guess it's not as expensive as having to make full-blown system calls.
Cosmopolitan defines linkable symbols for all the __NR_syscall constants so you can absolutely use inlining if you like to live dangerously. For example: https://github.com/jart/cosmopolitan/blob/91f4167a45f811fde6... There really isn't any performance advantage to doing this, because the SYSCALL instruction itself has something like a 2000+ cycle cost due to all the copying that needs to happen on the kernel side along with things like spectre mitigations which have made is slower. So the 8 cycle cost of having a normal read() function wrapper is the wrong thing to be focusing on. Cosmopolitan aims to address the performance cost of SYSCALL by making it possible to run your binary on bare metal, where your program becomes a kernel, and therefore needn't pay any context switching costs at all.
.apk files and .ipa files are secretly zip files too. With the addition of a few manifest files and other crufty things embedded in the zip, I bet you could add Android and iOS to the list of supported platforms too.
I've recently succeeded in building working .apk files byte-by-byte completely from scratch so I could maybe help some. Open an issue on https://github.com/akavel/hellomello if interested in some talking/advice and/or watch my talk on NimConf 2020 for an overview (I don't have a link handy as I'm on mobile).
The magic numbers file ends with a comment linking to a youtube video of numbers station playing very disconcerting and dissonant music, so we can add art/comedy to the list of features (which was already obvious!)
Normally when a browser asks for Content-Encoding: gzip, the http server needs to schlep the file into memory, compress it, and only then is it able to send the output to the client. Since redbean reads files from a zip archive, that file is already compressed. So what it does is it uses mmap() to load the zip executable in memory. Therefore instead of read()+write() it can just call writev() on the mapped memory area. That enables us to skip the extra copy. The kernel can just copy the gzipped file contents directly to the network wire without having to pay any userspace context switching penalties.
This is amazing!
There are a lot of low level things that I don't really grasp and, while going through the APE description, I saw this: "The other tradeoff is the GCC Runtime Exception forbids code morphing, but I already took care of that for you, by rewriting the GNU runtimes".
Can somebody, please, explain if allowing code morphing will increase security risks?
That's strange. I'd recommend filing an issue. Any details on how I can reproduce the error will be helpful, even if I have to temporarily rent one of those things. I'm on a mission to make binaries portable.
FWIW, it works on QubesOS, which is Xen based. Though it works only in my Fedora and Debian VMs, but not on my Archlinux-based one.
I think that's because I have WINE installed in the Archlinux VMs, since that's what gets started when I run ./redbean.com. But even after I remove WINE, it still doesn't work, I probably have to restart (i.e. remove WINE from the TemplateVM, not the AppVM) for it to work.
Author here. The name was chosen mostly out of playfulness to describe Windows since I seem to recall XENIX being the earliest example of Microsoft dipping their toes in the water with UNIX support.
Author mentions that the APE’s can be booted from the BIOS. Would this imply you could perhaps implement a lambda-like cloud service that could spawn/start the APE on demand? Edit, I’d guess you’d want to/have to, also have a tcp/ip stack in the APE as well (but perhaps just going for containerization of the APE is the better play and not have to worry about any of that..)
The trademark collision is going to test the joke of Red Bean Software:
> We're software producers of a different kind. Rather than market software the usual way, we openly deny its existence, and challenge our customers to do the same.
It is all pretty good, but why there is update of "static bool" variables from signal handlers?
In old-style C it must be "volatile int" (and only int), in new-style moder C it must be atomic-anything (bool is ok). But access to plain "static bool" from signal handlers & main thread is race and UB.
She says somewhere in her website that arm/apple silicon support is really easy to add, with no significant size or performance penalty, the same way she added qemu support.
Not sure the is a working example. But they talk of embeding qemu to create cross archetecture executables. But the Window's and actual compiled code would still be x86-64.
If you're getting `check failed: 0xffffffffffffffff != 0xffffffffffffffff (98)` make sure you're able to bind the default port (8080). You can run the symbolized version to see where the check occurs.
The webserver is novel but I am just wondering, how would it compare to something like Apache/ISS in terms on security/config/etc, or is this mostly just to move small sites around?
I tried to run it on an old 32bit pc with Linux. But it complained about not knowing the format of the executable.
But it would be awesome if I could get it to work somehow.
Alas, I wanted to try this out, but the format shenanigans triggered the Sophos "machine-learning"-based threat detection on my Windows laptop in mid-download.
Can it be extended in a way similar to CGI back in the day? I would just want to compile it with the CGI program rather than expect it to dynamically load.
You should file a feature request! We could crowdsource some ideas about the best way to do that. For example if you read the APE blog post, we've already got a JavaScript interpreter (duktape) checked-in to the Cosmopolitan codebase. I could easily link that into redbean and define possibly a nodejs-like API.
When I downloaded the file, it also had the date in the name. You have to accommodate that change in the instructions. Also be sure to run bash -c './redbean.com -vv' (also with the name as downloaded).
AFAIK a fat binary is just a binary with all dependencies included. It would still be compiled for a specific platform. The magic part about this is that it's a single file compiled once that can be run on most platforms. And it also adheres to the format of a zip file, so you can add, modify and remove assets as you please inside the actual file, post compilation.
No, fat binaries are not binaries with all dependencies included. Fat binaries are executable files that contain equivalent compiled code for multiple architectures. Some also facilitate running binaries across different operating systems. More: https://en.wikipedia.org/wiki/Fat_binary
The idea also reminds me of (executable) self-extracting ZIP archives that were common once upon a time.
You have a long history of taking HN threads into ideological flamewar and we've asked you to stop many times. The slack we cut users may be large, but it is finite, and this was the last bit. I've banned this account.
If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. I know you know where they are, but in case anyone is curious, they're here: https://news.ycombinator.com/newsguidelines.html.
Aren't most discussions on HN an ideological debate of one form or another? I think my error is being on the wrong side of the popular view here. Oh well, you cannot please everyone all the time :). In parting I thank you for the great, and not appreciated enough, work you do as HN mod. My best to you, sir.
Ideological flamewar is a specific form of internet hell and a clear failure mode for HN. I don't think it's hard to understand what it is or why we try to avoid it here.
Please don't respond to a bad comment by breaking the site guidelines yourself. That only makes the thread even worse. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and sticking to the rules when posting here, we'd be grateful. Note that they include:
"Don't feed egregious comments by replying; flag them instead."
I'll engage. I think your comments are clearly bigoted, and I'll tell you why.
What you do with that is up to you.
> Hmm, somehow I knew Justine is a 'transwoman' by this feat. Lucky guess I suppose...
You use scare quotes around "transwoman". Why is that? Is that a concept you
believe is not real somehow?
You are guessing that something technically impressive was achieved by a
trans woman. Why is that? Are women not capable of technical feats?
> Lucky guess I suppose...
This implies that women being technically inferior is something obvious,
something we all know but can't say.
> If all the high achieving women in tech end up being [yet-again-scare-quotes (and for some reason one word)] "transwomen", ...
This isn't a playful hypothetical question. You are reinforcing your claim that
women in tech are not high achieving.
> I think trans/self identifying people should be in a separate category so they don't exploit allowances made for under represented minorities.
> So when the dominant group starts taking the identity of underrepresented groups and coopting their protections
From these two statements it is clear you think that trans women are the
dominant group (men?) that are exploiting special allowances meant for women.
Because in your eyes trans women do not count as women. And they don't deserve
protection.
You are getting downvoted because the community recognizes the bigotry against
cis and trans women that is the backbone of your two comments.
>So it's a tiny virtual machine. Intriguing, but not really shocking.
Nope... it's native code, that runs on multiple platforms, all in one small executable.
How? Lots of very clever hacking, and trimming all the accumulated dreck out of the normal C runtime, while making it work across all the supported platforms. (Polyfill is the term she uses)
This is the most amazing thing I've seen in a decade or so. It is right up there with GIT in terms of the possibility space it opens up.
Admittedly they solve different usecases, where Redbean seems to want to serve a whole static site as a single file, which isn't currently possible with Caddy, (but you can run `caddy file-server` to serve the current directory as a static site), but it may become possible in the future with Golang's new embed package https://golang.org/pkg/embed/
Caddy is cool but it has a separate 33mb executable for each operating system. Redbean is a 128kb executable that runs on all platforms so it has fewer moving parts and is 1584x tinier. All the convenience of being able to "just add your assets to the zip executable" wouldn't have been feasible if we needed to repeat that process n times for each operating system.
That's actually a pretty interesting idea. I wonder if we could make a lite version of Caddy bundled with your website embedded...no need to mess with file systems or permissions or devices and folders to run a static site. Might need some massaging to become a truly pleasant workflow though.
The go std library offers the embed package. You can just embed your files and use the std lib's webserver and generate a single file binary.
No need for Caddy.
> Is that smile at the end of your first sentence meant to be condescending?
What? Of course not! You're reading way too much into that. It's simply enthusiasm. I read about Redbean and it being a single file webserver and the first thing to come to mind is Caddy, as "prior art".
Like I said, different usecases; Caddy is a general purpose webserver which has automatic certificate management, H2/H3 support, reverse proxy, file server, a strong plugin system, the list goes on. Of course it'll be bigger in size. Redbean is super cool and does have the advantage in file size and portability, clearly, but it is essentially a single-purpose tool. And there's nothing wrong with that.
Ok well this doesn’t work on Mac, so much for not needing to cross-compile on every architecture. It’s because of the OSX Gatekeeper but if majority of people need to self-sign or otherwise compile this binary I’m not sure if it is that portable be it if it’s only a Mac thing. This dumps the zip content directly on the wire skipping an extra kernel write. There have been attempts before to run the same binary on multiple architectures by inserting magic headers without having to recompile. There are shell/batch versions of the same thing [0]. The issue is such projects usually never get much traction since every programmer is wedded to their choice of OS and typically relies on APIs specific to each system. There are also “fat binaries” which accomplish the same thing if one truly wanted a cross platform single executable. There isn’t a huge need for them currently and not just because they’re “fat” (something this approach seems to tackle primarily) so I expect for the same reasons there won’t be many needs for this approach either.
I see a lot of people making the comparison between this and the OG Linux tools. I absolutely don’t think this qualifies as one, most old Linux tools are irreplaceable or at least were when they first came out and there were no alternatives. Serving a bundled zip file on a web server across many different platforms is not something people are dying to do. You can run a web server like nginx or Apache to index your files or if you really hate those why not a single line command which is supported on almost any platform:
Justine has built a c library that allows you compile a binary once and have it run it on any os or baremetal. The SAME binary. Quite frankly, that sentence doesn't even make sense to me.
Check out https://storage.googleapis.com/justine/cosmopolitan/index.ht...
As far as I'm concerned, this is literal magic. Look at the magic numbers: https://github.com/jart/cosmopolitan/blob/37a4c70c3634862d8d...
I could go on, but there's no binary portability comparison with any other language. And she has made some pretty neat optimizations.
Back in the day, I saw some pretty neat stuff with the ELF format, but this takes the cake.
Wow.
Edit: I'm editing because this is just so bloody absurd.
https://storage.googleapis.com/justine/printimage.html
$ ./printimage.com someimage.jpg
Like wow. And also video.
https://storage.googleapis.com/justine/printvideo.html
I'm struggling to put my shock into words. I've been around.
There's engineering. There's academia.
But this falls into straight-up wizardry.