It's not often these days that a project pops up where I start thinking about re-imagining our whole build/deployment strategy around it. This project easily hits that high watermark. I'm already thinking about how we re-architect a few existing projects around it to reap the benefits. Very excited to do some new projects with this as well and get to understand it in more detail. Many thanks!
PHP support would be beyond amazing, what would be required (top-level) to get that working/usable?
I think this is awesome too, but I’m at a loss to describe where I could replace already built services. Can you share some perspective on specific things that you see this can do that is amazing?
My industry routinely has to have electrician install additional controllers (basically just a Linux industrial computer) on customer sites because the ones already there won’t support our software. With this, I could imagine just shipping a binary on a flash drive that the customer can run on whatever they have on site. There is nothing stopping that other than it’s too hard to build our software for hundreds of hardware and OS combinations. With this you probably only need to build a handful of artifacts. It would save thousands per install.
If there were a plug-in system to be able to run different languages than Lua, that would be cool too, but maybe I’ll have to learn Lua.
That sounds like a good fit for redbean. How often are those industrial computers updated? I ask because backwards compatibility has always been a focus of the project. redbean runs on RHEL5+ (c. 2007) and Windows Vista+ (c. 2006) by default. See https://github.com/jart/cosmopolitan/#support-vector Lua is real nice and I offer support services too. See my email at https://github.com/jart/
Yea, I’m going to look at it more, but sometimes those computers are a few years old, many of them are on ARM too. Having windows supper would really open doors, because our software is designed to be the low cost solution and if we could even run it on several redundant office computers then even that would be sufficient for our business needs and though it sounds crazy I just think it might work.
I've been following this project but am curious about what are the actual use cases apart from being able to send a portable web site around?
If you're running a production web server you're not really switching OS's often. Something like nginx is battle tested so what would be the benefits of using redbean?
It is a marvelous piece of technology but I am struggling to see the use cases right now.
redbean was originally intended as a locally running web server, sort of like an Electron competitor, where you launch your GUI in Chrome, rather than linking a copy of Chrome into your executable.
We live in a fragmented o/s world. If you're someone like me then you've got a Macbook from the office, a Windows PC for games, a Linux workstation for compiling code, a FreeBSD server, an OpenBSD router, etc. When you're dealing with so many different systems, sometimes just having something as simple as a sed command that works reliably the same seems like an impossible ask. Now we've got an entire app platform that works on the lion's share of PCs/servers in a small 1mb file.
It's also a question of being able to distribute code. I used to work on the TensorFlow team. We were tasked with building an open source library that people on pretty much every platform imaginable would use. It broke my brain just how difficult it was for us to ship open source binaries that actually work and don't cause an avalanche of GitHub issues. Even just working on more than a few Linux distros felt like an impossible ask back then. Now that I've figured out how to do it for every distro and seven operating systems total in just one file, I wish I could go back in time and use tools like redbean and Cosmopolitan Libc to fix all the things with TensorFlow that I wish I could have done. It'd've been a different project.
redbean as an electron alternative is mind-blowing! I had the same question as the op, and this perspective does put me in fascinating directions. Like, this could be the most light-weight electron alternative, with the server acting as a escape hatch to sneak cross-domain communications and computation intensive parts of the app, etc.
How would this work? Does the user start and stop the server manually? One nice thing about an electron app is that you open and close it like a normal application.
Is there any possibility that this could be extended to be able to specify a preferred browser or ordered list of said? I ask this because part of my work involves accessing a large number of user computers and I've found that few users set their preferred browser, they will overlook the browser request to be the default for long periods of time or just dismiss it.
This is a great example of something that can be built with the new unix module. It's unlikely we'd add a high-level API for this. The good news is you don't need us to do that since the unix module gives you all the power you need, and if it doesn't, please file an issue and we'll fix it.
I did something like this before redbean. You can set it up so double clicking the app icon launches the server and opens the UI in a PWA window, not a browser tab. Then you can set it up so that the server exits when all application windows are closed.
my similar approach is one golang webserver binary that runs across multiplatform,and do ui in a browser,is redbean a similar idea but using lua instead?
I can see it as an Electron competitor which is itself a desktop app competitor. I can also see the value of having a cross platform technology albeit using web development.
I cannot currently see it as a traditional web server replacement but perhaps that was not the intention. Running the web server locally loses the benefits of single place updates.
I could see usage for web developers who would like to develop in Windows (natively without WSL) and then deploy it in a Linux server by just copying files via SFTP.
This is so amazing - I had to give it a shot. There have only been a couple of projects I’ve ever gotten excited about on HN and this is certainly one of them!!
I’m curious if any other users ran into issues with MacOS running 2.0? I may have just missed a step, but I started an issue nonetheless.
This is a high priority item. It sounds like you're using a recently released Mac OS X version. One of the great fears has always been the possibility of Apple breaking the UNIX system interface, like they did to the Go team in the past. So I'd like to get any issues resolved here as quickly as possible. I left a comment on the GitHub issue about next steps.
So many great features in this release. Love the .args and subprocess support. I have been toying with redbean on and off, and I have been wanting to see if I could kludge together a poor man's long running job (scheduled) runner (something like Celery).
Try hitting ctrl+shift+r in your browser. Sometimes browsers don't respect the cache headers which say 1 hour max and I haven't figured out why. This new release includes functions like setitimer() to help the primitives be there for anyone wanting to use Lua to build something like Celery.
HTTP has some strange rules about using stale caches so usually you want to add must-revalidate to your cache-control header. That ensures that the browser must revalidate once the cache goes stale.
Personally I've opted for "stale only" caching, so everything is served with Cache-Control: max-age=0,must-revalidate and a Last-Modified header and the browser will always make corresponding If-Modified-Since requests. This means significantly more requests per page, even if the responses are mostly 304 Not Modified, but getting to avoid all forms of cache busting makes developing a lot nicer.
It's definitely cheap to execute, the problem lies in the network overhead. With sub-resources (css, js, images) you can go from 1 request per page to 10 or 100 which is still negligible for fast connections (10 mbps+, http2) and servers with low request overhead - but the worst case scenario is high latency http1 connections where each request really matters.
However if you are serving clients with highly restricted bandwidth you're probably going to want extremely cacheable resources (public, immutable) and perhaps even a completely different site architecture.
Maybe that depends on the OS? Firefox on Linux does a normal refresh for me when doing CTRL+R but a hard refresh (without any cache) when I do CTRL+SHIFT+R.
This is standard at least across Chrome and Firefox on Linux and MacOS— (ctrl|cmd)+R does a normal page refresh, (ctrl|cmd)+shift+R does a "hard refresh" where it reloads every resource on the page without caching.
Also, in Chrome, if and only if you have the dev tools open, right clicking the reload button will give you a menu with the options "Normal Reload", "Hard Reload", and "Empty Cache and Hard Reload". The third option will ensure that requests initiated by JS or that otherwise weren't part of the page load also won't be served from cache.
Ok MacOs (Chrome), I'll usually open View and hold down shift. 'Reload This Page' changes to 'Force Reload This Page' and I like the visual feedback that it's actually doing what I intend.
But it hasn't always been easy to find documentation about it, thanks to a combination I think of "simplifying" everything and Google not delivering correct results since somewhere around 2010.
Thankfully now there is Kagi that actually makes a bug report if you provide them with an example that doesn't work.
I think I've been using that shortcut for as long as I can remember. In web development circles it's certainly common knowledge, as you always end up in a situation or two where you need to reload without cache, but you don't want to clear the entire browser's cache.
> Thanks to Linus Torvalds, we not only have a consensus on architecture, but we've come pretty close to having a consensus on the input output mechanism by which programs communicate with their host machines, via the SYSCALL instruction.
Could someone explain this? I cannot find any information online
It might help to understand API vs. ABI with respect to the C language and CPU architectures.
I put definitions here awhile ago, and mentioned the Actually Portable Executable project as an example of taking the ABI philosophy to an extreme, and ignoring APIs (source code).
It's unfortunately not explained well in many places, and even experienced C programmers don't understand the details. You won't learn about this in a C programming book, because it's not really part of the C language proper. It sits at an awkward spot between the language, the CPU, and the operating system.
----
The relation to Linus Torvalds is that historically Unix kernels were developed in a single tree and there was no stable ABI. For example I believe OpenBSD and NetBSD are still like this. But Linux is different in that the kernel maintained a stable ABI, and you can run different user utilities on top, without necessarily recompiling them against headers.
In short, a stable ABI is a must for portable (and backward-compatible) binaries. It does not matter where the stable ABI is. It is fine to have a stable userspace ABI only if you control the userspace. (e.g. Windows)
A syscall is a method for calling a kernel routine to do something outside the current process.
All i/o (network, filesystem, ipc), memory allocation, process interaction, signals, etc go through syscalls. open(), read() and write() are all syscalls.
If the syscall interface exists then the only thing left to do is to execute the program -- which depends on the cpu executing the machine instructions. If the machine instructions are for the wrong cpu then emulation may come into play.
I've been having a lot of fun with this developing tiny webapps using Fullmoon[1]. I love Lua, but I frequently bounce between a Windows PC and a Linux PC. Having redbean + Fullmoon has made it a breeze switching back and forth without having to deal with system Lua installs. SQLite and the thorough amount of built-ins[2] is also a dream. Lua also has a lovingly awesome hobbyist community, so having another outlet for me to leverage that ecosystem is great.
Redbean is amazing. I was able to run some TLS stress tests against it (where you setup a handshake and just disconnect). I was able to pump out about 1200 connections per second between my desktop and laptop on 1gb ethernet. I only had 5 handshake errors.
Half a percent is small but there should ideally be zero errors. Could you file a report so we can fix that?
Part of what makes the redbean TLS stack fast is I spent some time hacking on MbedTLS to improve its performance. One of things that's counter intuitive about crypto code is assembly can be safer in additional to being faster, since it helps guarantee the compiler doesn't add branches, for things like bit overflow carrying in arithmetic operations. One example is https://github.com/jart/cosmopolitan/blob/master/third_party... which I wrote to make the NSA curve much faster. I've been meaning to upstream into MbedTLS. Another thing I added is https://github.com/jart/cosmopolitan/blob/master/libc/nexgen... which greatly improves the performance of RSA multiplication using Intel's ADX ISA. But C crypto code can be beautiful too! For example, I found Everest (curve25519) particularly impressive, since they solve overflow in a different way, and as such, it's quite possibly the only large piece of C/C++ code I've ever seen that contains zero conditional branches and zero pointers. https://github.com/jart/cosmopolitan/blob/master/third_party... The closest thing I've found to measuring the performance impact of my changes is by recording how long the MbedTLS test suite takes to run. The speedup for suite_ssl is 13.11x faster, RSA is 1.91x, ECP is 1.86x, and ECDSA is 2.84x.
Oh phew! I've always aimed for five nines reliability in my career so hearing half a percent (one nine) while small is a big deal for my rep. I would encourage you to keep torture testing redbean. Let me know if you spot any opportunities for improvement!
Really impressive feature additions since the first time I heard of it. Is Lua backend-able? I've been looking for a langage since PHP, the node ecosystem seems too volatile to me.
Yes and some of the largest web infrastructures in the world run on it! I'm not sure if it's still true, but it's my understanding that CloudFlare runs it at the edge globally. I personally know of several other _very large services_ running it at massive scale.
In my experience Lua in nginx is such a pleasure to work with. It's all transparently async using nginx's event loop. You just write regular procedural code in Lua and the runtime handles yielding/resuming for you automatically. There's no special async/await stuff. Just write your Lua code and the runtime figures out the yield points internally. It's a breath of fresh air.
Thanks tomcam! There are still several sections that need to be completed (mostly around template processing), but getting feedback like this is always encouraging.
Lua also powers itch.io! [0] Leafo, the creator, has also built Moonscript [1], a language that compiles to lua, and Lapis [2], a web-framework for Lua and Moonscript. I am always fascinated by his productivity.
You're not alone. Maybe it's lack of knowledge on my part... but too much around node / npm, still seems like copy/pasting random incontations into the command line to try and solve whatever problem I have at a specific time. There is lot's of good in there, but somethings just don't work how you think they will.
For instance, we've been building a totally static site recently in node/npm/js/posthtml and posthtml has a concept of "local variables". A project called posthtml-expressions in theory allows you to put "expressions" into "HTML" modules (if's, loops, etc.), but it only works with "globally" defined variables. i.e. you can't define a variable locally to a component, you need to define it "globally" in the project, which IMO defeats the whole purpose of the project. Nowhere is this easily explained or defined in their documentation. The first thing I want to do is define a property/attribute inline to a component, otherwise I end up with word soup of global variables - $page_1_title, $page_2_title, etc., rather than <component title="xxx">. Bizarre. No idea what the point is if I can't declare variables "locally" to said component? And this is the feeling I've got of lots of NPM packages - there is just no cohesion between things. I love the work people have done, it's literally saved me hundreds of man hours, but I've also spent untold hours wrestling with things that seem obvious to me, that should work, which frankly just don't work how I expect. Maybe it's me?
If you like this, you should support Justine & the project, if you can spare a few bucks a month:
> Funding for the development of redbean was crowdsourced from Justine Tunney's [GitHub sponsors](https://github.com/sponsors/jart) and [Patreon subscribers](https://www.patreon.com/jart). Your support is what makes projects like redbean possible. Thank you.
Maybe Cosmopolitan Libc can be developed to become a full-featured glibc replacement like Musl and what LLVM libc is going to target? To be able to link with large C++ codebases it is still missing a ton of small cruft...
Wait, you reimplemented most of GNU Readline in a single file with no dependencies? Appreciate you started from someone else's library, but yours actually looks feature complete! Wow.
Could you join our Discord and help troubleshoot the issue with me? https://discord.gg/EZwQUAcx Normally when it fails to run, it's because WINE is installed to binfmt_misc or WSL needs to have binfmt_misc disabled. But it looks like it ran the script in your case. Could you try downloading https://justine.lol/ape.elf and letting me know if that runs? It should, since it's designed to work and is regularly tested on CentOS5. It's also possible you might have a different shell installed.
Most languages expect you to have a high quality web server between the outside world and your code. Nginx, Apache, IIS, etc are extremely well tested and secure. They can serve up assets fast, cache, and much more.
A popular way to host now is to have an nginx server or process that accepts requests from the outside and then makes a request to a less capable web server for the app internally to provide a response, also known as proxying. This provides a clear barrier with outside world in one very concise config file.
What redbean apparently has is a sufficiently high quality web server included, so they can handle requests from top to bottom in one package (vertically integrated).
Thank you for explaining this. To add numbers to the explanation, a Python ThreadedHTTPServer can do about 1,500 qps on my machine whereas NGINX does 400k (and redbean does 1.1mqps). https://docs.google.com/presentation/d/1WEtSyz5oyTTGe0Fi2Nhh... So one thing that's always helped Python be more secure and stable, is having NGINX as its advance guard which can store and forward requests. That way Python can do its job as quickly as possible. Because untrusted clients can do evil things, like open a bunch of really slow clients that send 1 byte every second. NGINX is outstanding at addressing that with its evio model. redbean does almost as good of a job. But poor little Python would need to preempt the entire server acquiring its GIL for each byte received, and that would cause a lot of contention.
In my GitHub profile photo https://github.com/jart you'll see me holding a stuffed toy I call greenbean which has been cute little work desk companion of mine for some time. Later on I came into possession of a stuffed red lobster. A Chinese friend of mine suggested the name redbean as a companion for greenbean. Later in 2020 I was writing demo programs to showcase the capabilities of Actually Portable Executable. When I decided I wanted to write an HTTP server, redbean was looking up at me on my desk. That's when the idea clicked and it's been redbean ever since.
It’d be fun to add raft support so that multiple readbeans could share state such as monkey patching updates and other useful distributed features. On bare metal this could be a whole stack in a tiny bin.
Just tried and am very impressed with APE and redbean. However, I had to turn off Windows Defender to even download (much less run) redbean, due to it detecting redbean as Trojan:Win32/Sabsik.FL.B!ml
Windows Defender quarantines the file within seconds of download, but even when I did catch the option to Allow, it never seemed to stick.
It would be nice to either find a way to avoid redbean being seen as a virus (I'm guessing it's the self-modifying code that bothers it), or ask Microsoft nicely to add redbean to the False Positives list.
However, I'm guessing it would be easy for someone to misuse redbean too, deploying what looks legit but with some added nefarious code. In that case, you'd need to submit checksums of legitimate releases to Microsoft (and other antivirus vendors) just to make deployment of the base redbean work without complaint. And anyone who adds to the archive for their own use (basically any real use of it) would have to do the same...
Not trying to discourage anyone. This is a great project! Looking for solutions to the false positive antivirus problem.
Author here. I would happily share information like hashes and build artifacts with Microsoft and other virus scanners if it would fix all these false positives. However I don't know how to do that (other than uploading to VirusTotal which I've always done) and Windows is just 1 out of the 6 platforms we support. I don't charge money for redbean. I don't have a commercial interest in distributing these binaries. I'm just an open source developer trying to help out. It's up to the community members who want to see it unflagged with Windows to help us make that happen, and I'm happy to cooperate in any way I can.
I love this project! I still remember that one of my first questions when I started programming was why there wasn't really a standard between OS for executable files. And while I always thought it should be possible, seeing it implemented and laid out is a completely different beast.
We're working on improving our bare metal support. Once we embed a TCP/IP stack your redbean will optionally be able to run as its own autonomous operating system with the absolute lowest network latency and minimal attack surface area. It's a idea whose time has come now that everything runs under a hypervisor in the cloud. If the hypervisor is the new operating system, then it'd be nice to not need a second operating system too.
I love your work! How do you find the time to work on readbean/ape/etc? I sponsor you on github, but surely donations pale in comparison to what you could earn at an engineering day job.
Do you plan on writing your own tcp/ip stack with cosmopolitan? Why not pull in the networking stack and syscall libraries from MirageOS?
It's not so much could but have. I used to work at Google Brain. I decided I'd rather be doing this instead. I make it work by living modestly, eating ramen, and avoiding doctors. Thank you for sponsoring me! It helps a lot.
I have really enjoyed your submissions in the past year. Just sponsored you on Github. I feel like your work is of sufficient excellence and utility that you shouldn't need to eat unhealthy food to get by. Have you considered applying for grants from the non-profit arms of big tech companies? I do think that if you can combine Python in redbean in a seamless way, it would really drive adoption and lead to more sponsors for your work. Anyway, thanks for many hours of entertainment watching blinking lights.
I don't know much about the subject, but some quick searching leads me to believe that the first step is to create a legal non-profit entity, since that looks to be a requirement for all these big companies to dispense grant money. I think it costs a fair amount of money in fees and legal help to make a 501c3, so maybe you can do a Gofundme to raise the money for that (or ask for help from lawyers). Once you have the entity set up with you and people you trust in control, you can write a basic grant proposal, which wouldn't look that different from the articles you've already written explaining the project and the benefits. Then you can customize this with an addendum that responds to specific questions for each grant request application. And then send it to any organizations you can find that seem like good fits. I'm not a lawyer (and this isn't legal advice), but if you can get enough funding, perhaps you can "hire yourself" as the first contractor to the organization (I don't know if this runs afoul of the rules). I guess it's a lot of work if you've never done it before. Your best bet would be to find someone who has done this a bunch of times (maybe someone associated with the Internet Archive or something) who can help put it together for you.
I’ve been part of over 5 501(c)3 and 501(c)6 startups and unless something drastic has changed in the last few years since COVID, it is not that expensive (well under 1k). Consult with someone who does nonprofit creation and they will let you know what 501(c)X that the effort would best be described as and absolutely shouldn’t break the bank.
I can't edit my comment (maybe I waited too long?) so I'll add more here...
My comment about not knowing if running redbean on bare metal was sarcasm was a comment about me, not about redbean or Justine, so no disrespect intended.
There's a lot of "turtles all the way down" today (a web server compiled to WASM so it runs in a browser running on a OS hosted in QEMU that's running on another host OS that's a virtual machine running in a linux container on top of a hypervisor running on an X86 simulator...) so I quite honestly couldn't tell if the idea of running redbean on bare metal was sarcasm or a joke.
But Justine says its true, so I guess it's not a joke. Consider me schooled.
I agree because the whole point of being able to run on metal is to have fewer turtles. How many vendors control the space between your code and your computer? Who controls your memory? I think those are questions worth asking. My goal is simply to help people get closer to the source.
That's correct. Redbean is already capable of booting on bare metal. We do what we must because we can. The executable file format redbean uses is a DOS Master Boot Record with a partition table, so it's technically a valid disk image you can `dd` too. If you're willing to trade away Windows support, then it can be an EFI image. The main blocker is figuring out how to get an e1000 and/or VirtIO driver in there with a TCP/IP stack. Right now Cosmopolitan bare metal support is only adequate for stdio applications, which use the serial port and read from the zip fs.
Recently, I took this approach to get proper ACPI S01x sleep on my Lenovo X1 Fold (a wonderful device BTW, once you take the time to work around Intel drivers bugs)
A battery discharge rate of 0.5%/h in sleep is just great... but I think I can do better: I'm now trying for 0.25%/h.
Imagine if you could immediately resume your foldable oled tablet, and it'd have only lost like 6% of the battery. With a 20% hibernate trigger, it would remain immediately available for over 3 days straight!
> That's correct. Redbean is already capable of booting on bare metal. We do what we must because we can.
I love the spirit :)
> The main blocker is figuring out how to get an e1000 and/or VirtIO driver in there with a TCP/IP stack.
Why? Is it for performance reasons or security reasons? (or both)
> Right now Cosmopolitan bare metal support is only adequate for stdio applications, which use the serial port and read from the zip fs.
I'd suggest you "think different", and use instead something like ppp to create a TCP/IP stack over a serial link.
Modern btuart implementations already routinely achieve >1Mbps on commercial devices. The GSI as seen on the Intel Serial IO devices support bitrates over 20Mbps.
This could buy you time until you find a better solution, if it's ever needed (which I doubt as back of the envelope estimations make me believe you'll hit other limitations before)
It's less about perf/security and more to do with simply needing the time to wrap my head around how to do it. Entering ring0 is a whole different world. My productivity depends on a responsive TDD workflow and it's hard to test kernel code. I currently do it using an emulator I wrote called Blinkenlights, because the existing ones (Bochs and QEMU) couldn't be integrated into my Makefile / Emacs build system. So I not only need to build the kernel but I need to emulate the CPU features the kernel needs too.
> It'll be nice to know that any normal PC program we write will "just work" on Raspberry Pi and Apple ARM. All we have to do embed an ARM build of the emulator above within our x86 executables, and have them morph and re-exec appropriately, similar to how Cosmopolitan is already doing doing with qemu-x86_64, except that this wouldn't need to be installed beforehand. The tradeoff is that, if we do this, binaries will only be 10x smaller than Go's Hello World, instead of 100x smaller. The other tradeoff is the GCC Runtime Exception forbids code morphing, but I already took care of that for you, by rewriting the GNU runtimes.
> Probably related to #399. The recommended approach would be to use a full emulator like Bochs. It's not something we use at the moment so we can't provide support on this. Although we do intend to have APE support ARM at some point in the future.
What would be awesome if fossil (the scm) could be integrated into this. This would enable to put you repository on a USB stick and you could use it (including issues etc) on any platform without installing anything.
Program 'redbean.com' failed to run: Operation did not complete successfully because the file contains a virus or
potentially unwanted software
At line:1 char:1
+ .\redbean.com -v
+ ~~~~~~~~~~~~~~~~.
At line:1 char:1
+ .\redbean.com -v
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : ResourceUnavailable: (:) [], ApplicationFailedException
+ FullyQualifiedErrorId : NativeCommandFailed
From cmd.exe I get
The system cannot execute the specified program.
I'll see if I can get it "unblacklisted", Perhaps my anti-virus got to it first?
Yes please help us get it unblacklisted with your scanner. Security vendors work by keeping an eye out for the strange and unusual, and I myself am strange and unusual. The last month or so has been particularly bad with the virus scanners. Possibly due to some random change I made to the APE assembly or performance optimization in the C library. For example, we've needed to make changes in the past where error messages containing the word "oldskool" needed to be removed because virus scanners thought that made it a virus. Also if you get a chance, please upvote the redbean binaries on VirusTotal. I always try to do that (I'm "howishexeasier") since it helps people verify that a binary came from me.
Traditionally the open source community worked around these issues by not distributing binaries and instead asking people to build the software on their own. I like the convenience of binaries because open source software is becoming increasingly fragmented and impossible to build. So then people use interpreters instead of compilers, which are slower. In any case, I don't think the source code workaround is going to last forever. Many companies are now focusing on applying the virus scanner model to source code too.
Microsoft Defender can be particularly difficult about unsigned executables, which I appreciate is annoying for cross platform developers that don't have that problem elsewhere.
I've submitted this as a false positive under our enterprise licensing.
The invalid system call issue is likely due to a recent Mac OS update. It turned out redbean was depending on a Mach system call when the intent was to only use the UNIX APIs which I believe will be stable since they're much more public. https://github.com/jart/cosmopolitan/issues/426 New 2.0.2 release available on https://redbean.dev which includes the fix.
Wow. Could you please open a GitHub issue with a `./redbean.com --ftrace` log? I only need the fifty or so lines of the log. Since that tells me which system call is likely the culprit. https://github.com/jart/cosmopolitan/issues Alternatively you could email it to me: jtunney@gmail.com Thanks!
It's an instant `Segmentation fault: 11` - although this might be because I'm on M1 (although it still happens under `arch -x86_64 bash` which forces it directly through Rosetta 2.) Seems I can't run `dtruss` or anything useful either to track it down further because of the new stringent SIP protections, sorry!
One thing that helps is the binary size is tiny and the memory usage patterns are tight. It's bloat that usually makes fork() go slow, since if you have a lot of little pieces updating disparate memory locations, then you're going to have a lot of page faults. Redbean was built with a singular first-principles vision. When I started working on this, all I had was an empty file and an assembler. As such, I spent a lot of time using things like linker scripts to organize exactly where in the binary code gets relocated.
>have a lot of little pieces updating disparate memory locations, then you're going to have a lot of page faults.
This is wonderful, just thinking out loud. Out of interests would something like ECS (Entity Component System) be useful or yield better memory usage since it should in theory trash the cache less ?
Just guessing here, but redbeam likely does not have to access the filesystem as the files are in the binary and thus saves on syscalls and context switches.
We ported Fabrice Bellard's QuickJS engine to Cosmopolitan last year. We ported Python3 too. (You can build Python3 as a 5mb single file Actually Portable Executable!) So integrating them into redbean is on the table. The big question is whether or not redbean should be multitenant with languages, or if we should ship separate binaries where you choose the redbean "flavor" that has your preferred one. I'm leaning towards the latter, since there can only be one REPL language. What do you think?
>> The big question is whether or not redbean should be multitenant with languages, or if we should ship separate binaries where you choose the redbean "flavor" that has your preferred one. I'm leaning towards the latter, since there can only be one REPL language. What do you think?
Would love to see Python support. This project and your libc sort of break my brain and I reverse engineered weird stuff for a decade professionally. I understand how it all works, but it is easy to understand something once someone built it. Some of the design choices and constraints were just not things I would have thought reasonable for a project like this.
I'm open to considering change. For example, the project has a lot less assembly than when it started. Please join our Discord and tell us the particular things you had in mind! https://discord.gg/EZwQUAcx
I did not mean it in “I would not have done it this way” way. Just “I did not even consider that a way to do it” in a good/great way. It is like… you had a clear end goal in mind and all of the things you wanted led you down this really amazing/cool path and the end result is just incredibly cool to see. It is the sort of project you can only really get done sticking to a plan (first principles really?) and not letting it get watered down along the way.
Thank you! https://justine.lol/sizetricks/ covers some of the hacks for anyone reading who's interested. It's certainly felt the last few years, since I discovered the hack, the universe had put me on a mission to make executables portable. Writing files like //ape/ape.S also felt like I was writing a chronicle of the history of modern software. It's been a fun ride!
I also think the latter is cleaner. I might also suggest that if you are going to integrate with Python, you should make it as "batteries included" as possible with the Python data stack (numpy/pandas/scipy/sklearn) so that you can easily add a redbean web app to an existing Python project without worrying about how those "heavy duty" libraries will work.
that seems sane to me with my novice understanding of the project. im most confortable with Python3 & have existing projects which I would try to port over as a test.
a python3 flavor could be experimental at first, I bet it would boost overall adoption & be an unfair advantage over nginx approach.
having multiple repl in one binary would remind me of containers in a very strange but cool way, that would be more a fun experimental feature a rabbit hole for others to explore.
> having multiple repl in one binary would remind me of containers in a very strange but cool way, that would be more a fun experimental feature a rabbit hole for others to explore.
It would also let the flavors of redbean compete for donations, which could be directed towards your language of choice (say python3 for you) while limiting the size and complexity.
I don't know if it makes any sense for this project but Caddy's approach to this is very nice. Would probably need a corporate sponsor for the build server though.
Lua has similar uses to those but has a uniquely simple implementation and very close relationship with C that probably makes it a far, far better choice for this. I'm not really a lua lover in general but this is basically exactly what its made for and it shines in this role in a way those won't.
It is a resource strapped project. I doubt they have the bandwidth to throw in additional languages. Plus, Lua is perfect for this niche: small, easily embeddable, fast, and simple enough to pick up in an hour. There are even a variety of languages which can transpile down to Lua.
That's not really similar to Redbean; it requires an OS-compatible Python be installed on the system (whereas Redbean is a single binary artifact that runs anywhere). Plus Redbean can substitute for nginx, whereas you should never try doing that with the built in Python server.
You can. Just create a folder for each hostname. Here's how redbean handles virtual hosting. If a request comes in for http://foo.bar/baz.html then redbean looks for the asset `foo.bar/baz.html` and if it doesn't exist then redbean looks for `baz.html`. This virtual hosting feature also works if redbean is used as an HTTP proxy. One thing I'm slowly but steadily doing is creating a gigantic redbean containing a painstakingly restored archive.org snapshot of the web circa 1999, with a Lua script for AltaVista, and then configuring FireFox to use it as a proxy so I can surf the old web.
For the curious, Fennel[1] is to Lua what Clojure is to Java. I'm curious, though, can you not add Lua libraries to redbean? My understanding is that the file is a zip and you can therefore add dependencies and (presumably) load them into the lua runtime on invocation. E.g. you don't need Justine to add it for you, you can add it yourself.
That was my first thought as well. If it can support Fennel as a first-class citizen, then I might just switch wholesale to using Redbean for any application development.
This strikes me as a terrible idea, relying on an idiosyncrasy of an arbitrary database to deny access to users.
For anyone doing something like this with a service that's useful to people, consider MaxMind's minFraud service. It's PAYG and you can choose three different levels of information at three different price points. It's targeted at fraud minimization, but you can hand it as little as an IP address and it'll give you what you need to know.
We go from decrying use of an arbitrary implementation detail of a database not fit for purpose, to recommending using a separate database for its intended purpose that just happens to be from the same vendor.
MaxMind's minFraud is $0.015 per request and requires a call to an external endpoint. Justified for finance and shopping apps, not great for a high volume free service.
And you don't have to deny based on this one signal alone, you could fall back to recaptcha or minFraud to reduce the false positive rate.
> MaxMind's minFraud is $0.015 per request and requires a call to an external endpoint. Justified for finance and shopping apps, not great for a high volume free service.
What fraud are you trying to prevent that gets abused in a free service? Most of the abuse I see for free services I host is DDoS, brute-forcing paths/authentication and port scanning, but all of those are trivial to deal with rate-limiting.
Any and all public form submissions will get spammed.
My wife ran a small business for awhile that catered to local customers only (i.e. not an online business, products didn't ship through the mail. She added a contact us type form which submitted to google forms as a quick alternate to phone calls for prospective clients. It was used almost exclusively by people trying to sell her things from overseas.
She wasn't even running ads or anything online, so how they ever found the website in the first place, I have no idea.
Just require submissions (from new users at least) to be manually approved before being shown publicly or having any other side effects and your will be set. A lot of people seem to like making spam into a much bigger problem than it is but how much do you actually get and how long do you really need to sort it out manually instead of relying on automated solutions that are GUARANTEED to have false positives. But since it's your time (not much) against other peoples frustration (which you might not even see so can easily ignore) you choose to make the world a worse place?
That's also relatively easy to protect. First defense is rate-limited, second is captchas, third is having a dynamic "secret" key fetching from some backend via JS that gets injected into the form and works as a "password" to successfully trigger the email to get sent (but displaying success messages even if it's not).
> but displaying success messages even if it's not
Please don't do stuff like this. Some of use use VPNs and aggressively block third party JS. These sorts of shenanigans render many sites unusable which is frustrating but simply not using those services is a price I'm more than willing to pay at this point.
> dynamic "secret" key fetching from some backend via JS that gets injected into the form and works as a "password" to successfully trigger the email to get sent
Are you describing nonces? A nonce is basically a "secret" (due to SSL encryption) number inserted into the HTML and submitted via a form. You don't need JS, just SSL.
This is a really bad idea. Bot detection is an incredibly complex problem, and simple rules like this are more likely to catch humans than (motivated) bad actors.
“(Motivated) bad actors” are a different problem to drive by bot spammers.
While I agree with all the people saying “this’ll block real people at significant rates”, for some things (like personal blog replies or small forums) that can be a perfectly sensible trade off for some sites.
Sure, if you are Netflix, or a government department website that people need to use, it’s a terrible idea.
For somebody who’s sick of deleting spam replies on their personal blog posts? Simple rules with known and acceptable unintended consequences might be a really good idea.
For a more specific check, you could query internetdb.shodan.io and see if the client IP has port 22 open, has the "vpn" tag or is any other services that you wouldn't expect to see from a visitor. We have a bunch of enterprise users that created rules to filter traffic out from IPs that are compromised or otherwise look malicious.
What sort of ”legitimate traffic” comes off AWS/Azure/DigitalOcean et al?
And how does the volume of that (and relevance of that traffic) compare to home add and mobile CGNAT ipaddresses?
Unless I’m selling something, dropping all cloud instance IP ranges doesn’t em like it’d block a lot of “legitimate traffic” to me, at least not enough to care about. If my choices were “spend time implementing a more sophisticated locking technique to reduce blogspam” or “write more blogposts”, I’d be triggerhappy blocking everybody using AWS o browse from and writing more content.
I use FSecure as a VPN, and I see it “do weird shit” at about the same rate as I see when I’m using a Hertzner box as a wire guard endpoint. (Most common thing there is I can’t order via menulog when I’ve got a vpn endpoint out of I am getting it delivered to, and I’ve just got used to switching off my phone’s vpn while ordering..)
Yeah, now consider living in a country which blocks stuff left and right (including vpn services) so that DO/Hetzner endpoint is the only endpoint outside you have.
Every time this masterpiece comes up on HN, I try to think of something to say that explains how profoundly good this project is and I never feel like I can do it justice. It's small, it's fast, it's useful. It makes so much other software look bloated by comparison.
In a world where a lot of developers on HN are too scared to run apt upgrade, you write this brilliant piece of tech. It's inspirational, really.
An example of what can be achieved and how much our community benefits from good software.
The Redbean 2.0 license is intriguing (snip):
"
Copyright 2020 Justine Alexandra Roberts Tunney │
│ │
│ Permission to use, copy, modify, and/or distribute this software for │
│ any purpose with or without fee is hereby granted, provided that the │
│ above copyright notice and this permission notice appear in all copies. │
"
Seems to assert the (FSF) four freedoms well, all is good in the world, it can be used and contributed to safely by all.
I don't know yet. This was only reported recently. Do you know if there's a form I can fill out to file an appeal? Abuses of DNS are very common around the world and we're starting to see a lot more of it in the first world. In the Cosmopolitan repository, we've always had a PNG file checked-in to cope with these kinds of situations. https://github.com/jart/cosmopolitan/blob/master/libc/dns/dn...
I always upload my binaries to VirusTotal and always have. Like what? Writing assembly? Using the RDTSC instruction? Having a run-length decoder to make lookup tables smaller? I must be a stone cold black hat hacker. Anyone who monitors the behavior of these programs can see they're fully transparent, don't interact with the system weirdly, and they don't have any telemetry. The truth is some virus scanner person wrote an overly broad regex somewhere that's raising false positives due to a completely random occurrence. It might not happen as much to people who use MSVC since it always generates binaries the exact same way. But does that mean we should accept an MSVC monopoly? Because MSVC is the kind of compiler that's been known to insert telemetry into your programs when it wraps main(). Why don't virus scanners complain about that? All I'm trying to do is innovate in this space and create a better alternative. There shouldn't be these kinds of roadblocks to competition. In fact APE doesn't even compete. It's designed to be a complementary format that bridges gaps between communities and helps reduce pain for developers.
It is really annoying little the anti virus industry seems to care for false positives of anything other than the standard compiler output. Even an otherwise perfectly normal .exe compiled from a .c but without a standard compiler-provided start routine makes you an evil hacker writing dangerous programs in my experience. Guess caring about removing unnecessary overhead and bloat is only something virus creators do these days.
Of course if you get these vendors to fix their detections for your program and get google "safe" browsing and virus total (aka google again) to stop slandering your it will only take time before the whole thing starts again - even without any changes to your executables.
And beacuse these are respected™ software companies making the virus allegations your users will of course more often than not tell you to fix your executables when the bug is in their third party anti-virus software.
I'm getting more and more tempted to just not provide any Windows executables for my open source projects :/
On an OS without a Python interpreter? I doubt it. This is a binary that runs anywhere. No interpreter or virtual environment needs to be installed for it to work.
PHP support would be beyond amazing, what would be required (top-level) to get that working/usable?