You can call me out any time you'd like. I think an actual database backend for this site is only a couple years old. Or do we actually have one yet? I know the API has one, but HN doesn't run the API. It's possible this thing is still relying on flat files.
Direct answer: no, probably no database. HN forwards info to Firebase which provides an API. But assuming Arc is still running on Racket, there aren't a lot of options for database bindings (https://docs.racket-lang.org/continue/) and flat files work well enough.
EDIT: Actually, you may be right. They already forward data to Firebase, which means they could just store the data in Firebase directly. Firebase is pretty great, so it's possible they did this.
One bit of supporting evidence: there seems to be no startup time. Loading GBs of data from disk takes a little while, so restarting HN would be painful. We'd notice it as ~30 seconds of downtime. Especially now that HN has grown so large. I haven't noticed this in a couple years (but I'm not omnipresent).
So it could go either way, and there are a lot of advantages to the Firebase approach.
I guess I take back everything I said. But only because a lightweight alternative exists, not because flat files are inherently bad.
Flat files FTW. Proof that a simple solution can scale. And not just in a "Why the fuck would you subject yourself to this torture?" kind of way, but in a "This actually simplifies everything" way. It's why Arc was so fucking fast: remember how impactful memcached was? Arc had that from day one.
Here's how it works. Everything is loaded into memory at startup. You can think of it like a bigass JSON tree. Anytime you change the tree, write the whole thing out to disk. Since all the data is loaded from file "foo", and you write out to "foo.tmp" followed by `mv foo.tmp foo`, there is no risk of data loss, ever.
So what about sharding? Well, yeah, that's a problem. But it's not a problem that HN had to deal with. You can see how effective it was to throw a single 64GB RAM monster at the problem rather than a cluster of servers. The moment they hooked up Cloudflare to HN, they got all the same benefits a big cluster would provide anyway.
People can be prejudiced against this simplicity all they want, but it's hard to argue with results. You don't even need to know SQL to start using it.
"But migrations --" well, Arc makes it super easy to define a "template", which is just a set of fields, and each field is whatever you want them to be.
(deftem item
id nil
type nil
by nil
ip nil
time (seconds)
url nil
title nil
text nil
votes nil ; elts each (time ip user type score)
score 0
sockvotes 0
flags nil
dead nil
deleted nil
parts nil
parent nil
kids nil
keys nil)
So yes, theoretically if you wanted to rename "kids" to "children", you're gonna have to process all the old data. But 1. then don't do that, and 2. even still, it's not hard to do that anyway.
It's sort of similar to mongoose record definitions.
But look, HN is proof positive that this system can work and can scale to a top-1000 website. 99% of webapps won't even come close to those performance requirements. And when you do approach them, you can find ways to make it scale.
My main focus has been updating Arc to run on JS, and to add keyword arguments. That way you'll be able to use the whole JS ecosystem too.
>"But migrations --" well, Arc makes it super easy to define a "template", which is just a set of fields, and each field is whatever you want them to be.
That's... just a struct or a map in any other language. Convenient but not necessarily a point in Arc's favor per se.
>So yes, theoretically if you wanted to rename "kids" to "children", you're gonna have to process all the old data. But 1. then don't do that, and 2. even still, it's not hard to do that anyway.
It's still an argument against using flat-files. Updating a site with a proper database isn't typically tied to disk read/write speeds - at best renaming "kids" to "children" means changing a field in a model and maybe a template - and that's a fixed amount of work regardless of the number of records you have.
>My main focus has been updating Arc to run on JS, and to add keyword arguments. That way you'll be able to use the whole JS ecosystem too.
You mean Node/NPM on the backend? That might be interesting. I'm not familiar enough with Arc to know (I've only played with Anarki a bit, but I have too many random projects on my plate as it is) but does it even have dependency management?
When you get the design right, there's no need to change it. :)
Jokes aside, I refuse to let Arc die. There were genuinely good ideas embedded in it, amidst the rough edges. People just never forced themselves to use it. It's like if Vim was invented but had no documentation, so no one ever realized it's powerful if you embrace it.
But yeah, I imagine my version will have a different name and a shiny website. Being a decade out of date does matter, especially for hiring purposes.
Ah, yeah. (It's odd that Arc went closed-source but kept improving. It's maybe one of the most important custom languages in the industry, relative to HN's impact on the world.)
