On: Probably the worst URL scheme ever

[jgrahamc]

There's no database.

EDIT: to all the people arguing with me. Read the source code to Hacker News.

    (= fns* (table) fnids* nil timed-fnids* nil)

    ; count on huge (expt 64 10) size of fnid space to avoid clashes

    (def new-fnid ()
      (check (sym (rand-string 10)) ~fns* (new-fnid)))

    (def fnid (f)
      (atlet key (new-fnid)
        (= (fns* key) f)
        (push key fnids*)
        key))

    (mac afnid (f)
      `(atlet it (new-fnid)
         (= (fns* it) ,f)
         (push it fnids*)
         it))

They are in memory. Which is why they expire randomly when the HN process is restarted.

[Robin_Message → jgrahamc]

That's probably true John. If I may mis-quote Greenspun:

"Any sufficiently complicated Lisp program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of an actual database."

[scotth → jgrahamc]

Depends on what you consider a database.

[kens → jgrahamc]

The fnids do not expire randomly due to restarts; they expire when there are too many or they timeout so memory doesn't fill up with these continuations. Personally, I don't like this continuation-based approach since "Unknown or expired link" is a really bad user experience.

Way back I wrote a bunch of documentation on the Arc web server if you want details: http://www.arcfn.com/doc/srv.html Look at harvest-fnids which expires the fnids.

The fnid is an id that references the appropriate continuation function in an Arc table. The basic idea is that when you click something, such "more" or "add comment", the server ends up with the same state it had when it generated the page for you, because the state was stored in a continuation. (Note that these are not Scheme's ccc first-class continuations, but basically callbacks with closures.)

(The HN server is written in Arc, which runs on top of Racket (formerly known as PLT Scheme or mzscheme))

Edit: submitted in multiple parts to avoid expired fnids. Even so, I still hit the error during submission, which seems sort of ironic.

[ahknight → jgrahamc]

There's always a database.

[gcr → jgrahamc]

Racket (arc's host language) keeps continuations on the filesystem, or you can write your own "stuffer" to do what you want with them (store them in a database or whatever). But you have to keep them somewhere or else (assuming the server uses continuations) you can't keep track of the user's path through your code as they click through links and such.

Racket does have an option to serialize the continuations, gzip them, sign them with HMAC, and then send all of that to the client so the server doesn't have to keep track of anything, but HN doesn't use it.

See http://docs.racket-lang.org/continue/#(part._.Advanced_.Cont... for a quick introduction.

[epochwolf → gcr]

HN doesn't use racket. It's a custom lisp based on scheme.

[gcr → epochwolf]

Sure it does. HN is written in "Arc":

https://github.com/wting/hackernews

Arc runs on Racket:

http://en.wikipedia.org/wiki/Arc_(programming_language)

See the "OS" section where it says "runs on the Racket compiler"

See also the Arc source code, https://github.com/Pauan/ar/blob/arc/nu/arc Note the "#lang racket" at the top.

[oh_sigh → jgrahamc]

Then where is the data that is associated with "b7VO4wED8MRumCeiX5fCnF" stored? How is that data requested? There certainly is a database, it is just most likely not a traditional database that most people think of.

[defen → oh_sigh]

where is the data that is associated with "b7VO4wED8MRumCeiX5fCnF" stored

In the Racket process that's running the Arc code for news.yc