On: A search engine that favors text-heavy sites and punishes modern web design

[swyx]

how on earth do you index so much on consumer hardware? my frontend developer mind is blown.

[foxfluff → swyx]

Wait till you learn that modern CPUs run billions of cycles per second. With multiple cores in parallel! And they can reach transfer rates of tens of gigabytes per second to RAM, or around a terabyte per second into L3.

[ThalesX → foxfluff]

And then you add a single HTTP request and everything tones down to the speed of the web. Or I/O. Or DB call.

[goodpoint → ThalesX]

More like: you add some javascript library and now the browser needs 5 seconds to run 10 MB of javascript.

[knuthsat → ThalesX]

You can still support millions of these requests per second if you just bake all of the dependencies directly in a small OS running on your fastest raspberry pi.

[paxys → swyx]

Consumer hardware today is simply what was cutting-edge and crazy expensive 5 years ago.