
There’s plenty of room at the Top: computer performance after Moore’s law - dsavant
https://www.techrepublic.com/article/mit-if-chips-cant-get-smaller-then-coders-have-to-get-smarter/
======
dang
This is a knockoff piece but it links to the paper:
[https://drive.google.com/file/d/1PFCcP5WAE-64DlrRoZZE5NhC3cc...](https://drive.google.com/file/d/1PFCcP5WAE-64DlrRoZZE5NhC3ccy3zKD/edit)

The paper's URL is
[https://science.sciencemag.org/content/368/6495/eaam9744](https://science.sciencemag.org/content/368/6495/eaam9744)
but the text seems to be paywalled. We've put its title above.

~~~
ncmncm
Paywalled? No problem: [https://sci-
hub.tw/10.1126/science.aam9744](https://sci-hub.tw/10.1126/science.aam9744)

------
Stratoscope
For anyone unfamiliar with the title's origin, it is a play on words of a 1959
Richard Feynman lecture, "There's Plenty of Room at the Bottom: An Invitation
to Enter a New Field of Physics".

[https://www.google.com/search?q=there's+plenty+of+room+at+th...](https://www.google.com/search?q=there's+plenty+of+room+at+the+bottom)

~~~
kens
And Feynman's title was a play on Daniel Webster's comment "There's plenty of
room at the top" when told that he shouldn't become a lawyer because the field
was too crowded.

~~~
DC-3
I wonder if the author of this article was aware of the etymology or if it's a
curious case of mutating references returning to the original?

------
harikb
Anecdotal - One of favorite YouTube live-streaming personalities, Jon
Gjengset[1] is a Ph.D student at MIT CSAIL. His talks are very detailed[4] and
captivating[3]

[1] [https://thesquareplanet.com/](https://thesquareplanet.com/)

Talks

[2] [https://youtu.be/s19G6n0UjsM](https://youtu.be/s19G6n0UjsM)

[3] [https://youtu.be/DnT-LUQgc7s](https://youtu.be/DnT-LUQgc7s)

Rust Details(my favorites)

[4] [https://youtu.be/rAl-9HwD858](https://youtu.be/rAl-9HwD858)

------
monocasa
I always interpret the end of Moore's Law to mean that was once a curve of
exponential growth, has shown where the inflection point is, and now we're on
the other side of the S curve. Yes there's still gains to be had, but they're
harder and harder in ways that aren't ameliorated by just increases in
funding. There needs to be changes in how we compute, and a rejection of the
idea that specialization is beat out by the economies of scale of generic
solutions.

~~~
jimmaswell
You want more ASIC's? Why do you assume there's an unmet need for more
specialized hardware?

~~~
ahartmetz
If you want something faster and more energy-efficient for any particular
task, you do want an ASIC. You can try to tell physics that you'd prefer
something else but physics is not a good listener.

------
Veedrac
> Unfortunately, Feynman’s “room at the bottom” is no longer plentiful.

There's plenty of room at the bottom. Another million times the transistors is
hardly implausible; a thousand times is practically a given.

> Although other manufacturers continued to miniaturize—for example, with the
> Samsung Exynos 9825 (8) and the Apple A13 Bionic (9)—they also failed to
> meet the Moore cadence.

This just factually isn't fair; transistor density has been on an unwavering
straight line since 1970.

[https://docs.google.com/spreadsheets/d/1NNOqbJfcISFyMd0EsSrh...](https://docs.google.com/spreadsheets/d/1NNOqbJfcISFyMd0EsSrhppW7PT6GCfnrVGhxhLA5PVw)

~~~
ncmncm
Imagine being a CPU architect and being told to find a good use for yet
another billion transistors. It is hard to get much benefit from the N+1th
multiplier when the ones you have are already starved for input, so it's not
hard to see why now we mainly get more cores and cache. Cache is good because
most of it is idle at every moment, so doesn't produce much heat. Cores are
easy to count, and people will pay for them even if whenever they all get
busy, you have to slow the clock after a moment. Often a moment is enough.

Productive uses for a billion transistors that can be mostly turned off most
of the time are hard to find.

~~~
ladberg
I feel like more and more transistors will be dedicated to machine learning
tasks. You'll eventually have phones and computers constantly running neural
networks to accomplish advanced but relatively mundane tasks (e.g. text-to-
speech, speech-to-text, facial recognition, human-seeming AI in videogames,
neural rendering (see NeRF from Berkeley), and tons of other random stuff.

All of it is parallel and benefits from a sheer increase in transistors and
density. Maybe in a few generations we'll see chips that are 99% ML compute
and 1% traditional CPU.

~~~
ncmncm
In those chips, I think practically all the transistors have to be active all
the time, so the density must be quite limited unless you have really serious
cooling capacity.

~~~
ladberg
I feel like there's a ton of unexplored area in the hardware space for
extremely low power circuits used for ML. As we know, you can do ML with very
low precision, which is why chips that can do fast INT8 and F16 are getting
more popular.

However, these lower power chips are still designed like classical computers
and are deterministic. What if you had something like a few thousand analog
multipliers in parallel computing matrix products? An analog multiply can use
far less power than a digital one at the expense of accuracy and determinism,
which might not matter for some ML tasks.

------
abnry
I was always jealous of the stories about programmers in the 80s who wrote
games in insanely small amounts of RAM. I kind of want that same kind of
challenge needed to be overcome in my lifetime.

~~~
jcadam
Expectations for what software should be able to do were quite a bit lower in
the 80s too :)

I started learning to code as a kid in the 80s on 8-bit machines. Seemed like
there was rapid progress in the availability of CPU power, RAM, etc., up until
the early 2000s and then things seemed to... slow down.

Which is about the time everybody started to focus hard on horizontal
scalability.

~~~
rayiner
Were they? My recollection is that software back then had more features.
Software today is neutered in comparison. (Compared Google Docs to Word
Perfect from the early 1990s).

~~~
siberianbear
Google Docs isn't a great example. In WordPerfect's case, WordPerfect _was_
the product and there was an incentive to make it great.

Google Docs isn't the product. It's free. You're the product. And it only
needs to be good enough to keep competitors from creating something so great
that it removes their first-mover advantage. It's clear that Google has given
up developing new features for it.

~~~
AlexCoventry
Grammar checking on Google docs had gotten wicked smart.

~~~
Dylan16807
I can only remember a single time the bad grammar marker on google docs wasn't
hilariously wrong, contorting a perfectly normal sentence into something
weird. And that one time it was catching a doubled word.

------
Ericson2314
Only at the scale of the big tech companies do the machine costs start to
seriously stack up against the labor costs. (Well, also smaller machine
learning shops, but they are in trouble anyways.) And I don't think they fsce
enough competitive pressure to be forced to take the massive foundational and
educational investments they would need to pull this off.

It won't happen with this generation of the industry.

~~~
ineedasername
The tooling infrastructure of code development itself needs to be optimized
(not necessarily in speed terms) to provide coders with the information &
tools necessary to write faster code. It seems like IDE's are currently
optimized to facilitate rapid code development, but not development of rapid
code.

~~~
est31
The issue is, most code only runs so few times that it doesn't really matter.
Only when the code requires so much cloud resources that it's worth to spend
an engineer's salary for a few hours/days to optimize it you should do it.

~~~
adrianN
Or if your software aims to be successful on end-user devices. Only because
you don't pay for the electricity or the faster machines, that doesn't mean
that no carbon is emitted for producing them.

~~~
est31
Slack is extremely slow and sluggish. Outside of the hn bubble, people seem to
care more about features than about speed or carbon use.

That being said, there are areas where users care about speed. For example,
FPS is very important to gamers, so games are often highly optimized.

~~~
adrianN
And cars are way too heavy. Just because users don't care about it, that
doesn't mean that it's not important.

------
P-NP
There is even more room at the top. The physical limits are as follows. A
computer "with a mass of one kilogram confined to a volume of a liter" can
perform at most 2:71 * 10^50 operations per second on 2.13*10^31 bits. See:

Lloyd, S. Ultimate physical limits to computation. Nature volume 406,
1047–1054 (2000)

Bremermann, H. J. Minimum energy requirements of information transfer and
computing. Int. J. Theor. Phys. 21, 203–217 (1982)

Bekenstein, J. D. Energy cost of information transfer. Phys. Rev. Lett. 46,
623–626 (1981)

~~~
reedwolf
From what I recall, that ultimate computer would have to be exploding to reach
10^50 operations.

~~~
P-NP
It would become very hot indeed. But that does not necessarily imply a runaway
explosion. Just like the center of the sun won't explode anytime soon.
Nevertheless, I'll take the much cooler model with only 10^44 operations per
second.

------
philipkglass
I quote from the Conclusions section of the actual paper linked by dang:

"As miniaturization wanes, the silicon-fabrication improvements at the Bottom
will no longer provide the predictable, broad-based gains in computer
performance that society has enjoyed for more than 50 years. Performance-
engineering of software, development of algorithms, and hardware streamlining
at the Top can continue to make computer applications faster in the post-Moore
era, rivaling the gains accrued over many years by Moore’s law. Unlike the
historical gains at the Bottom, however, the gains at the Top will be
opportunistic, uneven, sporadic, and subject to diminishing returns as
problems become better explored. But even where opportunities exist, it may be
hard to exploit them if the necessary modifications to a component require
compatibility with other components. Big components can allow their owners to
capture the _economic advantages from performance gains_ at the Top while
minimizing external disruptions."

(My emphasis on " _economic advantages_ from performance gains.")

Databases are still getting faster. You don't get a faster DB by buying a new
server and then running a 10 year old version of SQL Server or PostgreSQL on
it.

Language runtimes are still getting faster. You don't get better Java or
JavaScript performance by running a 10 year old release of the HotSpot JVM or
V8.

3D renderers, video encoders, maximum flow solvers (one example examined in
depth in the article): they're all getting faster over time at producing the
same outputs from the same inputs.

The key is _incentives_. There are probably people in your organization who
care a lot if database operations slow down 10x. But practically nobody in
your organization cares if Slack responds to key presses 10x slower or uses
100x as much memory as your favorite lightweight IRC client. (I'm trying to
make a neutral observation here. Looking at it from 10,000 feet, I get annoyed
when I see how much memory Slack takes on my own machine, but that annoyance
wouldn't crack the top 20 priorities for things that would improve the
productivity of the business I'm in.)

The incentives problem is also why the Web seems slow. Browsers too are still
getting faster. I used to run multiple browsers for testing and old Firefox
and IE releases were actually _much slower_ at rendering identical pages than
current stable releases. But pages are getting heavier over time. Mostly it's
not even a problem of people trying to make "too fancy" sites that are
applications-in-a-browser. It's mostly analytics and advertising that makes
everything painfully slow and battery-draining. I run a web site that has had
the same ad-free, analytics-free, JS-light design since the early 2000s. It
renders faster than ever on modern browsers. It's not modern browsers that are
the problem -- it's the economic incentive to stuff scripts in a page until it
is just this side of unbearable.

For certain kinds of human-computer interaction, people will pay a lot to
reduce latency. Competitive gamers will pay, for example. Sometimes people
will invest a lot of effort to reduce memory footprint -- either because
they're shaving a penny off of a million embedded devices or because they're
bumping up against the memory you can fit in a 10 million dollar cluster. But
the annoyances that dominate Wirth's Law discussions on HN -- why do we have
to use Electron apps?! -- are unlikely to get fixed because few people are
willing to pay for better.

------
SlippyCrisco
Yeah, (generally) we got lazy for sure.

~~~
xellisx
I remember in the Windows 98 days, there was a program called GuitarFX that
blew my mind. IIRC, it was only a couple MB in size, didn't use a lot of RAM,
and was fairly low latency. I think I was running it on a PIII 500Mhz with
128MB of RAM.

Now we have apps that are like 40+ MB, eat RAM and CPU cycles to do fairly
simple things.

~~~
rayiner
What’s mind boggling is the change in relative demand. Back in the 1990s, an
IRC or AIM client was a little background app that took a tiny fraction even
of a 64MB of RAM computer. But Slack takes a larger share of an 8GB if RAM
computer.

~~~
ncmncm
That's garbage collection for you. Typical GC programs want all the memory
there is, just to be able to sweep less often.

~~~
imtringued
From my experiments with the JVM that is not really true. Keeping memory usage
below 1GB is pretty easy. It's only getting tricky when you want to go below
200MB. You'll have to deal with the overhead the JVM introduces. As a result I
often just use lua for really simple programs and I am rewarded by creating
programs that use less than 2MB of RAM.

~~~
pdimitar
Which Lua kind/runtime you're using that give you such frugal programs? I'm
intrigued (pun with your name intended).

------
sillysaurusx
A bunch of you seem interested in taking up the challenge: having to write
smarter software, wistful for the old days when it was forced on you.

Turns out, you can! There's a big domain that needs your help, today: deep
learning.

A lot of the software is currently crude. For example, to train a StyleGAN
model, the official way to do it is to encode each photo as uncompressed(!)
RGB, resulting in a 10-20x size explosion.

There's plenty of room at the top, and never moreso in AI software. Consider
it! Every one of you can pivot to deep learning, if you want to. There's
really nothing special or magical in terms of knowledge that you need to
study. A lot of it is just "Get these bits from point A to point B
efficiently."

There's also room for beautiful tools. It reminds me a lot of Javascript in
2008. I'm sure that will sound repugnant for a majority of devs. But for a
certain type of dev, you'll hear that ringing noise of opportunity knocking.

~~~
ammanley
Very heartening to hear. You think some of us tired enterprise/product devs
that come from non-traditional (read: didn't study CS) background can make it?
Any recommended starting resources? I've heard supposedly good things about
fast.ai as a "guided tour".

~~~
sillysaurusx
Absolutely. I didn't study CS.

What you need is _determination_. There's no substitute for this.

If you have that, there are all kinds of resources. Here are a few...

Resource 1: a community. We've set up a discord server for AI dev. It has 360
users. At any given time, around ~50 people are online, of which ~10 are
skilled devs. Come join!
[https://discordapp.com/invite/x52Xz3y](https://discordapp.com/invite/x52Xz3y)

Resource 2: Find something fun _for you_ , and pursue that. I like generative
AI, so for me that's been GPT-2 and StyleGAN. Gwern has some lovely tutorial-
type articles on both.

GPT-2: [https://www.gwern.net/GPT-2](https://www.gwern.net/GPT-2)

StyleGAN: [https://www.gwern.net/Faces](https://www.gwern.net/Faces)

Peter Baylies' StyleGAN tutorial notebook is a hands-on resource. This was
actually how I started, nearly a year ago.

github: [https://github.com/pbaylies/stylegan-
encoder](https://github.com/pbaylies/stylegan-encoder)

notebook:
[https://colab.research.google.com/drive/179SPYbBC8pKDxVRjZep...](https://colab.research.google.com/drive/179SPYbBC8pKDxVRjZepIvIma8SITPJvU?usp=sharing)

(The original notebook was broken; this is a copy I've updated with some
fixes.)

Lastly, follow a bunch of AI people on twitter. Here are a few to get you
started:
[https://twitter.com/i/lists/1160386581850730496/members](https://twitter.com/i/lists/1160386581850730496/members)

The reason to follow them is, whenever you see something that seems
interesting or fun, tweet at them and say so! Ask questions. Ask how to get
started. Everyone is shockingly nice and helpful. My theory is, the software
is so crude and often hard to use, that we all like to celebrate together
whenever one of us gets it working, and we're happy to share that knowledge
however we can. (Twitter is a bit chaotic right now due to world events, but I
imagine it might return to normal within a couple weeks.)

And yes, you're right about fast.ai and other courses. You can go that route
if you like it. I found it more exciting to dive into the deep end, though,
and try to tinker with stuff.

~~~
YeGoblynQueenne
>> Absolutely. I didn't study CS.

But, have you achieved something singificant in the context of deep learning,
as per your earlier comment? There's no information about that in your profile
and a cursory glance at ddg and google results for "shawn presser" doesn't
turn up anything very relevant.

So, I have to ask: without having studied CS, what contributions have you made
in deep learning that are widely recognised?

I hope you agree this is a reasonable question to ask, and that you are not
offended by it. Otherwise, I apologise because it's not my intention to offend
you.

~~~
sillysaurusx
Certainly! Some highlights:

\- A Newsweek article [https://www.newsweek.com/openai-text-generator-
gpt-2-video-g...](https://www.newsweek.com/openai-text-generator-gpt-2-video-
game-walkthrough-most-tedious-1488334)

\- GPT-2 chess
[https://www.theregister.com/2020/01/10/gpt2_chess/](https://www.theregister.com/2020/01/10/gpt2_chess/)

\- ... which DeepMind referenced:
[https://twitter.com/theshawwn/status/1226916484938530819](https://twitter.com/theshawwn/status/1226916484938530819)

\- GPT-2 music [https://soundcloud.com/theshawwn/sets/ai-generated-
videogame...](https://soundcloud.com/theshawwn/sets/ai-generated-videogame-
music)

\- Swarm training (WIP) [https://www.docdroid.net/faDq8Bu/swarm-
training-v01a.pdf](https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf)

~~~
YeGoblynQueenne
To be honest, and again without having an intention to be harsh, but those are
not what I'd call "widely recognised contributions to deep learning". They're
mainly articles in the lay press and a honourable mention in a DeepMind blog
post. They certainly sound like contributions to Shawn Presser's reputation,
but "contributions to deep learning"?

To clarify, what I was hoping to see is, at best, an article published at a
reputable venue for AI research, a conference or a journal, or at a minimum an
arxiv article that at least looks like it was meant to be submitted to a
conference or journal. And at worst, a software tool that can be used in deep
learning research. But it seems to me that your achievements are mainly having
fun with and in one case finding an interesting use for tools that are already
available.

Again, I'm not trying to be harsh, neither do I want to say that all this is
not worth the trouble. But it should not be held up as an example of what
people can achieve without studying CS. Because, I think you'll agree, they
are kind of underwhelming when compared to what people routinely achieve who
_have_ studied CS.

~~~
barrkel
> But it should not be held up as an example of what people can achieve
> without studying CS

You really need to get out more!

~~~
YeGoblynQueenne
Excuse me, but is there a reason why you chose to reply to my comment with an
insult? And six days after I originally posted my comment?

------
mD5pPxMcS6fVWKE
I am old. Computer hardware advanced 1000 times since 30 years ago. But
computer software degraded at the same rate. We started with Pascal and C, and
ended with PHP and JavaScript ... ughh.

------
Exmoor
> "For tech giants like Google and Amazon, the huge scale of their data
> centers means that even small improvements in software performance can
> result in large financial returns,"

And really this is only for software that runs inside their servers. The costs
for software running on client machines is just distributed to the user.

If google makes a change that makes Chrome run less efficiently that means I'm
paying for it with my time and electricity. If I want it to run better I'm
shelling out money from my own pocket for new hardware.

------
yodelshady
FYI, I optimised the "python3" code by using numpy. Total runtime on a laptop
CPU was 1 s.

This is why all classes for scientists I've seen teach numpy.

(Incidentally: total time to "optimise" and TEST is less than 5 minutes.

Total time to rewrite in Rust: ongoing after 2 hours, with issues filed on the
best available crate for the task, whose documentation examples fail to
compile. Not even going to try in bare Rust due to the issues with > 32 length
arrays.)

~~~
pdimitar
This could also mean you're much better at Python than Rust?

------
code-faster
Yeah, but computer performance isn't a bottleneck in most applications,
engineer performance is.

