
The death of CPU scaling: From one core to many — and why we’re still stuck - evo_9
http://www.extremetech.com/computing/116561-the-death-of-cpu-scaling-from-one-core-to-many-and-why-were-still-stuck
======
ChuckMcM
So how many times do we get this story? Interestingly Herb Sutter wrote about
it here:

<http://www.gotw.ca/publications/concurrency-ddj.htm>

And then he updated that here: <http://herbsutter.com/welcome-to-the-jungle/>
and we commented on it here: <http://news.ycombinator.com/item?id=3502223>

Then the Extreme Tech guys rip off the graphics and write about it again.
(personally they should have given Herb a link but whatever)

Both Herb's article(s) and this one strike me like someone looking at an
approaching storm front and detailing how much water is likely to to come
raining down when it breaks. That is accurate information but ultimately
useless.

One answer is that existing architectures will work better on new materials
(carbon for example) because they can dissipate more heat (so keep your eye on
the research about doping graphene wires into silicon or creating diamond
substrates.

"Web 2.0" is all about new ways of computing which exploit parallelism. And
while I don't see a lot of benefit in Google inventing a new language to
express it (Go), the challenge is real. Folks have been designing chips which
are essentially parallel at the transistor level. Little of that research has
yet to percolate into the software architectures being proposed.

Its one thing to say "The gas tank is about 3/4 empty, start looking for gas
stations." and another to just go on and on about all the ways the remaining
gas in your car is going to be consumed down to fumes :-)

~~~
troymc
You wrote, '"Web 2.0" is all about new ways of computing which exploit
parallelism.'

I've seen many descriptions of "Web 2.0" but that's a new one to me. Or to
quote Inigo Montoya, "You keep using that word. I do not think it means what
you think it means."

Here's Tim O'Reilly's explanation: <http://oreilly.com/web2/archive/what-is-
web-20.html>

~~~
ChuckMcM
Web 1.0 was bigger and bigger SMP servers, Sun was that 'dot' in "Dot Com" and
"Grid" was the thing.

Web 2.0 is Beowulf type and other shared nothing clusters where the only
'fabric' between processes is the network and the parallelism and the service
API is an emergent property of the cluster of servers not of any one server.

Web 2.0 is where you can run a web application locally in some co-location
facility that is pulling its data off the S3 cloud at Amazon across the
country.

Web 2.0 is the difference between a SQL server that creaks under the load of a
million queries per day and a noSQL cluster that does billions of queries a
day.

A looooooong time ago I challenged Sun's executive management with the
question "What are we going to do when a 'big yellow hose'[1] runs through
everyone's living room?", Eric Schmidt (who was the senior director of the
Systems Group at the time) felt the challenge was a bit over the top since
getting 10 megabits of network bandwidth to everyone's house wasn't really on
anybody's road map (and we had just done a deal with AT&T which thinking that
_maybe_ in 10 years 30% of the households would have an ISDN line). I had just
come up for air after looking at how to build a network service that, like the
Andrew File System, didn't exist on any one server, it existed on all servers.
That was one of those 'oh wow' moments, sort of a 'we are all under-estimating
this' kinda thing.

So for me the explosion of bandwidth was the fundamental moving force behind
the evolution of the web, you _could_ assume that data could be on the far
side of the country and you'd have a chance of getting it to show to the user
before they died of boredom. And when that is true, what were the boundaries
of the system then? What were the invariants?

Working at Google, and now Blekko, is hugely exciting because the friction
between data sets is so much lower you can do awesome things. So my
'backplane' can be 4,000 sq ft of data center and I can fit a whole lot of
machines into that 4,000 sq ft, and I can easily give everyone of them a small
piece of a problem. Or the same problem where different things are assumed to
be true. And they can all return their answers and those answers can be
correlated, evaluated, formatted, and outputted in the blink of an eye. That I
contend is Web 2.0.

[1] At the time the long haul version of 10 megabit ethernet was a large
diameter yellow cable with marks where you could install vampire taps.

~~~
InclinedPlane
Web 1.0 was the web as a publishing platform with a sprinkling of limited
applications that do all of their heavy lifting on the backend. Web 1.0 was
about cgi-bin, it was about lycos and yahoo and mapquest, it was about
geocities and "under construction" pages and Fortune 500 companies with static
websites consisting of a handful of pages, none of them terribly interesting.

Web 2.0 was a dramatic reshaping of what was possible with the web. It was
about the web as a full-fledged application and communication platform. Web
2.0 was about AJAX, and web-mail, and wikis, and blogging and commenting, and
google maps, and sites with social features, etc.

In the Web 1.0 era if you wanted to share information with the world you
bought some hosting and you set up your own site where you put up a handful of
hand-edited html pages. If people wanted to have a conversation with you on
the web they would have to email you or make a comment on their own site. In
the Web 2.0 era you turned to one of many platforms (blogger, wordpress, web
forums, livejournal, etc.) and you started blogging, or making podcasts, or
making web comics, or doing whatever suited your fancy. And to have a
conversation people used the same medium, they commented on your blog posts or
they talked to you on a forum, or they commented on your flickr photos, etc.

Fundamentally web 1.0 is about static data from a handful of authors, web 2.0
is about dynamic data from a myriad of contributors.

Most of Web 2.0 runs on ordinary PHP and MySQL servers, the SQL vs. NoSQL
division doesn't play a part in it.

~~~
wisty
I think Web2.0 was when many things started going super-linear.

1 customer isn't just another customer, they are more SEO, more content, more
"Likes", more network effect.

It's also about standardization (which is also kind of super-linear) - if you
code to a standard interface, you no longer need to code to every interface.

I'm not such a fan of AJAX. You can make a punch-the-monkey game in AJAX, and
it's just web 1.0 all over again. You could be something like Facebook with
little more than static HTML. AJAX is a tool, not a revolution. It's a good
tool, but that's it. The same goes for NoSQL servers.

------
rogerbinns
For a perspective from CPU designers, checkout the talk given at Standford by
Bob Colwell on Feb 18, 2004. It is about half way down this page titled
"Things CPU Architects need to think about":

<http://www.stanford.edu/class/ee380/ay0304.html>

Bob lead the Pentium Pro effort all the way through the Pentium 4. The talk is
funny, fascinating, includes history (like how pointless that change was that
led to the Pentium FDIV bug), how the Itanium was approved, change, bugs and
many other things.

In the Q&A session at the end, one of the questioners is Dave Ditzel who
headed Transmeta.

~~~
agumonkey
defcon switzerland had a reverse engineer talking about x86 ISA and all its
weird cornercases/cruft

<http://www.youtube.com/watch?v=MJvsshovITE>

------
chipsy
If memristors pan out they're likely to be the "next step forward" for the
near future. There are some very big single-threaded wins to be had by
incorporating them into CPU design, even if it's a relatively brute-force
integration into existing architecture like making a very large, fast cache.

~~~
FD3SA
Precisely. These articles are inevitably too narrowly focused on the von
Neuman architecture, which at this point is relatively old technology.
Memrister computation architecture will be radically different, and orders of
magnitude more powerful once they pass the initial economy of scale hurdle.

~~~
sliverstorm
Maybe I'm just locked into an old model, but I still haven't found anything or
anyone who has actually been able to satisfactorily explain why they believe
this to be the case, beyond "Because it will be!".

~~~
InclinedPlane
Quite plainly, memristors are the most exciting invention in electronics since
the transistor.

Our current understanding of memristors is that they have the capability to
offer non-volatile storage at densities of flash or higher with speeds near
that of SRAM. This alone provides several very simple implications. First,
replacement of non-volatile storage systems with memristor based storage with
much faster transfer speeds than what we have today. Second, replacement of
main memory and CPU data and instruction caches with memristor storage. This
would allow for systems that can be kept in "hibernation" for indefinite
periods of time and transfer between being awake and being completely off in a
matter of nanoseconds. This would vastly improve battery life for mobile
devices and power efficiency for all devices. Additionally, it would vastly
speed up the typical "fetch, process, write" cycle of most computations,
improving overall computing power with equivalent logic hardware by huge
margins. Imagine if programs didn't need to be loaded in memory because there
was no difference between main memory and the "hard disk". Third, memristors
can be used to create FPGA like devices which approach the space and power
efficiency as well as performance of custom ASICs. That is a revolutionary
concept. Imagine if you had a bank of programmable logic the size of your
graphic card's GPU capable of transforming itself in a fraction of a second
between given hardware configurations. That alone would fundamentally change
computing as we know it. Fourth, potentially all of this stuff can be put onto
a single chip, which would dramatically lower the cost and further shrink the
size of a computer system, making them even more ubiquitous than they are
today. Imagine a sliver of silicon only a few square centimeters in size
containing an entire CPU, plus gigabytes of L2 cache, plus reconfigurable
logic with the power of a GPU, plus terabytes of non-volatile storage that
also serves as RAM.

Here's the kicker, all of these things are just trivial applications of
memristor technology, but the revolutionary stuff is even more astounding.
Memristors are capable of being used for logic on their own, much like
transistors. And it seems to be trivial to alternate between using them for
memory or using them for logic. Potentially you could create a single chip
containing terabytes or even petabytes of memristors and the components of the
chip could easily transition between configurable logic (in a different way
than the FPGA like devices mentioned above) and memory, allowing the chip to
match computing power to available resources and position computing elements
close to the data being used for optimal parallelism and minimum latency. Not
to mention that it would enable non-von-Neumann computing architectures. The
implications of this are far too heady for us to grapple with today, it
implies the potential for much more advanced machine learning systems and
vastly more powerful computational resources than we have ever dealt with
before.

~~~
sliverstorm
Thanks for taking the time. I am familiar with the argument that they would
replace SRAM, speeding up the chip. I am familiar with the argument that they
would be lower power. My frustration with the discussion has always been that
faster/bigger SRAM does not equal a fundamentally different computational
model or radically different architectures, which is always what memristors
are claimed to bring. Sure, it could be worth a huge speedup, but what does
that have to do with antiquating the von Neumann model?

I was not aware they could behave as both memory and logic; perhaps _that_ can
contribute to a fundamentally different structure.

------
ZephyrP
"For over a decade prophets have voiced the contention that the organization
of a single computer has reached its limits and that truly significant
advances can be made only by interconnection of a multiplicity of computers in
such a manner as to permit cooperative solution."

This now pithy statement was written by the famous Gene Amdahl in the year
1968; A time when computers ran at speeds that are dwarfed by today's digital
clocks, but also it gives us insight into a time when people were still
dealing with the same problems that we deal with today in developing faster
and faster CPUs.

The truth of statement may be something that functionalism or parallelism
advocates don't want to hear - the so called Parallelism Revolution will never
come, at least not in it's current incarnation.

The end of serial advancement, and thus the parallelization revolution was
"supposed" to happen in the 60s, and despite the considerable advances in
methodologies of parallelization, it did not come. The 90s brought us
standards and technologies like MPI which standardized procedures in
developing cooperative computing solutions, but still, it did not come. The
2000s sought to simplify the very act of programming by reappropriating the
ideas of programming back to the realm of pure mathematics - by representing
programs as a mathematical description of time and work itself, with languages
like Haskell and ML we sought to build machines which model math, and thus,
the parallel nature of computation within universe itself.

I feel the curious allure of parallel computing myself, a sublime glitter of
gold that is locked in the idea - It is irresistible for a curious individual.
To feel as if all the power of the world is in your hands in this moment (as
opposed to 20 years from now), to wipe away the frailty that underlies all of
computation today; We all would like to be able to lift a trillion billion
bytes into the heavens.

Theres only two problems.

The first problem lies squarely within our own human inadequacies, and it
could be argued that this is where parallelism fails deepest. It is certainly
true that parallelization is complex, but like all things, abstractions of
complexity are nessasary, and designing the abstractions in such a way they
are understandable to 'mere mortals' is a greatly undervalued aspect of
technology today. So, I would posit as a result of insufficient desire to
establish simplified abstractions of parallelization, to most programmers,
ideas like parallelism remains in the domain of machine learning and condensed
solids analysis - A kind of electronic black art, only used by those with
sufficient training to know what horrors they may wrought upon the world if
they're to make some trivial programming mistake. As a result (ceteris
paribus!) serial power will always be valued greater than parallel
computational capacity, which many have claimed to be the predominant driver
of commercial development of scientific ideas.

The second problem is more controversial and is much more material, but I
think time will prove it so -- Computer have managed and will continue to
manage getting faster at an alarming rate. Regardless of our preconceptions
about the mechanics of computation, I believe it is reasonable to say that
computers will continue to get faster at exponential rates, even after the so
called quantum limits of computation come into play. This is reasonable for
the same reason the Normal distribution manifests itself in disparate natural
phenomenon - The Central Limit Theorem. Anyone writing about 'CPU slowdown'
admits that people have been saying the same thing for 70 years (Even before
'real' computers), and I fail to see where they justify their reasoning that
somehow this century is different.

------
sid-azureus
Two words (Combined as one) : GreenArrays

~~~
mooism2
Link?

~~~
sid-azureus
didnt know which link you mean html link : <http://www.greenarraychips.com/>
conceptual link : chuck moore and green arrays' philosophy has always been
that you cant simply scale the cpu, you have to scale the whole computer. Make
them smaller and slower and less power hungry, but REALLY parallelise the
computer itself: VLSI cluster computing basically. i increasingly believe that
taking this philosphy forward is the right way to go, not intels "more than
moore" or whatever ARM/MIPS has up their sleeve, they are bound to hit the
same walls. Yes, the flagship greenarrays product the GA144 is not really a
competitor yet, just a DSP, but its 144 complete computers squeezed onto a 1cm
sq chip with a 180nm process. There is a lot of potential there imho.

~~~
mooism2
I meant conceptually --- I'd never heard the phrase before. Thanks.

------
rorrr
History has been crushing the predictors of "the end" for such a long time.
Why are they even trying?

~~~
getsat
Linkbait title -> page views -> ad impressions -> money, cash, hos

