

10 petaflop supercomputer in the making, will take US back to 1 on top500  - eerpini
http://www.ncsa.illinois.edu/BlueWaters/system.html

======
mortenjorck
This is going to sound somewhat shallow, but I hope this new installation gets
some decent visual design. Look at some of the classic supercomputers, such as
the fantastic Thinking Machines CM-1: <http://www.mission-
base.com/tamiko/cm/CM-1_500w.gif>

Now look at the current reigning champ in 2010:
[http://www.blogcdn.com/www.engadget.com/media/2010/10/mod-65...](http://www.blogcdn.com/www.engadget.com/media/2010/10/mod-656188tianjin1.jpg)

Sure, the visual design of a supercomputing installation doesn't have any
bearing on its actual utility. But having a certain presence commands
respect—even if the researchers using it know it's just stagecraft, there's an
undeniable attraction to working on something that isn't just a great
engineering achievement, but also _looks the part._

~~~
blhack
Huh, maybe it's because I already know what it does, but I actually think that
racks of machines are _really_ cool looking.

Stuff like this:
[http://www.google.com/images?hl=en&q=Marenostrum&gbv...](http://www.google.com/images?hl=en&q=Marenostrum&gbv=2&um=1&ie=UTF-8&source=univ&ei=jbPkTIXqMYnEsAPu69jVCA&sa=X&oi=image_result_group&ct=title&resnum=3&ved=0CDcQsAQwAg&biw=1066&bih=566)

is just _really_ cool.

~~~
mortenjorck
Well, yes, but that's the MareNostrum!

It may be a bunch of 1U blade servers, but the architecture plays a huge part
in building its presence: The location, the contrast between classical
ecclesiastical design and stark modernism, the glass, the color choices, the
dramatic lighting...

Compare that with a gray room with a drop ceiling and fluorescent lights. The
hardware may be the same, but the effect is not in the same ballpark!

------
zitterbewegung
I went to a presentation at my school about this computer. There are two main
goals to the project. One is to be a usable supercomputer meaning that it will
be much more userfriendly to actually use to write software for. Another is
the goal of actually getting 1 petaflop of sustained performance. Whats
interesting is they were able to get the building under budget and ahead of
schedule due to the housing crisis. They also want to make the computer much
more energy efficient and they aim to be more efficient than modern
datacenters.

~~~
eerpini
yes, they seem to be using the Power 7 architecture by IBM , which is quite
energy efficient compared to most other architectures, and also they are
planning to use UPC(Unified Parallel C) or similar languages for applications
running on the system, that a good step towards better usability. But that
does not matter as a super computer is always(mostly) used by a small group of
highly qualified scientists.

------
coffeenut
You know, they start waking up at around 1 Petaflop.

~~~
eerpini
I quite did not understand what "waking" up means, but it seems that Blue
waters will indeed have a sustained performance of more than a petaflop with
adapted applications !

~~~
booi
skynet'ing

~~~
Devilboy
Skynetting

------
revertts
For anyone wondering about the "back to 1" part: the Chinese very recently
(this week?) took the spot with their 2.5 petaflop Tianhe-1A. It's made from
Xeons, Fermis, and Feiteng-100s. Right now they're having trouble actually
utilizing the machine; very little of their software takes advantage of more
than a modest number of cores.

~~~
eerpini
mostly the problems with utilizing the system completely are the same ones a
common programmer faces when trying to use GPUs for general purpose
programming, the problem of transferring the data to be handled from the main
memory to the GPU's memory. Though Tianhe-1A has a very high peak performance,
the sustained performance seems to be comparatively low !

------
charlesju
What are some real world applications for this bad boy? Is this a for-profit
endeavor where real companies can rent this computer out to do stuff with it?

~~~
InclinedPlane
These things are most useful for research via numerical simulation. The uses
that would come closest to aiding for-profit endeavors would probably be
various simulations of systems still in the design-phase. For example,
simulation of a car crash, simulation of aerodynamics of an automobile or
airplane, simulation of the fluid dynamics of a jet engine, simulation of
hydrodynamics of a new super-tanker design, etc.

Such simulations are fairly common today, although generally super-computers
of this magnitude aren't needed for them.

------
nkassis
Talking about HPCs anyone at SC10? If you are I'd like to meet up with other
HNers at the conference. I'm stuck in a booth until 3pm (3445 ciena demoing
some brain imaging app over high bandwidth links) but it be cool to have a hn
meetup afterwards in some bar. Only issue will probably be choosing a bar from
NO large selection.

------
dstein
It is still kind of mindblowing that the entire Amazon cloud shows up as 231st
of the top 500 computing clusters (top500.org). Are all these other systems
just sitting around crunching proteins? They should start hooking these bad
boys up as EC2 mega-instances.

~~~
nostrademons
That list claimed that all of EC2 had only 7000 cores, which seems absolutely
ridiculous to me. I'm more inclined to believe that the list is full of shit
than to think that EC2 has so much less computing power than these other
supercomputers.

~~~
wmf
Amazon ran Linpack on 7000 cores and submitted it to Top500, so that's what
appears on the list. Maybe they could have done a larger run, but keep in mind
that regular EC2 machines are unsuitable for HPC; I can believe that they only
have 7000 cores in the compute cluster machines.

~~~
eerpini
that is still unbelievable ... they sure have more cores than that just in
EC2, and anyway looks like the new GPU instance offerings were not included in
the benchmarking, that would have shot the rank up by an order of magnitude !

~~~
jeffbarr
The Top500 benchmark for EC2 was run on some of our Cluster Compute instances.
It was definitely not run on the entirety of EC2 (what would we do, kick
everyone else off?).

~~~
SkyMarshal
Any chance you guys ran it on the new GPU cluster before opening it to the
public? Would have been cool to get the benchmark on that entire thing b/f
customers started running stuff on it.

~~~
jeffbarr
Not that I am aware of, but the AWS team is big (and getting bigger) and lots
of stuff happens that I know nothing about.

------
Jabbles
Does anyone know why there's a ten-fold difference between peak and sustained
power? I would have thought it'd be more like 50%, but I don't work with HPCs.

~~~
eerpini
The performance is measured with LinPack which mostly does FP operations and
is optimized to run on the architecture, when normal programs are run, it is
tough to run them at full utilization, this is because, a lot of time might be
spent in data transfer or I/O apart from the computation ... there would be
various other factors !

------
temugen
The buildings on our campus keep their lights off for the majority of the day,
probably to make room on the grid for this machine's energy usage.

------
jmtame
i was really excited when they started construction on the building that
houses those machines. it's finished now, but i was able to attend the less
than glamorous pre-open house* =]

* <http://bit.ly/dkJiaZ>, <http://bit.ly/9mEo2e>

~~~
sparky
It's much cooler now, but access is more restricted. I think you can still get
tours if you arrange ahead of time
<http://www.ncsa.illinois.edu/AboutUs/tour.html> . If you do, if you look
really hard in the far corner of the room you might spy the little Top500
machine I worked on this fall.

------
pjscott
The networking hardware can read and write directly to L3 cache. That's just
_cool._

------
jacques_chester

      The key component is the hub/switch chip.
      The four POWER7 chips in a compute node are
      connected to a hub/switch chip, which serves
      as an interconnect gateway as well as a
      switch that routes traffic between other hub
      chips. The system therefore requires no
      external network switches or routers,
      providing considerable savings in switching
      components, cabling, and power.
    

Sounds a bit like the design SiCortex had for their machines (before cashflow
interruption killed them).

~~~
sparky
Most supercomputers with multi-socket nodes have had something like this for
similar reasons. The earliest reference I can find is here
[http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefi...](http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA166610)
, but more modern examples of the idea are the Cray SeaStar
[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.150...](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.150.7293&rep=rep1&type=pdf)
or even SeaMicro's I/O virtualization ASIC if you squint the right way
<http://www.seamicro.com/?q=node/38> .

SiCortex had some neat ideas, it was a shame to see them go.

~~~
jacques_chester
Thanks, I learned some new things.

------
Devilboy
Anyone want to guesstimate the date of the first exaflop computer?

~~~
sparky
If you believe that the trend-line that has held surprisingly steady for the
last 17 years ( <http://top500.org/lists/2010/11/performance_development> )
will keep going, than by my crude eyeballing it looks like about 2019.
Granted, there are a lot of good reasons to think it won't quite hold.

