Hacker News new | past | comments | ask | show | jobs | submit login

This brings me back to something I discussed here in the past - are there scientific problems that are simply not compatible with running in the cloud, and require a supercomputer? I can't, off the top of my head, think of anything where the performance would degrade enough with the increase in communication latency between components, that the increase in available (on-demand!) resources wouldn't keep up.



Well, 3D physics simulations a la finite difference or finite element are not trivial to break up for parallelization, because the state of individual cells is mathematically dependent on the state of its neighbors, so communication latency between loop cycles would be a bottleneck on a distributed system without some kind of clever optimization.

Edit: for clarity, finite difference and finite element simulations both involve discretizing a 3D volume into individual cubes (I believe other geometries exist, but are less common) and running an update loop which performs calculations for each of millions or more cells. Of course, models with spatially small divisions, and/or those designed to handle high frequency wave propagation, require potentially enormous amounts of memory (e.g. climate simulation, or large seismic dataset processing). Depending on what math you're solving, there are heuristics which you may be able to use to update only a subset of active cells, or divide up the model to split among multiple machines with minimal loss of accuracy while retaining high precision.


Depends how Googles network is set up and how fast and over provisioned it is - how much budget you have for networking also comes into it (that's the power budget btw)

Brad Hedlund has some interesting articles on the challenges

http://bradhedlund.com/2011/11/05/hadoop-network-design-chal...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: