
How would you write a program that utilizes 10,000 cores? - tambourine_man
http://upthere.com/jobs/
======
lenjaffe
The main problem with running 10,000 cores is partitioning your program so
that inter-core communication can happen effectively.

You could have several very large thread pools, each performing a different
task in a processing pipeline.

You could set up a big SIMD system. Bu didn't the Connection Machine already
do this, to the tune of 64k cores?

Maybe you're trying to figure out how to apply the model in a large
distributed system where "core" is actually some cloud compute instance.
You're back to devising effective ways to communicate with all your cores.

In the end, massively parallel processing, beyond generalized horizontal
scaling tends to be designed for a particular problem domain. Some problems
lend them selves to vectorization, others to SIMD decomposition, others
shared-nothing multiprocessing with message passing between cores.

------
jamesbanner
Less of an article more of a recruitment drive? (indicated from "/jobs" sub-
directory.

Anyways I am interested in knowing why would you do that ? What kind of
problems need 10,000 cores to solve ? Could be easily partitioned with Big
Data or Hadoop instead of building 1 giant skynet ?

