In the mid 1980s there was a CPU called a "Transputer" [1] made some of the people who moved to ARM. These CPUs could be connected together in huge networks and directly talk to each other.
The network of CPU's could auto-discover its topology, but coding for so many CPU's was difficult. Some specific algorithms scaled well with the number of CPUs, but most did not.
I think the reason transputers didn't 'make it' is not because they were super hard to program (it was only a little bit harder than programming a regular computer), but because the price premium you paid for a transputer set-up was too high and x86 got faster very rapidly.
This is right around the time when the first 386 machines were launched and in a very short time we went from 12-20 MHz 286 boxes (and some 68K machines for the lucky ones) to 33 MHz 386 machines with a ton of RAM.
So the advantage that transputers had was eroded very quickly and I don't think INMOS was ready to match pace.
Now that we've reached the end of the line for that kind of speed increase we are seeing a renewed interest in multi-cpu fabric architectures, of which the transputer was an instance.
Yes, but in the areas I saw at the time, transputers lost out in particular to i860 for floating point systems and to m68k for things like our mostly-integer "streaming map-reduce" -- it seems! -- system. (Being able to house 68ks in a blade system, i.e. VME/VXI crates, helped.)
I was involved with a system porting it from an old transputer based system to a PowerPC&C/C++ based one.
Transputers never really scaled into this sort of league, they were more 8/16/32 type scenarios..
For the interested, there was also the Atari Transputer Workstation (ATW).
...Although interesting, they were expensive and didn't perform compared to the equivalent sequential machine. Given development tho, who knows where it could have gone.
transputers had four links and comms beyond that required routing in s/w. I wrote such a thing for a transputer machine.
Also, the auto-discovery wasn't really auto... That was the boot code probing for CPUs on other ends of links and propagating itself to connected CPUs, building a map of the network in the process (which is kind of cool).
This is "the" question. I suppose one could look at the Connection Machine: https://en.wikipedia.org/wiki/Connection_Machine The CM-2a had 4K or 8K processors and was reputedly very difficult to program.
The Connection Machine Lisp programming language described in Daniel Hillis' PhD dissertation was essentially going to be Lisp with parallel map/reduce but AFAIK was never done being implemented.
I think the big problem was that most of the SIMD algorithms were yet to be discovered at the time. For example this paper by Hillis and Steele was a very big deal but looks kind of basic today: http://uenics.evansville.edu/~mr56/ece757/DataParallelAlgori...
Guy Blelloch did a lot of work on Connection Machines and basically wrote the book on SIMD programming: http://www.cs.cmu.edu/~blelloch/papers/Ble90.pdf He also made a very nice programming language for parallel computing (NESL), as did Gary Sabot who worked at Connection Machines (Paralations). When you compare those to Hadoop or OpenCL it really is a wonder where we went wrong and what the designers of the latter were thinking (or not).
You program a machine like this much like you work with the unix command line when you string together a series of commands using 'pipes'. Only now the pipes are the comms links between the CPUs and the programs are the pieces of software executing on each of the CPUs, rather than that they all run on the same CPU.
So it's more of a data flow model than a function-call model, and every node will transform its input rather than that it will return results to callers.
This means that not every problem is easily mapped to such an architecture, but when there is a good correspondence between the topology of the network of CPUs and the problem then the throughput can be very good.
After all, there is no bottleneck where all of these have to access a shared resource (RAM) so when it pays off it pays off big.
High performance computing relies on message passing -- if not pure and simple -- normally with the MPI standard. There are applications which scale at least past petaflops and 1000 cores is unexceptional.
The developers of the PETSc system wrote a position paper eschewing the typical hybrid (MPI+threading) techniques for trundling towards "exascale" -- if with MPI's shared memory support, which doesn't require cache coherence. (Apart from remote memory access, MPI departs from actor-/CSP-like systems in supporting things like collective communication (and i/o), and dynamic process management.)
I've read the first paragraphs, so take my comment with a grain of salt.
There are 12 memory modules on the chip, so I assume each is shared by less than 100 cores, which is close to a beefy multithreaded server chip. Each core also has a small local program memory that's not shared between them. It'd be not unlike programming a networked cluster of processing nodes each with a separate chunk of memory, except that the cores could be assigned to neighboring workloads.
Effectively you have to design for message-passing between microservices as a replacement for function calls. Not only do you have to step back from cache consistency but you have to step back from uniform memory access altogether and head towards a streaming architecture. It's very alien, but that's the only way to really use a system which has such a high ratio of processing power to external memory bandwidth/latency.
These people have some grand claims in that regard:
- http://texasmulticore.com/
I keep failing to get plans to match up to talk to their CTO on how this would stack up against what a rebuilt (differently optimized) Erlang VM could do...
And, yes, I know: Not open source. No one needs to make that clever observation...
Cache coherency seems really hard to give up on, and even CPU-GPU cache coherency is becoming the expected norm, with even ARM delivering it.