It's important to look at this in context of history. A lot of what came about today started under the Strategic Computing Initiative (link below). It was a massive investment over a period of time that threw money at every smart person or neat idea you could think of hoping for scientific equivalent of moonshots. We got a bunch of them, too, so it worked imho. Thinking Machines explored the novel concept of massively parallel CPU's with languages & work distribution modified for them. Supercomputers of time had fewer, less-radical cores. Projects like Thinking Machines led to Massively-Parallel Processing architectures that combined hundreds to thousands of more-typical CPU's, custom interconnects, and special tools for programming. My favorite of those were Distributed, Shared-Memory (DSM) machines. Later, Beowulf clusters were invested to use commodity CPU's, boards, and networking to bring cost down. High-performance interconnects improved and got cheaper simultaneously. These trends led to commodity clusters from startups and semi-custom clusters from IBM (esp SP2), Cray, SGI, etc.
The reason for doing something so different & risky like Thinking Machines Corp is to try to create those ripple effects. Well, build and dominate a market is number 1 but government funders trying to get ahead wanted ripples as a backup. They got it, too. Also, the machine could pull off amazing stuff [1] with its architecture. Similar one's using 8-bitters piled into a chip were used for neural networks. One could probably re-created benefits of Thinking Machines same way for the problems that suited it.
Note: The book is well-worth skimming through given how it documents how modern computing practices formed. I had never heard of SCI. I thought the stuff was unrelated but turns out it was all connected. Documents lots of attempts, justifications, policy effects, etc. Gold mine.
This (along with the Burroughs B5000 architecture) are my favourite computers of all time. I have the book form of his thesis on my shelf right next to Knuth and Kenya Hara.
I feel like junior people should be given a tour of the greatest hits from the past when they start in industry.[1]
1. So they can judiciously steal the good ideas later on in their careers.
Sure. And Markov algorithms are much better than lambda calculus. See Simon Peyton Jones's recent works on supercompilation which is deeply rooted in Markov algorithms based Refal language.
These kinds of efforts were always doomed by Moore's law. Why build them if you can wait 5 years and then build something of equivalent power with commodity hardware?
If Moore's law really is ending I could see that changing.
I think the Anton machines for simulating molecular dynamics (https://en.wikipedia.org/wiki/Anton_(computer)) show that this has already changed. The 8 year old Anton 1 machines are still ~1-2 orders of magnitude faster than commodity clusters, and the now 3 year old Anton 2 is another order of magnitude faster still.
The way an ALU or CUDA core gets its data to operate on and where that data goes and the latency involved are the big deal for large computational tasks now. Specialized ASICs also can give a huge advantage. With specialization, you can eliminate overhead. People keep talking about reconfigurable arrays of hardware as well.
sure, its a time game. but I remember a 60GF (maybe it was 30?) result on the CM-2 in the early nineties. I think that machine was $40M, give or take. Lets also pretend that it was 10 years before we could have done the same runs on a commodity machine. You can imagine that for things of sufficient importance, there is an arguable cost utility. Of course that one machine ran quite a number of codes 24/7
But there is a second order effect, maybe a lot of that cultural history was lost before SIMD really caught on, but in general I think we gain quite a bit of collective experience by pushing the state of the art.
OTOH, maybe since games ended up driving the development, and HPC ended up trying to adapt their workloads to those platforms, you could argue that it was wasted effort.
But to echo my earlier comment, programming models to easily exploit substantial concurrencies is where the real sugar is.
Actually, I just remembered an example in your domain: Cavium Octeon II and III for networked applications. SDR, firewalls, VPN's, etc combine general-purpose algorithms with specific ones easy to accelerate in hardware. Also doesnt take what's in Intel/AMD CPU's that have tk run huge range of workloads. So, Cavium started with 16 MIPS-ISA cores, added TCP/IP offloading, encryption engine, compression engine, and so on. A PCI card on 45+nm could handle multi-Gbps processing with 15 cores idle. My Core Duo 2 isn't going to pull that off. Their 28-nm has 48 cores and "hundreds of accelerators" to pick. Heteroginity is not just future: it's the present in many markets.
Another example is the Automata processor that similarly represents protocols in an ideal way. Check it out. Crazy efficient in a way you can only do with atypical, hardware-level thinking.
Not to mention entire FPGA ecosystem that accelerates algorithms faster than Moore's Law systems while their own hardware is less efficient & cost more. :)
That certainly became true. I'm not sure it was especially true in the timeframe that we're talking about here. By the mid-eighties we were heading toward more standardized SMP designs but there were still a lot of exotic architectures out there--and it would be close to another decade before volume processors really became the norm in midrange to high-end servers.
Moore's Law itself was the result of HW engineers, fabs, chemists, and so on doing the same thing. You'd see professionals in the field thinking a certain thing was impossible with existing tech. The textbooks would even describe certain limits or practices. The person and text would be proven wrong within about a year, with new texts written, thanks to sheer volume of PhD's, cutting-edge equipment, and money thrown at the problem. It's truly an exceptional situation.
Far as Thinking Machines comment, I answered that one here:
Not really, the Tilera chips are MIMD with a coherent shared memory.
The CM was a SIMD, and each processor has its own independent memory. A Sun workstation submitted the (single) stream of instructions to the array of processors.
Having a 64K wide SIMD seems completely nuts, though since the processors are single bit, so if you're dealing with 64 bit values, that is "only" 1024-way SIMD. Which is still pretty nuts.
If you declared a 64-bit value, it went "down" each processor's memory, so that it would still be 64K values.
Interestingly, you could ask the machine to consider itself as having twice the CPUs (each with half the memory) and so have 128k, 256k, etc. processors.
Ah, come to think about it, if you had the floating-point accelerator chips, you could program it "slice-wise", which was kind of what you're talking about. Our CM-2 didn't have them, so I never did it, so... ask someone else :-)
A quad-core CPU is worth building, because it is general-purpose, not application-specific. Note that the story already changes with - say - 16-core CPUs. Those are not useful for many users so they are not as common.
It seems to me that the Connection Machine is somewhat analogous to massive-core GPUs, and that with the right tooling one could use it for general purpose apps.
Very well written thesis. The machine, however, was extraordinarily difficult to program, resulting in Thinking Machine's quick death once they actually tried to compete commercially with Convex and Cray.
It wasn't really an open market since the US Gov was the only customer. They liked to spread money around to keep some kind of competition going. There were a lot of fans of the CM architecture. It did take substantial work to get a code running, but once you did the price performance scaling and stability was pretty nice. Enough people had gone through that process that there were plenty of codes that would have like to have continued running.
The generally accepted narrative is that TMC upper management really screwed up. More tactically the CM-5 really stumbled, with a host of hardware, system software, compiler and environment problems. Right before they folded it was starting to be a really nice platform to use. But very expensive, and the 2-3 years from the CM-2 to that point were quite dark.
If you look at the Cray machines from the last 5-10 years they are pretty similar to the CM-5
- fat tree-like guaranteed delivery message network with credits
- commodity processors and the primary node type
- attached vector processors
- front end nodes running commodity OS
of course there are lots of subtle differences, but if you squint...of course the language environments aren't the same (except of course for parallel fortran :/)
Nope, USG was only ~1/3 the market for supercomputing back then. There were lots of industrial uses for supercomputing (e.g. computational fluid dynamics calculations to design tires that hydroplaned less, flame dynamics to create more efficient engines, finite element analysis to model parasitic effects in semiconductors, ..) However the market for the TMC was indeed almost exclusively the USG (or research departments at a few large corporations, and a few universities) because no one could figure out how to program it for something useful - as mentioned before, it was really, really hard to program anything more than a really constrained scenario on the CM.
I spent several years programming the CM-2, and it was a joy to program, and not difficult at all (for most things). I still miss C* in some ways.
Also, see the Inc. article about why they went out of business... it was REALLY bad arrogance about what markets to try to work. They categorically refused to do anything for "ordinary" business problems.
The Connection Machine had some cool art and animation usage.
Karl Sims developed an art installation using algorithms to evolve images based on user selections. It was to see for example in Paris at the Centre Pompidou.
As you can see on his page, the installation used a bunch of color graphics screens, a Connection Machine (2^15 processors). You can see the machine on the top image on the right.
The actual software was written in Starlisp:
I got to interact with the Pompidou installation, mid 90s, you could select your favorite rendering via a foot switch (one for each screen) and the Connection Machine would then render a new set of image based / derived from your choice. It was presented as images created by a "genetic algorithm".
Having a super computer in an art museum was a really powerful statement. (But I was probably more impressed by the whole thing that by the resulting generated images)
It was a brief period at Pompidou when they would, regularly, present computer based works, at the frontier between art and science. (The series was called "revue virtuelle", virtual review)
Interesting -- I had a VHS tape as a kid with Some of Karl Sims's work, but never knew it was exhibited at the Pompiduo as an interactive installation.
Every time I hear about these, I think about the Jurassic Park movie, which claimed to run the park using a network of Thinking Machines Corporation CM-5 machines instead of a Cray supercomputer that the book had used.
That was the Connection Machine 5, a very different (MIMD/fat tree) architecture compared to the ‘classic’ Connection Machine 2-type architecture (SIMD/hypercube) described in this thesis.
> the Jurassic Park movie, which claimed to run the park using a network of Thinking Machines Corporation CM-5 machines instead of a Cray supercomputer that the book had used.
In 1995, Gary Taubes (lately a writer on health and diet) wrote a profile of Thinking Machines Corp.
The Rise and Fall of Thinking Machines
A close up look at a doomed-yet-brilliant start-up computer company that never quite grasped the basics of business.
Funny -- been reading Taubes's "Why we get fat" which is super interesting. Read this article a while back and really enjoyed it. Taubes is a great writer.
this work was pretty thoroughly examined and realized by the resulting company, Thinking Machines.
the one part of the story thatI always wanted to hear about is why CM Lisp turned out to be an ineffective language. The 'thinner'*Lisp was usable for a surprisingly large number of things, but CM Lisp was alot more forward looking and interesting.
The 1980s were the era of clever supercomputing architechures. You had the onset of computed aided design of CPUs or boards; an individual, or small group, could design an d build a supercomputer. PC or Workstation CPU chips were not yet powerful enough to dethrone mainframes or supercomputers. I was science grad student at Stanford that time. On a monthly basis it seemed some new supercomputer company would try to sell us their clever new machine in the million dollar range: supercomputing for the price of a departmental computer.
But these super companies could only update their architectures every 3-5 years, whereas Intel was chugging out faster CPU every year or six months. It only took a decaded for a commodity cluster to beat a mini supercomter company in price-performance. Most these companies never made it to 3.0.
I used to hang out in school with someone who wrote assembly language on one of the Connection Machines. He told me once that there was a single assembly command for sorting 65536 items or something like that. (Probably MergeSort using the tree stucture.)
Well, much of "AI" in its modern form is statistical analysis. SIMD machines are great for linear algebra (integrated vector/scalar machines like 70's CDC machines, Cray and Convex through the 90's, most Intel and AMD processors today through their SSE and AVX instruction set extensions). Any statistical calculations that use linear algebra can be accelerated with a SIMD architecture.
True, and as hardware keeps evolving towards array-based setups like GPUs/ASICS with FPGAs and other hybrids, languages like J/APL/K fit them like a glove.
Using qdb from Kx or Jd for J, both columnar databases with billion row operations on capable vector-processing hardware is the modern realization of this type of computing.
These languages are not currently too popular, but I think a cousin of theirs will come on the scene and really light things up. Some current attempts at newer GPGPU targeted languages have popped up these past few years [1, 2, 3, 4].
The difference between these array languages and something like NumPy and other libs to add array processing and better numerics, is that the columnar databases or stores, have the array language built-in; you can abstract out, but if you program with them, there are no onion layers between you and speed and processing.
Well, they're both SIMD architectures. But I think the way you program them is/was very different.
I quite miss programming the CM-2 in C* -- it was a really nice environment, and really pretty easy to program to (contrary to the legend I've seen thrown about that they were hard to program). Once you understood the various library functions, it was usually pretty straightforward to get things done.
Connection Machine CM-2 Technical Summary (PDF)
http://people.csail.mit.edu/bradley/cm5docs/nov06/Connection...
The Essential StarLisp Manual, with Programming Examples for the Connection Machine (PDF)
https://omohundro.files.wordpress.com/2009/03/omohundro86_th...
Getting Started in StarLisp (PDF)
http://people.csail.mit.edu/bradley/cm5docs/nov06/GettingSta...