Distributed memory parallel computing library. Most common in supercomputers with a high performance interconnect where you want to run one program that coordinates separate instances of itself via message passing (“SPMD” - single program multiple data parallelism). Slightly different than distributed computing that generally doesn’t assume a tightly coupled system with a low latency, high bandwidth interconnect. Allows you to scale up beyond what you can do within shared memory spaces where you are limited to low core counts (hundreds) and low memory (<1TB) - MPI is what you use when you’re in the many thousands of cores, many TB of RAM regime. In practice, supercomputers require a mixture: shared memory for many core parallelism, some hybrid host programming model for bringing accelerators in, and then MPI to coordinate between these shared memory/accelerator enabled hosts.
