The human brain is also three-dimensional, heavily interconnected, and has built-in thermal management at every scale. Chips are much faster, but still operate on the essentially linear memory cells, and this limits how many matmuls you can do per second. If we can figure out true connectivity without doing tons of matmuls, then we should be able to massively cut computational demands of models.
