But the point is that there were super important algorithm advanced that allowed training bigger models. It wasn't all hardware... A big part of it was getting more eyes on the problems (via hype over imagenet results circa 2012) to figure out how to solve them.
