With DevOps you monitor [at least] four metrics: lead time for changes, deployment frequency, mean time to recover/restore service, and change failure rate. These indicators show if you're improving, stagnating, or straining. You can also track more specific service level indicators, the amount of toil to project work, and tech debt versus feature backlogs.
The "everything's on fire" metaphor also has a larger context. Sure, when people are getting woken up in the middle of the night because the site is down, shit's on fire. But also "we're constantly missing our deadlines" is shit's on fire, "our customers are not satisfied" is shit's on fire, "our overhead is way too high for what we're delivering" is shit's on fire. If you're out on the water and building a new boat because yours is on fire, it's a little late.
The "everything's on fire" metaphor also has a larger context. Sure, when people are getting woken up in the middle of the night because the site is down, shit's on fire. But also "we're constantly missing our deadlines" is shit's on fire, "our customers are not satisfied" is shit's on fire, "our overhead is way too high for what we're delivering" is shit's on fire. If you're out on the water and building a new boat because yours is on fire, it's a little late.