The difference between H.264 and "any video codec ever" that may have an infringement charge brought is that H.264 has a large, well-organized overlord in the form of MPEG-LA. They are actively looking for violations, and threats of suits from them are credible and realistic. "Any non-trivial program" may have an incidental patent infringement claim filed against its authors, but with MPEG-LA and especially a violator as big as Mozilla, you can expect a claim if you don't purchase a license ahead of time.
What I mean more specifically is that a group like MPEG-LA could be lobbied more effectively by Firefox to produce some kind of acceptable licensing model for open-source software and end-users if Firefox had stayed in the game instead of taking their ball and going home.
I think you're confusing MPEG and MPEG-LA, but either way you're mistaken.
MPEG claimed they were going to create royalty-free profiles of H.264 and then went back on that (possibly due to cunning political moves by Microsoft) and now claim to be evaluating a couple of ways forward for royalty-free codecs (a profile of H.264 again, or building on older MPEG 2 tech). It's fairly obvious that it was pressure from (primarily) Google & Mozilla that put RF codecs back on the table for them.
MPEG-LA reduced the uncertainty around their future codec licence payments, and as a result their maximum profits. Again almost certainly due to Google and Mozilla.
They didn't take their ball and go home, they negotiated and competed and as a result won valuable concessions that continue to have value as the next round of codec development begins.