>"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place."
Is "sloppiness or laziness" really the only possible attribution here? I'm not a big fan of Verizon but I'm a big fan of civility and empathy, two qualities which your blog post lacks. Outages are a really unfortunate fact of life. We've seen them recently with Google, AWS, Dyn - all companies where technical competency is generally not questioned. It's quite possible the cause of of this outage was some "perfect storm" scenario such as an eBGP router rebooted and came up with a stale or incorrect config. "Perfect storm" scenarios even happen at companies with very rigorous engineering cultures as we saw with the most recent Google outage.
Your attempt to shame an organization without knowing all the details reeks of immaturity and pettiness. Ditto for your willingness to turn this into yet another Cloudflare marketing opportunity. Have you forgotten about your own Cloudbleed incident? How would you feel if it a security company took that as an opportunity to shame you for "sloppiness or laziness"? Or some other company's CEO was offering to send people "Cloudbleed Support Group" T-Shirts on HN as your own CEO is doing in this thread?
Lastly RPKI isn't a silver bullet, RPKI authorities can also be misconfigured and attacked. This happened with the LACNIC incident in 2013. It's also worth mentioning that RPKI potentially creates new threats. But again it seems more important to you to use this as a marketing opportunity and promote yourself while throwing someone else under a bus while uttering pithy summations.
Also from your post:
>"And, in particular, we're looking at you Verizon — and still waiting on your reply."
Although Verizon is the 400lb gorilla in the room, their NOC and network engineers are still regular people with kids and families and feelings. They are also people who have had a really shit day today. Why you can't extend just a bit of human compassion and feel compelled to try to shame is quite inexplicable.
You may think that your blog post was a marketing coup but I see it as a massive failure in in both leadership and civility.
As a thought exercise maybe Cloudflare leadership could think about how they would like the community to react the next time they are at fault.
More than eight hours on, after utilising everything from what they were told was a Tier 1 support line to Twitter, they have nothing.
Even if we're kind to Verizon about the network failure, which was a global issue, they haven't done anything or said anything to suggest that Cloudflare should be treating them kindly in any way.
Not even a "we're aware, we're handling it".
Ghosting one of the world's largest (as in utilised) companies is not wise for administrative, technical or PR reasons.
Verizon have shown a complete lack of leadership.
>"Ghosting one of the world's largest (as in utilised) companies is not wise for administrative, technical or PR reasons"
Oh the Cloudflare marketing machine. Largest by "utilized"? What does that even mean? Cloudflare is not a Tier 1, a Tier 2, or a major eyeball network. They are pretty far down in the pecking order despite what your marketing department wants us to believe. There's always some fuzzy stat isn't there?
Being too inundated to respond to everyone on the day of outage is a human resource problem, plain and simple The fact that you have taken this so personally is kind of embarrassing. What this blog post, the opportunistic marketing ploy and finger pointing have shown is a complete lack of maturity on your part. You want to call out Verizon for their behavior yet your own behavior is unnecessarily aggressive.
What? I have said nothing personal.
> What this blog post, the opportunistic marketing ploy and finger pointing have shown is a complete lack of maturity on your part.
Ah. You seem to be confused. I am not affiliated with Cloudflare, and have not worked with Cloudflare at any point in time.
This is just an appeal to emotion. No-one is even calling out any individual people. With a company of this scale and responsibility, individuals shouldn't even come into the discussion, and there should be multiple levels of redundancy. Verizon, collectively, is being shamed.
Verizon should be compared to a power plant, not a SaaS provider or some 3-person dev shop.
Not at all, its an appeal to civility. The statement that Cloudflare made with "there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place' is the appeal to emotion here.
>"No-one is even calling out any individual people.'
No, that is a very clear attempt to call out a specific group of people who work in the network engineering department.
No, you didn't. You provided a vague conjecture for how the initial cause of the problem might not have been gross negligence, but offered no hypothesis for why Verizon isn't answering the Red Phone.
Verizon was in contact with people yesterday. I have spoken to two people from two other carriers who were in touch with them. And you are just parroting the idea that because Cloudflare didn't get a response that Verizon wasn't responding to anyone period. And that's just not true. The fact that you think there's some red phone that just anyone can call the NOC and magically speak to someone during a major outage shows you have no practical experience with thes things you are commenting on and criticizing.
If that's what it takes to be able to decide, then I guess we can safely declare that it wasn't negligence, because no-one that fits that description will ever publicly admit to it.