
North American Object Storage Service Impact - mlosapio
https://documentation.meraki.com/zGeneral_Administration/Support/North_American_Object_Storage_Service_Impact_(8-4-2017)
======
creyes
I designed/deployed a decent sized Meraki network about 4 years ago - at the
time it was one of the larger full-stack Meraki networks that exists. An 11
site school district with the edge, all idfs, ap's, phones and at a a later
point some of their cameras.

Meraki still thinks of themselves a startup, but they have these "uh-oh's" all
the time. Random bad firmware that turns off the 5g channel on the MR42's. A
DPI "upgrade" that blocked ALL SSL traffic (which at this point is basically
all traffic). Their solution was always to try "beta" firmware... in
production... in the middle of state-mandated online testing.

I was a huge advocate for them but at some point it's gonna be hard for me to
keep recommending them. They're so excited about new features but really fail
about 1) fixing bugs and 2) ensuring robustness. The "fail fast fail often"
mentality really shouldn't work with critical infrastructure

~~~
tgtweak
Ubiquiti (unifi) was very much like this back then too, and to this day still
breaks stuff every update. They're getting better now but running a large site
or multiple large sites was a constant game if whack a mole trying to figure
out which firmware works best on which equipment.

I have heard similar stories to yours about meraki and that's what swayed the
decision to just go ubiquiti since it's less expensive.

~~~
chrishacken
We removed almost every piece of Ubiquiti equipment from our network because
of the quantity and severity of bugs in their products. For example: Disabling
a port on Edgerouters will grey out the port in the UI, but it doesn't
actually stop traffic from passing through the port.

I also have no love for how some features are only available via command line
while others are only available in the UI. This also differs depending on what
product line you're using. Pick one strategy and stick to it.

~~~
tgtweak
What are you using instead now? Genuinely curious.

~~~
chrishacken
We're nearly 100% Juniper on the network side. The PtP and PtMP equipment we
were using from Ubiquiti is now 95% Mimosa. The only Ubiquiti products that I
like are Airfiber 24's and that's only because no one else offers an
alternative.

------
jacquesm
Whenever people say they don't need backups because they are 'cloud based' I
always wonder what they'll do when their precious cloud provider messes up.
The chances of this happening to Amazon, Google or Microsoft are small but
they're not 0, if it can happen to Cisco it likely could happen anywhere.

~~~
samfisher83
Do you think you will do a better job of backing up stuff compared to google,
msft, etc.? They have dedicated engineers and spend lots of money on this
stuff.

Think of it from a statistical perspective what is probability of you setting
up this back up system vs them?

~~~
icebraining
They may have better engineering, but they also have extra risks. My home
server will never ban me because it thinks I've violated its TOS, for example.

~~~
toomuchtodo
Nor will your own storage lock you out because you've annoyed a state actor,
while a cloud provider will roll over.

------
SoMuchToGrok
Not surprised to see this, I've run into many issues with Meraki devices over
the past 2+ years.

Their support team is amateur at best; at one point I had 6 Meraki engineers
working on a DHCP problem (yeah...DHCP) and their recommendation after several
weeks of troubleshooting - do a factory reset.

I have dozens of stories...don't even get me started.

------
bogomipz
>"The issue has since been remediated and is no longer occurring."

I would think that if you lost your data, unless they have restored your
deleted data the "issue" is still very much occurring for you as a customer.

Wouldn't remediation be that they have recovered your lost data?

~~~
bartread
That would certainly be my understanding of remediation.

------
bartread
FTA: "Your network configuration data is not lost or impacted - this issue is
limited to user-uploaded data."

Errrrrrr... so the issue was limited only to _data I would actually care
about_ then? Or did I misread?

That is a frankly extraordinary use of weasel words.

~~~
wmf
Yeah, you misread. Configuration data is what makes the network actually work
and that was preserved.

~~~
bartread
Fair enough. And I suppose that release was meant for people with an
understanding of the product, in which case that makes more sense. Still...
interesting choice of words. "Oh, it's only the user-uploaded data that we've
lost."

------
madsushi
Only losing user-uploaded data seems pretty mild, in this case. Meraki sells
cloud-managed network hardware, so you don't interact with it that often.
Network configurations, logs, traffic data -- any of those would've been much
worse to lose. The custom bits like voicemail greetings and IVR will likely be
the hardest to replace.

------
rsync
Does anyone familiar with their marketing language know how many "nines" they
had on their resiliency number ?

~~~
kyledrake
As many extra as were punched into that command that permanently wiped out a
bunch of AWS EBS data a while back.

It haunts me to think about how many people are using these services as their
single source of data. Fat fingers melt through 9s.

------
oneplane
And this is why outsourcing 100% of your stuff isn't the greatest idea. Sure,
the 'managing servers is not your core business'-story holds up most of the
time, but when you have no control of anything anymore, you no longer control
your services.

~~~
rhizome
I would think "business continuity" is a core function of, you know, business.

~~~
oneplane
As do I, but that's not what marketing teams advertise for ;-) I suppose doing
something like specific offloading makes for a great cloud case, but being
able to run the baseline at home will at least guarantee a degree of control
that will keep you going.

