

Verizon 4G LTE is down, nationwide outage confirmed - tilt
http://www.theverge.com/2011/12/7/2619137/verizon-4g-lte-outage-december-2011

======
mbreese
As someone who understands networking at the office / rack level (but not the
datacenter or nation-wide level), what are the single points of failure that
could bring down an entire nationwide network? I would think that things would
be setup by regions just to avoid this type of scenario.

At least the 3G networks are still up for voice/sms fail-over.

~~~
nknight
pquerna's guess is a reasonable one, though I don't know the authentication
mechanisms, either.

Here are some other wild guesses that may have little or no resemblance to
what Verizon is experiencing, but kind of show what can bring down an
Internet-like network:

* Some sort of configuration fat-finger that got rolled out everywhere without proper testing. You'd like to think that's impossible on this scale, but after having an insider's look at the operations of a couple major US cable companies, I put nothing past telecoms.

* Verizon probably runs some form of routing protocol internally. If a core router has gone insane, you could, conceivably, get route flapping and bring down pretty much everything, but that's not supposed to happen on carrier-grade equipment ( _cough_ see previous paragraph).

* In a closely related scenario, large portions of the public Internet have been brought to their knees at least once by a nasty BGP bug in a major vendor's routers that got triggered when somebody broadcast just the right set of options in their announcement. Picture a guy in upstate New York trying to route around a bad line card, and suddenly the US goes dark because half the routers crash when they got the BGP table update.

You can probably spot a common theme here -- a "decentralized" but homogenous
network is still vulnerable to the triggering of problems shared by all
points.

~~~
Nrsolis
The thing you have to understand about "wireless" systems like
CDMA/GSM/HSPDA/LTE is that there is little diversity in the vendors of these
types of systems and there are a bunch of relatively closed protocols that
govern how things like the base stations authenticate subs and do provisioning
of stations. It's not like IP; it's more like Novell IPX.

So even though LTE is ostensibly an IP-based system, there are a host of
dependencies on single vendor closed management systems and protocols that
could be the cause of any outage. A nationwide outage points to the failure of
one of these systems to properly handle letting subscriber stations onto the
network, not something simple like getting packets from here to there.

Think failure of the RADIUS/DNS like systems, not the routing infrastructure.

~~~
Maven911
Your premise is more accurate, you just need to replace IPX with 3GPP standard
protocols (vendors dont make their own protocols - everything is
standardized), and instead of radius/dns (which are used in telcos
nonetheless), its more likley HLRs,CSCFs, MMEs, or PDN gateways that can bring
an entire network to its knees, or a poorly implemented mobile backhaul
solution

------
darksaga
As a current 4G customer, I can say this has been the most frustrating 6
months. It's been well documented their LTE network has been sketchy from the
start:

4G launched in Dec 7th 2010 and this article ran a week later about issues
already showing up with handoffs between networks: <http://bit.ly/td1XQT>

Another article from April 29 2011 about an earlier 4G outage:
<http://bit.ly/v8zJUZ>

I'm not surprised this happened and will be returning to 3G. Clearly their 4G
is not ready for prime time yet.

~~~
potatolicious
I'm not too familiar with 4G tech - but why "move back" to 3G? If the LTE
network falls over, your phone switches to 3G and keeps right on humming along
more slowly, doesn't it?

Or is my concept of 4G modems out of date?

------
jtchang
I'm actually surprised at how resilient the cell phone networks are. For
something that is so heavily used I would expect it to go down more often.

Or maybe my definition of going down is different than others. There are
plenty of bugs all the time (dropped calls, misdialed numbers, etc). I'm not
convinced that whenever I dial a number and someone else picks up and says
wrong number that I did indeed dial the wrong number.

Ghost in the shell I tell you.

~~~
kalleboo
> There are plenty of bugs all the time (dropped calls, misdialed numbers,
> etc).

Precisely. Erlang was created to run digital phone switches, and so was
designed exactly for the scenario of "it's fine if certain calls crash, they
can always be redialed, but the system itself always has to be available".

------
williamle8300
that's interesting. a fake telecom spec (4G!) can't perform like a 3G network.

~~~
Maven911
Care to elaborate ? Are you referring to the fact that 4G was meant to be 1
Gbps but marketing departments hijacked the term to just mean "as fast or
faster then 3g"

~~~
williamle8300
yea pretty much. the whole term, 4G, is just a dubious marketing term meant to
mislead consumers.

------
cowmix
The $299 price tag for a Verizon Galaxy Nexus is looking like an even worse
deal now.

------
grp06
4G is down: 23 customers affected.

~~~
Karunamon
I know it's an overly snarky comment, but being from a state that has
literally ZERO 4g coverage, I gotta share the frustration. You have to live in
a huge city to even get the signal, and when you do get it, your battery
consumption is greatly increased. 4G as it stands for me is a huge pile of
care not.

~~~
imbriaco
You actually don't have to live in _that_ large of a city. The Raleigh/Durham
MSAs combine for about 1.6M in total population spread over a pretty sprawling
area and we have had LTE service for a few months.

I frankly don't get the complaints or the folks who are going to downgrade to
3G. If you don't have LTE available, the device will fall back to 3G. When you
do have LTE available, however, the performance can be unbelievable. I was co-
working with a colleague and we were getting 20Mbps down and 10Mbps up over a
little MiFi.

I'll hang onto my LTE MiFi, thankyouverymuch. :)

~~~
mrb
For comparison, the best I have measured with a 3G technology (HSDPA) was 10
Mbps down, while downloading from kernel.org on my laptop, tethered to my
phone over wifi, in a sparsely populated area covered by a strong HSDPA
signal.

------
alanh
(In the voice of the Apple fan in the … Samsung? LG? HTC? commercial: _Why
don’t you just get a 4G phone?_ )

