
EU's GPS satellites have been down for four days in mysterious outage - new_guy
https://www.zdnet.com/article/european-gps-satellites-have-been-down-for-four-days-in-mysterious-outage/
======
michaelt
I'm more familiar with the older US GPS system than I am with Galileo, but I
can make an informed guess.

GPS satellites broadcast "ephemerides" letting receivers work out where the
satellite is - and some other details like clock error and predicted effects
of the ionosphere.

You can make a rough prediction of these weeks in advance, but not precisely
enough to meet GPS's precision requirements. Up-to-date short term predictions
are needed to make things precise enough, so a ground-based 'ground segment'
or 'control segment' broadcasts updates to these every 30 minutes or so.

The 'ground segment' is comprised of ~20 monitoring stations (at known
locations, measuring the errors on signals from satellites), one 'master
control station' that aggregates this data and decides what update to send
next, 4 ground antennas that can send that update to the satellite no matter
what side of the globe it happens to be on, and a backup MCS.

This is what is meant by "ground infrastructure".

The design of GPS is informed by its cold-war-era-military origins, so there's
a backup master control station, and the satellites have an 'Autonav mode'
letting them operate for 60 days in even if the entire ground segment has been
destroyed.

Assuming the Galileo system has a broadly similar design, you can imagine the
criteria for a four-day outage:

* Failure of both the master control station and the backup, in a way that takes more than 4 days to recover from. Maybe their backup control station wasn't fully commissioned yet, wasn't fully redundant, or some IT problem hit both sites at once?

* Discovery of some fault severe enough they felt they had to shut down until it was fixed, and the fix couldn't be rushed out.

* Either satellites have no autonav mode, or it somehow failed (such as due to bad data from the ground segment), or it's imprecise enough they decided pushing people onto GPS was a more responsible option.

~~~
makomk
The rumour I saw was that the entire ground segment is dependent on a single
Precise Timing Facility in Italy, which went down and took everything with it.
Not sure what the holdover capability is like without the ground stations, but
I'd venture a guess it's not great,

~~~
michaelt
According to a report from 2009 [1] _" The Galileo ground segment has two
GCCs, each with a Precise Timing Facility (PTF) providing a very stable
reference GST to all elements in the GCC. The two PTFs are developed by
separate companies; this diversification is applied to reduce the possibility
of common failures in this important element."_

I suppose it's possible the second GCC hadn't been fully commissioned? Or the
switchover failed in some very hard to recover way?

[1]
[https://reports.nlr.nl/xmlui/bitstream/handle/10921/210/TP-2...](https://reports.nlr.nl/xmlui/bitstream/handle/10921/210/TP-2009-082.pdf?sequence=1&isAllowed=y)

~~~
antpls
If the two GCCs don't agree on the Time, which one is correct ?

~~~
michaelt
According to the document linked, each GCC has its own PTF comprised of _four_
Caesium atomic clocks. So between the two sites, they have eight atomic
clocks.

Presumably if one atomic clock seems suspect, they simply look to their other
seven?

------
stupidcar
Given that the project is funded by €10 billion of EU taxes, I hope we'll get
transparency sooner rather than later on exactly what's gone wrong, regardless
of who it might embarrass.

~~~
mtgx
If it costs that much you'd think they'd also offer encrypted GPS, too.
They'll regret it eventually anyway.

~~~
Dont_panic
Encrypted GPS do yu even make sense, what benefit an encrypted GPS will
provide. Eu will know where were you when you requested something and even if
your data was encrypted and i don't think you can intercept gps coordinates
and you have to intercept at least three for it to be useful.

~~~
detaro
Yes, encrypted GPS makes sense. You can broadcast additional information that
can be used for more precision only on encrypted channels, and have people pay
for the decryption keys. (That's part of how the better-quality GPS for US
military works)

Another way is selling online access to better-quality correction data.

Neither necessarily involves telling anyone where you are.

~~~
bluGill
Even the US military doesn't use the encryption channels much anymore. Turns
out that differential GPS is more accurate and you don't even need the
encryption channel to do it. We already know the next message the GPS will
send with high accuracy, the important part is detecting exactly when the
first bit arrives at the antenna you everything you need from that.

If you want to get more accurate you don't use the encrypted channel you put a
GPS in a fixed location on earth and have it send corrections to you. (called
RTK)

Edit: Note that I have no information on what the US military uses their
classified channel for. It is unlikely they get any accuracy information from
it anymore given what civilian GPS systems have figured out.

------
pgt
Not trying to be funny here, but I would not be surprised if an automated
Windows update borked something. Recently heard about an Antarctic base that
lost heating for 36 hears after a Windows Update triggered a several GB
download over the base's modem downlink.

~~~
eightysixfour
The heater required an active internet connection and available bandwidth?

~~~
Fordec
It was actually just a gaming PC with a Nvidia GTX480 belonging to a scientist
perpetually left turned on

~~~
pizzapill
Those NVIDIA fanboys. I have a AMD RX 480 which heats my room just as good.

------
t3rabytes
It's a system/service that's still in beta, I'd expect some occasional issues
until it's deemed "production-ready".

~~~
jhayward
Everything on orbit and on the ground is production product.

When you are several billion euros in, have 24 satellites on orbit, and have
been broadcasting with the ' __Valid __' bit set for 2 years you aren't really
in 'beta' as we use it.

They have been in production mode in all but name. Given the life/safety
implications, there are some rather rigid qualification checks that have to be
done to be 'fully certified'. That's what's been going on.

------
saagarjha
> In a statement published after this article's publication, the GSA blamed
> the Galileo outage on "a technical incident related to its ground
> infrastructure."

I wonder what this could mean?

~~~
mtmail
copying from
[https://news.ycombinator.com/item?id=20427544](https://news.ycombinator.com/item?id=20427544)

Guenter Hein, Professor Emeritus of Excellence at the University FAF Munich
told us, “As far as I know, it is a problem of the PTF [Precise Timing
Facility] in Italy – time has an impact on the whole constellation!”

~~~
saagarjha
But the satellites should be carrying their own clocks?

~~~
elfchief
They do (that's literally how GPS/GNSS works), but they drift, for various
reasons. One of the parameters that GNSS networks broadcast is a clock
correction (how far off each individual satellite's clock is, how fast it's
changing, and how fast how fast it's changing is changing), and those
parameters come from the ground stations.

It seems like a little drift shouldn't make that much difference, but keep in
mind that the speed of light in a vacuum (or air) is about 1 foot per
nanosecond, so for every nanosecond one of those clocks has drifted, that's a
foot of positional accuracy you no longer have...

~~~
Salgat
But GPS can run up to 60 days with no ground station contact at all.

~~~
elfchief
Nominally, yes, but the positioning error (according to the spec) is expected
to be up to 425m after 14 days, growing worse over time until it hits a
positioning error of ~10km at 180 days.

(later generation satellites have an 'autonav' mode that presumably makes this
less severe (by using SV-to-SV ranging and such so they're not completely
blind to changes), but Galileo may not have that capability or there might not
be enough SVs to use that capability yet)

------
benj111
So what would break if GPS went down for 4 days?

Sat nav obviously, Amazon package tracking, Grindr? (I assume that falls back
to cell tower triangulation???).

I guess theres some more serious things, any examples?

~~~
galangalalgol
this will be a fun thread! high frequency trading relies on gps disciplined
osscilators for the precise timing needed (ntp isn't good enough, ptp isn't
even good enough, though White rabbit should be). so hft would cease or go
wonky

trucking companies rely on it to enforce rest periods. so more truck
accidents?

turn by turn direction on phones would go out. wi fi is enough for nav, unless
your phone and the wifi are on different ntp networks maybe I had a phone with
broken gps and I didn't really notice.

~~~
a3n
American trucker here. Occasionally I will lose the GPS signal in my truck,
but my hours of service still counts down. The electronic logs are kept
onboard, and transmitted at some point to headquarters.

I suppose the Qualcomm unit might eventually be off by a second or three. But
I don't know if its clock is updated by GPS, phone or wifi. I'm guessing
phone.

We're also required to carry paper log sheets in case of Qualcomm outage. So
then it would be paper, watch and honesty. The honesty can be sanity checked
by speed an distance.

~~~
sethhochberg
I don't know much about Qualcomm's gear for truckers, but FWIW even if the
machine does pull its time data over the cellular network (possible on CDMA,
less likely - impossible? - on GSM), the cell tower itself likely relies on
GPS for its own time sync. Its very common for GPS to be used as a globally-
accurate source of time data even in applications that have nothing to do with
location.

~~~
toast0
GSM also provides a time reference. I seem to recall connecting to some
networks that didn't provide it though.

Base stations should be using GPS (and others) to discipline a local oscilator
that should be ok enough for a while in absence of signal. They can probably
also get a timing signal from their backhaul.

It works better with precise timing, but I think it's workable with best
efforts.

------
supermatt
I had someone come and measure boundaries for some land on Friday, supposedly
with centimetre precision. Does this mean that the markers they installed are
likely in the wrong place?

~~~
module0000
It depends... were there any markers already present? In a past life I worked
as a surveyor, and if a marker was present(it looks like a silver-dollar on
the ground, with a spike holding it in place, these can last 100+ years), we
would use that to measure from. If, however, they set up what looked like a
tripod with a oval-shaped dish on top, that's a GPS receiver, and could be
affected by this outage.

Link: here are some survey markers for sale, linking it so you can see a
picture of what they commonly look like:
[https://www.berntsen.com/Surveying/Survey-
Markers/Aluminum-S...](https://www.berntsen.com/Surveying/Survey-
Markers/Aluminum-Survey-Markers)

~~~
supermatt
No - they placed new markers (orange disks on long spikes) driven flush into
the ground, but they were walking around using GPS to do this (with supposedly
centimetre precision, which I understand is what GNSS is for). This is in
Eastern EU.

------
dfeojm-zlib
This is a good reminder to buy and use satnav modules supporting multiple
positioning systems and technologies: most of GLONASS, GPS, BDS, Galileo,
IRNSS, SkyHook, etc.

Disclaimer: Trimble Ltd. alum

------
ppod
They put the _Precise Timing Facility_ in _Italy_?

~~~
toyg
Casual racism, because fuck yeah.

Italy has a pretty big space-related industrial sector, and excellent academic
standards in some fields. It might not produce geniuses with the same
frequency of the past (Galvani, Volta, Marconi, Fermi and so on) but it's not
some random backwater.

~~~
martin_a
> but it's not some random backwater.

And nobody really thinks that.

My god, what a time to be alive, when you can't make simple jokes anymore,
because somebody somewhere always is offended and thinks there must be some
racist agenda in place.

Even as a German with a total lack of humour (and an Italian motorcycle with
strange quirks in the garage) I found this funny. Funnier than the Holocaust
at least.

~~~
toyg
_> > [Italy is] not some random backwater. > And nobody really thinks that._

As an Italian expat, I can assure you this is unfortunately incorrect.

 _> you can't make simple jokes anymore_

No, you can't make jokes based on lazy stereotypes that end up affecting the
lives of millions. It's a different thing. It might have been fundamentally
innocuous, back when populations didn't mix as much as we do today, but now
it's a real problem. If I go to a job interview and the potential employer
assumes Italian == lazy, I'm done for. That's the casual bigotry that these
jokes end up entrenching.

~~~
martin_a
And I've worked with and for Italian people and they were just as lazy or
hard-working as any other nationality. No reason to value them more or less.

