
The July Galileo Outage: What happened and why - shannietron
https://berthub.eu/articles/posts/galileo-accident/
======
outworlder
The most interesting part of the whole write up is that it sheds light on how
dependent of the base stations these constellations are. I used to think that,
other than sending orbital corrections and the occasional fixes, the
constellation would run itself. Apparently this is not the case.

So a localized outage can in fact affect the entire planet. This has also
implications in disaster scenarios.

Not sure how applicable this is for GPS.

~~~
nwallin
The GPS constellation is considerably more robust to a ground station outage.
If the ground station goes out, the satellites themselves are a distributed
network and they use each others' data to correct themselves. It can operate
in this mode for something like 60 days before it's unable to continue
correcting itself.

Of course, if the ground station failure was a soft failure, and instead of
ceasing to upload, it began uploading incorrect data, the GPS location results
would be arbitrarily bad to unusable. It's not clear to me whether the Galileo
outage was because of a hard ground station failure or a soft one. But given
that one of the contributing factors was that the backup system was not
online, it would indicate to me that this was a hard failure that GPS would
have been able to correct.

~~~
bradknowles
The other factor for the US GPS system is that there are actually over 24
ground stations, one for each time zone plus at least a couple of spares. A
local failure in one time zone doesn’t take out the whole fleet — they can
fall over to a backup ground station, and backups for the backups, etc....

------
guykk22
Just wanted to give you a little bit more info. I work for the Spanish company
that has developed the OSPF, GMV, and although I'm not currently involved at
all in this project, from what i know, the following statement: "around 5% of
the Galileo capacity is lost to software problems likely in the Orbit
Synchronization Processing Facility (OSPF), run by GMV." is not fully true.
The OSPF receives data from sources (From my limited knowledge, time stations
in order to generate the ephemeris), and it looks like the problem was not in
the OSPF code but in one of these time stations sending the data.

~~~
ahubert
Hi - thanks for this (author here). What I mean is, currently, around 5% of
the satellites on average broadcast NAPA status. I did not refer to the big
outage with this statement, I mean to ongoing developments. Or perhaps you are
referring to the same thing & in that case it is entirely possible that the
NAPA flag gets raised because the ephemeris is being generated from stale
data. Do you know? Thanks!

~~~
zmk_
So that's 1 satellite on average. Even though 5% means the same it makes you
think the number is much larger if you do not constantly think of the total
available.

------
ahubert
author here - if anyone has any questions or remarks, let me know!

~~~
efm
Thanks for writing this.

The title doesn't mention that Galileo is a Global Navigation System (GPS).

~~~
9nGQluzmnq3M
GPS refers specifically to the US system.

~~~
myself248
Which is funny, because I've heard it both ways.

For a while, as new systems were coming online, GPS was being used as a
generic term, and the US system was being referred to by its original name,
NAVSTAR.

But only purists would do that, and everyone else kept saying GPS to refer to
the US system, so the new term GNSS was invented as the generic.

I try to say NAVSTAR and GNSS to avoid the ambiguous GPS, just like I say
"gridiron" and "soccer" to avoid "football".

~~~
reaperducer
_But only purists would do that_

Perhaps the same sort of space nerds who still call Dish Network "Echostar."

/Waves hand.

~~~
myself248
Or track the ISS as Zarya.

/waves a trusty Omnipoint phone

------
dzhiurgis
Is 1cm accuracy part of Galileo system already operational? If so, why our
phones and other devices cannot utilise it yet?

~~~
Rebelgecko
Galileo has like a dozen different signals and I believe that most of them
aren't broadcasting yet. I'm not sure which of those signals is supposed to
give 1cm accuracy, but it could be one of the encrypted signals. Another
consideration is that the 1cm accuracy would only be possible with fancy DGPS
or RTK setups. 1cm accuracy is doable with GPS today, but unless you're a
farmer or land surveyor you probably don't own any hardware capable of
realizing that accuracy.

~~~
mehrdadn
I suspect this is an extremely naive question, but why would one signal give a
1cm accuracy and another one not? Shouldn't the satellite just tell you its
trajectory + identification info so you can calculate where you are based on
that? Or perhaps by signal do you mean the physical electromagnetic properties
(frequency etc.) are unsuitable for high accuracy, rather than the actual
information content it's supposed to carry?

~~~
petschge
To get good precision you need to remove the effect of electrons in the
ionosphere. These basically change the index of refraction and therefore the
propagation speed of the radio waves. The delayed signal makes your receiver
think it is further away from the satellite than it actually is. One way to
estimate this delay (instead of relying e.g. on the correction signal of a
differential-GPS base station close to you) is to receive signals from the
satellites at two different frequencies. The time delay is proportional to
f^-2. Knowing the frequencies and the delay between the two signals allows you
to estimate the electron content and and correct both signal to the undelayed
"infinite frequency" arrival time.

~~~
mehrdadn
Ahh I see! So they're referring to signals of different frequencies? Makes
sense, thanks :)

------
ken
The "instantaneous violations" link is dead -- I was hoping to learn what that
means!

~~~
ahubert
back up, thanks! It is on our experimental node. What is means is: the Galileo
service definition regards a 'dilution of position of >6' to be problematic,
but only if this happens for more than 77% of the time. The instantaneous bit
is "well it is >6 NOW".

------
madengr
Maybe instead of 24*7 work, some manager thought it was 24/7 work, and sent
everyone home.

~~~
jepler
we shoot for π uptime, or around 22/7...

~~~
toast0
Eight 8s

------
jtwaleson
When was the last time GPS had issues like this? I can imagine that during the
early years this happened there as well? Or maybe not, as military quality
engineering requirements were stricter.

~~~
toyg
Or rather _silence_ requirements were stricter. The military is not in the
business of publicising failure and weaknesses.

~~~
vel0city
This is a bit of the interesting aspect to this failure. Official channels
have overall been pretty weak on details on what all happened in this error.
Most public knowledge of what happened has been gleaned from the signals
coming off the satellites themselves which anyone can see. This live data
combined with understandings of how the system _should_ work is most of the
public knowledge of what has been going on.

I don't have a ton of knowledge into the deep technical aspects to GPS, but I
imagine we would probably have some similar clues of an outage of this scale
in GPS. Maybe a little less technical details of what is happening behind the
scenes, but knowing a high percentage of satellites entering a no guarantee
precision mode should be possible.

~~~
myself248
Exactly. And the galmon network already does that. (It should really be
renamed gnssmon at this point, it's far outgrown its Galileo-watching roots.)

You can run a station yourself if you have a Ublox 8- or 9-series receiver.

~~~
vel0city
As an amateur radio operator interested in space signals, it definitely
intrigues me. Does this galmon project collect data collectively as a wider
analysis of the satellite network, or does operating a node only really
benefit the group operating a node? I'd love to operate a station to analyze
GNSS traffic.

------
FeepingCreature
My current impression from reading this article is that SpaceX could create a
better GNSS than Galileo in a year as a side project - not because of their
cheap launches, but because of their diy development culture. I am stunned by
the level of organizational inadequacy of allowing such critical parts of the
system to be developed by not just a contractor but a chain of subsidiary
contractors, with feedback cycles that move at the speed of corporate
communication. My expectation now is that Galileo will never catch up to GPS
and will eventually be forgotten.

~~~
abdulhaq
Galileo is a strategic asset in case of dispute with the US. It is not in a
competition with GPS, it simply has to function as designed and continue to
operate independently of US desires. The US objected to the construction of
the Galileo system.

~~~
creato
The US did not object to the concept of Galileo, the US objected to the EU's
choice of frequency, which made it impossible to jam Galileo without jamming
GPS. The EU changed the design to avoid this, and I am not aware of any
follow-up objections.

