
A bug in SpaceX’s communication system hid ESA's messages - amaccuish
https://www.theverge.com/2019/9/3/20847243/spacex-starlink-satellite-european-space-agency-aeolus-conjunction-space-debris
======
Faaak
Original source of information:
[https://twitter.com/lorengrush/status/1168917747109191681?s=...](https://twitter.com/lorengrush/status/1168917747109191681?s=21)

My theory: someone must've turned off their pagerduty and didn't hear the
alarm.

~~~
londons_explore
They silenced it thinking they'd dealt with it, but it was a new alert of the
same type now odds had been updated.

------
cbanek
If you have one mission critical system, all your systems are mission
critical.

It's usually the link you don't expect or don't know about that screws you
over. Like an email server, DNS, networking. At any level, a complicated house
of cards can easily come down.

Thankfully, in rocket engines, simplicity is a design goal.

------
ineedasername
I'm not sure that SpaceX's excuse that it wasn't deliberate really helps their
image here. Deliberately ignoring it, for whatever reason, could be fixed with
a firm "you really can't do that", backed by lawsuits or complaints to the US
government (FCC? Airforce?) etc.

On the other hand, a culture of oversight that fails to adequately test its
emergency response procedures seems a more systemic problem.

~~~
jolmg
The article mentions that at the time they said that, the probability of
collision wasn't that high. So, it looks totally OK to say they didn't have
plans to move it, then, since it means they didn't believe there really was
any risk of collision if they both stayed in their current paths. At that
time, ESA's query was probably to ensure they didn't have any plans for
maneuvers that would raise the probability.

> “They said at that point in time they had no plans,” says Merz. SpaceX
> confirmed that it exchanged the initial email with ESA. At that time, the
> probability of collision was about 1 in 50,000, according to SpaceX, which
> is too low to require any preventive action.

SpaceX might have even interpreted it as ESA asking because they might have
their own plans for maneuvers and they just wanted to make sure that SpaceX's
movements stayed predictable.

It doesn't need to be that SpaceX was being negligent here.

~~~
ineedasername
At the time they said that, yes, probability of impact was low. But then the
probability got much, much, much higher. 10x the probability at which
maneuvers are typically made. And we can't chalk it up to what or how SpaceX
might have interpreted it. We know interpretation was not the problem. They
freely admit that they never saw further messages because of a bug in their
process, so we don't need to imagine their interpretation. The simply never
knew the risk was there. Even if we were going to that extent though, clearly
such an interpretation doesn't comport itself to how these things are resolved
within the community of satellite operators. Which, if SpaceX made a
misinterpretation, also shows a negligent disregard for how emergency
resolution functions here.

I hope their ubiquitous constellation of broadband-providing satellites works
well. I couldn't care less about the negligible impact it might have on visual
observations of the sky-- the society enriching effects of cheap ubiquitous
networks outweigh it. But they have to get basic things like safety protocols
right, not iterating without such things from a minimum viable product.

~~~
jolmg
> They freely admit that they never saw further messaged because of a bug in
> their process, so we don't need to imagine their interpretation.

No, what I said about interpretation was about the first query when it was a 1
in 50,000 chance. For the message they missed, we don't need to imagine their
interpretation because they can't interpret a message they never received.

> Even if we were going to that extent though, clearly such an interpretation
> doesn't comport itself to how these things are resolved within the community
> of satellite operators.

At the level of the community, I don't see a problem here. ESA tried to
contact SpaceX to agree on a course of action; SpaceX couldn't be contacted
because of a bug on their end, so ESA acted on its own to resolve the
situation from its end.

I can see a problem if they plan to be a significant portion of all space
traffic and they can never be relied on to be contactable in case of problems
arising from their satellites. However, this happening once doesn't seem like
negligence.

~~~
ineedasername
Their response to the first query was fine. We agree on that.

My criticism is with them having a system that wasn't tested adequately enough
to realize they would be blind to further messages. This _forced_ ESA to act
on its own when standard protocol, the responsible way to behave, is to work
together. The ESA's ability to act was a luxury not all satellites have.
SpaceX's inadequate testing could have had catastrophic consequences for both
parties. You seem to be using the fact that it all worked this time as
evidence that there was no real problem here. But the fact that one
responsible party avoided disaster doesn't excuse the irresponsible party's
problem with inadequate testing. It would be a bit like having a car drive on
the wrong side of the road, nearly hit another car, but that other car moved
out of the way, so really everything is okay. That doesn't quite parse for me.
I don't see how someone can come away from this without saying that SpaceX
needs to fix something, because SpaceX themselves admit to that.

------
mlindner
One point that's missing here, is this occurred on Labor Day weekend, a
Federal US Holiday. Everyone involved would have been off of work and they
were relying on the pager system. If a bug occurred in that then there would
have been no one to reach.

------
perilunar
Are there 'give way to the right' or 'both alter heading to starboard' type
rules for orbit? You wouldn't want both satellites to alter their orbits in
the same direction.

~~~
mlindner
No. The only thing codified in law is that the country of launch bears the
liability for the satellite. If there were a collision, the liability to ESA
would be due from the US government given that the satellites launched from US
soil.

------
thomersch_
Just a few months ago SpaceX fanboys were telling us those satellites were no
problem, because "space is so large". The problem is, they’re operated by
overworked, delusional people.

People are going to die, because hype is more fun than security.

~~~
nexuist
People are going to die from satellite collisions?

~~~
jolmg
Probably because the created debris would be practically impossible to clean
up later, and if we ever have consumer space travel, that debris might pose a
significant risk.

[https://www.youtube.com/watch?v=heESAW2addo](https://www.youtube.com/watch?v=heESAW2addo)

------
dmitrygr
> SpaceX CEO Elon Musk claims that each Starlink satellite is programmed with
> its own autonomous space debris tracking system, which it will use to move
> out of the way of potential debris if necessary.

Uh huh. Yup, and teslas already fully self drive. Do not believe Musk's claim
that anything he works on does X until you see it do X with your own two eyes!

