
Software error doomed Japanese Hitomi spacecraft - nimbs
http://www.scientificamerican.com/article/software-error-doomed-japanese-hitomi-spacecraft/?print=true
======
dwc
This is the key passage:

    
    
        The spacecraft then automatically switched into a safe
        mode and, at about 4:10 a.m., fired thrusters to try to
        stop the rotation. But because the wrong command had
        been uploaded, the firing caused the spacecraft to
        accelerate further. (The improper command had been
        uploaded to the satellite weeks earlier without proper
        testing; JAXA says that it is investigating what
        happened.)
    

Going into safe mode is a thing. It happens with NASA stuff, ESA stuff,
whatever. The spacecraft failed to stabilize and went into safe mode, and
that's proper. Whatever glitch in the systems, this would have saved it and
allowed for recovery.

But the uploaded command to shed rotational velocity was wrong. This is what
caused the loss of the spacecraft. I'm sure there will be a pretty heavy
postmortem on how this happened.

~~~
ww520
Isn't there a parallel simulator on the ground to try out the commands to see
the effect before hand?

~~~
rtkwe
> The improper command had been uploaded to the satellite weeks earlier
> without proper testing

All the available testing programs in the world doesn't matter if the tests
aren't run.

------
jupiter90000
It seems very strange that the 3 missions that would deploy an X-ray
calorimeter have all failed. I feel sad for the scientists who have been
hoping to get a working one into space for what sounds like over 16 years now
(based on the article).

~~~
ww520
Someone traveled from the future doesn't want those calorimeters up there?

~~~
desdiv
Or aliens with cloaking ships that work in every spectrum except the X-ray
one.

~~~
bitL
Or architects of the Simulation worrying X-ray spectrum could expose some
dirty hacks they are ashamed of.

~~~
serf
or time-traveling alien cloaking simulation architects.

sorry, had to.

------
dewiz
Interesting, I never heard about the south atlantic anomaly.
[https://en.m.wikipedia.org/wiki/South_Atlantic_Anomaly](https://en.m.wikipedia.org/wiki/South_Atlantic_Anomaly)

------
tlb
I'm surprised that they had to design custom inertial stabilization,
considering how many times it's been done successfully before. Was it NIH
mentality? Or did it have some requirement for more precise stabilization than
other space telescopes?

~~~
bayesian_horse
At a guess, I'd say that most satellites have different requirements due to
the way their weight is distributed and where the inertial stabilization is
done.

As far as I understand it, Satellite design is all about cramming the most
amount of features in as small and light a package as possible. That would
mean a lot of tight coupling and a hard time standardizing anything across
different types of satellites.

~~~
JshWright
While that's true for science payloads, when it comes to commercial
satellites, there are a handful of 'busses' that form the core of the vast
majority of satellites.

[https://en.wikipedia.org/wiki/Satellite_bus](https://en.wikipedia.org/wiki/Satellite_bus)

------
microcolonel
This is just depressing. How was this not tested? This is complete sign
reversal of a control output; you'd think it would show up immediately.

~~~
johansch
(I have absolutely no insight into the software development practices of JAXA
and their subcontractors, so I apologize if this is insensitive or uniformed.)

Is this another sign of how bad japanese _hardware_ -oriented companies and
organizations are at doing software? Like the organisational software crisis
at Toyota?

Or was it a fluke?

~~~
microcolonel
I hate to feel that way; but yeah most software I see come out of Japan scares
me. Outside of legally-critical software, such as automotive control systems:
it doesn't seem that there is much focus on quality.

I suspect it has to do with the scale of integrators in the country being much
smaller, and lacking in terms of B2B collaboration.

There are some jewels here and there, though. It's probably just a matter of
culture. Maybe they're also somewhat isolated from the English and Chinese
language leads in software development. I don't think Japanese English
language education is very effective. Their popular courses likely don't leave
an individual comfortable with technical reading in English.

~~~
sitkack
Software isn't valued as much in Japan, and the pay isn't that high.

~~~
johansch
Nintendo seems to care quite a lot (I worked at Opera when the two companies
built the Wii browser together. Wow. They were incredibly brutally
perfectionist, in a painful but somehow admirable way. I've never seen a
company doing so many iterations of some particular detail before that.)

But yeah, I agree, whenever a large Japanese company that is not Nintendo does
software (at least for consumers), you can pretty much expect a disaster.

(I know there's a small group of startups that do good stuff - I am not
talking about them.)

~~~
protomyth
I think there is an interview where Steve Jobs said as much in regards to Sony
and why they didn't own the market before the iPod.

~~~
johansch
Maybe the reason we think of asian hardware companies unable to build good
software is tainted by the fact that one big american hardware company (Apple)
managed to do so (build good software, I mean).

I mean, which other large american hardware-centric companies have managed the
transition to software well?

~~~
protomyth
I don't disagree, but I would imagine most base their view on the titans of
each country. Apple and Sony got a lot easier to compare in the 2000's and
Sony really did have bad software (friggin e-reader). Steve Jobs talked about
Sony a lot in interviews. I would bet if we talk South Korean products,
Samsung would be the archetype. Microsoft for their part makes nice hardware
with pretty good software to back it up.

I haven't used a PlayStation in a long time so I don't know the current state,
and I would guess Nest is not helping the US side of the equation. Apple isn't
exactly covering itself in glory in the last couple of iOS releases (never
mind the "California" series of OS X releases).

~~~
johansch
The PS3 software (based on FreeBSD) had quite nice usability but horrible
system level performance, particularly when it came to disk/filesystems. You'd
end up watching software update install progress bars more than play actual
games.

The PS4 has a weird UI that feels oddly half-finished, but does okay on system
level performance. It's sort of where the Xbox 360 was in 2005.

------
unchocked
It seems like the core error was in the inertial measurement unit: it would be
a common cause between the reaction wheel failures and the failure of the
despin burn.

~~~
sitkack
Seems like the kind of thing one would have multiple of, along with voting, it
should also contain a kalman filter. The software correcting the rotation
should have been run in a tighter feedback loop so that it would stop making
the problem worse.

Lots of these subsystems can be tested in a pure software simulator. Esp when
it comes to faults.

~~~
sitkack
Thinking about it further, the simulator should consume the logs from the
existing system and map those logs from existing simulation runs, use a form
of compressed sensing and pattern matching to figure out what is occurring
just from the logs and previous simulation runs.

------
lerax
Sad.

