Hacker News new | past | comments | ask | show | jobs | submit login
Hubble is back (nasa.gov)
768 points by perlgeek on Oct 27, 2018 | hide | past | favorite | 117 comments


They were able to recover the failed backup gyroscope by executing a series of attitude maneuvers while switching between operational modes on the gyro.

They literally shook the spacecraft and turned the gyro off-and-on.

Sometimes you gotta bang on something to get it to work!

I would love to buy a beer for the mission operations team member who came up with that idea!

https://en.wiktionary.org/wiki/percussive_maintenance ftw

edit: I have to add, i just found out about the extent of the definition.

>make a malfunctioning device or person work.

Packing a punch is not only useful to fix things, but often the best way to get you an answer. Or as my control theory professor used to say:

If you want to know how an unknown system reacts, first thing you do is to hit it hard.

What he meant was that applying the Heaviside function to the inputs of a system to determine the step response [1] is one of the first things we should do.

[1] https://en.wikipedia.org/wiki/Step_response

I remember that thinking from control theory classes, but I thought you use a Dirac delta as a hard shove.


(Actually, our control classes told us to do both to get a basic view of a system.)

There is a German proverb about percussive human maintenance, literally translated to: slight hits on the back of the head increase the ability to think.

a hammer is my go to tool for anything thats stuck. a good whack usually get things going ;-)

I wish I could do that to my software sometimes. Just bang my computer to get it working.

In that case, you may enjoy https://github.com/ajalt/fuckitpy

Or its siblings, including https://github.com/mattdiamond/fuckitjs

It is cathartic, even if it doesn't work.

You can - in Javascript for example:

setTimeout(function() { ... setTimeout(function() { ... }, Math.random() 1000); }, Math.random() 1000);

// winning!

Proof for P==NP is somewhere 30 layers deep in the bowels of a low-traffic javascript-rendered site.

They are the real heroes! This is such an amazing example of getting a job done.

Makes me wonder how long the fix will last, though.

Maybe they just grew up with Windows 95 where switching off and on solved 99% of issues.

Reading about the specs and the planned degradation of the gyros is a very interesting exercise in planned reduction of a service.

They know the gyros will eventually give out and they have contingency plans around what kinds of science they can continue to do even when they're down to 1 gyro.

Very impressive. I'm always discouraged hearing people say NASA is wasting money and science and that's just not true IMHO.

While people may think that this is wasting money, I see that this these teams gain unieque knowledge and experience on how to resolve issues on "machines far far away", and if we are ever to explore/move humans to a new planet, you want THOSE exact teams as 'helpdesk' that use anything and everything to bring things back to life and operation.

This is assuming generational knowledge can be sufficently passed down to be useful in future missions. I'd assume this is quite a niche/specialized skillset they developed for this particular machine. The utility of the knowledge has a limited timeframe which may not justify education of the next generation of scientists.

But regardless recording it would likely be helpful as most problems in engineerings seem to recur over time in different and new contexts.

We build on the knowledge of those before us. The people who invented the original programming languages are mostly gone and most of us probably don't know how to use the stuff created but future works were based off the findings of those before them and we all benefit from that.

Hubble has been around enough now that the next generation of stellar and space engineers are getting though university and into NASA and other space ventures.

Kids need to see the previous generation has set extraordinary goals to inspire them to set their own goals. What could be more inspiring than a middle school student seeing Hubble launch, being inspired to study and then after college get to participate in what inspired them in the first place.

I think its reasonable to assume AI will be able to fix these kinds of problems in the future. They could run simulations of all the possible failures and solutions ahead of time or during operation eventually

I think the creativity and imagination required to solve this kind of problems is not really within current AI reach yet. The ability to take existing ideas and combine them in unique ways to come up with something new is very powerful, and not yet easily replicable by AI.

Genetic Algorithms are pretty creative. I think a bigger problem is that we have no general way to turn “what we want” into a mathematically formalised goal that an AI can work towards — we can only manage specific cases like “how many components does this signal generator circuit use” or “make a new face based on these examples of faces”.

You can probably train them using dwim

The search space is so huge and the amount of training data we have is so little that AI, as we currently understand will be useless for this kind of stuff.

Wow lots of AI naysayers here. What kind of person downvotes this kind of comment? Reveal thyself!

You can achieve impressive results and still be "wasting money" if there were more cost efficient ways to achieve an equivalent result. I don't have an opinion on NASA but that could be the point of view those people are coming from.

The problem with having non-science people run an organisation of science people is self explanatory.

In the Netherlands they have outsourced the organisation and managment of health care to the doctors themselves (roughly speaking). It is often cited as one of the main reasons the Netherlands rank #1 in health care ratings over the years.

In the US, the American Medical Association is run by doctors and they artificially limit the supply new doctors by capping the number of medical schools. So sometimes having the fox guard the henhouse is not the best way to save money.

The AMA doesn't directly govern the number of medical schools in this country. That would be the LCME (Liaison Committee on Medical Education), which is sponsored by the AMA and American Association for Medical Colleges.

It's also important to note that the number of medical schools isn't really artificially limited by the LCME, as they will accredit most any medical school that meets the accreditation standards.

On top of that, medical students in the US have to undergo a residency before becoming fully licensed physicians. These residencies are accredited by the Accreditation Council for Graduate Medical Education (ACGME), a separate organization.

However, while the ACGME can accredit these programs, the majority of funding for training resident physicians comes from the federal government (department of health and human services, center for medicare and medicaid services), and is capped by the Balanced Budget Act of 1997 (modified in 1999 to accommodate the rural physician shortage). Today, this costs the federal government around $11 billion each year for all of the residents in the country.

The real question, then, is who we can point to as the bottleneck for new doctors in this country. I found this article enlightening, but it may be behind a paywall:


The AMA controls enough seats on that board to prevent new schools/seats from being created.

And the idea that an industry as profitable as medicine needs the taxpayer to fund its training is preposterous. This is just a deflection from the AMA's role.

a profession as wealthy as medicine needs the taxpayer to fund its training is preposterous

Most doctors already come from wealthy families (specifically, they have doctor parents) because in addition to the tuition of medical school, there are a variety of other expenses (board examination fees, the undergraduate requirement, interview expenses for both medical school and residency) that are difficult to fund without assistance. Also, compensation for physicians is not what you seem to think it is outside of subspecialized fields, and the expense of medical school (and the opportunity cost of residency) means that physician shortages in the general disciplines will only continue to get worse as medical students realize the only way they can pay back their loans is to subspecialize.

Also, medical schools are primarily limited by number of faculty, and it's difficult to find doctors for med school faculty roles... because they're so poorly compensated in comparison to other opportunities.

If you want more doctors, first provide incentives for doctors to teach, then provide funding for those who normally don't see medical school as an available opportunity, then incentivize people to go into general disciplines, then properly compensate residents so they're not throwing away ten years of potential income, then complain about how "doctors are overpaid but there's also a big shortage of them".

The rest of us have to compete in a free market for our skills. Why not doctors?

I agree that the undergrad requirement is absurd. But it is also just another part of the system that increases entry-barriers and leads to all of these problems, but makes the field more lucrative for a big segment of those who get through it.

Since I can't respond to my own child comment as it was flagged, I'll just note that when I made it, this comment said only:

   The rest of us have to compete in a free market for our skills. Why not doctors?
And the rest was added later.


You might find this USA Today article relevant and interesting. It's old, from 2005, but crimping the pipeline of doctors coming through the system has impacts that last decades after.


It doesn't matter how many medical schools or students exist, if there aren't enough residency positions for them. And the AMA does not control the number of residency positions.

Podiatry in the US has this problem in a big way.

Something similar happened in the UK with GPs, and it's been one of the least popular health reforms ever to take place.

(England, not UK)

Not sure why you got downvoted. The Lansley Reforms - where Primary Care Trusts were abolished and replaced by Clinical Commissioning Groups which are run by GPs and are responsible for commissioning most of the NHS services (about 2/3 of the budget) - are known to have caused severe disruption to English NHS services.

Netherlands, a high density country without the added costs of delivering healthcare to remote places, spends 30% more per head on healthcare than the UK.

That has a far higher outcome on healthcare than taking doctors away from patients to run hospitals.

Huh, coincidentally I just heard a couple hospitals in Netherlands are closing on short notice due to bankruptcy[0]... I'm sure there's a lot more backstory there, but it doesn't sound like #1-tier healthcare when that's happening. hehe ;)

[0] https://www.dutchnews.nl/news/2018/10/slotervaart-hospital-s...

Yep. I love space and science, but after working with NASA for a while, you see some of the problems. NIH syndrome is a huge problem, especially between GSFC and JPL. There are incompetent engineers who will have a well paying job at NASA for life. And then there the passionate, brilliant scientists and engineers the that dream and make it all work. A reduction in force of dead weight would go a long way but it will never happen.

From what I hear the original "startup" NASA from the 60s was a no deadweight organization that efficiently cranked out world class space flight.

That it since followed the path of every other monopoly bureaucracy was of course unavoidable.

> ...it since followed the path of every other monopoly bureaucracy...

This is not really true, but you are taking it as axiomatic.

It's a heck of a lot more true than false. Of coarse if you use a roller you need to cut in with a fine brush.

To better appreciate what NASA does, I think it helps to look at its achievements in a different way. Instead of (just) marveling at the big rockets or the rovers on other planets, consider the amount of paperwork that was needed to do these things. Someone had to get approval to move a large amount of high explosive across several state lines, obtain permission to detonate a large amount of this explosive material in a wildlife refuge, coordinate with foreign military agencies to ensure that "yeah that thing you see on your radar going on ballistic trajectory toward your homeland TOTALLY isn't WW3", and mate with a space station partially constructed by a foreign power with which you have sanctions that limit your ability to exchange technical information.

Even though SpaceX has to do some of this, they get to reuse quite a bit of NASA paperwork.

Interestingly the Indian Space Research Organisation seems not to have followed this path.

A professor of mine who used to work at ISRO says that the place is choke full of bureaucracy.

Could tweet Trump and ask him to remove the deadweight.

i think that's true generally of any organization.

pareto -- 80% of work done by 20% of workers.

there's also some other law about the work being done by the square root of the total workers.

I agree the 80/20 is true of most organizations. But producing nothing and keep your job for life as long as you show up for work is not common in private industry. On the private side, I see developers get let go all the time for inadequate work.

This is an oft repeated trope, and it couldn’t be further from the truth.

I’ve personally worked at large private firms where incompetent folks coasted for decades.

When management’s clout is measured in headcount, you get this (whether in the private or public sector)

Wasting money criticism with regard to SLS is valid, I think, but is more of a political reality than a NASA issue.

It's always a lead into cutting NASA spending so remind them that it's not a zero sum thing, cutting NASA's budget isn't going to cause lower taxes, higher spending on $THINGS_SHE_PREFERS, or a fatter personal wallet, it'll just be cutting NASA's budget.

Why wouldn't it mean those things? That's the essential nature of budgeting.

The money will probably be directed to another department.

That's one of the options in the parent post. My point is that the budgeting process is in fact mostly zero-sum. It's only not totally zero-sum because of the use of debt (which considered by itself is zero-sum over the long run) and the possible effect of particular spending on the broader economy (which makes it not perfectly zero-sum as a repeated process)

Tax changes and budgets are passed separately

From this link...


Each year astronomers from dozens of countries vie for precious minutes of Hubble's unrivaled view of the cosmos.

A review committee made up of experts from the astronomical community determines which proposed observations address pressing scientific questions and make the best use of the telescope's capabilities.

Each year more than 1,000 proposals are reviewed and approximately 200 are selected, which represents roughly 20,000 individual observations.

I feel bad for all the projects that got delayed. I wonder if they'll be rescheduled, or have to be resubmitted or are just out of luck.

Some of them might not actually need Hubble specifically. There are a fair amount of observation platforms, maybe they can get their work done on another satellite.

Perhaps, but given the competition for Hubble time, if you can get the data you need from another observatory you're generally better off doing so. For these competitive observatories you usually need to explain why your project actually requires those facilities.

To some degree yes, but most of the approved observation plans have already been sifted through the "can they get this done without Hubble?" filter.

I will never cease to be impressed by the sheer skill and quality of design at NASA, that allows them to keep missions like Hubble or Opportunity operating far beyond their planned lifetimes.

It amazes me that they are still in contact with the voyager probes. NASA does some amazing stuff.

Amazing. It's like they performed the Epley Maneuver on a satellite.


That reads worse that it actually looks like: https://www.youtube.com/watch?v=ZqokxZRbJfw

Amazing. HN community disagrees on almost all topics, except something like this. Best news in the past few weeks. Kudos NASA.

Didn't the NSA gift NASA two hubble class telescopes with a note along the lines 'bought a ton on bulk discount and never ending up using these two before the new model was released'?

Not really. They gave NASA two "telescopes" that have mirrors and a chassis and basically not much else. It will still require billions of dollars and years of work to turn them into space based observatories. One of them is actually being built (WFIRST) but isn't expected to be launched until the 2020s.

Also, overall there isn't a huge advantage to this hardware. It doesn't save much money as most of the cost of the telescopes is in the other components: the spacecraft parts (including the attitude control system which as we've seen with Hubble is quite important) and the instruments. The biggest advantage to the NRO gifts (the NSA doesn't have telescopes, that we know of) is not that they make building space telescopes cheaper but that they mainly just make those missions more likely to be funded.

Yes, but it was the NRO (I doubt the NSA owns any telescopes).

NASA has plans to maybe use one of them sometime, but because of how expensive the James Webb Telescope turned out to be doesn't have money to do anything with the second.

The wikipedia article gives some more context: https://en.wikipedia.org/wiki/2012_National_Reconnaissance_O...

NASA has gotten out of quite a few pickles by using Earthside clones/spares of spacecraft to troubleshoot issues with the ones actually in space. Even if NASA had the money to launch the spare telescope, they'd be wise not to.

The NRO (national reconnaissance/spysat office), back in 2012. They also gave them the mirror and other components for a third. They were hand-me-downs, presumably obsoleted. Things like that always make me wonder what the current state of the art is, let alone when you look at the things like the SR-71 Blackbird, first launched in 1964 -- 54 years ago!! On the other hand then you look at things like the F-35, SLS, Zumwalt, etc today and it somehow almost seems that we've regressed technologically.

Wasn't hubble itself a refit of a Keyhole 11 beta?

To my knowledge, no. It’s believed the two satellites NRO gave to NASA were KH-11 though.

They're past KH-11. KH-11 has the solar panels on side like Hubble. WRIST has the solar panel on the top which blocks solar radiation which is a newer design.

Interesting read.

On the other hand, I remembered that hubble was manually repaired by astronauts mutliple times back in the space shuttle era. I'm wondering that, do we have capabilities to perform in-space non-ISS-kind repairs now? Soyuz may be able to do in-space rendezvous but cargo space is limited. Soyuz+Progress composition (may be too complicated)?

No we no longer have this capability. Soyuz capsules are not really designed for EVA.

You can do EVA with Soyuz https://en.wikipedia.org/wiki/Soyuz_5#EVA_details

However that mission will unlikely be possible for both technical and political reasons.

Servicing HST with a Soyuz is basically impossible. Hubble is in a higher altitude orbit in a significantly lower inclination than the launch site for crewed Soyuz vehicles. To perform the plane change maneuver (of 23 deg.) would require a delta-V of 1.5 km/s (20% of the orbital velocity at the typical Soyuz destination altitude). That would require bringing nearly 4 tonnes of extra propellant onboard the Soyuz, which isn't feasible. Additionally, this doesn't include the added weight of the docking adapter and Hubble replacement parts that would need to be brought along.

It's conceivable that a repair mission might be possible with the soon to come into service US crew capsules (Dragon 2 and Starliner) since the rockets that launch them have more performance than the Soyuz, but even then it would be a pretty ambitious mission.

Yea I know you can do EVA from the Soyuz, but it's not really designed for the types of extended EVA missions required to repair the Hubble.

For those interested in learning the spectacular engineering that went into the Hubble: http://a.co/d/4kLahfW

That's definitely not what I expected to find at the link! Thanks for posting it. My father was an engineer on the Hubble, Hexagon and other reconnaissance satellites, and I'm sure he'll get a kick out of this "manual" as a gift.

For those not wanting to click on a shortened url, the long, non referral version is https://www.amazon.com/NASA-Hubble-Space-Telescope-collabora...

"so we just gave it a shake et voilà!"

- some NASA scientists, probably

“Last week the operations team commanded Hubble to perform numerous maneuvers, or turns, and switched the gyro between different operational modes, which successfully cleared what was believed to be blockage between components inside the gyro”

And here I am on Earth, barely capable of debugging JavaScript code.

It's also fun to compare programming to surgery, where they work for hours straight with no backup on a production system that is actively trying to die.

It has always amazed how hard it must be to debug an issue in a human being, you need to check A, B, and C, but if you check C your human will be scared since it could indicate your human has cancer, if you check B it will be expensive since it involves a lot of studies, you can also check with A giving some medicament but it also is expensive and maybe other people needed more. So you end up testing all three, but none gave you any feedback, the issue must be somewhere else, those tests took months and it doesn’t help you had been testing hundreds of other humans and for each case you must understand the whole context, so you have to start again, meanwhile your human starts to think you are not capable since you haven’t fixed his problem so he starts looking for a “second opinion”.

Perfect explanation for why for House, M.D. was so entertaining.


But also surgeons make misstakes. Lots of them. Or don't find the perfect solution in time, needed to save someones live.

This is exactly why I switched from medicine back to programming.

The universe is transactional.

and doesn't have a rollback to safe point.

You need to compile with --multiverse

Henry S F Cooper's book _The Evening Star_ describes similar solar-system-scale distance debugging of a race condition on the Magellan Venus orbiter. Pretty tense stuff; the stakes are high, the tools and insight to the actual problems are poor, and if you screw up badly enough you wind up with a dead spacecraft.

> And here I am on Earth, barely capable of debugging JavaScript code.

If as much education was required of software engineers as it is from people who work on the Hubble telescope then you wouldn't have any troubles debugging JavaScript.

If JavaScript was specified to the same degree as the Hubble telescope then the OP wouldn't have had a problem. Blaming gdubs's education is a bit problematic.

The ECMAScript spec is actually quite thorough and freely available to read online. In addition, it has a very large suite of tests to check for compliance.

I don't think there's any other language with as many independent production-quality implementations.

In what way do you think it could be better specified?

"I don't think there's any other language with as many independent production-quality implementations." - C++?

Yeah, it looks like you might be right. I just looked it up and there's a ton of C++ compilers I'd never heard of before.

Sure, but the number might fool you. Not every C++ compiler is standards-compliant, and at least on the PC side, many of the good compilers bought the frontend from the Edison Design Group and focussed on the backend.

Anywhere the spec says "implementation-dependent" or "behavior is undefined" could be fully specified. Some might say its impractical, but given the effort folks but into the hardware and software of a NASA project, you can really see the difference.

I would be willing to bet that there are many engineers on the Hubble project that have and struggled debugging java script themselves.

Nothing to do with education; it’s all about operating procedures. Hubble has a large and well funded team behind it that generally operates on much longer timelines. Everything is engineered and documented to a tee. Hubble mission spans around 30 years now, from concept and until today; very possible that for some people, Hubble is the only thing they experienced in their profession.

Obviously such measures would be very much impractical outside of NASA.

I shake my phone vigorously if the small motor gets stuck and stops buzzing.

And it works, doesn't it?

Yes, it has diagnostic software which allows you to run the motor constantly. I switch it on and shake it about until it gets unstuck, run it for a few more seconds, and it's good for another three months or so.

They've probably done a "bang it with a hammer" equivalent.

with your javascript you can cmd+r over and over again until you figure it out. with the hubble they had to simulate and hypothesize/imagine possible solutions. It's the same process just a different medium for solving problems and they generally get only 1 chance to press cmd+r

I love that they basically mashed all the buttons till it started working properly :)

Yay! Cheers to Hubble!

Because NASA is awesome.

Well, they are. Yeah, I am a fan. NASA is one of the few all good things from government.

Government can be better, and I hope we can make it better too.

What else to say? I love a good NASA win, geek, nerd, engineering story.

Url changed from https://twitter.com/NASAHubble/status/1056189182274625537, which points to this.


Hubble is back but nasa.gov is black. A blank black page. All because their web devs wanted (or were pushed into using) some crap front-end lib that has no fall-back for no javascript.

nasa.gov's accessibility is absolutely terrible for a government website.

So enable javascript?

Having the embedded software in the sky working well is a bit more mission-critical than having the website work without javascript.

How is requiring javascript an accessibility problem?

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact