

Heroku Postgres Official Maintenance - rdegges
https://status.heroku.com/incidents/510

======
pilif
So. Us normal users will get the security fix on Thursday, but Heroku is
somewhat special and gets early access? What does one have to do to become
special?

I don't like how this is being handled, no matter how serious the issue. I
could kind of understand what the postgres team was doing until this point
where they show that while everybody is equal, some are more equal than others
and get early access to security fixes.

Doing this does nothing but motivate the bad guys even more to find the hole
on their own.

The initial announcement got me scared already, but this just made it worse.

~~~
breser
This is not an uncommon practice at all. There are other projects that provide
early notifications to large vendors. Part of the point of that early notice
is so binary packages can be available. Packages can't get built, tested and
released alongside security announcements without someone getting early
notification. Users are in a much better position if the binary packages are
available when the announcement hits than if the announcement hits and the
binaries are available a week later.

If the issue leaks out early or someone independently finds it they're just in
that much better of a position to release the fix. Vendors will be closer to
having packages ready.

Yes, it's probably not entirely fair. But there's no way to tell everyone
without telling the bad guys too. The risk of telling select people to get the
ball rolling is worth it if the entire ecosystem is ready when the
announcement comes out.

Dealing with these issues is always going to be about managing the risk. You
can't draw a line in the sand.

~~~
tensor
I thought the point of the delay was so that people could not reverse engineer
the patch through the source or binaries? I suppose with Heroku being a
service it might be the case that people don't have access to the new patched
binaries. If that's the case, then there is no issue.

~~~
breser
I can't speak for the specifics of Postgresql or Heroku. But in my experience
the expectation is that you're supposed to control access to the information.
If you produce binaries you're not supposed to make them accessible prior to
the embargo date. If you can upgrade without revealing the binaries I don't
really see a problem with doing so.

------
thruflo
It may be de-rigueur to bash Heroku in the wake of the Rap Genius
drama/revelations but this kind of "downtime to apply critical security patch"
announcement is precisely why I choose them.

I spotted the announcement of the postgres vulnerability and thought "sounds
serious". Now I read that my production databases will be fixed as soon as the
patch is available without me having to engage with the details.

------
ryan_f
This is rather disconcerting. They are given us a window of 48 hours where our
site will be down for 60 seconds. On top of that we cannot schedule the time
with them.

60 seconds may seem like a small inconvenience but it is one still the same. I
host a few clients on their servers and I need to relay this information. I
sound like a jackass being that I can't give more specifics on the timing of
the server being down to my clients. How does Heroku not feel the same?

I am appreciative of the update but that is also what I pay them to do.

~~~
fein
> How does Heroku not feel the same?

Because they, as a hosting provider, have several orders of magnitude more
clients to please than yourself. The only way to be fair to everyone is that
everyone gets treated like shit (obviously embellished).

All you should be doing to CYA is telling your clients exactly why this is
happening and that its Heroku, and not you, that holds the blame at this
point.

The caveat would be those difficult customers (suits, exec managers, etc) that
you have to handle with kid gloves. For these people... I don't think there's
a graceful way to present the upcoming issue.

~~~
gingerlime
I doubt customers typically know or care whether you use heroku, Amazon or any
other provider. What they want is to see their system up and running.

That said, I wonder if with the heroku postgresql setup, this vulnerability
could mean that if even one db is compromised, it could allow access to other
databases too. Maybe heroku runs several postgresql db's on the same
virtual/physical hosts?. If that's the case, then heroku simply can't afford
to let even one database stay exposed because it would risk all others.

Whether or not this is the case, I don't know, but it kinda feels like running
your app on a shared-hosting account...

~~~
lambda
> Whether or not this is the case, I don't know, but it kinda feels like
> running your app on a shared-hosting account...

Remember that you _are_ running your app on a shared hosting account with
Heroku.

Now, it has somewhat more isolation than a typical shared hosting account, but
less isolation than using physically separate hardware.

Heroku is built on top of AWS, so your machine is running on the same physical
hardware, though a different virtual machine, as other AWS customers.
Furthermore, Heroku uses LXC to isolate its dynos; so you are running on the
same VM as another Heroku customers, albeit separated by a container barrier.
And finally, if you're using Postgres, then you're running on a shared
database service, which is pretty much exactly like what you'd get with shared
hosting.

------
gingerlime
I was just recently considering whether we should move our postgresql over to
Heroku.

On one hand, it sounds promising that they deal with important security
updates, and can do it even before the fix is officially announced. Very
impressive.

On the other hand, not being able to schedule the downtime to fit with your
app/userbase is quite annoying. I understand this could be a potentially big
security vulnerability that needs urgent fixing. But each organization is
different and their risks are therefore different. Some people might prefer to
take the risk of a few hours delay to apply the fix (being exposed) in
exchange for not having to experience unscheduled downtime of this nature...
It seems a little awkward to me that Heroku is taking this decision for its
entire customer-base in such a way.

------
neya
I'm yet to see an update/fix on Heroku's recent PR nightmare on the random
routing issue. Anyone?

~~~
joevandyk
They are about to push an update that lets you run larger dynos, which reduces
the risk of having all the requests to a dyno stuck doing something long-
lived.

~~~
redguava
I believe the larger dynos are mostly to better handle using Unicorn as the
webserver. Unicorn allows multiple workers to run per dyno, and has it's own
smart routing, so it compensates well for Heroku's routing system. The
downside to Unicorn was that the workers share a dyno's memory and often you
couldn't have more than 2 per dyno with 512mb memory. This change helps that a
lot.

------
falcolas
A good reason to take a few minutes and make sure your backups are up-to-date
and working, and that your application won't blow up in unexpected ways when
the DB is unreachable.

------
nevinera
>Due to the nature of this update, a scheduled time is not possible.

What, it takes place outside of linear time?

This crap is unacceptable. Even my $2.00 shared host tells me _when_ they're
going to take down my site without asking.

~~~
viraptor
More likely it takes around 5min per host if everything goes right. If they
have 100 hosts and get delayed on the 10th and 15th for some reason, all other
scheduled times would get shifted... And the upside is that they spot failures
as they appear, rather than "oops, the last 5 hosts we did at the same time do
not start up, there's some big issue".

</speculation>

------
JuDue
Just as an aside, how good is Heroku's Postgres performance, speed wise?

~~~
guywithabike
Don't ask for anecdotes -- test it yourself with conditions similar to those
that you'd be operating under.

~~~
JuDue
Hey, reviews and anecdotes save time. Everyone does it.

