Hacker News new | past | comments | ask | show | jobs | submit login
Heroku resetting passwords for “a portion” of their users immediately
340 points by vorador on May 4, 2022 | hide | past | favorite | 169 comments
Just received this email from Heroku. Given the timeline it seems like the breach is pretty serious.

"As part of our efforts to enhance our security and in response to an incident published on status.heroku.com, we wanted to inform you that we will begin resetting user account passwords on May 4, 2022. We recommend that you reset your user account password in advance here and follow the best practices below:

Minimum of 16 characters Minimum complexity of 3 out of 4: Uppercase, Lowercase, Numeric, Symbol Don't just add a letter or a 1 digit number to the existing password while changing Passwords may not be duplicated across accounts If you do not reset your password and your user account password is reset by Heroku on May 4, 2022, your existing password will no longer work. To log in to Heroku, you must reset your password by accessing the Heroku login page and clicking the "Forgot your password?" link . Please be aware that you may be required to reset your passwords again in the future. "

This is turning into a complete train wreck and a case study on how not to communicate with your customers.

For those of you that haven't been following, Heroku has been adding non-update updates to this security thread over the last couple of weeks, which began with the announcement that some (or maybe all) of their GitHub granted access tokens had been compromised: https://status.heroku.com/incidents/2413

Now, weeks later, we're hearing that all account passwords are being reset, and for some reason if you have been using an HTTPS-style log drain that you should reset any secrets related to it as well.

Heroku needs to come out and clearly state what they know about this situation, and more importantly what they don't know -- which is starting to sound like the answer is "a lot". It's not even clear they know how this all happened -- whatever door was left open might still be open. So if you've gone and rotated all of your application secrets (which you probably should do), be prepared to rotate them again when this is all over.

It's a small comms 101 thing, but the email is from "Salesforce Incident Alerts". Since the email's communicating to Heroku customers about a Heroku incident, the email should be from "Heroku Incident Alerts".

I know it's small, but some will skip the email because they don't use Salesforce software directly and wouldn't anticipate emails from a parent company.

Its small and deliberate. Setting the mental state of the customer to obfuscate the responsible party by throwing in Salesforce. A deliberate dark pattern.

Another small point that contributes to the poor communication - as of 4th May, the option to connect to Github is still there and the documentation hasn't been updated [1]. If you try to connect you get an obscure error that tells you nothing about the situation.

[1] https://devcenter.heroku.com/articles/github-integration

Yep, I would have 100% skimmed past that.

I just received an email back from an Incident Handler at Salesforce. I wrote:

> A statement that confirms whether or not config variables and secrets were accessed, or that you're not sure, needs to be sent out.

To which they replied:

> We currently have no evidence that Heroku customers’ secrets stored in config Var were accessed. If we find any evidence of unauthorized access to customer secrets, we will notify affected customers without undue delay.

Take that as you will, but it doesn't fill me with confidence.


> Law of No Evidence: Any claim that there is “no evidence” of something is evidence of bullshit.

What else can you say?

It's impossible to know if a vulnerability was exploited.

"We don't have enough information to determine whether or not this vulnerability was exploited. We are operating under the assumption that is has been." is what I want to hear. I do not want to hear "We have no evidence that the vulnerability has been exploited." which, of course, minimises the fact that it may have been and does nothing to communicate what assumption they're working under - i.e. that they're probably going to assume it hasn't been exploited.

TL;DR: I'd rather them be entirely up front about the fact that they can't tell if it has been exploited and advise you to assume it has been than them try to weasel out of saying their logs aren't good enough but "you'll probably be alright, eh".

> We are operating under the assumption that is has been.

This gets expensive quick.

Probably the reason people try to avoid security incidents

In this case it's already happened, time to spill the bag

"If you think safety is expensive, try an accident!" - Stelios Haji-Ioannou

Which really means that if you discover a vulnerability in your system, you assume that it was fully exploited.

Reality: You can’t prove a negative.

Why? I think it's pretty evident that there's no reason to believe there's been a security breach there as far as they understand as of now.

There was certainly a breach three weeks ago that they seem to have been investigating since. I am, like the commenter above, not filled with confidence about their statement, mostly because of the total lack of transparency so far.

The fact they’re only now sending additional notifications to rotate creds hints at something bigger than they initially announced, but really we have no idea since they never gave much detail in the first place.

There was a security breach. They don't know what the breach was, they don't know if the attacker has access to other systems, they just don't know.

No evidence of something is not particularly useful information if you weren't even looking out for it in the first place.

Github tokens were accessed and used to attack things outside of heroku. Something was breached. We don't know what, when it occurred and extent of the breach and I bet heroku doesn't know either.

So the original issue was described as a leak of github oauth tokens, and made it sound like the risk would be someone using oauth tokens to access github repos.

Resetting passwords implies something else may have been compromised (passwords, either hopefully encrypted), but is a pretty scary ask for them to make without providing more context here.

Trainwreck indeed.

I certainly hope that passwords aren't encrypted but run through an appropriately-expensive password hash.

Yes, I should have said hashed not encrypted.

hashing is one-way encryption tho

That's the point, take a look into salting + hashing passwords

Resetting the password resets your API key, which is different from the Oauth tokens.

I found the email I received about the logdrain today to be particularly confusing. "any secrets related to it" indeeed. The specific wording in the email was... "We recommend updating and refreshing the credentials used with those log drains as soon as possible."

I'm still not entirely sure if I've reset/rotated everything I need to, what is "any credentials used with"? Neither the email nor any docs it linked to was clear about exactly what they are suggesting be rotated.

That message wasn't very specific, while also they're not providing the context about the breach that one could use to fill in the gaps.

At this point it kinds of sounds like... everything there is was compromised?

Very confusing! To clarify for others, what they mean is that if there was a secret embedded in the log drain URL, rotate it. This is often the case for HTTPS log drains.

Example: https://datadog.com/logs?api_key=abc123

>If you used your previous password on any other sites, we highly recommend you also change your password on those sites.

This is the most concerning part of that email, as it implies more than an "out of an abundance of caution", but rather that they suspect their password DB has been compromised.

Thinking about it, it does sound the most likely as they were probably the same DB the customer oAuth tokens were stored in that were used to access Github repositories. But if they already knew the data was stored together why wait till now to reset passwords?

Just to verify - having TOTP-based 2FA enabled doesn't help in case of a password DB breach, right? Since the protocol is based on a shared password, which means an attacker would be able to generate valid tokens using the secret they got from the breach. (looks like there's work underway to make a breach-resistant alternative to TOTP[1])

This means that assuming the DB is using proper salt+hash, the main differentiator is the strength of your password. If it's a relatively short one that can be brute-forced/found via dictionary+small mutation, then attackers could possibly log in as you. If it's a strong password from a password manager, then that will likely have kept them from being able to crack your password.

Of course all this only has value if we assume that only the password db was breached. If they managed to access the place your env-var/secrets are stored, then all bets are off.

[1] https://www.mdpi.com/1424-8220/20/20/5735/htm

I don't understand enough about the protocol to judge your claim, but it's challenging my assumptions about what 2FA is for. If 2FA with TOTP does not protect you in cases where the attacker knows your password... what is it for? I thought that's what it was for.

The server also contains the secret, so if that secret is leaked then the attacker can generate new tokens. It protects you in the case that your password was stolen, but nothing else e.g. via phishing.

OH, I see, thanks. The password and TOTP secret are separate, but you're suggesting they may likely both be stored in the same place such that a breach could give attacker access to both. Tell me if I don't have it right.

It occurs to me that I know how to reset my password most places I log in to, but I actually have no idea how to reset the TOTP secret.

It might be stored separately, the issue is just with an uncontained breach I suppose.

Heroku used to be freaking awesome, back in 2012. Ever since Craig Kersteins left circa 2015-ish, the UX, QoS, and platform really seem to have taken a dive into the complexity and nonsensical deep end.

It's what happens when the product visionaries get bored and leave. Such a shame.

Don't want to comment too much on this thread as I don't know the exact details of the incident and pulling for the team that is still around as I know this can't be an easy time for them. It does seem like the issue is not good and may continue to be trickle of updates like this for a while. Hugops to the team.

A wow, and thanks for the praise but the credit goes to way more than me. The team around in the early days was unbelievable, I learned a ton on building products and developer experience from James/Adam (founders) in particular, though Heroku wouldn't have gotten there without Orion (the other founder) as well. Byron, PvH, Mark, Noah, Morten, Oren were absolutely huge in so many ways to the leadership and direction of Heroku. And I'm sure I'm going to get messages from 50 others there in the early days that I didn't name drop them, the collective team was an awesome team and pushed each other really well.

At around 2015 it did feel like there was attrition and the technical leadership and vision started to fade. It wasn't me, it was a lot of us moved on to the next thing. At the time it wasn't Salesforce taking control or one person, we'd all put a lot into it and various folks moved on. Adam/Orion/James gave an incredibly amount and were understandably ready to recharge. Still very proud of what we created at that time, what it did for developer experience, and personally (along with that original Heroku Postgres team) trying to do what I describe as unfinished business for creating the amazing developer experience of Postgres.

Wasn't trying to imply it was all you, only that some of the qualitative declines I've observed began around the time you departed.

I know it's not an easy business to operate, and I'm also hoping the ship continues to stay afloat. Heroku was hugely inspiring to me and many others in the early days of Cloud PaaS.


You're being downvoted because having a "product visionary" does not in any way protect your company from being owned. It happens. If your point is that previous leadership would have communicated the situation more clearly, then perhaps you have a point, but even then it's purely speculation and not particularly useful to the current discussion.

It's okay, thanks for the feedback.

Downvotes don't bother me, they are a welcome signal.

in this context "pwned" is probably a better fit; natch?

They sent out a notification about cycling shared dataclip slugs today too.

Is papertrail a log drain?

Yes it is. Note that OP's comment only applies to those who connected Papertrail to their Heroku apps manually instead of using the Heroku addon to do it.


How do you know that? And that it doesn’t need to also be rotated?

It does. My thinking is that Heroku rotated everything they could do on their own, and need users to handle custom drains. However, looking at the json output, updated_at dates for the heroku-managed drains are old. I have a question back them about that.

Related, my hope is that the drains, passwords, etc. are efforts towards doing literally everything possible before re-enabling integrations so they can do so with more confidence (vs breach damage control). Fingers crossed.

Fwiw, I skimmed over all emails I got until I saw this thread.

You and me both, buddy

This has gone from bad to worse very quickly.

The initial issue was supposedly a breach of customer repositories. Which sounds bad, but if you’re not storing credentials in code, then the worst case is that a potential hacker had access to download your code. Not great, but certainly not as catastrophic as some breaches have been.

Since then, Heroku has been acting beyond strange. Everyday they update the incident with essentially the same non-update, but written differently, with vague references to the same information they’ve sent 14 days in a row.

Now, they send another very ominous and strange update about “some” customers having their passwords reset. However, based on this thread and my own experience, it seems like every customer is getting this message.

What does this have to do with the initial issue? Were actual Heroku accounts compromised?

This behavior is either extreme incompetence wrt customer communication, or they’re preparing to announce a truly insane breach that may include everyone that has ever used Heroku.

They need to get their shit together and quickly.

> Since then, Heroku has been acting beyond strange. Everyday they update the incident with essentially the same non-update, but written differently, with vague references to the same information they’ve sent 14 days in a row.

There’s a simple reason for this: their incident response policy requires they not leave customers un-updated more than x hours/days. Their tooling likely reminds someone if it isn’t done.

+1. If you need another data point, the entirety of my Heroku presence is a single free plan toy app that has literally been untouched for many years at this point, and I got the password reset notification.

Just FWIW... I have an account I created in 2013 (based on a registration email.) It appears in my 1Password, imported from LastPass, so I probably logged in around/after 2016. No later than Nov 2018.

I don't have the password reset email, and logging in just asks me to add MFA, and then to accept 2020 terms.

For another data point, I got an email yesterday from "Salesforce" about a Heroku dataclip. I had recently deleted it, so maybe the email was queued up a couple days before. The email says, in part, "We recommend that you update and refresh the unique URL used with dataclips using Shareable Links as soon as possible."

The password reset mail is yet another train wreck.

I received this email. The reset password link in it is NOT https. If I manually change the http to https it turns out that the server, click.msg.salesforce.com, is returning a certificate that is only valid for click.s10.exacttarget.com

I think all this link does is redirects to the 'Reset password' page (i.e. https://id.heroku.com/account/password/reset), with email address pre-populated, then shows a button to email the user a separate reset password link.

So I think the lack of 's' in http in the original link doesn't matter.

If I'm wrong, could you please explain why (I am reasoning as best I can, and am not an expert, and keen to learn).

You are right, by itself it doesn't matter. All Heroku users should be smart enough to figure this out themselves.

However, Heroku should also be smart enough to figure out that all links should be https and servers should have valid certificates.

For a company to make such basic security-related mistakes while in the middle of a bad security incident doesn't look good, to put it politely.

It sort of explains how they got where they are.

With a non-HTTPS link, an attacker could trivially MITM the connection and forward it to a domain like id.hеrоku.com instead with a completely valid SSL cert with lock symbol. In case you can't realize, that domain is made up of unicode homoglyphs and would be nearly completely indistinguishable from the real heroku.com in some browsers. They could have a website resting on that domain that is indistinguishable from the normal heroku website and simply acts as a form proxy, relaying the real credentials provided by the user to the actual heroku website, performing the password reset successfully while storing the legitimate credentials permanently and additionally scraping the user's cookie to utilize their access to the panel post facto even if the user has 2FA authentication enabled. It's a monumental fuckup.

The real problem is having a link in the email at all. How many readers will check to make sure the email is from a non-unicode domain and that an https link in the email goes to a non-unicode domain?

I certainly couldn't tell the difference, wow.. id.hеrоku.com and id.heroku.com look identical to me. I see the problem now.

How hard is it for an attacker to do a MITM? Would the attacker need physical access to the user's wifi router?

pay for colocation with BGP access with a fake ID or compromise any BGP-announcing router in the entire world (I've found cisco routers with announce access, with default credentials, on networks that don't do any kind of prefix filtering many times before) and you can trivially MITM most any network in the world with very minimal impact. I recommend you watch to Kapela and Pilosov's talk from DEFCON 16 to see how easy it is: https://www.youtube.com/watch?v=oWdjsfsS_Do

or if you don't want to do it via BGP announcement you can just compromise any of the devices that are along the route between the client and the server. BGP is the rocket launcher of worldwide circuit compromise, but there's many other guns you can pick up

Thanks for the recommendation, was very interesting. 14 years old though, things would have changed since, I hope?

Nope, nothing has changed! You can still MITM with BGP in the exact same manner! BGP is like SS7 -- once you get access to use it, it is wide open and completely insecure and no effort has been made to use it. All its security is predicated on the idea that anyone with access will have too much on the line to abuse it, and responsibility for monitoring announcements for your own IP is considered your own problem. Ask around -- how many people in your social circle of admins keep track of rogue announcements? How many would notice if this were going on?

More colocation facilities in the US and Europe now do prefix filtering at egress - but this matters little, as there are still quite enough that don't. In contrast, there's virtually no prefix filtering in Africa or Asia (remember when an accidental BGP announcement from Pakistan killed YouTube worldwide?)

I guess a concern could be that since this URL is insecure, it could redirect you wherever assuming your network is compromised (either their malicious URL, or even HTTP for the legit Heroku site assuming you don't have HSTS on it).

Obviously, you've checked the URL and seen that it's legit after, but realistically you should expect a legit email to not be feeding you a potentially insecure link.

Right. If you're on top of things then using plain HTTP links here didn't make anything worse, but if you aren't it's yet another unnecessary gift to bad guys because it's yet another opportunity for customers to get phished.

Sorry, meant if the end user's network was compromised, not the server's.

Whether or not you're on top of your game, HTTP leaves you vulnerable to MITM attacks[0]. So this would leave the end user vulnerable to any of these attacks even assuming Heroku has everything else perfect.

[0] https://en.wikipedia.org/wiki/Man-in-the-middle_attack

Along with the vague status updates that provide no information, this has been the most shockingly bad part of the response from my POV, as well. This felt like a phishing email (I know it is not, but given that emails were likely compromised along with OAuth tokens, an attacker could totally be sending - even alongside a legit campaign running as announced on their status site). Would have expected a valid HTTPS cert on the links in an email as sensitive as this one!

Salesforce isn't a small new startup where this can be understood as a cost of doing business, they are a tier-1 provider to the largest companies in the world. Unacceptable (but still sending #Hugops to the team dealing with this mess!)

I use HTTPS-only mode in Firefox and this is incredibly common. I think a lot of services are sending all mail through a click-tracking services and a lot of these redirect via HTTP.

I assumed that was a phishing email and didn't proceed.

If it isn't a phishing attempt it would be really surprising to me that they would use http. BTW, https://exacttarget.com redirects to an official Salesforce product page, but that doesn't guarantee it to not be phishing.

> If it isn't a phishing attempt it would be really surprising to me that they would use http.

Why? Technical incompetence is a systematic problem that comes from the top down. This is what happens when mediocrity becomes acceptable and takes priority over correctness.

I would suggest moving all of your business from Heroku ASAP. These kind of mistakes strongly suggest nobody capable of intelligent thought makes decisions at Heroku, nor has for awhile.

> If it isn't a phishing attempt it would be really surprising to me that they would use http

Spammers are the first to adopt https and all the mail and dns security stuff. It never was a sign of legitimacy

The website seems to be down at least for me.

"Passwords may not be duplicated across accounts"

This concerns me. How are they checking that no other account has the same password? Wouldn't that imply a strong possibility they're either hashing with the same salt across all accounts or not hashing at all?

Hopefully some of the smarter folks here can tell me why I'm way off base...

Perhaps they assume that all of their users’ passwords are pwned essentially immediately, so they can check against haveibeenpwned.com.

More seriously, it ought to be possible to create a data structure that, with some degree of security, will efficiently state either that a password is probably in a set or that it’s probably not. A bloom filter can do this, for example. I would not do anything of this sort in production, since, even if it has the best possible level of cryptographic security, any such capability could be abused to precompute a dictionary containing at least a massively higher than average density of Heroku passwords.

IOW, the mere fact that Heroku can tell if a password is or was in use with decent accuracy is, in and of itself, a dangerous oracle.

“Sorry, this password is already being used by joseph.h.blow@gmail.com, please choose another.”

every so often i'll find a website that has logins that will say "email address / username not found" instead of "username or password incorrect"; or, when clicking "i forgot my password" it will say it doesn't recognize the email address instead of "check your email inbox, if you have an account here, you will receive instructions via email".

I don't even

You can remove username enumeration from login and password reset, but it's much harder to remove it from signup. And this is pretty much an all or nothing, there's no point giving cryptic error messages (in the case of forgot password) to users when hackers will just use the signup form anyways. Want a good example? Hacker News: "That username is taken. Please choose another." I just tried on github, generic error message on login, explicit "email in use" error on signup. Amazon? Same thing.

I'd wager that 99% of website allow username enumeration. The very few cases that don't, typically shield their signup with additional steps (captchas, mobile OTP (e.g. paypal, ...)). None of which are foolproof and all of which are a compromise not just from a security point of view but also from a usability point of view.

Until someone gives me a great way to do this with signups, I'm going to stick with the much more usable forgot password error message.

Hacker News has username enumeration anyway because it has per-user profile links which, like Reddit's, are cool URLs (ie the URL has the username in it, as a parameter, not some UUID minted by a machine at random).

Also, and again like (most of) Reddit, Hacker News is public anyway, it's outright weird to have a HN user, choose a distinct name for it when you didn't have to, and then be angry that HN reveals the existence of your user.

However, if sites have distinct username and email then getting one doesn't give you the other.

How about email validation as a first step and send the "this email is in use" message via email instead of on the sign up form.

Adds friction to signup, but would work when it's really important to not leak account existence.

I think this is a good idea. It's essentially what I mentioned with Paypal requiring OTP before the signup process starts up.

But as you said, it adds a lot of friction to something most companies want to be as frictionless as possible: signups. So it's a compromise. I don't expect many companies will opt for it, except, as you say, those where it's particularly important.

I stand by my reply, since my parent implied that username enumeration was rare (i.e. "every so often") and some awful security practice (i.e. "I don't even")

multiple salts can be stored, no? and they can still figure out the same password. Google does the same, even going back multiple passwords in the past ("this password was used previously X years ago").

How would you check that across accounts if you had per-user salts though?

Ah I see, that I don't know (and I misread the original, quite correctly concerned post, my apologies). I am a customer and I will follow up with them about this. If I were to guess, they mean same accounts within the same organization, in case account setup has been automated.

They could in theory hash the new password with every salt on every other account and see if there is a match... but that seems unrealistic, and if they can do it, it would imply that they are not using a slow hash algorithm such as bcrypt.

Yeah, that makes sense!

This is the response I got:

> Sorry, that part of the email wasn't clear. We don't compare passwords between Heroku accounts, this was more of encouragement so customers will hopefully practice good password hygiene.

It is easy to determine previous password. Just store the previous hashes with the salt and check it when the user changes the password. The too similar checks are easy when you require the previous password. You can compare the new and old passwords together.

I'm pretty sure they don't check. "May" now means "legal told us to ask you to PLEASE don't do this".

Related, I often see people trip on the sense of MAY in RFCs (https://www.ietf.org/rfc/rfc2119.txt), because in everyday use it has become "passive-agressive must".

edit: I'm ESL so might be reading way to much into this.

"may not" is kind of weird, and probably was a legal team decision. "passwords can not be reused" would imply security issues that an A+ certified fresh-faced intern would prickle at, but "should not be reused" is too mealy and "Do not feed the ducks"-ish.

Also while "MAY" in RFCs per that document mean something, press releases aren't RFCs and there may be significant amounts of money riding on the exact interpretation of what those words mean.

And "reading way too much into" is the bread and butter of lawyers, technologists, and mathematicians. Don't worry about it!

All you do is keep a history of Password + Salt. When you change the password you hash the password against all historical salts, if one of those matches then you know the password has been re-used.

Perhaps the haveibeenpwned.com API? Otherwise that does seem odd


Couldn't you freeze the current, compromised database and when a user tries to set their password, try to "log them in" to the frozen db?

What does that line even mean? Duplicated across what accounts? Obviously they aren't checking that every password across Heroku is unique.

That's how I read it. How else would the sentence make sense? Maybe they're saving a separate list of all passwords, saved with the same salt, but not associated with any account? Not sure how compatible that would be with the rest of their "security measures", but then again, their security policy is all over the place already.

not that they're using it, but this is something that a zk proof would be a good solution.

The fact that I haven't heard more about this Heroku breach makes me think that maybe fewer startups than I thought are actually using it.

I'm at a startup that's been on Heroku since its inception and this breach has got my team thinking about moving to AWS for the first time, even if our scale doesn't necessarily demand it yet.

Does anyone that has done this transition in the past have any advice?

There was a time when everyone I knew had a rails app on heroku, and I mean everyone. Now it's just one guy left. I don't know if this translates outside of my small social circle, but they all seemed to fade hand in hand, rails, and heroku in my local group.

Have been involved with 2 companies migrations from Heroku to AWS.

It all depends on your company resources. In my cases the companies were no longer startups and could afford to staff Platform/Infra teams and build out a bespoke solution on AWS. This was an area the company wanted to invest in.

If you're still a legit startup where money/dev resources is tight, I would recommend first looking into one of the Heroku clones/competitors.

We've been checking Render and Fly as they progress, they seem like the modern equivalents if you're not quite ready for AWS.

To be honest I've been on Heroku for a long time, but Render looks extremely appealing. Cheaper and more modern. The only downside is they don't have native support for PHP yet... still waiting on that to switch over.

Render is awesome and offers among the best platform-as-a-service experience available today, at a lower cost than Heroku. That said, you're still locking yourself into a black box (just like with Heroku).

Given the availability of tools like Coherence (www.withcoherence.com) that deliver a great developer experience in your own cloud, it makes sense to evaluate them alongside modern PaaS if you're migrating off Heroku today.

(disclosure, I'm a cofounder of Coherence).

Another thing to be mindful of with Render: currently there's no native high-availability Postgres / Redis.

We're also starting to look into this. We're already spending more than we need to, but depending on what this breach turns out to be, we very well may need to move to AWS to keep our enterprise customers happy.

Yeah, same here. I have accelerated my plans to move to either GCP or AWS. I am evaluating both and plan to move to Kubernetes... even though I'd been avoiding it as long as possible.

Check out Render

I'd already be on either fly.io or Render, but my company has a hard requirement for SOC2 and ISO27001 certification. Heroku has it, Render is working on it, but not there yet.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact