

Important Security Announcement from PagerDuty - Titanous
https://www.pagerduty.com/blog/july-2015-security-announcement/

======
andreasley
Andrew Miklas wrote in a comment: _The attacker gained unauthorized access to
an administrative panel provided by one of our hosting providers._

There have been several breaches in the last months where this was the main
cause and it's something almost impossible to defend against – unless you're
running your own datacenter hardware, which is very hard to get right.

Few providers properly secure their control panels with 2FA, even though these
admin panels are an attractive target and almost always provide full access to
the system.

~~~
rdl
Even 2FA as commonly implemented is kind of a bullshit solution (it's once per
30 days, it's not on every transaction, it's not specific to the transaction,
there's no enduring log, no concept of segregating various types of action, no
concept of callback auth/validation, often not multiuser/shared credentials,
etc.)

Consumer-style 2FA protects somewhat against brute force, against password
reuse by users who just don't care, and against theft of a whole password
list. It doesn't actually protect the protected resource very much.

------
sarciszewski
"Both our salts and pepper are 40 characters in length and are randomly
generated."

From their CTO (on Disqus in response to my question):

"The salt, pepper, and password were concatenated together to form the string
that was in turn passed to the hashing function."

[http://3v4l.org/cE359](http://3v4l.org/cE359)

40 + 40 + strlen($password) > 72

Uh-oh.

~~~
geofft
> 3v4l.org

PHP. This is a known weakness in PHP's bcrypt implementation. From Wikipedia,
"Many implementations of bcrypt truncate the password to the first 72 bytes."
I would hope that they're using a competent implementation that either
supports longer passwords or throws an error if it's asked to hash a longer
password.

Besides, bcrypt takes the salt as a separate parameter anyway. So it doesn't
really make sense in context that the salt, pepper, and password were
concatenated and passed _to bcrypt_. Perhaps he was talking about the SHA-1
stretching; perhaps they hash the passwords with SHA-1 before passing it to
bcrypt.

I don't think we know enough to conclude that they were definitely doing it
wrong, but it would be nice to know more details about the algorithm, though.

~~~
ircmaxell
> PHP. This is a known weakness in PHP's bcrypt implementation. From
> Wikipedia, "Many implementations of bcrypt truncate the password to the
> first 72 bytes." I would hope that they're using a competent implementation
> that either supports longer passwords or throws an error if it's asked to
> hash a longer password.

Actually, it's a known weakness in BCRYPT. PHP did not implement bcrypt, it
was ported in via crypt(3). Meaning that ALL versions of bcrypt have this
issue.

Some implementations error on > 72 bytes, but NONE of them accept longer
passwords.

> I don't think we know enough to conclude that they were definitely doing it
> wrong, but it would be nice to know more details about the algorithm,
> though.

Given what has been shared so far, there's enough signs pointing that the
chances are pretty high they did something wrong. 40 byte salt? Bcrypt only
supports a 128 bit salt. So either they did something silly custom (at which
point it's no longer bcrypt), they aren't actually using bcrypt, or they did
something silly like concatenate the salt + pepper + password and pass it to
the password field.

------
skuhn
I appreciate the level of detail that PagerDuty went into with this
announcement.

I presume they wanted to have all of the facts before they notified customers,
but it is totally unacceptable that they waited 3 weeks to notify me about an
incident with confirmed external intrusion and confirmed theft of customer
data (including my own).

~~~
tfigueroa
Well, what could they do? Surely we can't expect them to contact us in a
timely manner via email, SMS, and/or push notification.

------
jc4p
It took them 21 days to announce this? It's a in-depth post and kudos to them
for that, but I'd also like to know _when_ the actual intrusion was (on July
9th when they detected it? Months before?)

~~~
darkr
The email I just received said:

> Within hours of the start of the intrusion, we were able to detect and
> remove the attacker, and shut down the attack.

~~~
rdl
Who cares? It was compromised, and customers weren't notified for 21 days.

If this were some highly regulated banking/securities/government system, 21
days MIGHT be ok for public notice, but not for an infrastructure tool with
highly technical users.

People get hacked (not good), but responding badly is the true badness.

And now additionally we're in WTF hell because no one knows if linode was
compromised meaningfully (again), or if it was just a PagerDuty credential
which got compromised.

~~~
noir_lord
PagerDuty use AWS and Linode, be interesting to see which one was responsible.

[https://support.pagerduty.com/hc/en-
us/articles/202830570-Wh...](https://support.pagerduty.com/hc/en-
us/articles/202830570-Where-is-PagerDuty-hosted-)

~~~
sarciszewski
According to Julius Kivimaki (whom has hacked Linode countless times,
including under the Hack the Planet hacker collective), Linode is pretty much
an open door for blackhats.

~~~
ryanlol
Beat me to it.

------
autotune
This is why PagerDuty needs a decentralized solution for customers. If
products like Chef can have both Hosted Chef and Chef Server and Atlassian can
have both both Hosted and Self-hosted options there's no reason they can't
have the same so you don't have to put your employees potentially sensitive
on-call and server incident alert history in one basket.

------
kolev
I'm happy with OpsGenie [0] - not so expensive, fully-featured, and now -
safer compared to the overpriced PagerDuty. OpsGenie first implemented push
notifications while PagerDuty was relying on people not knowing the
competitive landscape and had a web-based Android app with SMS for years.
OpsGenie innovates now and PagerDuty follows. Research and don't go with the
flow!

[0] [http://opsgenie.com/](http://opsgenie.com/)

~~~
rdl
I love OpsGenie too, but this isn't really the best place for that discussion.
Maybe you should write up your positive experiences with OpsGenie as a blog
post and submit it to HN; particularly in the context of how to set up a
notification system for ops, rotation, etc.

~~~
kolev
I will, but what's interesting is that I keep getting bugged by PagerDuty
sales people and today, just before I read this, I got an email with their
"improved" pricing and "improved" service! Had to vent out, sorry!

~~~
rdl
My problem with OpsGenie is yet another company monitoring critical
infrastructure, often for customers using AWS, which is itself built on AWS.
Sigh.

~~~
skuhn
All of these services seem to take the same approach: AWS US-East, maybe use
two availability zones, call it a day.

PagerDuty had to learn this lesson the hard way (circa 2012), as they were
entirely in US-East during the catastrophic AWS outage in June. They proceeded
to go offline along with a lot of their customers, who subsequently were not
alerted.

Multiple region redundancy is not trivial, but it would certainly be a little
higher up on my to-do list if it was basically the entire value proposition of
my service. I was a little suspicious of PagerDuty's operational chops before,
but now I simply don't trust them.

~~~
rand670
They are multi-region: [https://support.pagerduty.com/hc/en-
us/articles/202830570-Wh...](https://support.pagerduty.com/hc/en-
us/articles/202830570-Where-is-PagerDuty-hosted-)

~~~
skuhn
They are now; as I mentioned, they had to learn that lesson the hard way in
2012.

