
HipChat security notice - el_duderino
https://blog.hipchat.com/2017/04/24/hipchat-security-notice/
======
problems
Doubt this will be a popular view around here, but using a 3rd party service
for internal business communications is just a bad idea.

I've seen companies posting root passwords, ssh keys, salaries, internal
financial details, etc in Slack and HipChat. Just waiting for a disaster to
strike, adding value for every additional company to the target. Maybe this
breach won't be the last straw, but it's a consistent risk.

You can run your own MatterMost or XMPP server quite easily and even lock it
down to behind VPN only to minimize security risks almost completely.

~~~
throwaway91111
In my experience, using irc or xmpp mostly results in people not using it
unless a) the team is largely technical or b) there's a common, easy interface
like gchat used to be.

~~~
deaddodo
Why are you giving your employees a choice in the matter of something so
important? Set up an XMPP server, tell them that's what is used for internal
communication. Period.

And if they're too lazy/dumb/entitled to download Adium/Pidgin and enter their
email address+password; well, you should probably find better employees.

~~~
Xylakant
Friends don't let friends use libpurple based messengers. Sadly, Adium
development is pretty stale and unresponsive to even major security issues
such as [https://threatpost.com/code-execution-vulnerability-found-
in...](https://threatpost.com/code-execution-vulnerability-found-in-libpurple-
im-library/124448/)

------
notum
Needless to say their (login) servers crashed from the pressure of people
resetting their credentials.

"Hey, you know what might be a good idea? Let's email all of the accounts at
the same time using an Appriver blast!"

Atlassian. I hate to hate you.

~~~
rubyn00bie
While I can see your point in this case I think it was the appropriate action,
their ops team should've just beefed up their resources in conjunction with
the email blast.

Only emailing a rolling amount of your customers becomes a shit show of
support, who do you email first? Who do you email last? How long do you wait
between groups? For who is security important, your biggest customers, highest
paying, most security conscious? It's a real shit show to know, and one you'd
absolutely get wrong, letting everyone know as fast as possible is the only
acceptable solution to a security breach.

~~~
notum
The servers should have definitely been prepared for the increased load.
Perhaps I'm overly optimistically using the plural form in this case.

Truth to be told I can only assume this was done in a short burst, given my
limited sample of (hopefully ever narrowing) circle of people who use
Atlassian products. But would distributing the bulk-mail over an hour (two,
three) using a randomized sample of their customer base really made a
significant impact to security or their support?

I wonder how I'd do it, really, if let's say, beefing up my infrastructure for
some reason isn't an option.

------
Retr0spectrum
The HipChat desktop client had a trivial MITM vulnerability which took them
several months to fix after I reported it. They never made any kind of public
notice about it, so I'm almost surprised to see them talking about security
here.

~~~
j_s
Where does that vulnerability report fit in with the Atlassian acquisition?
(circa spring 2012)

~~~
Retr0spectrum
It was first reported around this time last year, so "after".

------
ndrake
I wonder which "popular third-party library" caused the problem

~~~
reverted
More importantly, I wonder how much they were paying for this library, or to
what extent they were supporting it internally. Because if the answer is zero
and they weren't, I would put a lot of the blame on HipChat engineering.

~~~
lolsal
I'm not sure I understand you - You would blame the users of a third-party
library if the library was found to have a vulnerability and it was exploited
against the people using the library?

~~~
falcolas
IMO, frankly, yes.

If you use someone else's code, especially if you're not paying anything for
it, you get what you put into it: nothing.

The liability for this breach is ultimately owned by Atlassian, not the third
party library writer. To quote the most permissive license out there:

"THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED."

~~~
lolsal
To use an analogy, do you blame everyone that has ever used the linux kernel
whenever bugs/vulnerabilities are discovered in the kernel?

~~~
falcolas
I would certainly blame Google if their Android phones were backdoored,
especially if they tried to foist the blame off on the Linux kernel developers
- a much more apt analogy since they sell Android phones.

~~~
algesten
I would be very surprised if something as complicated as Android phones didn't
contain anything that can be back doored.

Obviously that is Google's problem, but I haven't seen Google (nor Atlassian
in this case) claim anyone else is to blame.

------
tbomb
Do they really need a captcha on BOTH username AND password input? I get that
they are different submit pages, so they are likely querying the database on
each page, but is that really necessary? I don't see any benefit from it, as a
user, while trying to log in to my account.

------
chime
My recent password update policy: Wait for a service to be hacked and then
change the password to a long hash generated by Keepass. I was thinking of
spending a whole day updating all the passwords for all the services I have
accounts on but at the rate sites are getting hacked, it won't be long before
I have created unique hash passwords for all the sites.

------
yarper
I miss IRC. All it needs is a few tweaks to bring it to 2017..

~~~
vurpo
Then you might like Matrix. From the IRC point of view, it's basically IRC,
but updated to become a 2017 protocol.

------
925dk
By the way, HipChat still has no two-factor authentication.

------
pram
Is this implying their database was leaked?

~~~
plange
Yes, possibly.

------
Sephr
I understand not disclosing the vulnerability itself, but if they won't even
disclose the affected library's name then they are being grossly irresponsible
or are under an NSL. If they are under an NSL and not just being irresponsible
that would mean the vulnerability is part of one of the stolen NSA exploit
kits.

If I use the library, and it is non-essential for my business, then I should
know what it is so that I can remove it.

------
blauditore
> If you are a user of HipChat.com and have not received an email from our
> Security Team with these instructions, we have found no evidence that you
> are affected by this incident.

Well, I didn't receive anything, but couldn't log in either. I had to google
this to find out what the hell is going on - error messages on the login page
are not helpful either, they just refuse login even after resetting the
password.

------
higon
Just got hit by this. Everybody in my team was using HipChat as a primary
online communication tool. So, was nice to see nobody in the room for a while.

    
    
      Fine, but I wonder why they didn't reset the API tokens while resetting password immediately. Are they managed by the different servers/services?

------
cypherg
well written blog post imho

------
925dk
"This weekend our Security Intelligence Team detected" ...

"Security Intelligence Team" ... yeah, because that team actually exists.

~~~
925dk
It's sitting right next to the emoji team.

------
925dk
Why are they force resetting everyone's password if they are bcrypt'ed?

~~~
cypherg
it's considered best security practice to do so

~~~
925dk
By whom?

~~~
Godel_unicode
Science.

You might find the below numbers interesting. Note that this performance is
only one workstation with 8x gtx980. Even the mighty bcrypt (sidebar, look at
the sha512 #s) won't save you if your password is bad. Now consider social
media mining to enhance the word list. Now consider that (anecdotally) I have
never done a hashcat audit and not had to have a conversation with someone
about choosing better passwords:

Hashtype: bcrypt, Blowfish(OpenBSD) Workload: 32 loops, 2 accel

Speed.GPU.#1.: 6398 H/s Speed.GPU.#2.: 6507 H/s Speed.GPU.#3.: 6513 H/s
Speed.GPU.#4.: 6643 H/s Speed.GPU.#5.: 6534 H/s Speed.GPU.#6.: 6512 H/s
Speed.GPU.#7.: 6689 H/s Speed.GPU.#8.: 6542 H/s Speed.GPU.#*.: 52338 H/s

[https://gist.github.com/epixoip/c0b92196a33b902ec5f3](https://gist.github.com/epixoip/c0b92196a33b902ec5f3)

------
jordache
WTF? I got the email from HipChat. It includes this sentence. Without
additional non-techy context

"HipChat hashes passwords using bcrypt with a random salt."

This is a good example of how not to do mass e-mails targeting the general
population.

~~~
akerl_
If they'd just said "We securely store your passwords", they would have tech
people with pitchforks and torches about how they need to name their password
hashing algorithm.

If they'd explained both ways, somebody would accuse them of the notification
being too long as part of a scheme to bury the details below the fold.

As a company reporting an issue, no matter what you do, the internet's gonna
nit pick

