
Zendesk was hacked - tedivm
http://www.zendesk.com/blog/weve-been-hacked
======
cing
Here's the email from Tumblr:

For the last 2.5 years, we've used a popular service called Zendesk to store,
organize, and answer emails to Tumblr Support. We've learned that a security
breach at Zendesk has affected Tumblr and two other companies. We are sending
this notification to all email addresses that we believe may have been
affected by this breach.

This has potentially exposed records of subject lines and, in some cases,
email addresses of messages sent to Tumblr Support. While much of this
information is innocuous, please take some time today to consider the
following:

The subject lines of your emails to Tumblr Support may have included the
address of your blog which could potentially allow your blog to be unwillingly
associated with your email address. Any other information included in the
subject lines of emails you’ve sent to Tumblr Support may be exposed. We
recommend you review any correspondence you've addressed to
support@tumblr.com, abuse@tumblr.com, dmca@tumblr.com, legal@tumblr.com,
enquiries@tumblr.com, or lawenforcement@tumblr.com. Tumblr will never ask you
for your password by email. Emails are easy to fake, and you should be
suspicious of unexpected emails you receive.

Your safety is our highest priority. We're working with law enforcement and
Zendesk to better understand this attack. Please monitor your email and Tumblr
accounts for suspicious behavior, and notify us immediately if you have any
concerns.

------
lawnchair_larry
First, thanks for disclosing this.

Second - and any incident response team will tell you this - patching and
removing the backdoor is not enough. You have to wipe that machine.

It's not uncommon for an attacker to leave multiple backdoors. Even if you
don't _think_ they got root, you have to wipe it completely.

~~~
jpswade
Destroying data is never a solution.

~~~
martinced
You got downvoted by several people because what you say is not related to the
person you answered to.

Typically when a breach-in happens you do two things:

1\. backup the entire disks for forensics purposes

2\. wipe everything clean, re-install the latest version of the OS + all the
latest patches, re-install the latest version of the apps.

Nowhere did parent suggest to "destroy evidence".

~~~
amw
Not that this guy said "evidence" either ("data" was the word), but otherwise
I agree

------
jcoder
"As soon as we learned of the attack, we patched the vulnerability and closed
the access that the hacker had."

Ok, so striking out so far. The machine is still running? With the same
software (patched) and user accounts? How do you know only 3 users were
exposed?

------
tomhallett
If you have a mid-sized Rails app, with say 2-3 developers working on it, a
full time security engineer would probably be overkill. Anyone have any
recommendations of services/consultancies to be able to tell "oh, someone is
hacking us right now" or "our page which has stripe.js on it has been
compromised"?

I'm hoping for automated tools, services to install on our servers, or
security auditors who have an out of the box package.

This is the only automated tool I know about: <http://brakemanscanner.org/>

~~~
diziet
Tinfoil Security (<https://www.tinfoilsecurity.com/>) has automated scanners
and seems to keep up to date with all the vulnerabilities quickly, like in
case of the rails vulnerability recently.

~~~
tomhallett
very cool - thanks!

------
DanBlake
CPanel was also hacked which is a way bigger deal imo- Led to thousands(!) of
other server compromises:

<http://www.webhostingtalk.com/showthread.php?t=1235797>

~~~
meaty
CPanel is always hacked so it's hardly news.

------
codenerdz
Given Zendesk is a Rails shop, Id love to hear if this hack was related to any
of the recent Rails exploits

~~~
mef
If it was, at it occurred after the vulnerabilities were made public, they
probably wouldn't say so as it would look pretty bad given the amount of
advance warning they had.

~~~
scorpion032
Are you suggesting most popular rails (if not all) apps are upgraded by now?

~~~
tlrobinson
No, he's suggesting the ones that aren't are run by incompetent people.

~~~
scorpion032
If thats the case, there are so many incompetent people.

------
emptyage
The three companies were Twitter, Tumblr and Pinterest:
<http://www.wired.com/threatlevel/?p=54338>

~~~
creativityland
So what kind of integration did Twitter, Tumblr and Pinterest have with
Zendesk? How much of risk are the users are with their passwords?

~~~
unreal37
No password data was stored there - so zero. No passwords, password hashes or
encrypted passwords were lost.

~~~
viscanti
But there's a greater than 0% chance that zendesk had an API token for at
least one of those services. That could easily allow a hacker to make
authenticated requests to those services to gain user info. The fact that
usernames and passwords weren't stored on zendesk doesn't mean much, if a
hacker can gain full admin access to those other services through an admin
token that might have been stored on zendesk.

~~~
kelnos
I seriously doubt any company (especially the three listed) would give Zendesk
admin access to their service. Why would such a thing be necessary, anyway?

~~~
jorts
I think he meant it the other way around. Having their API token would allow
the attacker to have access to all of Twitter/Tumblr/Pinterest's information
that's accessible via the Zendesk API.

------
TheOnly92
Apparently the 3 customers are Twitter, Pinterest and Tumblr.

[http://www.wired.com/threatlevel/2013/02/twitter-tumblr-
pint...](http://www.wired.com/threatlevel/2013/02/twitter-tumblr-pinterest/)

------
tstactplsignore
From the perspective of a complete server administrator novice, are all of the
mainstream "hacks" due to the complexity of these applications? For example,
if I were to setup a basic, updated Ubuntu Server LAMP stack with a MySQL
database, is this system vulnerable? I understand how to protect against XSS
and SQL injection and how to hash and salt passwords properly, but where can I
begin to learn about implementing basic, hard server security? Additionally,
how can I hope to secure my web app if corporations with entire security
departments are failing to secure theirs?

~~~
ams6110
_how can I hope to secure my web app if corporations with entire security
departments are failing to secure theirs_

Excellent question. The answer really is, if you are running a "web app" with
any kind of sensitive information, you need to have a security expert
configure and administer your systems, or become one yourself, and you need to
stay on top of every update.

Now, there are basic things you can do that will eliminate most of the casual
script kiddie attackers. Firewalls. Defense in depth. Keeping up with patches.
Google or any good book on computer security will provide this information.
But the sad truth is, if a smart, determined attacker has you in his sights,
you will most likely lose. And even the automated tools are getting better and
better.

I think we are starting to see a tipping point in the web. More and more high
profile sites are getting compromised. We have learned that China has been
inside US government, utility, and industrial systems for years. Right now I
would not trust any sensitive, personal information to any website or cloud
service. I think we are going to see some major, consequential attacks on
government, banks, and other commercial entities in the coming years. At some
point I believe we are going to need to rethink the cost/benefit equation of
having everything connected to everything.

~~~
SomeCallMeTim
> Right now I would not trust any sensitive, personal information to any
> website or cloud service.

A friend of mine put together an open source project that uses cryptography to
store your data in the cloud securely, so it's certainly possible. [1]

It's also possible to write complex software and not be vulnerable, though
99.999% of the time companies (start-ups and otherwise) seem more concerned
with an MVP and new features than security. If you design a system from the
ground up with security as a core feature, then you have a CHANCE of having a
system that won't be vulnerable to script kiddies every other week. On top of
that you need to be sure to protect against social engineering, but that's
another discussion.

I don't even know if it's possible to use something like Rails (or Ruby, even)
and be secure for the long term without having to deal with constant updates
and patches. On the other hand, I HAVE used complex systems that were designed
from the ground up to be secure and that simply NEVER turned out to have a
security vulnerability after the first few releases. (Anything by DJB, for
example. [2] Some of those tools have gone 15+ years with no vulnerabilities.
Compare the constant sendmail or bind security exploits, numbering in the
hundreds at this point, to DJB's qmail and djbdns.)

Until it's a priority, it's always going to be an afterthought, by definition.
People will use Rails or the framework du jour, despite the fact that such
frameworks are designed with the same "get it done and release ASAP"
philosophy that most commercial sites are developed with, and then everyone
wonders at security holes. Sigh.

[1] <https://tahoe-lafs.org/trac/tahoe-lafs>

[2] <http://cr.yp.to/>

------
Argorak
I cannot access the page from germany: it immediately redirects me to the
(german) front page. (Firefox 19, german edition)

------
businessleads
Wow, all three customers?

~~~
mparlane
Does anyone know how many customers they do have to put this into perspective
?

~~~
jonlarson
Homepage says more than 25,000 companies

------
tomjen3
Anybody knows a good alternative?

Leaking my personal info is one thing, loosing customers personal info is
simply far too boneheaded to begin with.

~~~
TallGuyShort
Nobody is unhackable. They've dealt with this honestly . What makes you think
any other service could make further guarantees?

~~~
amw
Seconded. The sheer fact of the matter is that the state-of-the-art on offense
is outpacing the state-of-the-art on defense. These guys didn't expose any
password data, and evidently, not even the contents of the support cases (just
the subject lines). That's small potatoes.

------
OGinparadise
All kinds of sites, some supposedly super-secure, have been hacked.

Now a serious question: What does this say for storing everything, including
tax filings, in the Cloud?

~~~
dirtyaura
I'm curious why you picked tax filings as an example of information that would
be catastrophic to leak.

Here in Finland tax information is public and it doesn't seem to be that bad.
As you guess, yellow papers embarrasingly make yearly rankings of rich people
and they lose some anonymity. But anybody that was in any way interested in
their life likely knew that they were rich.

I don't say that this Finnish practice of having public tax information is a
good one, but it just shows that it's not the end of world.

Leaked Tumblr support information that allows associating Tumblr blogs to
email addresses can cause a similar level of agony that leaked tax
information.

~~~
jlmendezbonini
>I'm curious why you picked tax filings as an example of information that
would be catastrophic to leak.

I'm assuming that the original poster is from U.S. One reason why tax filling
are considered sensitive is that they include the individual(s) social
security number (SSN). This number is meant to be private and, since it's
often used as an identifier, it's frequently used for identity theft. See the
wikipedia article below.

[http://en.wikipedia.org/wiki/Social_Security_number#Identity...](http://en.wikipedia.org/wiki/Social_Security_number#Identity_theft)

