I called American Express Australia to report the defect & I was transferred through to the American call centre.
The CSR to whom I spoke transferred me through to a different department, after I explained that I didn't have an account. She did ask whether "I received an email" which I assume was some sort of inquiry as to whether I had been phished.
I then spoke to an online services rep., who after asking for my card number, listened to my report. She then put me on hold.
(The call had taken 10 minutes by this time).
After a few more minutes on hold, the CSR came back on the line, asked me to repeat the information, and confirmed for the umpteenth time that I don't have an American Express card. I explained that it wasn't my find, but that it had been published online & so was by now _very_ public.
(15 minutes by this time, most of that on hold listening to advertising for American Express, including some ironic praise for their website).
CSR comes back on the line. She's spoken to her 'technical team' who assure me that there's nothing insecure going on because it's all over HTTPS. So I politely walked her through the process - visit the page, add ?debug to the URL, click the admin link & behold: lots of should-be-secure stuff.
At this point she thanks me profusely, & asks that I hold while she speaks to her supervisor. Back to the American Express ads ...
(20 minutes at this point).
The CSR came back on the line, thanked me again, & said that her supervisor had taken a screenshot of the issue & escalated it. Job done.
So, yeah, I can totally understand the frustration experienced by the guy who discovered the vulnerability. But it certainly wasn't impossible for me to report the issue, & I'm in Australia.
(I didn't mention that there was a pregnant pause after she clicked the Admin Home link & saw the admin page in all its glory. I think the only sound was, as Scott Adams put it, the sound of eyeballs getting really big.)
Do remember that for every real serious problem such as this, American Express and any other large company receives a hundred or a thousand calls for non-issues like phishing or minor issues like a compromised card number to cancel and reissue. The system is optimized for these common cases; the edge case of a real public vulnerability will require extra effort and it's not a failing of Amex that the system is so.
I agree in principle, but a core part of their business is maintaining the integrity of a system. When you combine this with some of the statements they have made regarding their password policy, it makes me as a paying customer very concerned about their approach to cyber-security. THey have excellent fraud prevention, but I'd really like to see them make some strides in this area.
If they didn't filter aggressively, that would mean that their more technical staff would be deluged with crap.
And then quit.
I was operations/development at an ISP once, had three layers of acceptably competent support techs between the customer and I and a serious reputation as a fire breathing dragon if a non-issue was escalated to me - but -still- regularly lost half my day dealing with escalated "urgent problems" that were neither.
You misinterpreted me. What I said is that the extra effort needed to report this vulnerability isn't a fault of the Amex customer service system. Of course it's a faulty deployment process that allowed this to happen in the first place. But as far as escalating a trouble report from an ordinary consumer, what happened does seem pretty reasonable.
Flip it around: what if Amex (or any other large company) made it easy to escalate everything to a technologically capable supervisor right away? Those supervisors would be deluged in uninformed, irrelevant, and just plain wrong security reports. Filtering out the signal from the noise in the security landscape is a monumental task in itself. As tech savvy hackers, we always think we're entitled to say "I know what I'm doing so escalate me over the idiots", but how does a company or CSR tell whether that's actually true?
The dozen or so hackers in this thread that expect that "security vulnerability" is some magic keyword that gets you talking to the head technical honcho of the security group have probably never answered phone calls for a big company. Phone support for somebody like AmEx is a huge burden of cost and manpower; the structure of the tree has been set firmly in place since the 1980's to take care of the most common 90% of issues using the least-paid person available. I'm sorry, if you're in the long tail you will just have to expect to wait extra. That goes double if you are not a cardmember (read: paying customer).
I am surprised that the above person in Australia got through at all, and that the CSR had latitude to try to spend time replicating the issue. In my opinion, for a credit card company, 20 minutes and a positive conclusion for a matter as rare as reporting a webapp vulnerability is a success.
Sure, none of you should be thrilled about the situation because as technically-oriented people with generous motives the system is not set up to serve you. But that's not a failure of the system, except maybe from your own individual perspective. Believe me, AmEx has done the cost-benefit analysis and they are saving boatloads of money by having those rare well-intentioned hackers listen to some hold music, because it is too expensive to sort you out from the thousands of loonies that got a phishing email. Security breaches are an acknowledged risk and they are already prepared to absorb their effects on multiple levels.
In a way it is a failure of the system, in that it is much easier to simply post the vulnerability on your blog or a full-disclosure mailing list than 'officially' report it. This could potentially cost them large amounts of money.
Why would finding a vulnerability give you the moral imperative to waste so much time reporting it? Especially if you're not a customer or otherwise affected by it? I know I wouldn't.
This is why companies like Google have a security issue submit form. Sure, some lower-wage people will be filtering it, but at least they will have had training to separate the important from the unimportant problems. And for a bank, security is even more paramount.
> Sure, none of you should be thrilled about the situation because as technically-oriented people with generous motives the system is not set up to serve you. But that's not a failure of the system, except maybe from your own individual perspective. Believe me, AmEx has done the cost-benefit analysis and they are saving boatloads of money by having those rare well-intentioned hackers listen to some hold music
Which is why we shouldn't jump through their hoops. If we do we let them get away with it. If we didn't they'd be forced to pay more attention.
The well-meaning person in this thread did them and us a disservice by going so far out of his way.
Erm, no. Dude if I spend my time figuring out vulnerabilities to your system and don't exploit em, instead help you close them, and I am not even a cardholder... I will not jump through any hoops for any amount of my time. They owe me, I owe them shit. I am being kind and generous by not exploiting or giving the exploit to others, or using it to fuck up AMEX reputation.
Especially true if I want my anonymity preserved.
Jumping through hoops? Dude he got in contact with someone. Instead he got a "piss off" response.
> I for one think it's a seriously unrealistic expectation to think that AMEX or insert large corp here will handle security vulnerabilities over twitter.
a) Agree. b) That said, I think the fact that the person on the other end of the American Express Twitter account was accepting to talk to the guy over DM, and thereby actually /was/ willing to handle a security vulnerability over Twitter, is the most damning argument against this guy's rant; he insisted on using a "modern protocol", but apparently telling someone using Twitter, when they were perfectly happy to let him do so, was not modern enough: he insisted it be on his terms or no terms, e-mail or nothing.
No, he did not appear to want to discuss it over twitter either. They offered that.
I assume he was looking for a specific email address and perhaps a PGP key. Sure, that would be nice. But using the telephone is a pretty common method of transmitting important, time-sensitive information.
As someone who does community management/marketing, I take claims like security issues very seriously. If someone at 4pm messaged such over Twitter to my startup, I'd call the CEO and all engineers immediately, regardless of the time. I don't think I'm a doorman there, but rather the first line of defense/listening.
To be fair, he asked repeatedly for a proper security contact and claims they don't publish one for whatever reason.
If the only way to contact you is through clueless support people who have a script that doesn't include your option, yeah, that's a problem. But usually it's the customer who is screwed by this. This time, it bit the company instead.
That said, you can always ask if someone knows a security contact on BugTraq. Someone there will probably know.
Well, I didn't get a "piss off" response, I got a nice warm thank you. Perhaps he could have persevered just a bit? I had to do a bit of hand-holding to get the CSR to whom I spoke to understand the problem, but once she did, it was easy.
At first glance you made some progress and that makes him seem unrealistic in his demands for electronic communication, but what makes you think that the issue was reported properly upstream? I think that you got lucky to find someone who understood that it was a real problem, and unless they have an internal escalation procedure in place, there's a decent chance it will die with her or her supervisor. There's really no way of knowing if your report had any effect or not. Other large corporations have measures in place to handle vulnerability reports, it seems like a problem for a large CC provider to not have a clear procedure in place for handling these issues.
I have to agree with the others here, while you may have done what you believe is the "right" thing, you have absolutely no idea if that avenue of inquiry went anywhere, and based on my experience working in an enterprise, I would guess that even if it did go anywhere from there it would take weeks for meetings to get scheduled, and months for people to get assigned to actually do anything about it.
I disagree. If you want to be a black hat and exploit or sell the vulnerability, then fine. But if you're going to claim to be a good guy, you need to make more than a half-hearted effort to do the right thing.
In this case, the exploit is so simple and obvious that he could have fit it in a twitter DM (which is a method of communication that was specifically offered to him)
Why should he? The company was trying to be hard to reach to control costs. It cost them in another way. Life is tough.
This is financial infrastructure, there's a higher bar. I think the government should mandate that we give the hacker who discovers the hole 2% of the company's profits for the year... If not, nationalize them and rid us all of the useless frictional costs.
Someone signed up for their Wells Fargo account with my email address. For weeks, I tried to get in contact with Wells Fargo about the problem. I spent a lot of time on the phone with the 'security team' and nothing ever came of it. In the end, I kept receiving all of this person's banking information.
It wasn't so much a security problem so much as it was annoying to get this person's banking info all the time. (I assume I would have needed more than access to email to get into his account.)
In the end, after several phone calls and then tweeting at Wells Fargo, it was the Facebook reps that were able to get someone to call me and sort the problem out. The rep who called even verified that the first three of my social didn't match the account holder, so it wasn't identity theft.
Completely annoying but in the end, it just took finding that one person that understood and cared to help.
Some years ago when I was doing more stuff in spam and phishing I came across a phishing site for a small US bank. The list of phished card details was available through the interface and it was clear that there were some real people local to the bank who had given their name, address, card number, PIN, SSN, ... everything.
I decided to contact the bank. After filling in the form for contact on their web site giving all the details of the site, I did get an email back and eventually I got someone on the phone. This person (who said they were in charge of bank computer security) thanked me and said that they were going to try to deal with it (I had also contacted the school district whose computer was hosting the site to get it shut down).
I then told this person that there were real account details on the phisher site and would they like the list of people's account numbers so they could inform their customer/shut down their debit card etc. The bank officer replied, "No." As far as they were concerned the people who were that stupid got what they deserved.
I was flabbergasted, but couldn't do much to make the bank do something.
So, using the names and addresses of the people from the phishing site I managed to track a couple of them down (they were small businesses whose business addresses were available on the web) and phoned them up so they would be alerted. They took it pretty well considering that some weird British guy was calling them from France to tell them their US bank account details were at risk.
Verified by visa is hideous security theatre. I have no idea why banks fail so hard at security. They're actively targeted by criminal gangs; they stand to lose money if they get it wrong; they have money and expertise to get it right. Yet they all suck.
Sounds like they were trying to avoid liability. If you know person X has had his account hijacked, and you do nothing, you're probably liable under some law or another. If you don't know the exact identities involved, you can feign ignorance and probably get away with it.
Local? Just local?
I'd be as noisy about it as I could, and I would have informed the people who's info had been compromised as to just what the bank said when you offered them a list of compromised accounts.
Wouldn't you want to be informed if your bank was intentionally leaving your personal info and financial well-being at risk?
There's usually a procedure for reporting lost cards which results in immediate blocking, if you really want to secure those numbers. In Lloyds it's actually pretty strict - I found some wallet one day on a street but without any contact information - called up the bank responsible for the card so that they can contact the owner with my phone number, but they wouldn't proceed before cancelling that person's card. On on hand side I can understand that action, on the other I feel bad for causing that person to request a new card when I was already standing on the street he lives on.
Like I said - I understand why it's done and it seems to be a method of forcing some cards to be blocked ("I found cards with those numbers..."). Unfortunately it causes some issues if you actually intend to return the wallet/card to the owner.
yep, credit that the guy partially tried - but prefacing your first interaction about a serious issue by "I'm not available [to contact through most of the usual communication methods]" is sort of self-defeating.
He's reporting a vulnerability on a website. It is absolutely reasonable to expect to be able to report it through email, and utterly ridiculous of AMEX to refuse. That's where the conversation ends, not with "well, you should spend your time fighting through these costly and obsolete mechanisms so you can do us a favor".
Here's something I learned from AMEX last week ... if one of your cards gets compromised and you cancel the card, AMEX will continue to allow charges to flow through that old "canceled" number to your newly issued number if those charges are coming from a "trusted recurring entity". I discovered that charges were continuing to flow through a number that I'd canceled due to it being compromised even though I thought it'd been nullified. AMEX explained that their policy is to allow these charges to continue, and it took a number of months before I caught the problem because the charge was coming from a business I continued to have business with. Apparently the person that stole my number had setup a recurring charge with this business as well. To their credit, AMEX removed all of these charges even though they spanned a number of months ... but it caught me completely by surprise that a number I though was canceled was still allowing charges to flow through it.
That's standard practice for all cards, it's not just AMEX (I believe the authority is based on the account rather than the card). In the UK, there's no easy way to cancel a recurring payment on a card other than contacting the entity taking the payments. If they refuse, you can complain to the card provider and they will eventually sort it out, but payments will still go through in the meantime.
Moral of the story: Don't let anyone have a recurring payment authorisation on your card.
Even if you do, you're still liable for any charges which hit the account after it's been closed, at least according to every closed account letter I've received (in the UK, not sure what the process is in other countries). Plus you can't close an account until the balance is clear.
I have an AMEX card that expired in 2007 and it is still successfully charged by AWS each month. Apparently, it's a big pain to get customers to re-enter new payment details when cards expire, as a result I believe merchants are often allowed to charge to cards that have long since expired.
Not just AMEX. I had an account drained (and indeed sent quite negative) after I explicitly cancelled a Msstercard to stop two such entities who would not cancel my accounts from continuing to charge me for services I wasn't using.
This also happened to me. The problem was that the fraud was coming from one of their "Trusted Entities" (Best Buy). So on day 1 I had $500 worth of fraudulent charges, and on day 2 I had to call them back and let them know of more fraudulent charges.
The author should have contacted the email addresses given in the DNS WHOIS (email@example.com, firstname.lastname@example.org) and the obvious aliases (security@...).
However I can understand and sympathize, it's enraging how hard it is to get into contact with a person of any kind at certain companies (KLM/Air France, I'm looking at you). I understand they want to save money, but if you run a business, you have to be contactable in one way or another. And snail mail as the last option really doesn't cut it in the 21st century.
I did get contacted by a Twitter account after venting there, but after DM'ing (160 characters?!) my request, they couldn't help me either.
Really, why not just provide an email address? If you have someone listening and responding at @KLM in any case, why not also accept emails instead of the crippled communication possible through Twitter?
When a major company, especially a financial services company, is subject to public security vulnerability disclosures like this, it should really make other companies stand up and take notice. There is absolutely no excuse for these kinds of vulnerabilities to exist on a production system. When Citibank was recently hacked by simply changing the account number in URLs, that should have been enough for other financial institutions to do an internal security audit to make sure they weren't susceptible to anything similar. Don't wait until it's too late. For the sake of their customers I hope this is resolved swiftly.
It seems the bigger the company is the more irresponsible they become. In UK in the bank I use, you can activate protection of your debit card / current account (usage analysis, higher insurance), but to do that you need to register with Experian (credit rating company). The process for that is: put your recent bill, bank statement and photocopy of ID in an envelope and post it to them via normal mail.
I decided to ignore that great offer and keep my account secure in traditional way. Apparently ignorance with regards to the internet sites is not what causes big companies to act in stupid ways. It's the whole mindset...
I'm pretty sure normal mail is generally quite secure. Sure, there's very little barrier to someone opening your envelope, but perhaps because the ratio of sensitive stuff vs. letters to grandma is so low, I'm not aware of it ever happening much.
There's one big difference in those letters though. The letter to grandma will be addressed to a person. The letter with documents for Experian will be addressed to... Experian, which is a known company dealing with money and personal data.
Not to mention that that's a trivial security mistake. ActiveRecord makes it very easy to just "read" the id, and ignore whether or not the user actually has access to it, or just guessed the id. Any operation using an id needs to be checking if you actually have rights to the object. Yes it requires an extra SELECT before you UPDATE or an extra condition (my ORM doesn't do that), but it's secure.
Next time, I would try reaching their Public Relations group for help. PR people are almost always accessible by name, phone, and email -- they're usually on the bottom of every press release that goes out. They also have good internal channels to every part of the company and know who to contact.
Unrelated, it looks like someone at AmEx finally improved their crazy, broken password system at least, this used to be the password requirement:
"Your Password should contain 6 to 8 characters . at least one letter and one number (not case sensitive), contain no spaces or special characters (e.g. &, >, , $, @) and be different from your User ID."*
Now it's this:
"Your Password must be different from your User ID, must contain 8 to 20 characters, including one letter and number, may include the following characters: %,&, _, ?, #, =, -, cannot have any spaces and will not be case sensitive."
I'm not so sure about the origin, but it's commonly used in design/UX to showcase one primary or "Hero" product, and refers to a large space front-and-center above the fold on a page. Think apple's site putting up a huge iPhone image on release day (http://www.sprint.com is another good example).
I think this likely started in physical product sites and the lingo just stuck.
Hero;- An entity which is idealized for possessing superior qualities in any field.
It is someone / something that is promoting your company or product, and is a 'hero of the product X' being promoted. Kind of like how, if we mention it to our friends, we become 'heroes of hacker news' ..
The utter lack of a mechanism to report bugs, particularly security bugs, seems far worse.
I've encountered this problem frequently when interacting with various organizations. The pervasive availability of bug-tracking systems and/or bug-reporting email addresses makes the absence of one quite conspicuous.
I've many seen organizations applying spam filtering on their email@example.com address, leading to tons of reports ending up in spam boxes without being noticed by the company. The researcher doesn't receive any feedback on his responsible disclosure and multiple reminders, and finally submits the vulnerability to a full disclosure list.
This is crazy... when you go to the admin panel https://www.americanexpress.com/us/admin/ you actually get access to user cookies (session ids) which probably allow you to hijack their session (haven't tried it in case it's going to be traced back...)
Surely a DM message to the AskAmex account, with some actual details written in clear English, not jargon or "hacker lingo stuff" would have been more suitable? Or asking someone on here like Thomas to make a phone call?
I understand the argument between full disclosure and responsible disclosure, but if the author could have DM'd it on Twitter. Or posted it on Twitter wholesale, since its now public anyway.
AMEX made it incredibly difficult for this guy to report the issue to anyone who had the slightest clue as to its severity. Banging his head against the wall until someone finally clued in would not have fixed that communication issue. Full disclosure just might.
All the more reason to make as clear and straightforward a declaration as possible. Not "I have vulnerabilities", but a DM saying "American Express is leaking customer information at this URL and it is imperative this is reported to your security department." It's their problem to escalate if they don't understand, but you have to give enough information to make escalation possible.
How much more plain than "Who can I contact regarding security vulnerabilities in your system" can you get? When she asked what kind of vulnerabilities, would saying, "unsecured admin panel and xss allowing for session jacking and spoofing" really have been more meaningful than what he said? Even saying "unsecured admin panel" on twitter would have sent people scrambling for it. He was attempting responsible disclosure before he turned to full disclosure.
All you guys (not targeted specifically at you here) that say 'He tried it in a clear way': Call one of the lesser technical inclined people in your family/among your friends. Tell them you've just read about a security vulnerability and wonder if they could describe what that is to one (possibly less technical inclined) people in their family/among their friends.
That's essentially what you're looking at if you throw these words at a corporate marketing (with some links to support) drone that needs to fill in his/her supervisors to make anything special happen.
Indeed. Perhaps it would have been better to publish the exploit anonymously. It's definitely not too useful in terms of street cred -- while we all appreciate discovery of security bugs, it takes no skill and is a common script-kiddie method to just try /admin, etc., after URLs, so there is not much value professionally in having your real name tied to it, but significant risk and harassment issues come into play, especially in a big thing like this where the ire of angry and/or fleeced customers can easily be manipulated and misdirected.
Except, you can't leave your valuables on the street and then arrest someone for breaking and entering when they're stolen!
Agreed, that would be theft:
Theft by finding occurs when someone who chances upon an object which seems abandoned takes possession of the object but fails to take steps to establish whether the object is abandoned and not merely lost or unattended
if you're not meant to be there, you're guilty of an offence.
But how would I know? If someone's private property is not marked as such in any way, would I be a trespasser if I wander into it? Let's say it's part of a field or a forest, not a building with doors ...
Can trespass to land be committed without fault? The answer should be obvious but I have found it surprisingly difficult to track down. I am referring, not to cases of involuntary entry onto land (there are clear cases saying no liability if you get pushed or fall unconscious), but to the sort of case where you (without carelessness) cross over someone's boundary in the bush (maybe more likely in Australia than the UK!) without knowing it
The complaint indicates that AT&T's publicly accessible endpoint is a protected computer under Title 18, United States Code, Section 1030(e)(2). a protected computer is basically any computer used for interstate or foreign commerce in the US, or outside the US if it affects the commerce thereof.
the issue hinges on intent - if you know that you're exceeding authorized access to obtain something of value. 18 USC 1030 was created in 1986 by the Computer Fraud and Abuse Act and is often panned for being incredibly broad.
Should have? Maybe. But, with corporations that size it's unlikely. I'm not saying this fall entirely on him, but I feel he didn't exactly do his fullest before puling the trigger on the full disclosure.
I don't consider telephone contact for security vulnerabilities to be that unreasonable. They should support PGP encrypted email, yes, and have a page about how to report incidents, issue tracking numbers, etc., but it took me ~3 minutes on the phone to get the right info for Amex corporate security.
Unfortunately, I've had this kind of difficulty far too often when reaching out to large companies with disclosures. Most recently, the only thing that worked was blasting off an email to all the internal people I could find through google: the CTO, vp of engineering, and head of support were on the list, as were a few lower level employees. The lower level got back to me right away, eager to cc the CTO on their response =)
I empathize with the developer, but this disclosure is wildly irresponsible.
It's a pain contacting live representatives at any large corporation. When you're dealing with the financial industry, you should grit your teeth and find a way to do it anyway. If you have no choice, publish a warning about the exploit, but don't release all the details without a long warning period.
No. It's about time we stop letting the financial industry get away with incompetence. Every other software vendor would be raked over coals for not having a publicly available security disclosure email address and utterly failing to properly route a request via Twitter.
Responsible disclosure exists so that vendors have an incentive to respond to vulnerability reports in a timely manner. In fact, it is the responsible thing to publicly disclose vulnerabilities so that AmEx learns to implement a proper security reporting process.
No. I agree with almost everything you wrote, but this sort of disclosure doesn't punish the company, it punishes its _users_, and doesn't give them an easy way to make the causal connection. Unless this story is picked up by the mainstream media, how are any victims of this exploit to know that it happened because AmEx is incompetent, instead of e.g. because credit cards are risky?
FYI, this is one of the few good uses for LinkedIn. If you need to access the engineering department, the ordinary external avenues are usually going to fall flat, and that only becomes increasingly true as the target organization expands. However, hopping on LinkedIn you can find an engineer or someone who at least has engineering buddies within AmEx and similarly monolithic corporations in seconds.
As an Amex cardholder, I can attest to the fact that getting in touch with their service reps, should you happen to not have your card on hand, is a pain in the ass. They have many obstacles in place to prevent talking to non-members.
I agree, but this is a complete fail by AmEx. They don't even have a way to report or check on phishing emails from their contact page. THAT would have been the way I would have tried to get in contact with them to help them out. Hopefully, if nothing else, they'll get some sort of scam alert response.
Note to self: It's really hard to automate good customer service.
So 90 comments and no mention of "didn't he try emailing firstname.lastname@example.org". That would be my first step, not harassing a marketing account on Twitter. Marketing campaigns are often run by third-party companies. Whoever gets security@ emails, not so much.
If you want to inflate your ego, post to full-disclosure; don't annoy people on Twitter and blog about it.
The story was posted an HOUR ago! They're a bank! Imagine the number of criminals swarming over their website by now. You'd think they'd react quicker. Or maybe the bosses there aren't aware of the implications of this disclosure.
Ugh, it would just be easier to sell the vuln than try to inform one of these clueless dinosaur companies about it. I know why companies like Amex build these giant fortresses around their communications, but they should be more cognizant of the damage that can cause.
Wow. This is a huge vulnerability. I hope they fix this very soon. The cognitive dissonance going on with that twitter conversation makes me think he was talking to a bot. Also I love the "These cookies are secure" bit on the admin interface.
I don't think this is anything dangerous. All the data is static, its just some sort of demo. It doesn't matter who goes to the page, they will always get the same data, it never changes. I'm not a customer so can't try once logged in. If I was to wildly speculate, I'd say honeypot.
This is dangerous! Someone has left the debug=true in the config somewhere. Anything could be possible on the site, not just the script injection in the url and the debug page, but a lot of other stuff as well. When the debug flag is true on our sites, we have a link which will authenticate us as an admin without any credentials for example!