
Segment Security Incident - sandis
https://segment.com/security/bulletins/incident090519/
======
erichurkman
> For a small subset of customers (13), the unauthorized party was able to
> gain read-only access to their workspaces and click around in their accounts
> for up to a few minutes. These customers have been notified.

That implies that Segment employees, or a subset, have unfettered access to
view their customers' accounts? That it didn't require positive customer
assent to gain access?

I like Box's model for this: [https://community.box.com/t5/Working-with-Box-
Support/Box-Pr...](https://community.box.com/t5/Working-with-Box-Support/Box-
Product-Support/ta-p/20124#grantaccess)

~~~
pouta
Only 13 customers affected? I guess I won the haveIbeenpwnd lottery today.

Here is the email I got:

" We are writing to notify you of a security incident that happened between
August 26 and August 31, 2019. We became aware of it on August 31, resolved it
immediately, and have been assessing the scope of impact since then. Based on
that ongoing assessment, we have concluded that the incident involved your
workspace. We are very sorry for any issues this may cause.

What happened? Between August 26 and August 31, 2019, an unauthorized party
compromised a Segment employee’s account and used it to gain unauthorized
access to Segment product usage data. Upon detection, we took immediate
action, disabling and deleting the account and removing unauthorized access.
We also reported the incident to law enforcement.

What information was involved? The unauthorized party accessed information
about how Segment users interact with the Segment product, as well as first
name, last name, email address, and IP address for your Segment users, and the
Segment write keys for your workspace. No Segment customer passwords were
compromised.

No personally identifiable information relating to your own customers was
accessed. As a result, no action is required from you at this time.

More information For more information, please visit oursecurity bulletin page.
This page contains a detailed timeline about what happened and information
about what actions Segment has taken in response. It will be updated with any
new developments.

If you have any further questions, please reach out to support@segment.com. We
again apologize for any inconveniences this incident may cause.

Sincerely, Coleen Coolidge Chief Information Security Officer"

~~~
sp332
That message doesn't say the attacker got access to your workspace or clicked
around in it.

~~~
pouta
From above:

"Based on that ongoing assessment, we have concluded that the incident
involved your workspace."

------
zaroth
In all the SaaS type applications I’ve ever developed, the privileged “Admin
Panel” functionality is always on a private network accessible only via VPN
access, which is authenticated via a password + private key. Then the Admin
Panel itself has its own separate login, which is username + password.

Ideally the private key for the VPN is a hardware token, but I will admit in
most cases it’s simply a file on the drive.

The Admin Panel has zero third party JavaScript nor client-side analytics.

I’m not sure this buys quite as much security as it might appear, because
there’s probably any number of ways to piggyback on a valid session.

For example, there was a bug bounty Tesla paid out when a researcher was able
to establish XSS by mangling their car name, which was read via the API and
showed up in an internal dashboard as unescaped HTML.

So I think the biggest attack surface for Admin Panels remains a hostile
client seeding XSS into unescaped fields, and CSP helps a great deal with
this, but at the very least I don’t see any reason why these panels should be
on a public URL.

You can say, well, they should have MFA on the panel, but I can’t shake the
feeling that these simple measures avoid a huge number of attackers looking
for low-hanging fruit.

It’s like putting SSH on a non-standard port. It’s theoretically meaningless
but practically it’s a huge improvement, if only because it helps attack
signals stand out against the script kiddie noise.

~~~
vajenetehais
Could you elaborate about the architecture? I'm really interested in learning
this.

At which level the interface between admin panel and apps is? Is it a
dedicated app for the admin panel and just a front end to an AD or database?
Like there is no real admin panel on the app itself.

~~~
zaroth
Yes, the user/client facing app is a separate website from the admin panel
site.

They share much of the same underlying data access and business logic code,
but the screens that let you see all the clients, change SKUs, monitor
billing, impersonate a user, generate reports, etc. just would not exist on
the public-facing site.

Underneath both sites/apps are talking to the same database(s).

------
nrmitchi
Between this and CircleCI, this sounds like a targeted credential-stuffing
attack against accounts on Segment. If this is the case, it sounds like the
only two cases that have been detected are Segment (detected on August 31st
through unknown means), and CircleCI (detected on August 31st through an
automated email).

Has the risk of of other Segment accounts having been compromised through the
same channel (but have yet to be detected) been investigated?

~~~
seanieb
Was it Segment that CircleCI was referring to?
[https://support.circleci.com/hc/en-
us/articles/360034852194-...](https://support.circleci.com/hc/en-
us/articles/360034852194-Security-Incident-on-8-31-2019-Details-and-FAQs-)

I hope not, since there's a 5 day gap between CircleCI being notified and the
rest of their customers.

~~~
nrmitchi
Based on the wording, it's my belief that CircleCI was referring to Segment,
but I have 0 inside information to confirm this.

Even if it was though, Circle discovered this issue on their own through an
automated notification of an action taken; they do not seem to have been
notified directly by Segment. It doesn't seem fair to assume that _if_ Circle
is referring to Segment, that either company did anything wrong in their
response here.

------
teamspirit
> Removed privileged access controls internally, adding accounts on an as-
> needed basis

I believe this is critical. Access controls really should be set as needed,
per user, and fine grain. While it can be time consuming and may not even make
sense at the beginning of a project, doing everything this way from the start
should help to keep an organization better secured in the long run.

------
eranation
"Enforced mandatory Multi-Factor Authentication (MFA) for all employees when
accessing Segment-owned workspaces and performing administrative actions in
the Segment app."

This should always be step #1, don't wait for an incident.

~~~
tptacek
Really, what step #1 should be is getting all these applications behind an
SSO/IDP, and then using the policy controls in the IDP to enforce MFA for
users.

We've surveyed startup dir/security's (and the like) and this is almost
universally in everyone's top 3, and the leading contender for #1.

~~~
eranation
Definitely better, I 100% agree.

------
superzadeh
It is a bit strange, they mention that write keys have been accessed, but they
don't seem to have invalidated them, or even recommend to update them?

    
    
      Information about how Segment customers interact with different aspects of our product, 
      *including customer write keys for Segment*, integration names, workspace names, 
      and how customers interact with our user interface.

~~~
bradleybuda
They address this in the post: write keys are considered public (after all,
they appear in JS code on customers’ sites).

------
sunaden
I'm just awaiting the conspiracy theories including both CircleCI and Segment

~~~
yen223
CircleCI claims their leak happened through a “third-party analytics vendor“.
Perhaps that third party was Segment?

------
umvi
> we took immediate action, disabling and deleting the account that was
> compromised

Since this was a privileged account, how can they be sure the attacker didn't
use said account to setup more ways to get back in? That's the first thing
Kevin Mitnick always did when he pwned a box: setup alternative routes to get
in, in case his original door got closed.

~~~
throw777luck
There is a saying from pentesters about shells: "Two is one and one is none"

~~~
moloch
Always hack with a safety shell!

------
dmix
> This data is used by our product, marketing, and customer success teams

It always seems to be the marketing analytics data that's wide open for
'hackers'.

You could spend all day building a secure DB and application architecture then
have the marketing team upload analytics for everything onto some random
insecure service.

Maybe the marketing/data teams need to get some security lessons as part of
their training the way programmers learn?

~~~
anonymousjunior
Segment actually does a pretty fantastic job onboarding new personnel and
funneling them through security training. Recently gave a great talk here at
the Bay Area OWASP meetup about how they've gamified security awareness
training with an internal leaderboard and random CTF challenges.

------
Nextgrid
This is why I’m against third-party analytics in any app/service. Someone
compromising the analytics provider can get a lot of data from app end-users
even across different apps that use the same analytics service.

------
zer0faith
This is why you NEVER expose elevated access to the open internet.

Why all the down votes... this is a legitimate security issue.

~~~
ejcx
It's not rooted in reality and a gross oversimplification of the problem.
GSuite is exposed to the open internet. Dropbox is exposed to the open
internet. All of these services have permission models with "elevated access".

1) Your idea has been shown not to work (it's called perimeter security) 2)
With the huge shift to cloud services you need a new solution anyways

------
wildtomato
Looks like the whole site is down. Anyone save a copy?

~~~
jand
Night it be that you employ some ad blocking on network level?

At least my adblocker (browser) did not like what it saw and blocked the
article.

------
mindfulplay
Seeing way too many "incidents" these days ....

I would like at least one company to post an "incident" reveal in a more
honest way:

" Due to our carelessness and relatively insecure practices, we had improperly
disclosed user accounts to a moderately savvy hacker. We realize this is our
fault.

If you'd like to help and given that we have your attention now, it would be
valuable if you can help pentest our servers: the attacker used a simple SQL
attack based on an unpatched server via CVE-3245. Are we missing anything
else? Please let us know.

Thank you."

~~~
umvi
Yeah except in this case an employee account was likely compromised by
spearphishing/social engineering/(or worst case keylogger). That can be _very_
hard to defend against.

Good security is _not_ easy, and _not always_ due to "carelessness".

It's an expensive, onerous, never ending, and _ever evolving_ process to get
right. Most, if not all, companies do the bare minimum security they believe
is necessary; anything beyond that is a waste of money and computing resources
(if you believe otherwise, I have some retina scanners to sell you...)

~~~
zerkten
> Most, if not all, companies do the bare minimum security they believe is
> necessary; anything beyond that is a waste of money and computing resources

This why we continue to have incidents and vulnerabilities which could have
been prevented, or better mitigated. Most often these companies do not even
know how to make a correct assessment of their risk. They move forward with
this idea that it's a waste of money and resources, yet waste everyone's time
with clean-up, or just go out of business as a result. Even with minimal
security training and limited curiosity, this incident could have been
avoided.

~~~
sgarman
Customers don't want to pay for it. You can easily run yourself out of
business building a more secure system. We need to get people and customers to
care first to make the economics work.

~~~
zerkten
Plenty of these measures are just basic professionalism. Some are relatively
inexpensive (enabling MFA everywhere by default given the number of MFA
options.)

Other changes are mildly annoying to developers, ops, and support (e.g. re-
requesting production access.) Since developers hold sway in most
organizations, convenience often trumps security. None of these measures put
anyone out of business.

If I had to attack something I'd go for the limited resources to help smaller
organizations scale security appropriately. There are tons of resources for
large dev teams, infosec specialists, etc., but there is very little that
targets small organizations effectively.

