

Bonsai Hosted ElasticSearch Is Down - sstarr
http://status.bonsai.io

======
ihsw
How were they 0wned? Lack of MFA, rogue API key, or something else? Are full-
access accounts being handed out willy-nilly instead of IAM accounts?

AWS Multi-Factor Authentication (MFA):

[http://aws.amazon.com/iam/details/mfa/](http://aws.amazon.com/iam/details/mfa/)

AWS Identity Access and Management (IAM):

[http://docs.aws.amazon.com/IAM/latest/UserGuide/IAMBestPract...](http://docs.aws.amazon.com/IAM/latest/UserGuide/IAMBestPractices.html)

Managing your AWS API Keys:

[http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSG...](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html)

Go a step further with your AWS API keys and use AWS' API access logging
(CloudTrail):

[http://aws.amazon.com/cloudtrail/](http://aws.amazon.com/cloudtrail/)

Don't get burned. Check your stuff out.

~~~
toomuchtodo
At this point, MFA for master AWS accounts should be _mandatory_.

~~~
earless1
MFA for all console accounts is the only right answer. If machines require
credentials to do specific task or perform API calls then roles should be
used.

~~~
toomuchtodo
> If machines require credentials to do specific task or perform API calls
> then roles should be used.

Even then, if the data must be considered highly valuable/immutable, then
versioning/delete protection should be enabled for the S3 bucket(s) in
questions. This requires the MFA token to be _in the API call_ for the delete
to succeed.

------
bdcravens
Full-blown AWS console compromise - this sounds similar to what happened to
Code Spaces
([https://news.ycombinator.com/item?id=7909791](https://news.ycombinator.com/item?id=7909791))
Is there a new vulnerability?

~~~
nzadrozny
It wasn't a management console compromise, it was an old API key that got
leaked.

------
dorfsmay
Any word on how those accounts are getting compromised?

Have they been complacent (easy password to guess, keys easy to be compromised
(maybe in a public github repo)), or could there be some whole in the AWS
secutiy model?

~~~
count
If there were a hole in the AWS security model for this, I think it'd be
pretty obvious pretty quickly, given what happens when US-East takes a dive...

This happens constantly, and it's almost always through lack of best practices
(as mentioned in higher up comment - IAM, MFA, etc.).

------
huntermeyer
This brought my app down. [http://jrdevjobs.com](http://jrdevjobs.com). Our
shards were all missing from Bonsai. We looped through each model and saved
it.

We're back up.

------
huslage
AWS needs to improve usability of IAM so that it gets broader adoption. The
learning curve is non-trivial.

~~~
earless1
What particular issues have you had with IAM? I find that the wizard is a good
starting point for understanding the policies.

------
jayzalowitz
This sucks... I am happy we just put our search cluster on elasticbeanstalk
atm, but I wish we had more services like this running.. good news is new
security practices will hop up everywhere because of this.

------
grandalf
Not to focus on this when they are experiencing downtime, but Bonsai has been
one of the least reliable service providers I've ever used.

~~~
vegardx
Good that it's pretty easy to change elastic search provider with little
downtime. I'd recommend checking out found.no. We've been pleased with
performance and stability. Heck - I can't recall any downtime at all.

------
kitwalker12
our site [http://www.violetgrey.com](http://www.violetgrey.com) went down
because of this. Luckily we were able to reindex pretty fast before their
backups kicked in. Any ideas on how to have fallbacks in such cases?

