

Hologram: taking EC2 Instance Roles everywhere - nnx
http://tech.adroll.com/blog/ops/2014/12/22/announcing-hologram.html

======
ryanfitz
For working with AWS apis, securing access keys, and granting least privilege,
the first step to take is to make completely separate AWS accounts per
environment. Make a Bigco-Prod, Bigco-Test, Bigco-Dev AWS accounts. Developers
only access the Development account, so even if they connect with a highly
privileged key / role and a bug wipes out your services, only the Dev account
is effected and production keeps on humming. AWS makes this extremely easy to
do, they offer consolidated billing to pay one bill for multiple accounts.
Takes no more than 30 min to set this all up and I would recommend this for
even a company with just 1 developer.

~~~
Judson
I'm still not sure why it has taken so long for IAM to support this kind of
configuration, using VPC as the isolation point.

Our use case is running packer[0], which requires a large IAM profile[1],
without needing to create a separate account for AMI creation. It would be
awesome to issue credentials that limited Security Group creation/deletion,
Instance Creation/Termination, etc to a single, VPC.

[0]: [https://www.packer.io/](https://www.packer.io/) [1]:
[https://www.packer.io/docs/builders/amazon.html](https://www.packer.io/docs/builders/amazon.html)

~~~
onedognight
Why not just build using packer's chroot support? No instance creation or
security groups and much faster.

------
sciurus
I like the angle of how this reduces the diff between development and
production. I also agree that AWS keys as managed at many companies are a
security risk. However, there's an alternative to Hologram that is built-in to
IAM and might provide even more security: multi-factor authentication and the
Security Token Service (STS) [0].

MFA requires you to authenticate using both your AWS keys and your MFA device.
In other words, an attacker who gains access to your laptop won't gain access
to AWS. They'd have to get access to your phone or keyfob too.

When writing your IAM policies, you can require that users authenticate with
MFA before they can perfom an action. For example, with a policy of

    
    
        {
          "Version": "2012-10-17",
          "Statement": [{
            "Effect": "Allow",
            "Action": "ec2:Describe*",
            "Resource": "*"
          }]
        }
    

a user can list EC2 instances using their keys. If we add this to the
statement

    
    
            "Condition": {"Null": {"aws:MultiFactorAuthAge": false}}
    

they cannot. Instead, first they have to use their keys and their MFA device
to request temporary keys from Amazon's STS. Then they can use their temporary
keys to list instances until those keys expire, after which they have to
authenticate again to get new ones.

You shouldn't have to modify existing programs to work with STS. Instead, take
out any explicit authorization code, just like you would do to use IAM roles.
Then, wrap invocations of the programs with a tool that talks to STS and
configures your environment. I've written one in ruby [1], and it should be
easy to reimplement if you prefer another language.

Hopefully Hologram can add support for multi-factor authentication in a future
release. Right now, it looks like stealing someone's ssh key is all it would
take to get access to AWS.

[0]
[http://docs.aws.amazon.com/IAM/latest/UserGuide/Using_Managi...](http://docs.aws.amazon.com/IAM/latest/UserGuide/Using_ManagingMFA.html)

[1] [https://github.com/lonelyplanet/aws-
mfa](https://github.com/lonelyplanet/aws-mfa)

------
23david
Interesting. Not sure about the requirement for "an LDAP server with
sshPublicKey attributes on user records to store their SSH public keys."
[https://github.com/AdRoll/hologram](https://github.com/AdRoll/hologram)

I know you can put anything into LDAP, but why treat it like a metadata
service? Wouldn't it make more sense to store keys in something like etcd so
they're accessible via simple REST api calls? In my experience, the less LDAP
does, the happier everyone seems to be.

~~~
epall
Yeah, we connected with LDAP because that's how AdRoll operates, but we'd like
to support a more universal backend eventually. That support just didn't make
the cut for open-source release. With a bit of code, you could easily plug in
a .txt file or a SQLite database, or even etcd if you really don't want to
move a file around.

------
alimoeeny
I was also wondering how popular LDAP is? If you are thinking of using
Hologram, or can benefit from it, would you please comment and mention if you
already use LDAP or some other alternative?

~~~
23david
Definitely don't set up LDAP for hologram... but if you already have LDAP set
up, it could be worthwhile to modify your setup to support hologram.

But then again, if your org is large enough to have LDAP set up, getting LDAP
policies changed to support something like hologram would generally be a non-
starter in my experience.

In large enterprises, the LDAP service is a core security service that is
rarely if ever modified except for periodic maintenance and upgrades.

Messing up the LDAP service for any reason will compromise or disable all
authentication company-wide. It's not fun when the CEO comes over and says
that his wifi and email logins aren't working anymore...

------
mahmoudimus
I built something very similar to this using iptables and a simple flask app.

Not even close to full-featured, but anyway, here's a link for those who are
interested: [https://github.com/balanced-ops/docker-
host/tree/master/iamp...](https://github.com/balanced-ops/docker-
host/tree/master/iamproxy)

I think there's a large gap still to be filled with IAM Roles. Hope to see AWS
invest more resources into this.

------
alimoeeny
Great news, and very good work, thanks for sharing. One thing that I am
concerned about (or have questions about), is how would you make sure you are
not exposing the service to outside, so that when the laptop is on a coffee
shop network others can query the credentials? I mean, it often happens that a
developer needs to expose a webserver they are running to outside to showcase
something or demo or .... It is not very hard to make a mistake and expose
everything right?

~~~
earless1
On the Ec2 side security groups can be used to only allow access for the
company IP. Then VPN can be used to secure connections from coffee houses back
the the office.

~~~
silversupreme
That's what we do at AdRoll, and it seems to work pretty well for us. :)

------
STRML
I love this! We use LDAP in exactly this fashion so this looks like it may be
more-or-less a drop-in addition to how we work. Thanks for such a cool
project!

------
cddotdotslash
I'm really surprised Amazon hasn't developed something like this already. One
of my biggest pain points is creating new keys for every project to use
locally and then making sure I remember to delete them when I'm done. Not to
mention having to recreate them if I do future updates. Really good service
here, though, I'll have to check it out!

~~~
epall
Yeah, when we spoke with Amazon, they already had a couple of labels for what
hologram is, so it sounds like we at AdRoll are not the only AWS customer who
has had to roll it ourselves. Hopefully we'll be the last!

------
hamburglar
This looks very interesting. Any comments on production-worthiness? This isn't
your target scenario, but could it evolve into a component for using IAM-style
credential management for companies running their own servers instead of using
AWS? I'm always interested in new techniques for removing secret keys from
server configs.

~~~
silversupreme
We've been running Hologram in production at AdRoll for about 2 months now
with no issues, and we've eliminated permanent key usage entirely.

We've had a couple of requests internally for a server-oriented agent, and
it's something we want to investigate eventually. I'll make a Github issue for
it and at least we can have a conversation.

