
Show HN: Managing SSH Access to AWS EC2 Instances Using SSM - elpy1
https://github.com/elpy1/ssh-over-ssm
======
leg100
Seem likes an awfully laboured equivalent of GCP's OSLogin [1]. To SSH into an
instance:

    
    
      gcloud compute ssh <instance_name>
    

GCP takes care of generating a key pair. The GCP linux images configure
OpenSSH and PAM to check you have relevant IAM permissions. IAP [2] is used in
place of a bastion if the instance doesn't have an external IP address.

[1]: [https://cloud.google.com/compute/docs/instances/managing-
ins...](https://cloud.google.com/compute/docs/instances/managing-instance-
access)

[2]: [https://cloud.google.com/iap/docs/using-tcp-
forwarding](https://cloud.google.com/iap/docs/using-tcp-forwarding)

~~~
nostrebored
Session manager doesn't require inbound access iirc. An agent on the machine
handles the request via a connection to the SSM service. Your request for a
session goes to the SSM service and a websocket connection is established from
the machine you need to remote into. A cool solution.

~~~
dkersten
I assume it also logs the access in the SSM audit logs.

------
vegardx
You can also use ec2-instance-connect[0] to enable users to generate ephemeral
keys that are valid for a few minutes, authenticated with IAM. The benefit is
that it's all native after you've pushed your keys using either aws-cli or
mssh.

It has more or less the same benefits that SSM has, and you can use the same
method with ProxyCommand to establish authentication before connecting. You
can also chain it using ProxyJump to use a bastion host. Wrapped in a tool
like aws-vault you can quite easily work with 2FA and such.

[0] [https://aws.amazon.com/about-aws/whats-
new/2019/06/introduci...](https://aws.amazon.com/about-aws/whats-
new/2019/06/introducing-amazon-ec2-instance-connect/)

~~~
marcinzm
Does ec2-instance-connect use the generic system account or does each AIM
user/role get it's own instance account? Asking from an auditing and logging
perspective, as in is there a way to tie back a command which was run back to
an individual human directly?

~~~
vegardx
It has to use an existing system account, yes. You do lose a lot of auditing
abilities this way, since you cannot capture the content. For me this is more
about establishing a secure way to grant people access to instances, without
something like LDAP or SSH CA.

LDAP sends passwords in cleartext over an encrypted channel, I don't like the
idea of someone else on the host I'm connecting to being able to read my
password. Naturally you can use key based authentication with it.

I know that some people are required by law or regulation to log all these
things, but personally I don't see much benefit of such audit logs. All bets
are off when someone has local access to a machine. And there's this whole
debate about using MitM-proxies for logging and security and the potential
security implications they pose.

All that said, step-ca[0] and vault[1] looks really promising to provide some
kind of provider independent way to authenticate in a secure manner. You also
have teleport[2] but I haven't looked much into it.

[0] [https://smallstep.com/docs/cli/ssh/](https://smallstep.com/docs/cli/ssh/)
[1] [https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-
cert...](https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-
certificates/) [2]
[https://gravitational.com/teleport/](https://gravitational.com/teleport/)

~~~
marcinzm
Thanks for the links, Teleport seems especially interesting.

Yeah, we are required by contractual obligations to log these things or at
least sudo commands. Don't disagree that it's less than useful but only so
many things you can argue about. The logging we can do with auditd but we need
unique instance users for that to be useful.

~~~
pquerna
(disclaimer, I was a founder of ScaleFT - acquired by Okta)

This is exactly why many customers use Okta's Advanced Server Access:
[https://www.okta.com/products/advanced-server-
access/](https://www.okta.com/products/advanced-server-access/)

It does certificates as a credentialing mechanism, but also full separate
accounts, lifecycle management of accounts and sudo files.

------
jbergknoff
We've been using SSH over SSM at work, and it's been great for tying access to
IAM credentials.

Using the SSM tool as a ProxyCommand for OpenSSH is brilliant. I can SSH to an
instance (turning on agent forwarding, etc.) or SCP files back and forth, all
the same way that I'd do it with a regular host. But that host doesn't need to
be exposed to inbound SSH traffic!

I was also able to get the ProxyCommand working in a Docker container, so I
don't have to install the AWS CLI or the SSM plugin on my (Linux) host system,
with an .ssh/config entry like this:

    
    
      Host i-*
        ProxyCommand docker run -i --rm -v $PWD:$PWD -w $PWD -u 1000:1000 -v ~/.aws:/tmp/.aws -e HOME=/tmp -e AWS_PROFILE <docker image ref> ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'
    

A few problems in practice:

* We often use our bastions to do port forwarding, e.g. to connect to an RDS instance inside VPC. As far as I know, we can't accomplish this with just SSH over SSM, so we still keep our bastions.

* User management: SSH over SSM doesn't do any user or key management for you (which makes sense). It establishes the SSH connection and you're on your own from there. We use Keymaker [1] to dynamically create user accounts on the EC2 instances, and populate SSH keys according to the user's IAM profile. This works fine, but maybe there's room for simplification.

* Connecting based on instance ids (`ssh i-...`) can be awkward. It would be amazing if we could alias these somehow, like set up a CNAME that points to `i-...`. I wasn't able to get OpenSSH to follow a CNAME and then use the ProxyCommand. Maybe it's possible, though.

[1] [https://github.com/kislyuk/keymaker](https://github.com/kislyuk/keymaker)

~~~
ReidZB
I haven't used it or looked into what it supports, but I did remember this
announcement which may interest you: "New – Port Forwarding Using AWS System
Manager Session Manager": [https://aws.amazon.com/blogs/aws/new-port-
forwarding-using-a...](https://aws.amazon.com/blogs/aws/new-port-forwarding-
using-aws-system-manager-sessions-manager/)

To make the ProxyCommand setup a little easier, perhaps you could use `aws ec2
describe-instances` with the appropriate filter? For example, the article I
linked uses:

    
    
        INSTANCE_ID=$(aws ec2 describe-instances \
                       --filter "Name=tag:Name,Values=$INSTANCE_NAME" \
                       --query "Reservations[].Instances[?State.Name == 'running'].InstanceId[]" \
                       --output text)
    

to grab the instance ID from the Name tag. You could set up a shell script as
a ProxyCommand to do this automatically, although beware that ProxyCommand's
%h will always lowercase the input...

------
sakisv
It's hard to overstate how much of a difference this makes for managing
people's access to servers without having to manage access keys. Especially if
someone leaves the team/company you don't need to roll out new AMIs or run
your puppet/ansible, it's just a `terraform apply` to update the IAM policy
and you're done. Absolutely love it.

One thing that I found annoying through the web interface was the lousy
support of special characters. Every now and then either some character
combination or a big wall of text would make the browser-based terminal almost
useless. I have yet to make time to configure the cli access which I don't
think will have these issues.

------
williamdclt
Very funny, we received a ticket from a security audit a week ago asking us to
get rid of our bastion and use SSM instead. Having never heard of SSM, being
in a rush to a deadline and lacking skills with ops, we kinda panicked. 10min
later, we had found the SSM interface in the AWS console, found it extremely
easy to use and had deleted our bastion :)

Note: the README says "I found myself in a situation where I couldn't easily
copy a file from my machine to the server". Note that you can use S3 for that
(`aws s3 cp ...`), though you should be careful with permissions on your S3
bucket

~~~
vegardx
You can always just pipe things over ssh. And now that ssm supports local port
forwarding you can easily use netcat to send files back and forth.

------
klohto
Solving the SSH over SSM is sadly useless right now. The whole point of why we
began using SSM was logging.

1) We soon discovered that using bash (and not sh that SSM signs you into)
doesn’t format the logs correctly and makes them unreadable 2) Using SSH over
SSM doesn’t log at all

Both of these problems are on AWS github but as always, they don’t respond.

~~~
marcinzm
Does SSM generate unique users for each login or does it use a generic user?
The later would make any logging outside of SSM difficult since there's no way
to tie it to a specific IAM user except time windows.

~~~
klohto
You can specify the behavior. There is no unique user for each login, by
default you are dropped to generic ssm-user. It's possible to set up a log-in
into specific user based on the IAM username.

> The later would make any logging outside of SSM difficult There is a way to
> tie IAM user to SSM. You have to request SSM permissions and AWS logs that.
> AWS logs pretty much every single API call so it's just matter of storing
> and auditing correctly.

~~~
marcinzm
>There is a way to tie IAM user to SSM. You have to request SSM permissions
and AWS logs that. AWS logs pretty much every single API call so it's just
matter of storing and auditing correctly.

If you're using a generic ssm-user (and going to ssh so there's no ssm logs of
the commands) then wouldn't you not be able to distinguish the commands (say
in auditd) of two people logging in at the same time?

I'm reminded of RDS which does not store enough logs anywhere to reconstruct
which IAM user/role was running specific commands when using IAM
Authentication unless you create individual users in the DB.

~~~
klohto
> If you're using a generic ssm-user (and going to ssh so there's no ssm logs
> of the commands) then wouldn't you not be able to distinguish the commands
> (say in auditd) of two people logging in at the same time?

But that requires auditing directly on EC2, right? (You can indeed sync the
logs somewhere, but that diminished the beauty of out of box logging)

> I'm reminded of RDS which does not store enough logs

AFAIK, it's not possible with their current implementation to accomplish this.
We just swallowed that it won't work and for the sake of security created
seperate users.

------
jon918
This is an example we've been working that creates an Okta-managed user who
can get in to a tagged EC2 instance using Session Manager. No
bastion/sshd/security group ingress rules required.
[https://github.com/symopsio/terraform-okta-ssm-
demo](https://github.com/symopsio/terraform-okta-ssm-demo)

------
vermon
I've been using this script for a while and it works great
[https://gist.githubusercontent.com/qoomon/fcf2c85194c55aee34...](https://gist.githubusercontent.com/qoomon/fcf2c85194c55aee34b78ddcaa9e83a1/raw/59286cadad054018012fe22ed23379729cb893a9/aws-
ssm-ec2-proxy-command.sh)

------
crucialfelix
Still need to figure out how to do this with an ephemeral fargate task. I'm
guessing that I can build an image that adds SSM.

Otherwise my developers can't get a Django shell in production for
emergencies.

------
tilolebo
Hasn't this been solved by using SSH certificates instead of keys?

~~~
marcinzm
SSM doesn't require you to manage keys or have a bastion/vpn/public access.
It'll also log access attempts, in some cases log commands, and ties into IAM
permissions (ie: these people can't access prod but can access test, etc.).

------
booi
This is great. We usually overload the bastion server with NAT duties but SSM
+ hosted NAT makes bastion servers a thing of the past?

------
BrandonBradley
I've been rolling SSM out at work; 10/10 would do again.

------
m1keil
that's pretty great. been using ssm for a while but wasn't aware of AWS-
StartSSHSession, which solves all the quirks I had with ssm in the first place
(like loosing the abilities of scp, tunneling and such).

------
billman
Can anyone give any experience using this method vs. AWS Secrets Manager?

------
wheelerwj
thanks!

