
EC2 Instance Connect - DVassallo
https://aws.amazon.com/about-aws/whats-new/2019/06/introducing-amazon-ec2-instance-connect/
======
ranman
I made a little twitter thread here (shameless ploy for followers) (I work for
AWS):
[https://twitter.com/jrhunt/status/1144402767890436096](https://twitter.com/jrhunt/status/1144402767890436096)

* Works on amzn linux 2 - installed by default on newer versions

* otherwise: $ sudo yum install ec2-instance-connect

* The SSH public keys are only available for one-time use for 60 seconds in the instance metadata.

* you can send up your own SSH keys `aws ec2-instance-connect send-ssh-public-key`

* cloudtrail logs connections for auditing

* doesn't support tag based auth but it's on the roadmap

* plans to enable it in popular linux distros in addition to amzn linux 2

Install local client:

$ aws s3 cp s3://ec2-instance-connect/cli/ec2instanceconnectcli-latest.tar.gz
.

$ pip install ec2instanceconnectcli-latest.tar.gz

$ mssh instanceid

~~~
forty
Is there an easy way to integrate that with ansible style scripts (where it
needs to ssh to many instances at once)? Or is it planned?

~~~
jonesetc
should be a pretty simple little connection plugin:
[https://docs.ansible.com/ansible/latest/dev_guide/developing...](https://docs.ansible.com/ansible/latest/dev_guide/developing_plugins.html#connection-
plugins)

------
sandGorgon
About time. This is the other thing that GCP does so well and I was puzzled
that AWS still couldnt do - just add more than one keys to a EC2 instance
through the API
([https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-inst...](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-
connect-methods.html#ec2-instance-connect-connecting-aws-cli)).

There are tons of support questions about "how can I add multiple SSH keys to
my EC2 instances".

Now if only AWS brings in "projects". That's the last usability edge that GCP
has.

~~~
_wmd
Note the GCP equivalent requires some permanently running crapware inside your
VM, the OpenSSH hook EC2 are using is much simpler

~~~
sandGorgon
GCP open sourced the code here -
[https://github.com/GoogleCloudPlatform/compute-image-
package...](https://github.com/GoogleCloudPlatform/compute-image-
packages/blob/14f700ecc1adc0fa7b0e450126065f7f45347d2a/packages/python-google-
compute-engine/google_compute_engine/accounts/accounts_daemon.py)

is the AWS code opensource ?

~~~
cthalupa
It is.

Client: [https://github.com/aws/aws-ec2-instance-connect-
cli](https://github.com/aws/aws-ec2-instance-connect-cli)

Server: [https://github.com/aws/aws-ec2-instance-connect-
config](https://github.com/aws/aws-ec2-instance-connect-config)

------
bdcravens
I’m puzzled. Many of the comments here seem focused on browser-based ssh which
isn’t new, or even the most significant thing here. Using IAM instead instead
of passing around .pem files feels like a huge improvement.

~~~
ilogik
that also isn't new: [https://github.com/widdix/aws-
ec2-ssh](https://github.com/widdix/aws-ec2-ssh)

This will lookup your username in AWS iAM, and if it has the right
permissions, it creates an account and copies the public ssh key associated
with that user.

------
jmb12686
Google Compute Engine has had this functionality for years (at least the
browser based SSH). Furthermore, Google's free Cloud Shell feature is
fantastic.

Over the years, AWS has put their focus entirely on "Enterprise" customer
functionality as opposed to "developer friendly" capabilities.

~~~
peteretep
And that’s why I’m willing to give them my money. Google’s cloud offering
needs to be considered a hobby like all their non-search activity

~~~
heybrendan
Huh? A hobby?

Transcript from Alphabet Q1 2019 Earnings Call; April 29, 2019 [1]

> Sundar Pichai, CEO Google:

> We are also deeply committed to becoming the most customer-centric cloud
> provider for enterprise customers, and making it easier for companies to do
> business with us thanks to new contracting, pricing, and more. Today, 9 of
> the world's 10 largest media companies, 7 of the 10 largest retailers, and
> more than half of the 10 largest companies in manufacturing, financial
> services, communications, and software use Google Cloud.

> Some of the companies that we announced at Next included: The American
> Cancer Society and McKesson in Healthcare; Media and Entertainment companies
> like USA Today and Viacom; Consumer Packaged Goods brands like Unilever;
> Manufacturing and Industrial companies like Samsung and UPS; and Public
> Sector organizations like Australia Post.

> Finally, to support our customers' growth, we also announced the addition of
> two new Cloud regions in Seoul and Salt Lake City, which we plan to open in
> 2020. These new Cloud regions will build on our current footprint of 19
> Cloud regions and 58 data centers around the world.

This doesn't seem like just a hobby to Google.

[1]
[https://abc.xyz/investor/static/pdf/2019_Q1_Earnings_Transcr...](https://abc.xyz/investor/static/pdf/2019_Q1_Earnings_Transcript.pdf)

~~~
peteretep
Meh, I care about actions and not random words

~~~
9nGQluzmnq3M
> our current footprint of 19 Cloud regions and 58 data centers around the
> world

Do you understand how expensive this action is?

~~~
toast0
How many of those data centers were already there for their advertising
business?

------
jtwaleson
This is bad news for ScaleFT, which provided this service via bastion servers
(although not IAM based).

Rackspace managed AWS environments use this for high compliance systems.

The problems it solves are a) that login attempts are logged on a separate
system for compliance and b) user management is handled in a centralized way.
Both are handled with EC2 Instance Connect.

~~~
forty
Another competitor to this is hashicorp vault, which does both certificate
based ssh and AWS authentication. (I find the certificate based approach
better though)

------
cesnja
There's also SSM Session Manager [1]. Not exactly ssh, but you get mostly the
same features with ssh access completely disabled and the whole session being
logged to S3 bucket or some log aggregation service.

[1] [https://docs.aws.amazon.com/systems-
manager/latest/userguide...](https://docs.aws.amazon.com/systems-
manager/latest/userguide/session-manager.html)

------
rukenshia
I'm a bit confused about this one.. we have been using SSM Session Manager for
quite some time now and this looks like it does the same. We also export all
logs during the session with SSM and you can see which user initiated the
session. What am I missing here?

~~~
gauravphoenix
SSM session manager is basically a HTTP wrapper over a shell. You have to use
browser for SSM which mostly works until it doesn't. I had trouble sometimes
copy pasting to it.

This new service is basically a managed SSH so things like port forwarding etc
will work. With SSM you can't do port forwarding etc because it is not SSH
aware.

~~~
CSDude
But this needs to expose SSH? SSM is great (although its not fast enough)
because it eliminates our jumpers

~~~
gauravphoenix
Yeah but then a lot of people have use cases for SSH. The solution is targeted
towards replacing jump boxes.

------
crankylinuxuser
TBH I'm still waiting for console serial access via IAM. All my bare metal
machines have that. And it's absolutely essential when you bork networking.

And as much it pains me to say it, Azure has that feature.

~~~
cthalupa
I really really really try to not need serial console access to my machines. I
try to only rarely need SSH access.

But when you've got some sort of bug or issue that you're not getting any
metrics out of, no logs recorded, no kernel crash dump, nothing sent over
netconsole, nothing showing up on the instance console screenshot... Sometimes
serial console is what you need.

But, for the borked networking case, I'd recommend not modifying your
networking on live instances. Make your changes on a test instance, figure out
what works, and add it to your configuration management ;)

------
different_sort
I'm honestly a little disappointed here, I feel like there is not fully baked
but it is so close.

Unlike SSM, Instance connect goes direct over SSH - so you either need to be
inside of your AWS network, on a bastion host that can route to your AWS
network, or use a public IP address.

It would be great if they combined this functionality with the HTTP wrapping
capability so that I do not need to expose SSH/route to SSH ports in any way
but can also use IAM policy to control which unix user a given IAM principal
can land in the host as (Example use case would be I would only want a certain
class of user to land as a user with sudo/root access).

This is still valuable to my use case, and we'll go ahead with it using the
bastion approach most likely until they hopefully integrate this with their
HTTP SSH wrapper.

------
unixhero
Great new feature!

But _sigh_ I just built a PKI infrastructure provisioning system using a
gigantic shell script, maintenance user with sudo permissions and ssh access
where a master node would command a fleet of slave nodes.

I guess all of my work was for naught since this seems to cover some my needs
for user and ssh key provisioning.

Oh well, it'll work elsewhere on all other clouds. And I guess I should
release it publicly, it's just not pretty enough yet. Every time I do,
gremlins come out of the bushes complaining that the code isn't elegant enough
for them.

~~~
serpix
This happens so often in Devops that a good rule of thumb is to really think
it long and hard before doing any handrolling.

Today's dev ops stacks move so fast nobody is an expert on a single stack for
longer than a week.

------
jimktrains2
Do you still need to create users manually on each machine? There have also
been many tools out there to pull the ask key from IAM and use it via
authorizedkeyscommand previously, but my problem is always creating the user
accounts, especially if you don't want to keep a separate list in
ldap/Kerberos (or similar, like active directory).

~~~
ricksebak
Looks to me like this would have all users use the default ec2-user or ubuntu
user accounts.

~~~
jimktrains2
That's what I thought it was saying too. That's a mess from a compliance and
best-practices point of view :(

Or am I missing something and this would follow the PCI DSS?

~~~
lenova
This is what I'm wondering as well. Does the fact that everything is logged by
what an IAM user does work as compliance, or are individual user accounts on
the operating system still required?

------
the_duke
Well, it was about time...

IAM based auth was long overdue.

------
Pulletwee12549
I hope this helps all those people that somehow manage to lock themselves out
of their instances.

------
ravedave5
This is great! Giant step using iam over ssh keys.

------
SteveNuts
Not complaining, but wow that took a long time. Even Linode has had a web-
based shell for many years.

Either way, this will be very nice to have.

~~~
dbaggerman
AWS already had a web based shell via SSM. This allows native SSH connections
using credentials tied to AWS IAM users.

------
fortran77
Interesting, but I connect via ssh from Windows 10 PowerShell. I wonder why
this isn't a standard use-case. I suppose I can get it to work as long as it's
"openssh" or something compatible

------
itzsasi
how to connect the instance from a windows machine using ec2-instance-connect?
is that possible?

------
msoad
is this also xterm.js based? the WebGL based engine will make it even better!

~~~
dahfizz
Not gunna lie, it makes me sad that we need some huge, fancy graphics engine
just to emulate a 25 year old technology. Why are people so obsessed with
their browser? No matter how much JS you layer on, it'll never be as fast as a
terminal.

~~~
argd678
I had a VT102 for many years, it wasn’t fast.

------
peterwwillis
I mean, this is definitely cool, but we should also try to stop using ssh so
much. There's a long list of reasons why using ssh leads to bad things (but
not an anti-pattern - I wish people would stop using that phrase to mean
anything that _sometimes_ leads to bad things) so I just hope this
functionality doesn't exacerbate its use.

~~~
tomcam
Ugh, I'm a new full stack guy. Could you enlighten me about the evils of SSH?

~~~
peterwwillis
SSH is wonderful. The problems are more what it enables, and what it lacks (or
isn't designed to do).

For example, managing ssh keys for an individual is gloriously simple, but
managing them for a large organization is a huge headache. You want to use ssh
certificates, but even those are implemented in a weird way, and really you
should use an SSO system for auth. (This makes that easier/better, so, yay?)

When people start sshing into production servers, they end up making local
changes. They focus more on the "pets" aspect of managing systems rather than
as "cattle". They have to install a litany of extra software to diagnose and
troubleshoot bugs, rather than expose system metrics and tightly control the
app environment and its operation.

Remote access to production app servers is basically a backdoor waiting to
happen, and may violate corporate security policies. When you have local user
access to a Linux host, it's almost guaranteed you can privesc to root.

Finally, almost everyone I have ever seen will either force-ignore/auto-accept
host key changes, or just accept them blindly, because IPs and hostnames may
change, and there may be multiple environments you haven't logged in to, etc).
This completely defeats the purpose behind mitm protection, which is the main
_intent_ of using SSH, though these days its other features may be arguably
more of a reason to use it.

And for the tech hipsters out there: "it isn't serverless!!"

~~~
cthalupa
>They have to install a litany of extra software to diagnose and troubleshoot
bugs, rather than expose system metrics and tightly control the app
environment and its operation.

There's a lot of things that require more than easily exported system metrics
and logs to troubleshoot.

While I've played around with using PCP's perf plugin to try and remotely do
things with perf, generate flamegraphs, etc., it doesn't work nearly as well
as just SSH'ing into the thing and running perf directly, especially if the
perf data file is going to be large. I don't see how you could do serious
performance engineering work without SSH access.

But, I think I'm nitpicking here, because I generally agree that there should
be very little to no reason to login to servers via SSH day to day.

~~~
jameshart
in the ‘pets vs cattle’ analogy, it totally makes sense that even with cattle,
occasionally you bring one in for a checkup by a vet to see if you can detect
any problems that might affect the herd. Ssh into a production box to check
everything is working as expected and take some readings. Sure.

On the other hand, I tend to lean more towards a ‘wild animals’ model, where,
sure, you can tranquilize one and bring it in to look over, but once it’s got
the smell of humans on it, it’s doomed if you let it back out in the wild
again.

Once you ssh into a production box, it is forever tainted. Sure, poke around
in it, install some perf tools to run some diagnostics, learn what you can
about its behavior. But then, rather than putting it back into the wild to
serve traffic, out of mercy, you should destroy it and replace it with a clean
instance.

~~~
cthalupa
I don't disagree with this, but, at the same time, if I haven't made any
actual changes to the application, I'll generally not worry about manually
taking it out of service, because autoscaling will be getting rid of it soon
enough anyway.

