
Golang SSH Security - transitorykris
https://bridge.grumpy-troll.org/2017/04/golang-ssh-security/
======
eropple
I am...not a fan of Golang, as I have made pretty clear around here on
occasion. But I'll give credit where credit's due, and this is a good decision
on the part of the people maintaining x/crypto/ssh. Not the tooling vendor's
awful response--I'm pretty sure I know who it is, and if not there's two of
'em because I've had these conversations before--but the maintainers are doing
the right thing. This probably shouldn't have gotten out the door without host
key verification in the _first_ place (and that ties back into the reasons why
I do not like or trust Golang or its community when it comes to tools that I
have to consume), but it's better to bite the bullet and fix this _now_
instead of letting it fester.

(The "PGP is too hard for discussing security issues" thing, though, is total
nonsense. Can't be doing that.)

~~~
grey-area
Doesn't the writer call out hashicorp specifically by name in the article?

 _I used to respect this vendor and recommend their tools to others. I am
having to rethink this a lot. I am no longer happy to recommend Hashicorp
products to others._

~~~
eropple
Oh, ha, I glazed over that paragraph! But yeah, it was totally Hashicorp I was
thinking about; this sort of reliability-thoroughly-optional thing shoots
through a lot of their tools and keeps me pretty far away from any of the ones
that might touch live environments. (Packer and Vagrant are fine.)

~~~
RRRA
Which one of their tools are we talking about here? Terraform?

~~~
syscomet
blog-post author here: all of them written in Golang which use SSH.

Packer, Terraform, Vault, all of them.

Using a bastion host in your configurations doesn't help if you invoke the
tooling on your laptop, since the connection to the bastion host is done with
the SSH package, again with no host-key verification.

~~~
Piphe
And Vault is supposed to be a super secure root of trust. Hmm...

~~~
canes123456
Damn, I was planning on using Vault eventually. This attitude towards security
gives me pause about put all my secrets into it. Is there any alternatives?

------
avar
Oh man:

    
    
        > I am bemused by an approach to accepting
        > security reports which is to go through the
        > motions of having PGP public keys available
        > for people to use to report issues but upon
        > receiving such a request ask for it to be
        > submitted without PGP because digging out
        > the keys is too much of a hassle.

~~~
bradfitz
I was the one who replied with that, asking him to resend without PGP, for two
reasons:

1) both of our MTAs do STARTTLS. And gmail is only TLS.

2) 99% of the PGP-encrypted emails we get to security@golang.org are bogus
security reports. Whereas "cleartext" security reports are only about 5-10%
bogus. Getting a PGP-encrypted email to security@golang.org has basically
become a reliable signal that the report is going to be bogus, so I stopped
caring about spending the 5 minutes decrypting the damn thing (logging in to
the key server to get the key, remembering how to use gpg). But I recognized
him as a knowledgeable person from the Internet, and I knew (1), so I just
asked him to send without PGP to save me 5 minutes.

Even if I'd used PGP, I would've just replied cleartext anyway to our
security@golang.org list, except all the MIME would've been garbled and
unreadable.

In summary, the PGP tooling sucks (especially in gmail, but really everywhere)
and it's too often used by people who are more interested in using PGP than
reporting valid security issues.

~~~
scrollaway
> _I recognized him as a knowledgeable person from the Internet, and I knew
> (1), so I just asked him to send without PGP to save me 5 minutes._

I understand where you're coming from but this bit of logic I don't get.
Especially if you recognized him, then clearly it's _worth_ spending the five
minutes.

The five minutes you personally save, someone else has to pay them. The person
who originally sent you the email now has to read the response, process it,
send you another non-encrypted email and/or enter into a meta-debate with you.
They end up mentioning it on their blog and now we're all having this meta-
meta-chat about it on hacker news, wasting several man-hours on it.

Was it worth it?

~~~
bradfitz
I'd say it was worth it. This thread was more fun than using gpg, even if the
time savings is a wash.

------
3pt14159
I love Digital Ocean, but they do the same thing with their API. I wrote to
them about it years ago, even talked to some developers there, and the general
explanation is the same: Screw around with cloud-init to get the public key.

If you use the DO API to provision servers my feature request is here:

[https://digitalocean.uservoice.com/forums/136585-digitalocea...](https://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/9307569-return-
the-droplet-s-ssh-public-key-as-part-of-api)

Please upvote it or at the very least copy the cloud-init script to help
provision your servers.

~~~
problems
For hostkeys on DO you can probably get a script to run that'll request a
signed certificate from a server you own. The signed cert can be validated
fully by clients.

Or go with a convergence style system and probe it from multiple locations.

Or just give up and go with TOFU - if you never get an error even on different
connections, you probably haven't been mitm'd.

~~~
DorothySim
> For hostkeys on DO you can probably get a script to run that'll request a
> signed certificate from a server you own.

Or just embed the signed host certificate in cloud-init.

------
mbertschler
While I really love the stable nature of Go and its standard library, I am
happy that this breaking change was put out there in the interest of security.

This issue hit me while building a tool for internal use at my employer. I am
using the glide vendoring manager for this project, added another dependency
which triggered an update of all other dependencies. At that point my tool
broke and forced me to actually think about host key verification.

~~~
TheDong
This isn't the standard library though, and if it were then it wouldn't have
been changed.

~~~
syscomet
It's a special case though. The golang.org/x/ packages are experimental but
also candidates for promotion to the standard library. Eg, "context".

But this sort of issue is exactly the sort of real-world review and hardening
which justifies having a namespace for stuff to go _before_ it becomes stdlib.

~~~
kardianos
No, x repos are just eXtra.

/x/exp is expiramental.

------
adtac
I'm really impressed with the quick response from the golang team. The fact
that they didn't mind introducing breaking changes shows that their priorities
are right.

------
YZF
It's amazing how many people out there consider MITM as something they don't
have to defend against. If you're a developer you have to assume your system
will be MITMed. It doesn't matter if you're on the Internet or behind a
firewall. Trust on first use is not a good solution because someone can tailor
their attack against that first use.

~~~
ge96
What are some clear signs that this is happening to you? I track the url
requested on my server(s) and some of them don't make sense/looking for
exploits like wordpress login exploits. I don't know I have SSL/A+ according
to Qualys, I'm kind of drawing a blank where MITM happens. I've seen/read
about it before. Heartbleed? No I don't know... really tired, but interested,
gots to Google.

edit: I'm still on LAMP stack for clarification. Too bad to hear about Golang
though I'm still looking to learn it I hear a lot of great things about it.

~~~
F30
MITM attacks are designed to not be detectable, so the solution is to have
tooling which prohibits them – exactly what is lacking in this case.

For web, using TLS (SSL) is a good start. This could be improved further by
using HSTS, HPKP, DANE etc. (not sure if A+ already implies them anyway).

For SSH, you need to have an out-of-band way to get the host keys or use
something like an SSH CA.

~~~
ge96
What does that mean out-of-band? I'm not good with SSH, I'm not using 2-factor
key based, also in general it makes sense to have separate servers right? Like
one server that transfers request to a server closer to another country. Not
Cloudflare but your own thing assuming you rented from different datacenters
in the world. Sorry not related to the question.

edit: literally band? Like another wavelength/connection?

~~~
MasterIdiot
A different network connection (With physical infrastructure you might
actually use a completely different network) that you can use to move private
keys or any other sensetive information without exposing it to the Internet.
We use such a network to monitor all of our infrastructure, and for iLo.

~~~
ge96
Thanks for your input. I have to look into this more. Have to plan stuff out
for the future when growing past the single vps "instance"

------
niftich
Golang seems to follow the 80/20 rule from the outset (or perhaps an even
smaller proportion), which is perfectly fine. Some other languages'
standardlibs try to offer a complete treatment of a particular problemspace
from the start which is tricky to get right on first attempt. Those are the
instances where developers complain about complex APIs, uneven abstractions,
or the like.

However, one of the artifacts of a popular language having a lean-and-mean
standard library is that custom code proliferates, and the Go community's
distaste for frameworks (as opposed to libraries) means that the it's not just
the business-specific edges of the code that's unique in each implementation
(as you'd expect), but also a good amount of the plumbing and domain-specific
control code and their immediate callers. In some other languages, where
there's more of a culture for using a dependency to intentionally simplify
your problem space in exchange for ceding control, this style would be derided
as NIH.

The vendor's response here is a function of not only the vendor's own
rationale and priorities, but also of the above developer philosophy. This is
surprising to me, given that Go is an opinionated language, and yet
opinionated third-party code driving your logic is frequently discouraged by
its community.

On the other hand, the language maintainers' response was measured, proper,
and commendable. They made a breaking change to an experimental API, and
improved their product in the process.

~~~
e12e
This isn't 80/20 - this is broken. If you trust your network, use rsh or
telnet.

~~~
comex
Telnet does not support port forwarding, file transfer, host key
authentication (trusted network ≠ trusted clients), etc., and is not installed
by default on most systems. rsh only supports file transfer out of those, and
is dead enough that it's not even in Homebrew.

Nor is there any reason _not_ to use SSH just because you trust your network,
except possibly performance of huge file transfers.

~~~
e12e
Well, I'd rather run something I know has limited security (eg rsh+/etc/hosts-
allow over a closed vpn) than ssh without key verification - because then you
throw out trust, and encryption without trust buys you very little. Or
kerberized rsh/telnet.

As for port forwarding, if you trust the network, presumably you don't need
forwarding? And for file transfer just use zmodem, or run rsync/ftp?
(remember: you trust the network..).

I will admit that there's a minor convenience to be able to use one
client/api/interface - but I'm not sure it's worth the tradeoff of suddenly
not knowing if you should be trusting ssh to be actually _secure_ , rather
than just _convenient_.

Clearly the maintainers of the go package feel the same way (after some gentle
prodding).

------
mitchellh
Hello! As the blog post clearly states, the vendor is HashiCorp. As the
founder of HashiCorp and someone who participated in the initial report we
received on this topic, I'd like to state our point of view from my own mouth.

I'd first like to be up front about exactly which of our software doesn't
perform host key verification, since we have a lot of software and this CVE
doesn't apply to most. There are three places that were identified as
affected: Packer and Terraform with SSH provisioners, which both create a
machine resource and can perform SSH connections to setup the machine; and
Vault’s SSH backend in Dynamic Key mode performs SSH connections from the
Vault server to hosts (other modes do not).

Any other usage of our software is unaffected.

We’ll discuss each of these cases in detail, since the details matter to
understand our thought process and response.

Vault:

The SSH secret backend has three modes that can be used for generating SSH
credentials: certificates, one-time passwords, and dynamic keys. Only the
dynamic key mode ever actually makes connections to other machines, but more
importantly, our documentation has _always_ recommended that the dynamic key
mode only be used as a last resort because of its various (documented)
drawbacks compared to the other modes. With the addition of the ability to
generate SSH certificates (which was on our roadmap for a long time and added
in 0.7, prior to both the original report and the blog post), we did not
explicitly mark the dynamic key mode as deprecated in our documentation, but
we probably should do so.

Given that it is not recommended for usage (but maintained for backwards
compatibility), we chose to warn users of this additional drawback of the
dynamic key method, and documented the lack of host key verification
([https://github.com/hashicorp/vault/commit/251da1bcdc27678fea...](https://github.com/hashicorp/vault/commit/251da1bcdc27678feaa477f087c8d010223d7e8c)).
As we stated in our response to the reporter, "It isn’t something we want to
hide (and we’re not trying to) and we will document this."

Terraform/Packer:

Terraform and Packer support the ability to use "provisioners" to bootstrap a
machine. In both, the provisioner is run very shortly after the machine is
initially created, representing an extremely small window of attack. Neither
support connecting to a pre-existing machine via SSH under normal use cases
(you can make it happen through some advanced configuration trickery with
Terraform, but it's abnormal). Because of this, we didn't register this as a
high-priority issue.

However, we admit that this can be improved and we likely should've been more
reactionary in our response. I apologize for that. We have added plans to
improve this to our roadmap, covered in a couple paragraphs.

As the blog post states, the reporter suggested parsing console logs to
determine the host key. And, as the blog post correctly says, we don't want to
do this. There is a combinatorial explosion of complexity in supporting this,
we have experience with this (due to Vagrant supporting this type of
behavior), and we've found maintenance of this sort of functionality to be
difficult to support over time. We came to this conclusion though only because
there is a viable alternative: SSH certificate authentication. If a viable
alternative didn't exist, we may have been forced to take the more complex
route.

SSH certificate authentication was introduced many years ago and is broadly
supported. This type of auth also provides authenticity to a first-use
connection. We mentioned in our response email that this is something we're
open to doing instead. I admit that in our response to the reporter, we
explicitly said this "is not a priority" but shortly after decided to schedule
this work for the next major TF release. We should've followed up again, but
didn't.

And that's where we're at currently! I hope this helps make our response to
the report and our future roadmap around this issue more clear.

~~~
p4lindromica
Why do you think there is "an extremely small window of attack?" The mitmproxy
is likely between you and the internet somehow, meaning ANY ssh connection
will be MITMed. It is not like the MITM is waiting for you to spawn a machine
before it pops into existence.

------
risyasin
Well. I haven't really started to learn golang yet. But sure that this
breaking change indeed convinced me to do it. I have implemented an automated
ssh session in another language there was absolutely no host key checking or
tofu implementation even worse that they designed the api not to allow that
manually. That was frustrating. But obviously the golang language designers
and the entry owner and myself sharing the same concerns obviously. Thanks for
writing about this

------
aceperry
Really wonderful and thorough report. I'm not at all a security expert but
manage to learn quite a lot from reading this post. Kudos to the author for
giving context and background on the issues. If more security reports are
written like this, the whole industry would benefit greatly.

------
joneholland
Why do people keep saying "the vendor"? It's hashicorp.

~~~
fancy_pantser
Calling software companies "vendors" is a fun little lost piece of Americana.
Like calling marijuana "grass".

------
baby
A few things:

1\. how can an experimental library (x/) get a CVE?

2\. what is "hostkey verification"? Probably the fingerprint check you usually
get when you ssh into a machine + the blocking warning you get when the
fingerprint of the machine suddenly changes.

3\. if this is what "hostkey verification" is. How is it so hard to implement?
create some sort of fingerprint out of the server's public key; prompt the
user for input; cache the result.

~~~
lobster_johnson
Every host has a public/private keypair that it stores locally. The client
stores the last confirmed (or manually added) host key in ~/.ssh/known_hosts.
Host key verification is checking the host key against this list.

A lot of people ignore the importance of this part. It is also a bootstrapping
issue: At some point you need to get the key into known_hosts. Most people do
this by confirming it the first time they connect. That's usually fine for new
servers, but what about old servers, e.g. if you're a dev who just joined the
company? Most companies don't seem to have a secure way of seeding clients
with the initial host keys. I've never seen it.

We solve this with a script that gets them from Puppet (via PuppetDB, its
repository of host "facts"). That script has to hard-code the host key of the
Puppet host, of course.

Another solution is using the X.509-style certs to sign each host with your
own CA, but I haven't looked into that workflow. Edit: It's actually a much
nicer workflow. You can sign users, too.

~~~
uiri
_That 's usually fine for new servers, but what about old servers, e.g. if
you're a dev who just joined the company? Most companies don't seem to have a
secure way of seeding clients with the initial host keys. I've never seen it._

One way to do this is via SSHFP (ssh fingerprint) records in DNS.
Unfortunately they do not seem to be particularly widely supported nor widely
used.

------
DanielDent
I wrote about this a while back, and also proposed a solution which doesn't
involve parsing console output: [https://www.danieldent.com/blog/ssh-requires-
a-chain-of-trus...](https://www.danieldent.com/blog/ssh-requires-a-chain-of-
trust/)

------
ak217
FYI, you can also instruct cloud-init to use a particular key pair by
supplying it in instance metadata/user-data. This avoids the need for hacky
scripts extracting public keys from console output (which may also be delayed
by a few minutes after the instance starts).

------
jakewins
When we built the new set of drivers for Neo4j, we decided to allow three
modes: No encryption, Trust on First Use and Trusted Signature - there's no
way to establish an connection without trust.

This was a terrifying decision, because of ease of use concerns. Having done
so and shipped it, TL;DR: It worked awesome, outside of some early kinks in
TOFU that we worked out - and now everyone can sleep well knowing there's not
a single install that thinks they are running an encrypted setup when they
really aren't.

Anyone that came back asking for a flag to disable host key verification
seemed happy with our argument for why that's not really much different from
just disabling encryption.

See "Trust" here: [https://neo4j.com/docs/developer-
manual/current/drivers/conf...](https://neo4j.com/docs/developer-
manual/current/drivers/configure-connect/)

If you're interested in doing this as well, we wrote code to do it in Python,
JS, Java and C#, it's all Apache licensed:

JS: [https://github.com/neo4j/neo4j-javascript-
driver/blob/1.2/sr...](https://github.com/neo4j/neo4j-javascript-
driver/blob/1.2/src/v1/internal/ch-node.js#L106)

Python: [https://github.com/neo4j/neo4j-python-
driver/blob/1.2/neo4j/...](https://github.com/neo4j/neo4j-python-
driver/blob/1.2/neo4j/bolt/connection.py#L463)

Java: [https://github.com/neo4j/neo4j-java-
driver/blob/1.3/driver/s...](https://github.com/neo4j/neo4j-java-
driver/blob/1.3/driver/src/main/java/org/neo4j/driver/internal/security/TLSSocketChannel.java#L68)

C#: [https://github.com/neo4j/neo4j-dotnet-
driver/blob/1.3/Neo4j....](https://github.com/neo4j/neo4j-dotnet-
driver/blob/1.3/Neo4j.Driver/Neo4j.Driver/Internal/Connector/ITrustStrategy.cs#L28)

------
bradknowles
Phil may be grumpy, but he is not a troll.

------
asveikau
Good thing they didn't write this in C, or their library would have real
security trouble. /s

------
bogomipz
Could SSHFP records not have been an option here? Especially combined with
something like DNSSEC?

~~~
F30
Maybe, but then the library would have to support those.

This isn't as much about the concrete technology as it is about the
willingness to implement any host-key checking.

------
chrisper
I wonder if the devops company was this one:

[https://gravitational.com/teleport/index.html](https://gravitational.com/teleport/index.html)

EDIT: Actually, it looks like it's Hashicorp.

