
Urgent and Important – Rotate Your Amazon RDS, Aurora, and DocumentDB Certs - jeffbarr
https://aws.amazon.com/blogs/aws/urgent-important-rotate-your-amazon-rds-aurora-and-documentdb-certificates/
======
Nitramevfank
Here's probably a silly question: Shouldn't this work automatically? I just
assumed they would have an intermediate CA or whatever it's called and have
that certificate be signed by some widely trusted CA.

Or have they done it in a different way for security reasons?

~~~
inopinatus
Not silly at all. Certainly there are some database clients that will access a
system default CA trust store. ISTR the PostgreSQL JDBC client may have this
behaviour, although perhaps depending on which Java runtime & packaging is in
use. And there'll be others besides.

And behold, Amazon are themselves a globally trusted root CA! (Have a peek in
/etc/ssl/certs, on must current 'nixen I would expect to see the four Amazon
roots). So why not have the Amazon root sign the RDS CA certificate as an
intermediate CA, so that clients who _could_ trust an RDS-issued cert that
way, _will_ magically trust it?

I've met some of the RDS team and they are smart cookies. If you will agree to
my use of cookies, then I can think of at least three good enough reasons (and
just to be clear, this is speculation and extrapolation, not insider chapter
and verse):

1\. Changes in behaviour during another potentially disruptive change are the
province of the unwise, the unwary, and the otherwise soon to be fired for
cause.

2\. Not all clients can use a default trust store; even different clients for
the same database may differ. For example, the PostgreSQL C client (as used by
the command-line tools and many language bindings) cannot access the system
certificate store; the bundle must be configured explicitly. For one client to
have a magic behaviour that others do not, in the realm of cryptography and
trust, is going to make security admins and auditors very unhappy.
Inconsistency is the enemy of policy, and policy makes the DevSecOps world go
round.

3\. Not all clients support a certificate chain, so the RDS team already has
to maintain a bundle of the regional intermediates. Making the RDS CA an
intermediate just adds more complication to that activity, more special cases
to document, and more potential for confusion in client configurations.

So these are security reasons, although they're more from a policy and human
factors and software inadequacy perspective than anything to do with, say,
large prime numbers. It's quite possible the service team have more reasons
besides; I would certainly agree to their use of cookies or indeed any baked
product. For me, those reasons alone would suffice, and I am not surprised to
see that the new RDS root is another standalone self-signed CA.

~~~
securityty2020
Side point: +1 on not silly at all.

Re: anything w/ security (esp how it works), if you don't know ASK. ASK. ASK.

PLEASE ASK.

Security is incredibly complex and minor/subtle mistakes can destroy companies
and other potentially catastrophic consequences (especially to your
employment.) If you're not sure about something, ASK. End-users are bad enough
when it comes to security, but engineers should always feel empowered to "make
sure" about all matters of security, large or small.

Obviously it's best to to be familiar with best practices, but if you're not a
security "expert" and you're unclear about something, ASK.

There no dumb questions in security, just dumb practices. The dumbest thing of
all is not asking your question because you think it's "obvious."

tl;dr: ASK!

~~~
scurvy
> Security is incredibly complex and minor/subtle mistakes can destroy
> companies

For better or for worse, that's not the case. These days, security mistakes
just end up in 2 years of free credit monitoring. Ironic case in point,
Equifax.

I'm not saying you shouldn't take security seriously. But it's not fair to say
that insecurity could destroy your company. Mainly, most end users just really
don't care (or have a choice).

~~~
spenczar5
It depends on the company. What you say is true for a consumer website, but
much less true for a SaaS product which depends on customer trust. For
example, imagine an AWS or Salesforce or Github security breach that released
very private business data.

~~~
scurvy
Of the 3 mentioned, I doubt they'd lose more than 5% of revenues.

------
dperfect
In the spirit of asking silly questions (as encouraged in some comments here),
here's mine:

My small SaaS company's PostgreSQL RDS instance and app servers are in a VPC
with security groups configured to only allow connections from the app servers
to the DB (no public access to the RDS instance). My client (ruby-pg) on the
app servers is connecting via SSL, but not currently with certificate
validation (though I believe the cert date still needs to be valid [?], hence
the need to rotate the PostgreSQL server's certificate).

In this scenario, how important is certificate validation? I understand the
theoretical risk of clients not being able to fully trust that they're
connected to the database I intend, but from a practical standpoint, it seems
that if an attacker is able to poison the VPC's DNS and trick the app servers
into connecting to something else, I'm already hosed and cert validation
wouldn't do much to help me. Am I missing something obvious and very
dangerous?

~~~
luhn
AWS claims that VPC traffic is immune from MiTM attacks. [0] (Although they
strangely don't talk about this much, and I can't even find official
documentation of it.) So within a VPC, you can forgo certificate validation.

However, in general certificate validation would protect you. The attacker
wouldn't have a valid certificate, so your application would refuse to connect
to the malicious endpoint.

[0] [https://kevin.burke.dev/kevin/aws-alb-validation-tls-
reply/](https://kevin.burke.dev/kevin/aws-alb-validation-tls-reply/)

~~~
arkadiyt
They do talk about it in their Advanced Networking Study Guide:

[https://twitter.com/0xdabbad00/status/995833462660612096](https://twitter.com/0xdabbad00/status/995833462660612096)

~~~
stingraycharles
So basically they intercept all ARP communication and add VPC header
information to all Ethernet packets, which should rule out any MITM scenario.

Of course the keyword here is “should”, and depending on your threat mode you
may or may not want to have an additional defense layer of SSL certificate
validation.

------
sturgill
I created a new RDS instance via Cloudformation just last week and immediately
had the notice that I should update the cert. Looks like the new cert will be
the default on 1/14, but I thought that was really weird.

------
koksik202
Great to see Jeff Barr present on Hacker News reminding customers to rotate
certs

------
mabbo
> Next, I update my client applications to use the new certificates. This
> process is specific to each app and each database client library, so I don’t
> have any details to share.

This is how I lost the last two days of work. Learned a lot at least.

------
fovc
Is it possible to simply concatenate the two `bundle` files (2015 and 2019)
and feed that to client apps? That would allow updating the client first
without changing the db, then updating the db without breaking the client.
(Then step 3 would be to update the client to just use the 2019 bundle)

~~~
tingletech
The new file installed in a client will work with servers that use rds-ca-2015
and servers that use rds-ca-2019.

------
kureikain
Just think this may important to check if any of your app is using TLS.

    
    
      SELECT s.pid, s.ssl, s.version, a.client_addr, a.usename, a.datname, a.query
      FROM pg_stat_ssl as s
      JOIN pg_stat_activity as a ON a.pid=s.pid;
    

You can see `t|f` in `ssl` field.

------
insomniacity
> Regions – Rotation is needed for database instances in all commercial AWS
> regions except Asia Pacific (Hong Kong), Middle East (Bahrain), and China
> (Ningxia).

I wonder why?

Are these CAs subject to... shall we say, additional goverment oversight?

~~~
jontro
It's because those regions were created after the 2019 certificate was
released. Hence there is no 2015 cert for those.
[https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Using...](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html)

~~~
insomniacity
Fair enough - I didn't realise they'd launched regions that recently!

~~~
jontro
The Ningxia region might not support encryption. That one was released in
2017. Unsure about this one.

~~~
jon-wood
Yeah, Ningxia is a weird region. It’s run under license by a Chinese company,
and KMS isn’t available, which by extension means most encryption isn’t either
since that’s all built on top of KMS these days.

~~~
dlgeek
KMS launched in Ningxia (and Beijing) in June: [https://aws.amazon.com/about-
aws/whats-new/2019/06/aws-kms-n...](https://aws.amazon.com/about-aws/whats-
new/2019/06/aws-kms-now-available-aws-china-beijing-sinnet-and-aws-china-
ningxia-nwcd/)

~~~
jon-wood
Oh, fair enough, last time I looked into the China regions that didn't exist.
Reading the announcement I'm loving the seeming contradiction between " AWS
KMS is a managed service that makes it easy for you to create and control the
encryption keys used to encrypt your data, and uses Chinese government-
approved hardware security modules to protect the security of your keys." and
the very next sentence "The service is designed so that no one, including KMS
service operators, can retrieve your plaintext master keys from the service."

Something tells me those master keys aren't quite as secure as you'd expect in
other regions.

~~~
sneak
The US government also has their own set of security requirements for AWS
regions in the USA that serve the US government.

