
Heavy Rotation of Relational Hashicorp Vault Database Secrets in Spring - usr42
https://secrets-as-a-service.com/posts/hashicorp-vault/rotate-dynamic-relational-database-connection-in-spring-at-runtime/
======
mp911de
Thanks for your blog post. Introducing rotation of configuration without
downtime (interruption) is a quite challenging topic. The most challenging
parts are breaking up static assignments and considering concurrency. If we
reduce the problem to external integrations such as databases and message
brokers, it's easier to grasp the scope.

If we look at what happens after credential expiry, we experience a quite
diverse field. Some databases just keep working on existing connections as
these remain authenticated. In other cases, databases validate the security
context on each operation, which requires disposal of connections.

While injecting updated configuration is possible in some cases, it's not an
approach that works for all technologies. Rather, we should come up with an
approach that helps to understand and solving the problem for all components
(or at least the majority).

The root of all this is that configuration binding and the components being
configured just do not expect rotating credentials. We need to work around
this fact. While this approach works for Hikari, it might not work for other
pool implementations and that is the problem I have with the posted approach.

Instead, I would recommend a Proxy-based approach that can work with the
majority of components. A pool is a DataSource after all and with the usage
pattern of opening and closing connections, an implementation can track
resources in use. So instead of exposing a HikariCP pool directly as data
source, it would make sense to expose a proxy that routes getConnection()
calls to the currently active and valid pool implementation.

Once credentials expire (or are about to expire), the implementation would
fetch new credentials, pass these to a DataSource factory function and obtain
a new, healthy pool. Also, once all connections of the previous pool are
released, the old pool can be closed.

For technologies that do not expose a releasable connection (MongoDB,
Cassandra), we require a different release mechanism, probably timeout-based.
Still, the general proxy idea (proxying the Mongo or Session object) would
remain the same.

~~~
usr42
Thanks for your feedback.

I agree that the post does not offer a generally applicable solution for the
rotation of configuration. Unfortunately, it looks like in this moment of time
there is no such solution available (at least not that I'm aware of). On the
other hand, the reason for writing this post was not to solve this issue but
to help developers of Spring applications to use Hashicorp Vault for
generating dynamic database credentials.

A solution for the more generally applicable, but also much more complex
problem of configuration rotation without downtime would likely also solve the
issue of the blog post. But as long as this solution is not available a more
specific approach could at least address the challenges for some of us. The
quite narrow scope of the blog post is bound to relational databases and
HikariCP and I hope that I could point out that this still addresses a lot of
use-cases (at least as long as SQL databases are not dead ;) ).

Additionally, I would like to increase the awareness that Spring itself is
currently not automatically addressing the issue of rotating expiring secrets
provided by Hashicorp Vault. Because Spring provides so many out-of-the-box
production ready solutions for a lot of use-cases a lot of developers don’t
know and even think about the edge cases which could still create pain.

I’m really interested in more details about the proxy-based approach you were
writing about. If I’m able to build a more general solution with this approach
I would be really happy to share this approach in another blog post. Or do you
know if it there already something planed in the Spring backlog to make this
easier?

~~~
mp911de
The approach is currently a design draft to approach credential rotation from
a generic perspective. It spans over a couple of components and we need to
identify first on which level we want to address the requirement of
credentials rotation. So it will take a bit until we have a more clear
picture.

~~~
usr42
So there will be a generic solution provided by Spring?

I'm really looking forward to this. Is there I way I can be kept up to date
about the progress or how I could potentially participate?

------
usr42
Author here. Happy to answer any questions or hear any feedback about this
post.

