
Securing Microservices - prabaths
https://medium.facilelogin.com/securing-microservices-with-oauth-2-0-jwt-and-xacml-d03770a9a838
======
ejcx
If you're using microservices and care about security, do yourself a favor and
use a monorepo.

A lot of improving security is about changing things in small ways but across
the entire fleet. If you have microservices without a monorepo you oftentimes
need to make the same changes in potentially hundreds of places.

This makes it a lot easier to do things like enforce standards for repos. Code
coverage. Testing. Unsafe function use. Repo sprawl makes microservice
security very challenging, and it isn't mentioned in this blog post. Losing
track of services and leaving specific services behind is not good.

~~~
adrianratnapala
> If you're using microservices and care about security, do yourself a favor
> and use a monorepo.

This seems like a strong reminder that "microservices" aren't really about
having lots of independent little systems but are a different way of factoring
your one big system.

~~~
stephengillie
It's like FizzBuzz - do you handle the 3 first or the 5?

~~~
reificator
You handle the 15 first to avoid the accumulator requirement.

~~~
stephengillie
My 15 falls through both the 3 and the 5 paths. So I must do 3 first, or 15
will be BuzzFizz.

~~~
jackweirdy
I think the point was that you do

If n%15==0

First, then the others. But it’s a pedantic point anyway (like this one)

~~~
sk0g
Or you can print fizz and buzz in independent statements, or it against
printing the raw number, and then print a new line for each number in the end.

------
mlakewood
There is a new standard forming for providing identity with this kind of
architecture called SPIFFE. Check it out at
[https://spiffe.io](https://spiffe.io). Its basically mutual TLS but with
identity baked into the certificate. Along with what the certificate looks
like, there is a reference implementation called Spire, to generate and
distribute the certificates.

~~~
suniljames
SPIFFE's next SF community day is 3 November. To learn more about this event
and other project updates, join the Google Group
([https://groups.google.com/a/spiffe.io/forum/#!forum/announce](https://groups.google.com/a/spiffe.io/forum/#!forum/announce)).

~~~
prabaths
I assume SPIFFE is more useful to system to system authentication without the
end user context - like how Netflix uses short-lived certificates to secure
interactions between microservices ([https://medium.facilelogin.com/short-
lived-certificates-netf...](https://medium.facilelogin.com/short-lived-
certificates-netflix-fd5f3ae5bc9)) ?

~~~
mlakewood
Thats the primary motivation and main focus for SPIFFE. Providing service to
service identity. However because its not breaking any of the standards its
potentially applicable in other contexts. The SPIFFE SVID (the certificate
standard) doesnt do anything wierd or different with TLS certs (which is
actually a strength) it more sets out a way to use the current existing cert
infrastructure to provide identity.

------
deathanatos
> _A signed JWT is known as a JWS (JSON Web Signature) and an encrypted JWT is
> known as a JWE (JSON Web Encryption). In fact a JWT does not exist itself —
> either it has to be a JWS or a JWE. It’s like an abstract class — the JWS
> and JWE are the concrete implementations._

This is backwards; a JWT is the payload of (usually, IMO) a JWS, sometimes a
JWE. But not all JWSs/JWEs are JWTs, so JWE/JWS cannot be called a concrete
implementation of a JWT.

> _Both in TLS mutual authentication and JWT-based approach, each microservice
> needs to have it’s own certificates._

JWT doesn't, to my knowledge, make use of certificates. I'm less clear on the
JWE cases, but JWS's only carry the algorithm used to do the signing, and the
signature. You have to know/figure out what key signed it to verify it.

Further, if you're using the HMAC algorithms, you're definitely not using a
cert.

~~~
prabaths
In fact JWT is an abstract concept - I have written a blog about that in
detail. Please find it here - [https://medium.facilelogin.com/jwt-jws-and-jwe-
for-not-so-du...](https://medium.facilelogin.com/jwt-jws-and-jwe-for-not-so-
dummies-b63310d201a3)

HMAC is not recommended - as it will be symmetric key. In fact you will find
more details in the above link...

~~~
deathanatos
Forgive me if I don't find your own blog a decent citation. Re-reading your
first article a bit closer, I can see how you might interpret it, I think (and
perhaps I was not charitable enough during my first read), but I still think
it is perilously close to mixing JWS being a concrete _serialization_ of a JWT
(this is correct) with JWS _being_ a JWT (this is not).

JWS is simply a format that contains a signature for an _arbitrary_ (that is,
not _always_ JWT) payload. See the "typ" header in the JWS RFC (in fact, see
the entire RFC)[1]; were a JWS always a JWT, we would have no need for "typ".
In fact, the RFC for JWS never mentions JWT in the normative parts of the
standard — it is only ever mentioned during examples, since JWT is perhaps the
primary consumer of the JWS standard. It calls this out, explicitly:

> _While the first three examples all represent JSON Web Tokens (JWTs) [JWT],
> the payload can be any octet sequence_

JWT is the concrete thing in that it is a JSON document encoding a set of
claims, and comes either signed (wrapped in a JWS) or encrypted (wrapped in a
JWE). JWT's RFC[2] states this fairly directly:

> _The claims in a JWT are encoded as a JSON object that is used as the
> payload of a JSON Web Signature (JWS) structure or as the plaintext of a
> JSON Web Encryption (JWE) structure_

While I see what you're getting at with "JWT is an abstract concept" — that
you need to wrap a JWT inside a JWS or a JWE — that does not mean that _all_
JWSs are JWTs, and while the text can certainly be interpreted as "you must
wrap a JWT in either a JWS or a JWE", I feel it toes the line too close to
"all JWS's are JWTS", particularly at, "A signed JWT is known as a JWS".
Having a JWS doesn't imply that it is a JWT; for example, the ACME protocol
uses JWS, but not JWT. The distinction here is subtle, but important, I
feel.[3]

> _HMAC is not recommended - as it will be symmetric key. In fact you will
> find more details in the above link..._

Not recommended by who? For what reasons?

Your article never mentions HMAC AFAICT (it mentions MAC in the process of
describing JWS, but no further). And yes, use of HMAC implies a symmetric key,
but that isn't necessarily insecure: it just means that anything that wishes
to validate JWTs signed with that key must have the key to do so, and thus,
must be trusted with that key. If you have a single service (say, an "auth"
service) that is responsible for validating JWTs, this works fine, and is a
great tradeoff for the additional complexity that signing w/ RSA keys brings.
E.g.,

1.

    
    
      client  -- login --> auth_service
             <--  JWT  --
    

2.

    
    
           client  -- JWT+cmd --> foo_service
                                              -- is this JWT valid? --> auth_service
                                             <--      yes, it is    --
                  <-- success --
    

Here, the HMAC key only needs to reside on the auth service, so it is
reasonably well contained. The tradeoff, of course, is that we need to make a
network request to auth service to validate JWTs, but we don't need to deal
with RSA. For some setups, this is perfectly acceptable. (Swapping out RSA
keys _directly_ here results in no more or less "secret" stuff on any given
node.)

The big advantage to RSA keys, of course, is that any service can verify JWTs
without being able to issue them, but if you want to swap out the secret (the
private key), you'll need to touch a lot more places, or have some
infrastructure to distribute the public key.

[1]:
[https://tools.ietf.org/html/rfc7515#section-4.1.9](https://tools.ietf.org/html/rfc7515#section-4.1.9)

[2]:
[https://tools.ietf.org/html/rfc7519](https://tools.ietf.org/html/rfc7519)

[3]: Doubly so since I feel there is a lot of ill-will towards JWT, and many
misconceptions about it. It's a good format, IMO, but the messaging around it
needs to be crystal clear if people are going to ever stop fearing and start
understanding it.

~~~
prabaths
Okay - rereading your comments - looks like you have misinterpreted this one.

"A signed JWT is known as a JWS (JSON Web Signature) and an encrypted JWT is
known as a JWE (JSON Web Encryption)"

This is a correct statement. This does not mean JWS is a JWT all the time.

This is well highlighted in the blog link I shared with you:
[https://medium.facilelogin.com/jwt-jws-and-jwe-for-not-so-
du...](https://medium.facilelogin.com/jwt-jws-and-jwe-for-not-so-
dummies-b63310d201a3)

"Yes, you read it correctly, the payload of a JWS necessarily need not to be
JSON - if you’d like it can be XML too."

------
spydum
Some good stuff in here: have been spending a lot of think time on this
problem, so this was helpful to validate some thoughts. but you lost me when
you went talking on about XACML. Does anybody use XACML still? I feel like
that is a dinosaur left behind with SOAP?

~~~
prabaths
I agree XACML has lot of complexities. But if you look at the recent
developments, you can now have both XACML request and response JSON based -
and the communication between the PEP and PDP in a RESTful manner. Also -
there is a standard coming up to have a JSON representation of XACML policies.
BTW, this blog only presents an architectural model - it can be any policy
language. Recently I found Netflix uses the same model for policy distribution
but instead of XACML, uses PADME. For me more than the language, the issue
XACML having is maintainability, auditability and governance. There are tools
around to support that. Even PADME does not solve these problems.

~~~
davegardner
I've done a fair bit of reading around policy-based authorization but have
never heard of PADME. For the life of me I can't find anything about it in any
Google searches. Can you point me at a reference for any information about it?

~~~
prabaths
This was discussed at a Netflix meetup. The official site is www.padme.io.
Also you can find the video recording of that meetup from
[https://www.youtube.com/watch?v=dim85J5cLq4](https://www.youtube.com/watch?v=dim85J5cLq4)
\- OPA and PADME are discussed from 33:49. Also check
[http://www.openpolicyagent.org/](http://www.openpolicyagent.org/).

------
t1o5
The article has pretty much summarized security, but did not talk about JWT
revocation techniques. Some may argue that short lived JWTs do not need
revocation and will expire and the refresh of the token can be blocked by
authorization server. But what about long lived JWTs ? For mobile apps which
logins once and keeps the login unless explicit logout. In cases such as
those, how do we revoke a rogue JWT ?

~~~
ianstormtaylor
I think this is the biggest drawback to "stateless" JWT's that most people
gloss over when championing. If you need to be able to revoke stateless JWT's,
you probably shouldn't be using stateless JWT's in the first place, because
you'll have to re-add state to the system to handle revocations. And at that
point why not just go with the simpler mental model in the first place.

And since giving users (or you) the option to "log themselves out of other
computers" on a fine-grained basis is often an important feature to provide,
it essentially means stateless JWT's aren't a great solution for sessions in
most web apps.

That's just what I've gathered personally. If anyone has a counter argument
for this case I'd love to hear it!

~~~
zeveb
> If you need to be able to revoke stateless JWT's, you probably shouldn't be
> using stateless JWT's in the first place, because you'll have to re-add
> state to the system to handle revocations. And at that point why not just go
> with the simpler mental model in the first place.

Because you can speed up the normal app path by using stateless access tokens
and only require online validation to issue new access token.

> And since giving users (or you) the option to "log themselves out of other
> computers" on a fine-grained basis is often an important feature to provide,
> it essentially means stateless JWT's aren't a great solution for sessions in
> most web apps.

The 'stateless' tokens may have arbitrarily short lifetimes, e.g. one or five
minutes.

An app may issue multiple requests per second or minute; there's a speedup if
those requests don't require online validation. Moving validations from
multiple per second to once per five minutes saves a huge amount of system
load.

It's an economic tradeoff between the costs of incorrectly giving access and
the cost of performing validation for every access.

~~~
ianstormtaylor
Hey @zeveb, I appreciate the response!

But the grandparent actually mentions the short-lived approach, and is asking
about cases where you want to have longer-lived tokens. And where you still
want to be able to revoke them.

Stateless tokens can definitely speed things up. But if you end up needing to
add state back into the equation for revoking tokens, then I'd argue that
_most_ web apps would be better off sticking with the much simpler mental
model of non-stateless tokens. That is, until they decide that their lowest-
hanging performance fruit is the extra validation work, and that it's worth
complicating their architecture to remove it.

~~~
zeveb
> But the grandparent actually mentions the short-lived approach, and is
> asking about cases where you want to have longer-lived tokens. And where you
> still want to be able to revoke them.

If that's actually what you want, then of course you want online-validated
tokens. But I think generally it's _not_ what you actually want: you actually
want short-lived self-validating access tokens and online-validated tokens.

Note that talking about an online-validated token's lifespan is a little
silly: there's no good reason for it not to live forever (until revoked).

------
chmike
This article illustrates well the authentication problem with HTTP. These
methods looks very complicated and not properly adressing the problem.

Jwt is a nice thing, but the associated information (req./resp.) is not
signed. We have to use TLS with certs, but TLS is P2P security based on
certificates and is not part of HTTP. It’s just a secure pipe.

The reason we can’t have a correctly designed security model and protocol is
because HTTP was not designed for that. One cannot sign an HTTP req/resp
because the headers may be rewritten or modified, etc. Same problem with SMTP.

Because of this design property, attempting to get properly conceived
authentication system on top of HTTP is like trying to force a cylinder into a
square hole. They simply don’t match or we endup with big gaps.

~~~
parasubvert
This is vastly overstating the case. HTTP was deliberately designed to enable
orthogonal approaches to security, it’s why it has endured in the face of
evolving to TLS 1.3 and oauth2 headers - we learn more about security as an
industry but the core semantics don’t change.

HTTP also has clear semantics of which headers do and do not get to be
rewritten, some web APIs do require signature of those headers, but in
practice it has been too much a burden on developers to get this right
compared to using subject/CA verrified TLS (which is hard enough).

The combo of HTTPS+Oauth2+JWT is pretty reasonable in practice, and in some
language ecosystems (Java and Spring Boot for example) requires little code to
implement: [https://spring.io/guides/tutorials/spring-boot-
oauth2/](https://spring.io/guides/tutorials/spring-boot-oauth2/)

Dsig of each request/response is what we used to do in the WS-Security days
and was terribly slow and tricky.

------
hiimcharlies
> API Gateway intercepts the request from the web app, extracts out the
> access_token, talks to the Token Exchange endpoint (or the STS), which will
> validate the access_token and then issues a JWT ( __signed by it __) to the
> API Gateway.

What is "it" in this quote? Will JWT be signed by API Gateway?

Otherwise.. a great article! Made my understanding of security princinples in
architecture like that MUCH clearer

~~~
prabaths
It should be signed by the STS - which is trusted by all the downstream
microservices. The STS, who validates the access_token, in the response can
send back this signed JWT to the gateway. The STS of the access_token and this
JWT can be the same or two different ones, based on the use case...

~~~
hiimcharlies
Thank you! I'll ask some more questions tommorow after I sleep on it if you
don't mind.

------
anoncoward129
Rather than managing a PKI for your JOSE (and all the revocation headache that
comes with that), why not just go with JSON Web Keys
([https://tools.ietf.org/html/rfc7517](https://tools.ietf.org/html/rfc7517))
and rely on your Web PKI for safe delivery of public keys?

------
darethas
I was a little disappointed with the Newman book because arguably the hardest
thing to get right with a microservices system is what to do when things go
wrong, when things fail, specifically partial failures. I also don't remember
much about security in the book, so I am thankful for this article.

------
ChicagoDave
This is an excellent article. It's not easy to manage, but if you get OAuth 2
right, you should be secure.

