A lot of improving security is about changing things in small ways but across the entire fleet. If you have microservices without a monorepo you oftentimes need to make the same changes in potentially hundreds of places.
This makes it a lot easier to do things like enforce standards for repos. Code coverage. Testing. Unsafe function use. Repo sprawl makes microservice security very challenging, and it isn't mentioned in this blog post. Losing track of services and leaving specific services behind is not good.
This seems like a strong reminder that "microservices" aren't really about having lots of independent little systems but are a different way of factoring your one big system.
First, then the others. But it’s a pedantic point anyway (like this one)
We implemented our services like this a few years back. It worked really nice. But in everything that I have read I have never seen any references to this practice. I didn't even know it was called "monorepo". I was just assuming everybody was using multiple repos and we were weird.
Most people aren't at that scale, but IMO, many benefits people get from monorepos you also get by using GitHub/Gitlab with master projects mapping in git repos via sub-modules.
Anyway, it really sucks to work on extremely large monorepos when you don't have access to the same resources as Google and Facebook. For this reason I'm personally always hesitant to recommend monorepos as the be-all-end-all.
Facebook also maintains a set of custom extension for it , and there is an interesting talk about the reasons beyond their choice .
(work for FB)
> For this reason I'm personally always hesitant to recommend monorepos as the be-all-end-all.
May be not a monorepo, but at least trying avoid having too many repos.
What kind of scale are we talking, and what issues do you get?
I've explored lots of different options, and hope to look at mercurial at some point, but am not hopeful.
I was functioning as an architect at a pretty large company and we used spring boot. My team wrote a number of internal starters (and our own parent Pom) that all other teams would use and set it up so that the services would build/test/deploy after our base pom and starters would release. It obviously takes a bit of time and tooling to do that, but it was working quite well for us and still kept us from needing to update the same thing manually in 20 places (that is until we’d need to release a breaking version).
I think a monorepo works well for company cultures that have a lot of internal code, it works less well when you have a highly decentralized mode of operating (the whole point of microservices IMO) and a lot of shared externally written open source code. Repo sprawl isn’t an issue if you have known orgs - the security team’s CI/CD checks them all and files issues or PRs to them all.
* If you have 100 services in a monorepo, then it needs completely different toolchain like bazel/buck (all new) or cmake/qmake/etc, to find out the whole dependency graph, with deep integration with the SCM to rebuild only changes and downstreams, avoiding a 2-hour build, avoiding 10gb release artifacts, scoped CI/commit builds (i.e. triggering build for only one changed folder instead of triggering a million tests), independent releases, etc
* Some more tooling for large repository and LFS management, buildchain optimization, completely different build farm strategies to run tests for one build across many agents, etc
* Making sure people don't create a much worse dependency graph or shared messes, because its now easier to peek directly into every other module. Have you worked in companies where developers still have troubles with maven multi-module projects? Now imagine 10x of that. Making sure services don't get stuck on shared library dependencies. Should be able to use guice 4.0 for one service, 4.1 for another, jersey 1.x f/a, jersey 2.x f/a, etc etc. Otherwise it becomes an all-or-nothing change, and falls back to being a monolith where services can't evolve without affecting others
* Does not mean its easy break compatibility and do continuous delivery (no, there is database changes, old clients, staggered rollout, rollbacks, etc. contracts must always be honored, no escaping that, a service has to be compatible with its own previous version for any sane continuous delivery process)
Imagine monorepo like: building a new Ubuntu OS release for every Firefox update, and then work backwards, doing it for every single commit. I'm not even scratching the surface of anything here. It changes everything - from how you develop, integrate, test, deploy, git workflows, etc. This is why big monorepo companies like Facebook/Google release things like bittorrent-based deployments, new compression algorithms, etc - because that's the outcome of dealing with a monorepo.
I may go as far to say this, after many many journeys:
Monorepo with monolith - natural, lots of community tooling, lots of solved problems.
Multi-repos with multi-services - natural, lots of community tooling, lots of solved problems.
Anything else without the right people who have already done it many times, and you're in for a painful rediscovery journey of what Google/Facebook went through, and this does not have as much knowledgebase/tooling/community/etc as other natural approaches.
Changes are hard (tm)
Execution order changes with each change requirements.
Should i commit this now? What if other team mate executes now?
It's not easy.
This is backwards; a JWT is the payload of (usually, IMO) a JWS, sometimes a JWE. But not all JWSs/JWEs are JWTs, so JWE/JWS cannot be called a concrete implementation of a JWT.
> Both in TLS mutual authentication and JWT-based approach, each microservice needs to have it’s own certificates.
JWT doesn't, to my knowledge, make use of certificates. I'm less clear on the JWE cases, but JWS's only carry the algorithm used to do the signing, and the signature. You have to know/figure out what key signed it to verify it.
Further, if you're using the HMAC algorithms, you're definitely not using a cert.
HMAC is not recommended - as it will be symmetric key. In fact you will find more details in the above link...
JWS is simply a format that contains a signature for an arbitrary (that is, not always JWT) payload. See the "typ" header in the JWS RFC (in fact, see the entire RFC); were a JWS always a JWT, we would have no need for "typ". In fact, the RFC for JWS never mentions JWT in the normative parts of the standard — it is only ever mentioned during examples, since JWT is perhaps the primary consumer of the JWS standard. It calls this out, explicitly:
> While the first three examples all represent JSON Web Tokens (JWTs) [JWT], the payload can be any octet sequence
JWT is the concrete thing in that it is a JSON document encoding a set of claims, and comes either signed (wrapped in a JWS) or encrypted (wrapped in a JWE). JWT's RFC states this fairly directly:
> The claims in a JWT are encoded as a JSON object that is used as the payload of a JSON Web Signature (JWS) structure or as the plaintext of a JSON Web Encryption (JWE) structure
While I see what you're getting at with "JWT is an abstract concept" — that you need to wrap a JWT inside a JWS or a JWE — that does not mean that all JWSs are JWTs, and while the text can certainly be interpreted as "you must wrap a JWT in either a JWS or a JWE", I feel it toes the line too close to "all JWS's are JWTS", particularly at, "A signed JWT is known as a JWS". Having a JWS doesn't imply that it is a JWT; for example, the ACME protocol uses JWS, but not JWT. The distinction here is subtle, but important, I feel.
> HMAC is not recommended - as it will be symmetric key. In fact you will find more details in the above link...
Not recommended by who? For what reasons?
Your article never mentions HMAC AFAICT (it mentions MAC in the process of describing JWS, but no further). And yes, use of HMAC implies a symmetric key, but that isn't necessarily insecure: it just means that anything that wishes to validate JWTs signed with that key must have the key to do so, and thus, must be trusted with that key. If you have a single service (say, an "auth" service) that is responsible for validating JWTs, this works fine, and is a great tradeoff for the additional complexity that signing w/ RSA keys brings. E.g.,
client -- login --> auth_service
<-- JWT --
client -- JWT+cmd --> foo_service
-- is this JWT valid? --> auth_service
<-- yes, it is --
<-- success --
The big advantage to RSA keys, of course, is that any service can verify JWTs without being able to issue them, but if you want to swap out the secret (the private key), you'll need to touch a lot more places, or have some infrastructure to distribute the public key.
: Doubly so since I feel there is a lot of ill-will towards JWT, and many misconceptions about it. It's a good format, IMO, but the messaging around it needs to be crystal clear if people are going to ever stop fearing and start understanding it.
"A signed JWT is known as a JWS (JSON Web Signature) and an encrypted JWT is known as a JWE (JSON Web Encryption)"
This is a correct statement. This does not mean JWS is a JWT all the time.
This is well highlighted in the blog link I shared with you: https://medium.facilelogin.com/jwt-jws-and-jwe-for-not-so-du...
"Yes, you read it correctly, the payload of a JWS necessarily need not to be JSON - if you’d like it can be XML too."
"JSON Web Token (JWT) is a compact, URL-safe means of representing claims to be transferred between two parties. The claims in a JWT are encoded as a JSON object that is used as the payload of a JSON Web Signature (JWS) structure or as the plaintext of a JSON Web Encryption (JWE) structure, enabling the claims to be digitally signed or integrity protected with a Message Authentication Code (MAC) and/or encrypted."
"JWTs are always represented using the JWS Compact Serialization or the JWE Compact Serialization."
A JWT will only exist as a JWS or JWE. It does not exist by itself - its an abstract concept.
Regarding HMAC - its not recommended for the context of this article. It's not a recommended approach to do authentication with shared keys is in a distributed environment.
In other words, there's not a long-lived self-validating JWT. If for some reason that were required, you might rotate the shared secret or signing key.
And since giving users (or you) the option to "log themselves out of other computers" on a fine-grained basis is often an important feature to provide, it essentially means stateless JWT's aren't a great solution for sessions in most web apps.
That's just what I've gathered personally. If anyone has a counter argument for this case I'd love to hear it!
Because you can speed up the normal app path by using stateless access tokens and only require online validation to issue new access token.
> And since giving users (or you) the option to "log themselves out of other computers" on a fine-grained basis is often an important feature to provide, it essentially means stateless JWT's aren't a great solution for sessions in most web apps.
The 'stateless' tokens may have arbitrarily short lifetimes, e.g. one or five minutes.
An app may issue multiple requests per second or minute; there's a speedup if those requests don't require online validation. Moving validations from multiple per second to once per five minutes saves a huge amount of system load.
It's an economic tradeoff between the costs of incorrectly giving access and the cost of performing validation for every access.
But the grandparent actually mentions the short-lived approach, and is asking about cases where you want to have longer-lived tokens. And where you still want to be able to revoke them.
Stateless tokens can definitely speed things up. But if you end up needing to add state back into the equation for revoking tokens, then I'd argue that most web apps would be better off sticking with the much simpler mental model of non-stateless tokens. That is, until they decide that their lowest-hanging performance fruit is the extra validation work, and that it's worth complicating their architecture to remove it.
If that's actually what you want, then of course you want online-validated tokens. But I think generally it's not what you actually want: you actually want short-lived self-validating access tokens and online-validated tokens.
Note that talking about an online-validated token's lifespan is a little silly: there's no good reason for it not to live forever (until revoked).
Jwt is a nice thing, but the associated information (req./resp.) is not signed. We have to use TLS with certs, but TLS is P2P security based on certificates and is not part of HTTP. It’s just a secure pipe.
The reason we can’t have a correctly designed security model and protocol is because HTTP was not designed for that. One cannot sign an HTTP req/resp because the headers may be rewritten or modified, etc. Same problem with SMTP.
Because of this design property, attempting to get properly conceived authentication system on top of HTTP is like trying to force a cylinder into a square hole. They simply don’t match or we endup with big gaps.
HTTP also has clear semantics of which headers do and do not get to be rewritten, some web APIs do require signature of those headers, but in practice it has been too much a burden on developers to get this right compared to using subject/CA verrified TLS (which is hard enough).
The combo of HTTPS+Oauth2+JWT is pretty reasonable in practice, and in some language ecosystems (Java and Spring Boot for example) requires little code to implement: https://spring.io/guides/tutorials/spring-boot-oauth2/
Dsig of each request/response is what we used to do in the WS-Security days and was terribly slow and tricky.
What is "it" in this quote? Will JWT be signed by API Gateway?
Otherwise.. a great article! Made my understanding of security princinples in architecture like that MUCH clearer