Application security should be everyone's responsibility. Architects, developers, and operations.
Too many times have I seen architects and developers completely ignore it to make their jobs easier, leaving it to operations/infrastructure to implement. It's easy to twist the arm of business people with a "I can't ship feature X if you want me to look at security Y".
If everyone took this seriously perhaps we would have fewer issues.
I agree, I was just making a point that different teams have different priorities and thus different scope. Saying "PodA can only talk to PodB over mTLS" is very different to "Users need to login using oauth". Who is going to build the product if product team is working on the service mesh?
you can implement mtls (and almost all the other service mesh features) without service meshes, and it's usually better because of lower overhead, less total complexity (see for example the fat client libraries in use by google, netflix, etc.). but people don't want to think about this so leave it to infra teams to plaster a service mesh over everything.
You can, but it's absolutely a pain in the neck. Services need to load the certs from the filesystem on boot-up and trust the certs provided by other services. To manage trust, you need a certificate authority. Now you need to load the certificate authority's cert, and you need to manage rotation of certs. You need to help developers set up laptop-local certificate authorities and get them to issue certs so that you have Dev/Prod parity. You need to ensure that developers are enforcing modern ciphersuites, not doing bullshit "insecure-skip-verify" kind of toggles that make their jobs easier (because remember, their job isn't security, it's shipping features), not accepting self-signed certs or other certs not signed by the certificate authority. You need to make sure all this stuff is put in the testing suite to make sure it keeps getting maintained, and you need the files for these tests marked in CODEOWNERS to be under InfoSec control to ensure nobody rips them out just because they're inconvenient. And you need to copy this for every single service you run in production and every single development team.
You know what else you can do? Write your own web server (/sarcasm). I mean, who needs nginx? Probably writing your own will have lower overhead and less total complexity, not running a bunch of features that you don't use. And probably it will not be anywhere close to as good as a battle-hardened web server used by millions of engineers that gets regular support.
Personally I think it's debatable whether services really need mTLS within a private network. It's mostly a question of what scale you're running at; probably there's higher benefit-to-effort-ratio InfoSec projects to tackle. But if you do decide you need it, unless you can prove that the overhead is unworkable for your requirements, really you need to bite the bullet and put in a service mesh.
Yes, traffic between generic service and the mesh entrypoint is clear text BUT since the proxy is in a sidecar of the generic service pod, it shares the same "localhost" by mean of Linux network namespaces, so it's virtually isolated (if there isn't a bug) from other code running on the same node. When it exits the pod localhost, traffic is already encrypted.
Too many times have I seen architects and developers completely ignore it to make their jobs easier, leaving it to operations/infrastructure to implement. It's easy to twist the arm of business people with a "I can't ship feature X if you want me to look at security Y".
If everyone took this seriously perhaps we would have fewer issues.