I really dislike that all "Microservice communication" guidances encourages RPC, synchronized communication. Service Meshes make the nightmare easier to bare. It makes bad design easier to implement than good design.
It creates temporal coupling between the services that becomes really hard to manage at scale, and relies on bells and whistles from the underlying orchestrator to keep your system healthy, instead of relying on the inherent robustness of your system design.
Couldn't agree more; to me, communication between microservices should be async most of the times, as otherwise availability of services is impacted by those of others.
This raises the question how services can update their own state (database) and send messages to other services e.g. via Kafka, without relying on distributed transactions. I found the outbox pattern comes in handy for that: services write messages into an "outbox" table within their own database, thus not relying on availability of any other resources besides that database. From the outbox table, events are propagated to receivers asynchronously. There's different ways for implementing this, most efficiently via change data capture (CDC) on the outbox table.
We support this pattern in Debezium (see https://debezium.io/blog/2019/02/19/reliable-microservices-d...) with a routing component and now also an extension for the Quarkus framework (Java stack for implementing cloud-native microservices with JVM and GraalVM).
Feedback on all this would be very welcomed. Outbox to me hits the sweet spot between not relying on availability of any other services, giving instant read-your-own-write semantics for services (as opposed to writing to a queue/commit log first and reading back from there) and putting eventual consistency to communication with other services, where it's typically easy to accept.
I agree you should do this from the user's perspective, but what about when a message handler (which is already async and not blocking the user) needs to call into an api? You could rewrite it to use queues for the request and response, but then you're convoluting control flow.
It creates temporal coupling between the services that becomes really hard to manage at scale, and relies on bells and whistles from the underlying orchestrator to keep your system healthy, instead of relying on the inherent robustness of your system design.