Seems like you're trading away haproxy as a centralized SPOF for brand new custom code, which itself is a centralized SPOF; and also needing a new watchdog daemon to do what the haproxy instance would have done. Would be interesting to understand what the problem was that forced that complex arrangement, because it's not obvious to me at the moment.
1. In the central LB setup, if the LB server dies, you're service dies. Not in this setup (HAProxy is deployed side-by-side with each instance).
2. Elasticity. Imagine you have a shopping cart microservice, a search microservice, and a users microservice. Each of these requires its own HAProxy instance. Every time you spin up or spin down a new instance of these microservices, you need to configure the central HAProxy to pay attention to these new instances.
3. Health checks don't work well in DNS. In the centralized load balancer infrastructure, you end up relying on DNS so that your users microservice can talk to the shopping cart microservice (for example). DNS requires client polling and has propagation delays, so if one of your shopping cart microservice load balancer dies, it takes time for all the other microservices to figure out where to connect to.
A central LB works well if you have just a single microservice. But when you have dozens of them, you're suddenly managing dozens of load balancers dynamically, and it gets to be pretty chunky to manage at that point.
Except most people deploy dual HAProxy servers connected via heartbeat, which share a floating IP. There's no SPOF. Deploying new microservices is as simple as managing the HAProxy config file via Chef or Puppet.
You can definitely do that. There is more programming involved, since you need to figure out how to tie a new instance deployment into Chef/Puppet/etc to update HAProxy. You also need to figure out how to get it to update quickly. Finally, you'll need to figure out how to deploy your dual HAProxy server with heartbeat setup automatically every time you deploy a new type of microservice, and update DNS appropriately. It just means that instead of deploying Baker Street as part of your microservice push, you're deploying a) your microservice b) your dual HAProxy setup and/or new HAProxy config and c) DNS.
Lots of ways to solve this problem; our big focus here is on simplicity. If you have the Chef-fu and time to do all the above, you could definitely make it work.
