An overlay network on top of wireguard would be really nice. For example you are running a wireguard network on 169.254.0.0/16. So every peer which is assigned an ip address within this range is by configuration of the network allowed to forward packets to another peer in the 169.254.0.0/16 network. So the only things needed to be implemented would be:
* an internal routing system to forward packets on some way to the destination
* a concept on how peers are found and how they build a secure channel (pre-shared key?)
Edit: A better way would be to have multiple shared secrets for every server. So you could basically assign roles to every server. So if a server has the keys "db" and "middleware" he can communicate with every same in the network for forwarding but the final destination can only be a server which has also one of the keys "db" od "middleware". Maybe such a server would have 2 virtual ips within the subnet, one for it's role for db and for middleware.
Did you look into these as alternatives?
Any connection flakiness is probably due to NAT or firewall issues and is going to occur in any P2P network layer since they all use a toolbox of common techniques such as UDP hole punching.
Does that mean that all your nodes have to be accessible to the public internet?
So someone on the internet doesn't necessarily know the node is reachable from the internet if they try and scan it for example.
Edit: IIRC only one end of the connection needs a stable endpoint as well. IIRC WireGuard supports mobility (changing IP addresses) for one end of the connection.
I've been interested in setting up a private network similar to what you describe and your comment has piqued my interest in finally building it
From my experience file locking on a distributed filesystem is either not implemented correctly or has piss-poor performance -- and databases use them
I haven't dug fully into but definitely will later today.