I agree. Always choose unix domain sockets over local TCP if it is an option. There are some valid reasons though to choose TCP.
In the past, I've chosen local TCP sockets because I can configure the receive buffer size to avoid burdening the sender (ideally both TCP and unix domain sockets should correctly handle EAGAIN, but I haven't always had control over the code that does the write). IIRC the max buffer size for unix domain sockets is lower than for TCP.
Another limitation of unix domain sockets is that the size of the path string must be less than PATH_MAX. I've run into this when the only directory I had write access to was already close to the limit. Local TCP sockets obviously do not have this limitation.
Local TCP sockets can also bypass the kernel if you have a user-space TCP stack. I don't know if you can do this with unix domain sockets (I've never tried).
I can also use local tcp for websockets. I have no idea if that's possible with unix domain sockets.
In general, I choose a shared memory queue for local-only inter-process communication.
> I can also use local tcp for websockets. I have no idea if that's possible with unix domain sockets.
The thing that makes this possible or impossible is how your library implements the protocol, at least in C/C++. The really bad protocol libraries I've seen like for MQTT, AMQP, et. al. all insist on controlling both the connection stream and the protocol state machine and commingle all of the code for both. They often also insist on owning your main loop which is a bad practice for library authors.
A much better approach is to implement the protocol as a separate "chunk" of code with well-defined interfaces for receiving inputs and generating outputs on a stream, and with hooks for protocol configuration as-needed. This allows me to do three things that are good:
* Choose how I want to do I/O with the remote end of the connection.
* Write my own main loop or integrate with any third-party main loop that I want.
* Test the protocol code without standing up an entire TLS connection.
I've seen a LOT of libraries that don't allow these things. Apache's QPID Proton is a big offender for me, although they were refactoring in this direction. libmosquitto provides some facilities to access the filedescriptor but otherwise tries to own the entire connection. So on and so forth.
Edit: I get how you end up there because it's the easiest way to figure out the libraries. Also, if I had spare time on my hands I would go through and work with maintainers to fix these libraries because having generic open-source protocol implementations would be really useful and would probably solve a lot of problems in the embedded space with ad-hoc messaging implementations.
If the protocol library allows you to control the connection and provides a connection-agnostic protocol implementation then you could replace a TLS connection over TCP local sockets from OpenSSL with SPI transfers or CAN transfers to another device if you really wanted to. Or Unix Domain Sockets, because you own the file descriptor and you manage the transfers yourself.
> Local TCP sockets can also bypass the kernel if you have a user-space TCP stack. I don't know if you can do this with unix domain sockets (I've never tried).
Kernel bypass exists because hardware can handle more packets than the kernel can read or write, and all the tricks employed are clever workarounds (read: kinda hacks) to get the packets managed in user space.
This is kind of an orthogonal problem to IPC, and there's already a well defined interface for multiple processes to communicate without buffering through the kernel - and that's shared memory. You could employ some of the tricks (like LD_PRELOAD to hijack socket/accept/bind/send/recv) and implement it in terms of shared memory, but at that point why not just use it directly?
If speed is your concern, shared memory is always the fastest IPC. The tradeoff is that you now have to manage the messaging across that channel.
In my experience, for small unbatchable messages, UNIX sockets are fast enough not to warrant the complexity of dealing with shared memory.
However, for bigger and/or batchable messages, shared memory ringbuffer + UNIX socket for synchronization is the most convenient yet fast IPC I've used.
On Linux you can use abstract names, prefixed with a null byte. They disappear automatically when your process dies, and afaik don’t require rw access to a directory.
> Another limitation of unix domain sockets is that the size of the path string must be less than PATH_MAX. I've run into this when the only directory I had write access to was already close to the limit. Local TCP sockets obviously do not have this limitation.
This drove me nuts for a long time, trying to hunt down why the socket couldn't be created. it's a really subtle limitation, and there's not a good error message or anything.
In my use case, it was for testing the server creating the socket, and each test would create it's own temp dir to house the socket file and various other resources.
> In general, I choose a shared memory queue for local-only inter-process communication.
Do you mean the sysv message queues, or some user space system? I've never actually seen sysv queues in the wild, so I'm curious to hear more.
Depends on the user-space stack, but OpenOnload doesn't. But, this topic of user-space acceleration of pipes created over Unix sockets comes up here periodically... some of my previous comments:
In the past, I've chosen local TCP sockets because I can configure the receive buffer size to avoid burdening the sender (ideally both TCP and unix domain sockets should correctly handle EAGAIN, but I haven't always had control over the code that does the write). IIRC the max buffer size for unix domain sockets is lower than for TCP.
Another limitation of unix domain sockets is that the size of the path string must be less than PATH_MAX. I've run into this when the only directory I had write access to was already close to the limit. Local TCP sockets obviously do not have this limitation.
Local TCP sockets can also bypass the kernel if you have a user-space TCP stack. I don't know if you can do this with unix domain sockets (I've never tried).
I can also use local tcp for websockets. I have no idea if that's possible with unix domain sockets.
In general, I choose a shared memory queue for local-only inter-process communication.