Part of the problem is that IPsec deals in authenticating peers and authorizing them to the IP addresses they sport, but IP address numbering is fluid and difficult to tie to authentication or authorization databases. All too often sites permit large prefixes to all clients with certificates from some CA.
Another part of the problem is that IPsec protects IP, not TCP and friends. This means that part of a TCP connection's packet flows may be protected by SAs with one peer, and later by SAs with... a different peer -- this may sound strange, but given the issues w/ authorizing IP addresses (see above) this is very much possible.
The right answer is to tie the upper level protocols (e.g., TCP) to IPsec policy automatically so that for the entire lifetime of a connection the local and peer IDs are "latched". This has been specified  (disclaimer: I'm the author), but not implemented. API-wise this would manifest as socket options you could set and/or get to specify or learn local credentials, peer name, quality of protection, etc.
No, it's not.
Some IPsec implementations are kludgy and lack proper granularity/control over their configurations. It is also very flexible, so you can of course misconfigure it to the point where the whole setup becomes a royal clusterf#ck, but doesn't mean it is "extremely difficult to use correctly". It works really well for a wide range of common scenarios, each of which is not that hard to configure once one knows what they are doing.
SG ("security gateway", aka VPN) is reasonably easy to use, especially if you assign users /32s (for IPv4) and inject those into you IGP -- then you cannot have the attack I mentioned.
Transport mode is very difficult to use correctly and safely, or even at all, in any network with more than a few nodes or with fluid IP address assignments.
For transport mode, the best answer, really, would be to use BTNS  and connection latching  so that you can just use IPsec from the application (via socket options or moral equivalent), performing channel binding to application-layer authentication, or else certificate chain validation and authorization at the application layer. But this hasn't been implemented. Solaris/Illumos has some IPsec-related socket options, and it will create and/or latch SPD entries (what traffic to protect and how) policy, but not SAD entries (how to authenticate peers) nor PAD entries (how to authorize peers) -- i.e., half-way to RFC5660 :(
Honest question, why is this a problem? If I really care about protecting my application traffic I am going to use TLS.
I skimmed the RFC, and it mentions applications that "rely on IPsec for session protection". So is it targeted more towards protecting HTTP traffic and other insecure protocols that are still in use today?
One might say that you'll use TLS because IPsec doesn't cut it for application traffic. Well, OK, so what traffic is IPsec good for? (ICMPs.)
If you're using TLS and IPsec then you're wasting resources by using two layers of session cryptography.
If you're using only TLS you leak some plaintext that you wouldn't with IPsec (e.g., port numbers).
If you're using only IPsec then you're not scaling to Internet scale.
> I skimmed the RFC, and it mentions applications that "rely on IPsec for session protection". So is it targeted more towards protecting HTTP traffic and other insecure protocols that are still in use today?
That was an allusion to iSCSI, which requires IPsec and uses only IPsec for transport security.
HTTP could use it too, but the chances of getting IPsec deployed at Internet scale (with BTNS or WebPKI or whatever) are roughly zero. iSCSI, on the other hand, could really use this. NFSv4 too.
Now, to use BTNS and connection latching in iSCSI or NFSv4 you'd need to use channel binding, but now we're getting a bit off-topic.
Let me stop you there. The only reason it sounds strange is because you are building a presumption that IPSEC has a primary role in protecting second hop TCP traffic. We have TLS and certificates for a reason, and when employed properly (verify certs, TOFU, GSM suite, TLS 1.2...) the problems you go to mention are mitigated.
If you use IPSEC for what it is made for, it is neither hard to understand nor difficult to implement.
I don't agree with you. You'd have to review a lot of history to make that sort of assertion about the purpose of IPsec, and you'd have to ignore transport mode. Transport mode clearly exists to protect end-to-end... In transport mode the purpose of IPsec really is to protect all upper level protocol packet flows covered by local IPsec policy.
Also, since both, transport-mode IPsec and TLS are end-to-end, using both would be a serious waste of resources -- in practice few ever use transport mode, because any non-VPN, non-BITS/BITW uses of IPsec are just ETOOHARD to deploy and scale.
Of course, encrypting multiple times at different layers, but only once end-to-end, is not wasteful.
One nice thing about having the data path in kernel is that you aren’t context switching to userspace and back, nor doing the associated copyin/copyout to move data between address spaces. This can be further leveraged to do inline crypto offload, with Chelsio’s T6 NIC and Intel QuickAssist chipsets. This means it’s quite possible to do 100gbit IPSec networking with commodity hardware. With KTLS it may be possible to do that with other VPN technology too though!
It may also be helpful to think of IPSec as layer 3ish while TLS is 4ish.
if you have a failure before phase 1 is negotiated you generally have no idea what is wrong. generally it is either because a) the other side has not configured their firewall or b) there is some kind of mismatch in the phase 1 parameters. however, it is generally impossible to tell which one it is because either IPSEC protocol doesn't support status messages or it is not configured because of 'security' to generate status messages. also, in the case where there is a mismatch apparently it is very difficult for the operators at the other end to extract the information from the logs about what is wrong :/
you get a similar problem with phase 2. if there is something wrong the setup will just fail and you will only know there is a problem with phase 2 but not the reason why.
compare this with TLS and I can generally identify what the problem with a failed handshake is from either the status messages or by looking at the client/server hello.
IPSEC is also more a pain to deploy than TLS. for TLS protected connections we just whitelist 2 IPs and run our integration on two boxes for high availability. really simple. for IPSEC now we either have two buy two pieces of expensive CISCO hardware to give us a HA VPN connection or we need to duct-tape up two strongswan boxes with automated failover to give us HA. not fun.
i understand why people might want to use IPSEC if they want to encrypt all the traffic between two networks transparently. but for encrypting individual point-to-point connections it is massively overkill. i suspect people end up using IPSEC for point-to-point encryption because they see it as difficult to add TLS support to existing software but don't know about options like stunnel (https://www.stunnel.org/).
This will be great, power of XDP in simple interface.
transform. `man ip-xfrm`
EDIT maybe "X" to mean "cross/transform" is an American-ism. Going for the first time in the US, it took me a while to compute that "Ped Xing" street signs should be read "pedestrian crossing" :D (and immediately think: why not a simple evocative symbol of a road with someone walking on it?!)
Kindly, the pedantry department.
It took me years to figure out that "user space" literally means a regular program running on the computer. Like, when you go "gcc -o hello hello.cc" that means "user space".
Whereas "kernel space" means downloading the kernel, make, copying it to /boot (or installing it in a VM), rebooting the computer, switching back to your stable kernel, repeat. So much more work, and security bugs mean that some attacker can access kernel memory space and methods (eek!).
Kernel space means the code is implemented by the kernel and is executed in this privileged context (with its own memory region). Other kernel level actions (such as the actual methods for most disk and network access) can be used and instead of copying memory over to user space (because user code can't access kernel memory for security reasons).
Because of this, all system calls (kernel routines) that return data use copying extensively. Every byte you read from the network or disk through a syscall is generally at least read into kernel memory then copied to user memory.
As you can imagine, providing higher level routines where more is handled in kernel space eliminates a lot of memory copying, and thus can achieve better performance.
kernelspace==privileged execution, i.e. has access to all the resources of the machine, all RAM, all devices, etc. The kernel by definition requires control of the machine because its primary job is to manage resources on behalf of the userspace applications.