Yeah, but for ICMP and so forth, ports aren't a thing. So you don't really have a universal 5-tuple.
For a router or other middlebox, or an OS kernel, to do things like outbound-initiated-flow firewall-rule exceptions correctly, it must keep N different flow-state tables, one per transport-layer (L4) protocol; where each flow-state table's "primary key" is over a set of columns unique to that table / L4 protocol.
TCP and UDP just happen to be both the best-known L4 protocols, and to both use {srcIP, srcPort, dstIP, dstPort} as their "primary key" for flows; but this doesn't hold for other L4 protocols.
(Which is in turn why L4 protocols "must" be handled in kernel-land, for kernel firewalls, traffic-shapers, etc. to work: L4 flow-state doesn't have a universal schema for these services to work with; and because these services are implemented in static-compiled languages, they have to be built with compile-time knowledge of each known L4 protocol, so that they can have concrete implementations for each L4 protocol written or generated for each service. There's no way to just bring in (through some hypothetical FUSE-like "userland L4 protocol server" abstraction) more L4 protocols, and expect those kernel facilities to work with them. [And all the same goes for ASICs in L4 network routers — only moreso.] Which is why we got the L4 protocol ossification we did. Modern protocols like SCTP and QUIC being implemented on top of UDP, is a direct result of there being no universal 5-tuple!)
You can handle L4 protocols in userspace. You can bind to a particular IP protocol number. You can even handle L3 in userspace.
Obviously if you do this you lose the ability for multiple applications to handle different "ports", unless you do the multiplexing in userspace as well.
You have a unique definition of "handle" that doesn't seem to include "your OS's kernel packet filter keeps working to pre-filter these packets based on an L4 understanding of them before handing them to userspace, or after being handed them by userspace."
Which, if your machine is acting as something like a router/NAT/firewall, is kind of... the entire point of the box being there in the communication path.
But the structure of those spaces can be different! The only structure IP imposes is that every packet has a source and destination address. It's up to each protocol whether it has port numbers (like TCP, UDP, and SCTP), or not (like ICMP and IGMP), or some other mechanism for identifying flows.
> TCP and UDP just have completely separate spaces of socket addresses
But so does SCTP, and ICMP and IGMP and ... -- so rather than enumerate the protocols we can just describe this property of IP.