why is that? we need the CPU to handle page fault interrupts, in order to populate the RAM. But assuming the page is already in RAM, there's no reason any of the memory accesses actually need to go through the CPU. (hardware can already raise interrupts; if the MMU can raise a page fault indicator, then you might be able to bypass CPU entirely until a new page needs to be loaded)
moreover, if we have support for mmap at the MMU level, we can cut the CPU bottleneck for disk access entirely. the disk controller can already handle DMA, but there's simply no way for things that aren't the CPU to trigger it. DirectStorage is an effort for GPUs to trigger it, but what if we could also trigger it by other means?
Okay, lets say page-faults are off the table for some reason. Lets think about what can happen even if everything is in RAM still.
* MMAP is still on the table: different processes can share RAM at different addresses. (Process#1 thinks the data is at memory location 0x90000000, Process#2 thinks the data is at 0x70000000, but in both cases, the data is at physical location 0x42).
* Physical location 0x42 is a far-read on a far-away NUMA node. Which means the CPU#0 now needs to send a message to a CPU#1 very far away to get a copy of that RAM. This message traverses Intel Ultrapath Interconnect or AMD Infinity fabric (proprietary details), but its a remote message that happens nonetheless.
* Turns out CPU#1 has modified location 0x42. Now CPU#1 must push the most recent copy out of L1 cache, into L2 cache... then into L3 cache, and then send it back to CPU#0. CPU#0 has to wait until this process is done. If CPU#1 wants to modify the data again (or even read it), it may require messages from CPU#0 (who is now the owner of the data, according to simple MESI models).
Modern computers work very hard to hold the illusion of a singular memory space. Eventually, these details are turned into a consistent memory model and become well-ordered sequential operations. The CPU is a good place for that.
---------------
That's how stuff works _today_. If you wanted to make a new programming model that's incompatible, that's fine. (CUDA does it: GPUs don't have as many virtual-memory features as a CPU. And __shared__ memory has a different model than L1 cache.)
But if you invent a new memory model that does things differently, it means that it won't work for the vast majority of code. Which means you need to bootstrap a new programming environment (much like how CUDA bootstrapped a new community from scratch).
Or the mapping info for your process is shared with that coprocessor (the OS just keeps this around in RAM anyway). Heck, you could have an OS-provided code that needs to be loaded so that it can properly resolve that mapping (which is what the TLB does by the way - it has a well-defined structure and when it’s missing from the cache if I recall correctly it’ll fetch some info directly from RAM until it gets to a point where it has to generate a page fault).
I don’t disagree that the memory model becomes more complex. For one CPU cache invalidation becomes really tricky. So do memory coherence rules.
I’m less clear how mmap matters here. That’s just a mechanism the OS uses to hand out views into the page cache to the process - if you’ve solve the virtual-physical mapping (which you have to do) then mmap is not relevant.
You’re spot on though that the particular design decisions are critical for this to be successful - pick the wrong point on the complexity/cost/perf curve and your solution will definitely be DOA.
Pointers are virtual addresses but memory is accessed physically. All of the means of translating virtual to physical are in the CPU. If you are proposing throwing out virtual addressing, I imagine you won't get a lot of support for that idea.
I think the point was more like the following situation.
You have a linked list. Each memory location, since it’s user-space, stores the virtual address to the next location. How do you offload a program to process this “in situ”? You’d need to translate these to physical addresses. Parent is 100% correct about the challenge this poses.
moreover, if we have support for mmap at the MMU level, we can cut the CPU bottleneck for disk access entirely. the disk controller can already handle DMA, but there's simply no way for things that aren't the CPU to trigger it. DirectStorage is an effort for GPUs to trigger it, but what if we could also trigger it by other means?