Hacker News new | past | comments | ask | show | jobs | submit login
Google Posts Experimental Linux Code for “Device Memory TCP” (phoronix.com)
8 points by mfiguiere 11 months ago | hide | past | favorite | 4 comments



As a "how HW works" noob, how is this different from DMA?


If I understand this correctly, it's RDMA for GPU/accelerator memory.


1) If you read Nvidia GPUDirect RDMA page closely (https://docs.nvidia.com/cuda/gpudirect-rdma/index.html), you can find this paragraph:

"Traditionally, resources like BAR windows are mapped to user or kernel address space using the CPU’s MMU as memory mapped I/O (MMIO) addresses. However, because current operating systems don’t have sufficient mechanisms for exchanging MMIO regions between drivers, the NVIDIA kernel driver exports functions to perform the necessary address translations and mappings."

This change address this problem in a accelerator-agnostic way.

2) GPUDirect RDMA has the limitation that two devices must share the same PCI Express root complex.

The change eliminates this limitation: the data transfer is at the lowest level of the PCIe tree, so you can isolate smart nics and GPU/TPU/accelerators with a PCIe switch and not saturate the PCIe root complex bandwidth.

(From https://lore.kernel.org/dri-devel/20230710223304.1174642-1-a... "Advantages")


Nifty. Thanks for the deep info - makes a lot of sense.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: