
How PCI express devices talk - sacheendra
http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-1
======
luckydude
This is pretty cool, thanks for the intro.

The funny thing is this is _almost_ what I wanted the system interconnect to
evolve towards back in the early 100Mbit ethernet days. After AVB and I
wandered around the valley convincing the card people that you could signal
over copper at 100Mbit, and they bit and started building cards, I started
thinking about the future.

It was fairly clear that the hardware guys got it at 100Mbit, they could keep
the packet format and keep making the interconnect faster and faster. The hump
they had to get over was that the packet was the thing that had to stay the
same, you can use whatever tricks you want to signal at the next speed level.
So I was thinking wouldn't it be cool if they really pushed it so that it was
viable for system interconnect. I half heartedly pushed for it inside of Sun
but never got anywhere. At the time I was pushing for ethernet out to the disk
drives (with power over ethernet so you just plug in more storage, easy peasy)
but I definitely had the switched interconnect in mind as a replacement for
the system bus.

So this seems like that vision, a switched network, but at a slightly
different level than ethernet, it's got more end to end stuff in it.

I sort of wonder if we'll get there at some point, where the system
interconnect is sort of like Formula 1 ethernet and the networking cards are
like commercial car ethernet where some of the Formula 1 stuff has been
trickled down to high volume stuff.

Eventually we'll run out of ethernet addresses though I bet anything we could
just have a rollover, start reusing any address that is more than 20 years old
(or for that matter, just have a block of addresses that are domain specific
and are reused in each computer's internal interconnect).

------
jwbensley
If the author sees this message, thank you! Just the kind of tutorial I was
looking for!

I'd love to see one that built on this a little and showed a packet from
kernel space in one device being sent over a PCIe NIC and then bring received
by another device, and DMA'd from the receiving NIC into kernel space etc. I
already know that whole process more or less but it would be nice to see it
and have my understanding confirmed :)

------
ktpsns
I love the comic on the right
([http://xillybus.com/media/comic-300px.jpg](http://xillybus.com/media/comic-300px.jpg)).
Never heard of this software yet. Amazing that it is part of the Linux kernel
(in terms of part of the Ubuntu distribution, for instance).

