For me SO_REUSEPORT was only a solution to be able to immediately reuse a port when a process crashed in some odd way and the OS was not releasing the port immediately.
When I read that multiple processes can use the port it finally made sense to me how nginx is able to spawn so many workers. If they are listening to port 80 or 443 with SO_REUSEPORT, each worker would get a new client in a round robin fashion, effectively spreading the workload without depending on the main parent process to pipe the data to them (and back to the client), which is how I thought that it was working.
The funny thing about this submission was that when I read the title I somehow thought I was reading it on Stack Overflow and was expecting a noob asking this question and didn't know what kind of answers I was about to be confronted with and if I should get excited to see a really good one.
Then, while in the article, I liked how it started to raise so many issues, fanning out into these special cases which really made it a valid question to ask what it does mean to listen to a port, how complicated it can become.
Sadly the article does not go into any detail, like showing snippets of C code from the networking stack and explaining how it is really working, how each requirement (or kluge, for that matter) is being satisfied, but it's OK, it's an article which touches the subject in an enticing enough way while having an excuse to not deep-dive into the details by keeping it in the form of a dialog between two students.
I'm not sure about nginx, but the traditional way to share a listen port is to listen on the port, and then fork. This shares the socket with the child, and if you fork several workers, they can all accept on that socket, and the OS will figure out which one wins (which may or may not be round robin). You can select/kqueue/epoll/whatever if you prefer, as well.
> ... like showing snippets of C code from the networking stack ...
I think this point deserves extra attention. I've been finding that in situations like this it is often much faster to read the implementation source code than try to work it out from documentation and trial-and-error. Something in this (quite realistic) vignette is that there is a lot of potential to mis-learn when the Linux source code for this is probably quite short and readable.
This is a particularly useful skill for terse languages where the source code is easy to read. And a good time to feel a bit of empathy towards the poor souls who are forced to use closed source software.
In fact, I now know what I'll be skimming through tonight ... [0]. Not short, but a lot more useful than guesswork.
Aside from the code I was wondering how much of this is the spec and how much is the specific implementation of the network stack they're using. Is this the same on MacOS? On Windows?
Typically, the IP stack places the port in TIMEWAIT after the listening process has exited, in order to ensure that packets that were in flight on the network at the time of exit has a chance to dissipate before the next program starts.
But, when you say "I am OK with this co-existing", you can listen to a port that is in TIMEWAIT, and quick restart is then possible.
> When I read that multiple processes can use the port it finally made sense to me how nginx is able to spawn so many workers.
It’s the socket descriptor that matters. When the listen call unblocks and the thread accepts the connection it gets a new socket descriptor, it’s through this socket the subsequent read/write happens for the connection.
• Instead of `.postbody p::first-letter { margin-left: 18px }`, probably use `.postbody p { text-indent: 18px }`. (Admittedly text-indent has the downside that it is inherited by any block or inline-block children, so caution is required. Y’know, maybe in the general case ::first-letter margin-inline-start is actually handier when you’re deliberately just applying it to paragraphs, so you don’t need to worry about potential block children or inserting a `p > * { text-indent: 0 }`.)
• `hyphens: auto` will generally do nothing unless the document declares its language (which this one doesn’t), e.g. <html lang="en">. As for whether you want it anyway… eh; greedy line-breaking (which is all that any browser implements at this time) and hyphenation don’t go particularly well together, you end up with significant quantities of gratuitous and unhelpful hyphenation.
• `font-family: Georgia` is unwise, as you can’t depend on people having Georgia. (Windows should, but other platforms probably won’t, though some may alias it.) You should certainly include a fallback of `serif`. (And in this specific case, the site is already using a serif, is Georgia even worth changing to?)
• Some of the numbers are a bit odd, but I’m guessing that’s because you just implemented it in dev tools and copied it verbatim. Me, I’d use em for the text-indent (and a larger value, at that, probably 1.2–1.4em), unitless for line-height (e.g. 1.5, which is a smidgeon more), em for max-width (though if the whim took me, I might devolve to ch occasionally), and 0 for margin-top and margin-bottom (or possibly 0.5em—half a line or so actually works quite nicely on a scrolling-based medium, though I wouldn’t do it on a paginated screen or any but the thickest double-sided paper).
I've no real knowledge of networking, I read this and have no idea what it's actually trying to teach. I say teach because I have read comments hear about it teaching but I can't actually speak to that. I am still as lost at the end as I was at the start, probably more so.
Some of the things that you can learn from reading this were:
- More than one program can listen to the same port on the same IP address, depending on how things are initialised
- Multiple IP's can be listened to for the same port
- TCP and UDP ports are separate, so even without allowing port re-use you can have two processes listen to the same port
And lots more! For me, this was a really engaging way to learn the nuances of how the concepts of processes, ports, protocols, sockets and IP addresses relate to each other. Perhaps the format isn't ideal for everyone. Also, I have probably learned all of these concepts before, so maybe that helps.
Anyway, I bookmarked this for the next time I forget it and need to learn it again :D
The article is aimed at low-level network programmers. I learned the subject from Stevens [0] which is quite old but is still one of the standard references.
I'm wondering if there's a good summary of this somewhere. 784 pages seems a bit much to get to know the basics of networking.
Maybe it IS that complicated, but I'm wondering if it's worth it for me as a developer that only sometimes touches on these issues, to read through it.
It's not that complicated. For a basic TCP socket application, you need a couple dozen lines of code. Explaining how it works isn't going to require a 1000 page book. Of course networking and network programming is a deep topic, but IMO once you get the basics down, you're soon at the stage where "you'll know it if you need it" w.r.t. more advanced stuff and can learn it on demand.
I skimmed that book and it does go quite deep, but also has lots of asides into C programming and Unix in general.
For example, at one point it introduces you to the idea of wrapper functions to simplify error handling. This has nothing to do with socket programming or networking. There's BSD networking history, which might not be worth your time. There's introduction to Unix commands that are outdated if you're on Linux, but "be aware that some vendors place these commands in an administrative directory, such as /sbin". There's a section on Unix standards; would you like to know what POSIX stands for? Would you like to know about 64-bit architectures? Would you like to learn about memset() and memmove(), or fork() and exec()? How about fcntl() and close(), or posix signal handling, or waitpid()? Here let me introduce you to select() and poll() and nonblocking i/o. And pthreads...
It's by no means a bad book but it's about "Unix" and "(C) programming" almost as much as it is about network programming. Depending on your background that may or may not be a bad thing but I prefer materials with narrower focus.
You can follow the story as a step-by-step tutorial, while running every serverN.py examples at the same time they say they run them and then use telnet and/or netcat like they say they use it in the story.
If you really want to understand this it'll be easy enough, connect to irc.libera.chat, join say ##linux or even better #networking and ask any question that comes to mind.
I'd advise anyone to do it; this little story is one of a kind.
Does anybody else dislike the socket interface abstraction? To me, an individual TCP port + IP is, effectively, giving me a 'virtual uart'. Instead of saying COM4 or whatever, you say tcp port 12345 IP address 6.7.8.9
Similar thing for ranges of IP address, except in that case, the 'listener' has to figure out which 'virtual uart' to act upon. In that case, the OS splits the 'virtual uart' assignment with the server listening to a range of address.
Sockets seem unnaturally messy to me. Maybe I'm ignorant.
Sockets are ridiculously complicated as APIs go. STREAMS [0] was a much nicer networking API but sockets won because in the late 90s sockets were faster (supposedly) and STREAMS became untouchable because they were part of the SCO IP fiasco. Now that SCO is history it might be worthwhile to revive STREAMS and explore it again.
The COM4 endpoint only exists inside that one computer. There is one, and only one COM4 in that one computer; anybody who wants to read or write from/to it knows how to identify it.
A socket is reachable from other computers, so it must have a network address of some kind associated with it. The process that owns the port may want to refuse connections from another computer, so there has be a step in which the receiver accepts an attempted connection.
COM4 can essentially never reject either a reader or a writer.
Yes. It is bad. And it’s warped programmer brains for decades now. If you want to have some fun and design a better interface, go grab a direct kernel-bypass layer 2 NIC interface and learn how much better things could have been. Unfortunately, the sockets api isn’t going away. But, on the positive side, it’s one of the most portable OS interfaces out there.
I switched from hardware engineering to software engineering, and the socket abstraction (as well as the TCP bytestream abstraction, which appears to have been part of the inspiration for how sockets work) seemed absolutely nuts to me. Kernel bypass style queues make a lot more sense from first principles.
I remember some of this on Windows when I was implementing a prototype DNS that could saturate a 10G link.
There were two components, Receive-Side Scaling and some special user-mode thing. RSS would load-balance 5-tuples onto different cores somehow, triggering interrupts on the right core for that stream. If you were careful with the driver stack you could have zero-copy all the way to user mode, where there was some magic that let you dispatch into user mode and read packet data from a ring buffer.
Gave me an appreciation for how a single machine is made of lots of pieces, and how frustrating it is to chase perf.
The socket API is slowly going away though. On iOS devices the socket API simply isn't good enough to deal with complicated things like transitioning from cellular radio to Wi-Fi. Apple has discouraged the traditional socket API and encouraged their own CFStream/NSStream API for a long time.
> To me, an individual TCP port + IP is, effectively, giving me a 'virtual uart'.
It's not, though. A single destination can have an arbitrary number of sockets/uarts connected to it. You can server on port 80, and accept a connection from 127.0.0.1 and from 192.168.100.23 and from 10.98.54.4 and from anywhere else on the internet and have them all operating simultaneously. And in fact each destination port can have many connections to the same IP and port combination, discriminated by the source port which is a number the programmer generally never sees (unless you bind it specifically, it's automatically assigned).
It's exactly that kind of almost-right-but-not-quite mental misrepresentation that the article is trying to address.
I don't think you fully understood my point. What I'm saying is that when any comm happens between network endpoints, it's essentially identical to a uart stream.
The fact that a tuple (my IP, my socket, your ip, your socket, protocol) is used instead of COM4 is just a detail.
UARTs have 7 vs 8 bit, parity negotiation, speed negotiation and all other manner of arcane config that has to be exactly perfect for anything to flow.
I'm not sure you quite understood my point? I'm not claiming that uarts are the end-all, be-all. What I am saying is that, even with the features of 'sockets' e.g. being able to be a 'server', ultimately, when data transmission happens, it's as if you have a virtual uart.
I don't care about the specific uart-level detail here.
write() and read() work on a connected socket fd just as they work on a uart. So I don't really understand what the problem here is?
Of course you have datagram protocols and recvmsg() & co, which are useful precisely when you're dealing with something that does not looks like a uart.
Since I read from left to right its major:minor to me. The major one is the IP, since it can have 65535 ports. Its unfortunately not the same with dates. I prefer ISO 8601. Major:minor:micro (or whatever you call it with 3).
Go is very much a descendent of Plan 9. It’s written by Rob Pike of Bell Lab / Plan 9, is heavily inspired from languages such as Alef https://en.wikipedia.org/wiki/Alef_(programming_language) which ran on Plan 9 and, if I recall correctly, Go’s original compiler was basically a modified compiler for one of this Plan 9 systems languages (to enable Pike et al to speed up the release
of version 1.0).
- the OS maintains a hash map mapping port/IP/protocol/protocol-version to lists of sockets
- there's a scoring algorithm that determines which socket wins, based on specificity of how the socket(s) were bound
- if multiple sockets are bound with SO_REUSEPORT, things are routed based on a hash of the source/destination address/port, presumably so packets from a single client aren't split between different handlers - essentially, simple zero-backpressure load balancing
- read the article for nuances of when and how to use this, especially performance considerations about pre/post fork
- consider adding TL;DR segments like this, whether you're writing discovery fiction like the OP, or technical deep-dives like the link in this comment. Different readers will prefer different formats, but everyone benefits from having the option to consume an abstract before reading lengthy nonfiction prose!
How does ICMP routing work? I always wondered about that. There's no port in ICMP, how does OS understand which ping executable receives ICMP echo replies.
Thanks. It seems that's not exactly correct. ICMP has type, code and checksum fields. Rest of data is type-specific. But echo-request and echo-reply types do have identifier and sequence fields indeed.
Fundamental? I see it the other way -- this is an illustration of how neat, fundamental concept ("When you bind to a port, it puts a pointer to your socket in that table.") is actually full of random hacks and ad-hoc changes. The linux kernel is full of random logic like this, and this article could be continued (there are at least network namespaces and sys.net options to consider).
Don't get me wrong -- all of this functionality is useful in some context, and it is there for a reason. We cannot radically simplify the kernel without making it less useful. But it is still somewhat sad there is so much essential complexity even in simple things like "listen" call.
TCP and UDP don't really dictate any of this behavior. Most of these hacks are the result of design choices in Linux and the socket API - abstractions on top of TCP/UDP that aren't the only option.
In my experience, the network layer is often way better-designed, more reliable, more interoperable, and less hacky than most OS abstractions built on top of it.
Well only sort of. As far as I understand, nothing about TCP means that ports can be reused, or that 0.0.0.0 has special meaning, or that processes can’t listen on the same port, or IPv4 to IPv6 delivery etc. These are all operating system design choices, some of which stem from the sockets API.
I’m pretty sure you could design a spec-compliant alternative which didn’t do any of these things, if you really wanted to.
I have to admit I'm not a fan of "discovery fiction" as presented in this post, but reading Michael Nielson's linked post and some of his examples (e.g https://michaelnielsen.org/ddi/why-bloom-filters-work-the-wa...) I think the idea is great if executed well.
The goal isn't to embed information in meaningless dialogue, but to walk the learner through an entire thought process rather than just presenting the finished result. Or to use one of tech's favorite expressions, to construct an idea from first principles.
I think the 'port' concept inherits so much abstraction, it changes the meaning and how it really works.
When I started out, a port was (and still is) just a designated memory location, and address, where some process (like a NIC driver) can write byte(s) into it. 'Listening' on a 'port' just meant the memory location is checked by a piece of code either through polling or interrupt driven. The interrupt is triggered by the same process that placed the data into the mem location.
Abstraction married to time is a is the enemy of knowledge.
Thank you so much for confirming this - I always wondered whether that's all it actually meant beyond the abstraction. TFA doesn't actually explain it despite the title(!)
I suspect the concept of port might not have been needed had we started with IPv6. As it currently stands we allocate each physical host a /64. That would mean that even if we by default used port 1 or whatever we would basically never run out of possible service end points. An IP address + port is just a way to multiplex an address.
It is kinda fun to work directly at IP level and not use TCP or UDP at all. I highly recommend giving it a try.
> It is kinda fun to work directly at IP level and not use TCP or UDP at all.
Possibly dumb question: if you're not using ports (because you're not using TCP at all), what does routing look like, in terms of this article? How does the kernel decide which traffic goes to application A and which traffic goes to application B? Or is all traffic directly visible to each application, if they aren't bound to a port? Can you bind to a specific IP, or just to an interface?
The normal way to "work directly at IP level" (or even below) is to use a RAW socket. The kernel will deliver all traffic matching the IP (or other things - raw sockets can essentially own the whole NIC) to your socket, and from there it's all up to you. The kernel will only look at the IP headers, the rest of the packet (TCP/UDP headers specifying ports, for example, if any) will be delivered to the raw socket.
Not sure of the implementation details, but in principle the same logic about specificity of socket binding could be done by the kernel when deciding which socket to actually send the data to. Any other logic (CRC validation, packet ordering, confirming packets have been received, MSS etc.) are all disabled, you just get the raw packet as it was received on the wire, based on the IP headers (an the same happens when you send a packet on a raw socket).
There are many more questions besides you've ready asked; basically, who or what is providing TCP-like transmission reliability and synchronisity, who's doing ICMP , who's doing fragmenting, and why does one need to reimplement better part of the stack
What would be a good search term to dive into "work directly at IP level and not use TCP or UDP at all"? Sounds interesting but not sure how to get there from just a basic familiarly with sockets.
We do not allocate each physical host a /64; we (usually) allocate each network segment (broadcast domain) a /64. Of course, for p2p use cases that ends up with a single-host network.
Most of that address space is used to allow for stateless autoconfig, though. It would be certainly possible to fit in port multiplexing, but you'd have to make a trade-off in increasing collision risk, depending on how you do it. Or go back to static addresses or DHCPv6.
If a packet hits a pocket on a socket on a port,
And the bus is interrupted as a very last resort,
And the address of the memory makes your floppy disk abort
Then the socket packet pocket has an error to report!
— Gene Ziegler, "A Grandchild's Guide to Using Grandpa's Computer" (1994)
I made extensive use of the SO_REUSEPORT flag a couple decades back to build agents that on the same machine would have their own UDP socket which received data from a local piece of software. Basically, by using that flag each socket received a copy of the buffer, so that each agent could act upon the contents ("is it for me? - if yes then read this field and act accordingly, otherwise ignore"). Very handy, despite all limitations of UDP. It allowed me to keep the code very tight and specific for every agent, although network buffer structures were all in common so upgrading the protocol to all agents was easy.
With some adaptations, it worked both on Windows's POSIX stack and Unix.
My recollection of SO_REUSEPORT behavior is that it will deliver a datagram to one listener, rather than all of them. The kernel will distribute datagrams fairly between listeners over time, so none are starved. This makes it super easy to share load over multiple servers, but doesn't get the same datagram to multiple listeners.
The freebsd page doesn't claim that it delivers everything to every listener. It says that's possible if multicast or broadcast is configured. That won't happen by default.
> This option permits multiple instances of a program to each receive UDP/IP multicast or broadcast datagrams destined for the bound port.
Yes, that was the case. Also I probably confused SO_REUSEADDR, which in Windows is implemented quite differently, with SO_REUSEPORT.
Anyway, I can confirm that on Windows (2000) I could open N programs whose receiving socket was bound on the same address:port and all of them would receive a copy of the transmitted datagram. If say I run 5 copies of the same agent, all of them would display the same data. Those were stand alone applications without any shared memory, so apparently the system made a copy of the buffer for each socket. I'm not sure however if I tested this particular behavior also under Unix, since I needed it only on the graphical interface, but on Windows it definitely worked.
I’m gonna come out and say it: I think ports were a mistake in TCP.
It makes way more sense to think of an IP address as the entire “address”+port tuple. When you look at it like that, so many aspects of computer networking get simpler. First, we think of every computer on a network as having a range of addresses - usually 65536 of them. But that’s an arbitrary choice that could be adapted in lots of ways. Every connection directly connects two IP addresses - which is way simpler than what we do now. DNS should associate a name with an address - including what we currently call the port. That way we wouldn’t need special application specific port assignments. And then we wouldn’t need SNI and all that - because computers could have a bunch of fully process-isolated web servers, just listening on different incoming address+port tuples.
Obviously we can’t change tcp/ip, but parts of the tcp/ip infrastructure already work this way - but we’re lacking the language to talk about it.
This explanation doesn’t sound simpler at all, it sounds like the complexity (each computer runs multiple services) is just obfuscated by making it part of the IP address. It would make routing a nightmare since the router now needs to keep a table of all services running on all hosts to know where to route traffic.
Routing would be no different. Routers already work with IP address prefixes. In the scheme described, a computer's IP is just a /32 prefix of a 48-bit address, and serves as the router for the 65k addresses.
The biggest difference is language - we don't have to think about addresses and ports being different ideas. Our APIs and textbooks get simpler.
And from an address perspective, its better because we end up with much more flexibility around how our networks are designed. Want to use 1 public IP address with 10k web servers? Now you can! Want a single machine to handle 10M outgoing socket connections? Sure - just allocate more addresses from your pool.
You can already do some of this stuff right now, but its awkward to reason about because of how ports and addresses interrelate. It'd be simpler and easier if we just think of the (address,port) tuple as a single address entity.
The main difference is that 'ports' live under the protocol layer (TCP/UDP/...) - TCP port 22 and UDP port 22 are not in the same address space
In many way's it's a very historical thing, back when TCP/IP was one of many competing network architecture (early 80s) most of the competing architectures did it the other way (there was only one port at that level and protocols ran under that). Is one better than the other? probably not much - TCP/IP won (IMHO) because they understood datagrams vs. virtual cicuits - they didn't plan on charging for TCP or IP level retries.
> DNS should associate a name with an address - including what we currently call the port.
This is something I really agree with. For an example that does naming right, Google's internal BNS addresses[0] in fact resolve to a host IP address plus port. Much more convenient because conceptually now a single piece of information is needed to identify a specific running instance of an application on a machine.
Unfortunately web browser, web apps and websockets can only use a few ports, so if you run a service that you later want to interoperate with a web app of some sort, you don't really have a choice. In a way, ports are already becoming obsolete for some applications.
Web browsers are actually quite flexible about what ports they can use. However some ISPs and “enterprise” networks will filter traffic on non-standard ports. That’s the real problem.
I've always viewed (and I'm talking about IPv4 here) that the IP layer allows two computers to exchange data, while UDP and TCP allow two processes to exchange data. In other words, IP names a computer, and UDP/TCP name a process. It's probably not really needed for IPv6, but the concept was carried over since it was already there.
IPv6 I think tries to do this with link-local addresses. It doesn't make things simpler. It probably is supposed to but I've yet to really understand it.
Ports are needed. Destination ports map to "listening" processes on a single machine; the abstraction is needed because more than one process on a system can use the TCP/IP stack.
> It makes way more sense to think of an IP address as the entire “address”+port tuple.
There's already a term for that - they call that a socket.
SNI has to do more with the "Host:" header of HTTP and the behavior of HTTP servers than anything else. Web servers started using that due to IPv4 address exhaustion and the need to host multiple "sites" on a single IP. IPv4 scarcity created a lot of twists in the end-to-end design, including this and NAT.
I think it would be cool if DNS gained the explicit capability to resolve names not only to IP addresses but also destination port. Not sure why it didn't. Certain apps can use SRV records or whatever but it's not standard.
In IPv6 each host has 2^64 addresses and the kernel just advertises its configured prefix to its router which then delegates routing to it. Instead of one address multiplexed onto 65535 ports, since every host now has 1.844674407371e19 addresses just use those.
That does not seem easier to think about, which address was this process on, you now have 1 interface with possibly 100s of addresses, need to keep track which address is for what application..
Seems more logical to just do the IP+port tuple. Maybe having TCP and UDP on separate port "namespaces" is the confusing part here and someone above mentioned could be simplified.
It's already typical to have multiple IPv6 addresses per interface, due to privacy extensions. You can see which IP address an application is listening on with netstat in the same way you see the port number. You would need to allocate addresses differently—each host would get a /112 instead of a /128 so that the port number can be placed in the lowest 16 bits, and the tooling (e.g. "ip addr") would by default show the 112-bit prefix assigned to the host rather than hundreds of individual /128 addresses.
The real casualty from merging IP addresses and ports, however, is that it would no longer make sense to listen on a certain port on "any" address/prefix (::/0); if there is more than one interface or prefix assigned to a host and an application wants to accept incoming connections from each of them it will need to listen on multiple sockets, and open and close those sockets as prefixes are added or removed.
SRV solves so many issues. It was codified as a standard 22 years ago. Maybe because it was invented at Microsoft, and the "year 2000 M$" was evil, the free software/open source people weren't interested in it and now it's mainly used by Microsoft products; as far as I can tell from my own experience with it.
Yep, that would be fun, and could obsolete DNS for some use cases. Though having variable length addresses in headers would be a headache for a lot of things.
Not a blog post, but https://howdns.works/ is a really brilliant (animated) webcomic explaining how DNS works through a story. Each panel has some title text as well.
That was fun, but more about Linux than ports in general. I was expecting something more along the lines of a computer checks the port of every incoming packet and if it matches something it is expecting it does something. Though this was much more interesting, and now I'm curious about even more specifics of how Linux handles this.
It talks about mapping a port (and ip address, and protocol) to a process, but on Linux (and I think other unixes as well) that isn't quite accurate. It maps to a file descriptor, and that file descriptor can potentially be share with other processes (for example by forking, or passing it over a unix domain socket). One common way this is used is to pass a socket file descriptor from the old process to the new process when restarting a service.
> Tim’s attention turns to the code, and he puts his finger near the screen to read over it. Liz, who hates a smudged screen, says “easy there!” and pushes his hand back. “I wouldn’t touch it”, he protests. Making an exaggerated show of keeping his hand a safe distance back, he points to the [...]
Okay, that was pretty good. I've long since lost track of the number of times I've had this exact exchange.
There doesn't seem to be a Manga Guide to TCP/IP yet.
Personally, it's divisive for me too --- if I'm looking for specific information, this gets in the way; if I'm looking for "edutainment", then it fits.
I think this style of writing can work - but it works best if the story is something other than two people just talking about 'how do you think this thing works?'.
Like if you can pick any story, why make it about two people in a study hall, studying and directly asking each other the question? It could be a story about some rogue hackers, international spies, or the characters could be ducks rather than humans, or you could tell it from the perspective of a sentient computer or application... So if you are going to make it into a fictional story, you might as well make it a fun fictional story (either by the premise or by a character!).
I think that the process of discovery is engaging, and far more relatable than a man page. Maybe the prose went a little further than it needed to, or as you suggest not far enough. I’m just saying that the mechanism of storytelling exploits some attributes of our evolutionary biology that make learning a concept more relatable and memorable.
I agree with the process of discovery - but would personally prefer that to be communicated in the first person (which I personally find clearer and more enjoyable to read).
As in telling the true story which is probably “I wondered how ports work, then I did this, then I wrote this script, then it had an unexpected result”, which still reads as discovery to an audience (but is the authentic and true story).
IMO Readers will find a true story interesting if told with authenticity and passion.
For fiction, you generally need ‘an ununsual thing’ in the story that happens that breaks it away from normality.
I think this treads the awkward space inbetween - it’s made up (so no authenticity) but that authenticity isn’t replaced by something ‘unusual’ that drives the story forward.
> Maybe the prose went a little further than it needed to, or as you suggest not far enough.
I think this is right - although I don’t think there is a happy medium between those points. I think you can go in either direction (further or less), but straddling the middle is the tough bit.
I wonder if it's like recipes where the author writes in an elaborate style to prevent theft of their material. "I discovered this pizza recipe when I was hitchhiking through Naples after the death of my father. I was trying to find myself...". Instead of just typing out the recipe.
W...what? I own copies of the Databases and Microprocessors books, and haven't seen anything like "up-skirt shots of teenage schoolgirls". What are you referring to?
Well done - thanks. A good dialogue, reminds me a bit of those written by Plato (okay, apart from the difference in topic); I wish I had more inquisitive students like Liz.
Right!? This just shows by example the basics of calling the socket syscalls via Python. I still do not know how these syscalls work or what it really means to be listening to on a port.
Had to read quite a bit before getting to the core insight:
> “So when you listen on a port, you’re really listening on a combination of a port, an IP, a protocol, and an IP version?”
> “Yeah, unless you listen on all local IPs. And if you listen on all IPv6 IPs, you also listen on all IPv4 IPs, unless you specifically ask not to before you call bind.”
> “Right. So the operating system must have, like, a hash map from a port and IP pair to a socket, for each combination of TCP or UDP, IPv4 or IPv6.”*
> “To a list of sockets”, Liz corrects. “Remember how I could listen on more than one?”
> “But it also has to handle listening on all ‘home’ IPs, and to be able to find a socket listening on IPv6 from an IPv4 IP.”
"and an IP version" is more of a matter of interpretation though. I'm pretty sure you can model that behavior in terms of binding to an IPv4 address and binding to its associated IPv6 address. Rather than binding to the same IP with different IP versions.
And it's worth mentioning that IP doesn't have ports. You don't take a port and then divide it into TCP and UDP use. TCP and UDP each independently build their own version of ports.
thanks for this summary (the lecture was pretty boring to me). As i understood it, the article explains what happens when you listen to same or similar combinations of a port-ip-protocol-whatever...
but i didn't learn what does it mean to "listen" on a port.
how does two programs communicate by this way?
The port number is one field in a TCP or UDP packet. When you listen, you register with the operating system (or whatever controls the network card, I guess) saying that you want a particular port. When a packet comes in (on the network card), it will be routed to you (on your socket) if it's addressed to you (by including a port number you're registered for).
You can think of the port number as the second half of your IP address. As far as the networking goes, an IP address and a port number are basically the same thing. The port number is just the lower bits of the "combined IP address".
This article has taught me the concept of "discovery fiction" and, I'm sorry to say so, but I already hate it.
I get the idea of presenting concepts in a more natural flow and smuggling in some spaced repetition, but the whole story just felt extremely forced and artificial to me, sort of like a drawn-out sequence of expospeak.
There has to be a better way to include spaced repetition and discovery in teaching.
You can see there's no dialogue; the idea is to frame an article as "how might you discover/invent this concept on your own", rather than just directly explaining how a thing works. You are given a fictional goal, and you "discover" the subject matter by building iterative solutions.
This example that explains kerberos is entirely dialog
https://web.mit.edu/kerberos/www/dialogue.html
Really helped me understand kerberos in a way other articles and technical documents had failed to.
One thing I've learned about pedagogy is that it's a fool's errand to try to create material that works for everyone, because everyone learns differently. In fact, the meta story here is supposed to be about Liz and Tim having different styles of learning -- Liz learns by exploring off the beaten path, while Tim prefers to stick to the course content. Neither are wrong.
Ok, I didn't realize this aspect of the story, this is actually pretty cool, I agree.
I also like that the story encourages experimenting and testing out your mental models, like Liz did.
It's just the particular way they discover things that feels forced to me. The story is all about them coming to their own conclusions and thinking up their own experiments, but - it being a story - you know, it's actually all guided beforehand.
Maybe what irks me is that this concept of a character discovering "on their own" some ostensible deep truth (which is really just the author's personal opinion about something) has been used for a lot of worse reasons in other stories - even though, in this case, the "truth" is perfectly harmless and beneficial.
Anyway, it's always easier to criticize than to create, so I won't say I really have better ideas of how to do it.
>> I've seen somewhere that "everyone learns differently" might not actually be true.
Even if we assume that people all learn the same way, good learning integrates new facts or concepts into ones pre-existing mental model of the world. Not everyone has the same mental model - of this I am certain. Sometimes new information just hangs on the existing model, and sometimes the existing model needs to be updated. This can make the process seem like everyone learns differently, since different explanations can make more or less sense depending what's already in their head.
On top of that, I think some people have (maybe inherently) very different abilities in things like visualization, memorization, vocabulary, etc... So yeah, I think everyone learns differently even if at some neuronal level it's all the same.
This is saying that VARK method of describing learning styles is incorrect. This does not necessarily mean that there aren't differences in what are the best method's for different people learn. One issue with VARK is that it involves self-identification, as was the case in all studies mentioned in the video. There's no reason to assume that the way people prefer to learn is actually the best way for them to learn, it may be that people choose what seems right to them rather than what actually is. Another issue is, as is mentioned in the video, the domain you are studying also has an affect and multi-modal approaches are most useful. Whether learning styles exist or not, some information intrinsically is better presented as a diagram than a wall of text. And no matter if some people learn better in certain ways, it seem universally true that people learn better if information is presented to them in multiple forms instead of just one.
Which is all interesting, but mostly just disproves VARK and similar approaches to describing differences in how people learn. There are still so many different ways to teach someone something, they're just much more holistic ways of teaching than the simple VARK split. That some people learn better from X course of teaching and others learn better from Y course of teaching still seems likely to me(admittedly, just pulling from personal experience and the anecdotes of others on that ). That we don't have a neat way to categorize that might just mean it's too messy to do so, or could mean we just haven't figured out the right way to look at it yet.
I think “everyone learns differently” is an overblown claim and gets repeated ad-nauseum. It is an escape hatch to avoid deeper discussions of successful strategies, what it means to teach and learn. A sort of a deus-ex-machina of arguments. There are more universals about learning than differences. It effectively surrenders the failure of teaching to “Well, it must be because everyone learns differently”. This way, we can never learn to teach or even debate about it.
This is actually a failure of discourse in many areas of life – "X is hard to solve or haven't thought about it, so it must be that X is different for everyone". Nutrition, Product Reviews, etc. I've learned over years that anyone that claims "X is different for everyone" is most likely exhibiting a defeatist attitude.
The universals about learning, which I've taken away from the discussion, is that active learning and recall is generally effective, while passive learning (e.g. re-reading) is less effective. A good research paper that covers this (which I've mentioned in that discussion) is called "Improving Students’ Learning With Effective Learning Techniques: Promising Directions From Cognitive and Educational Psychology" [PDF]: https://pcl.sitehost.iu.edu/rgoldsto/courses/dunloskyimprovi...
As others have pointed out (e.g.: https://news.ycombinator.com/item?id=30324567 ), the style in this article is not what I had in mind when I coined the phrase "discovery fiction". It also has nothing to do with spaced repetition or having a "more natural flow".
Sorry to drag you into this! I meant “inspired by” in the literal sense, not as a claim that it would fall under your definition. I mentioned so when I tweeted it, but not in the post itself https://twitter.com/paulgb/status/1492923557801869318?s=21
For myself - and I know others respond very differently - I usually want everything in an essay to serve the overarching point. So I find fictional asides pretty distracting. There are exceptions: Imre Lakatos did it well in "Proofs and Refutations", and Douglas Adams did it well too. But it's tough to pull off!
My first exposure to programming was a book my dad gave me called The Little Schemer, and I absolutely loved it. It's like discovery fiction without the fiction. It just asks a question and then immediately answers it; mostly composed of code with minimal explanation. It feels like a very natural progression of being exposed to something new, getting familiar with it, and starting to explore all the nuances.
I can imagine TFA re-written without the dumb story being really good. It makes use of spaced repetition, and it does a good job of introducing one piece of information at a time. But I agree; the story part of it is stupid.
I wholeheartedly concur about The Little Schemer, though well into adulthood when first exposed to it. It works so well due to combining a Q&A format with a respectful "tone" that never talks down to the student-reader. The authors' subtle sense of humor contributes too.
To be sure this article isn't in the same league at all. Great teachers are scarce no doubt because few are gifted with the requisite raw talent. It implies you're right, it's hard to produce top-quality written teaching material, after all excellent examples are uncommon, even vanishingly rare.
Trigonometry The Easy Way, was brilliant! I wouldn't categorize it as forced in the manner of the OP. The book creates a fantasy backdrop to connect trigonometrical concepts in a fun way. I really enjoyed the book and it made trigonometry really stick in my brain!
I personally just hated the long-form-journalism style of "okay, at the start, you have to wade through a ton of irrelevant crap about the settings and the coil-bound notebooks they're using".
>I personally just hated the long-form-journalism style of "okay, at the start, you have to wade through a ton of irrelevant crap about the settings and the coil-bound notebooks they're using".
Yeah, I'm really glad this was the top comment. That style of journalism has become all pervasive today, and the grotesque self indulgence of it is honestly sickening. I refer to it as "college term paper journalism", because they all read like a sophomore English 102 essay.
I'm convinced lots of sites do this just to push out the actual content below the "please login / buy subscription to continue reading" point, or if it is a free article at least make you scroll through as many inline ads as they can.
It's way too long since I read it, so I can't really compare, but I remember that the whole topic was actively discussed and reflected in the novel. That's something different.
Like, Sophie being a character in a novel was an important plot point of just that novel, if I remember correctly.
Haha entertaining. I rather like this style. I skipped till the first code and then the interludes were short. But I think I would have been annoyed if the first code block was not demarcated clearly so I could skip to it. The prelude was too long for me to read since I didn’t know if there was going to be payoff.
agreed. it's not useful for me as a more "tech experienced" person and having taught many people just breaking into the industry i don't think it would help them much, either.
people are often helped by metaphor (e.g. "a port is like a mail slot in a huge mailroom...") but i feel like this "discovery fiction" thing is like an inverse version of that... it allows your brain to be creative when thinking through the topic but just focused on the wrong part of the explanation.
It'd be ideal to have a very succinct and clear 3 sentence articulation. Followed by maybe a couple of examples.
If we're exploring the mysteries of the Universe, stories help, but our own arbitrary creations? They are just tools. They should be simple, robust, anti-complex and as mundane as we can make them. T
I've read the whole thing patiently even though I am not sold on the format of such stories -- but it was still informative.
The only thing I thought at the end was: "yeah, the whole thing should be torn down and redone from scratch". Which of course isn't ever happening, right?
I was very interested by the question, and closed the page after skimming through the first screen: why is there any need to scenarize? why would I want to read about some guys in a bar? Can't we make it interesting to talk about what it really means for a device to "listen on port x"? In my experience (listening to Collège de France podcasts), any topic, even the most obscure and narrow field of knowledge, say "Late Babylonian scriptures as seen by Ibn Amhoud during 8th Century" (invented), can be made a thrilling experience if the speaker is really passionate and knowledgable about it. But there is never any need to start by a conversation at starbucks: one just dive in head first. </rant>
So on one hand, I absolutely hated the writing style as well... on the other hand, its some guys blog and if he likes this style and it makes him happy to write educational stuff like this, even if its not an ideal format for us then should we really judge?
Brings to mind Elements of Style. Place yourself in the background, don't overwrite, don't overstate
"Write in a way that draws the reader's attention to the sense and substance of the writing, rather than to the mood and temper of the author. If the writing is solid and good, the mood and temper of the writer will eventually be revealed and not at the expense of the work. Therefore, the first piece of advice is this: to achieve style, begin by affecting none — that is, place yourself in the background."
I agree that it is probably not the optimal way to write an educational piece. However, in our year of 2022, the amount of content available online is absolutely staggering. If there weren't at least some pieces out there that tried to scenarize concepts like this, I would be disappointed - it's an experiment worth attempting.
I think the idea is that people who have had done their Computer Networks homework at the college coffee shop with their friends Tim and Liz can identify with the article. It's a niche audience but it's the only reason the author can imagine anyone would be interested in it. It's unappealing to me because despite having (re-)imagined this whole scene himself, the author can't really imagine why the reader would want to know, why I would want to know. If he could, then he would have motivated it that way instead.
Its making a point about embodied learning, curiousity, and work as a healthy part of life. Imo, the framing is a hell of a lot more interesting than the implementation details of port binding.
This is horrible to read. I gave up after the first screen page. When I'm trying to extract knowledge from a post or an article, I don't want it padded with useless prose that I need to skip when parsing the text. Like an invasion of this fluff style of "long read" that has become fashionable in what used to be journalism.
Interestingly, the life stories surrounding recipes may be a form of copyright protection.
> In the U.S. copyright law doesn't protect "a mere listing of ingredients," but "where a recipe or formula is accompanied by substantial literary expression in the form of an explanation or directions... there may be a basis for copyright protection."
This is from an article about a tool developed to extract recipes from their blogs:
It would be improved by changing the framing from a discussion to singular persons internal dialogue showing progression and train of thought. I’ve used this before in a work context to explain complex debugging issues.
“Hmm that doesn’t work, what about X”
<code>
“Oh but that didn’t do what I wanted. What if we added Y and Z”
Not a networking expert, but like classic literature (and some modern stuff). The writing, especially the dialogue sounds like a primary school first writing assignment.
> Anyone care to comment if the fall into either group?
I know a thing or two about networking and didn't really learn anything new from the article.
And I don't really like the style, but I don't feel like my expertise with networking plays much into it.
Why didn't I like it? Hmm, first of all, I skipped the italic blurb and jumped straight in. So at first I didn't know I was reading fiction. It sounded like story time, as in a story of something interesting that actually happened, and for which you may need to give a bit of context.. I don't have a great example in mind but think something like the 500-mile email (https://www.ibiblio.org/harris/500milemail.html). That's the kind of story I expected I was getting into. And then I was disappointed that there was no story.
Once I got over that and looked back.. well, one obvious thing that's missing for me is motivation. “Yeah, I know that, but how?” Liz says. Why does she care? Why should I care?
I immediately got the same vibe that I get when I come across someone with an X-Y problem and they're the stubborn kind who refuse to explain why or what they're actually trying to find out. Or when I come across someone who's curious but trying to pass the burden onto someone else (kinda like a help leech except that they don't even have any concrete problem they need help with). It wouldn't be the first time I've said on IRC something like "Sorry, I don't know. If you really want me to read the kernel code for you and tell you how it works, I might do that later tonight but I figured you could satisfy your own curiosity." Someone wants to know something but there's no real motivation for the next person to care. I guess I kinda feel how I imagine Tim feels: he can guess what the OS might be doing, but he doesn't have motivation to find out more.
And then there's the fiction-fluff that doesn't really do anything for me. Liz and Tim aren't interesting, there's not much personality, and even if they had personality, there's not much reason to care; they're just random nobodies. The setting isn't interesting. I don't care if there's a coffee shop because the coffee shop also isn't a key element of the story; arguably, there are no key elements. There isn't anything exciting going on, Liz just wants to know how Linux demultiplexes TCP and UDP. So all the fluff feels superfluous and forced.
It could work if there was an interesting story. A reason to build something, a reason to find something out, a reason to poke the kernel around a bit. And in that case, you wouldn't need so much "fluff"; the narrative could support and enrich the story (explain things and continue to add motivation) rather than just pad it. Writing that sort of educative story is really hard though.
When did it become normal to waste readers' attention on technical articles:
"In the corner of the student union building there is a coffee shop, and in the corner of the coffee shop are two students. Liz taps away at the keyboard of the battered hand-me-down MacBook her brother gave her when she moved away to college. To her left on the bench seat, Tim scrawls equations on a coil-bound notebook. Between them is a half-empty cup of room temperature coffee that Liz sporadically sips from to stay awake.
Across the room, the barista looks up from his phone to glance around the shop. ..."
It was the weekend, and I had fun writing it. Nobody is forcing you to read it. But I'm certain that if I wrote it in the style you wanted me to write it in (which is how I write most of the things I write!), you never would have seen it :)
Of course, just as nobody is forcing you to read my complaints :)
> you never would have seen it
On the contrary -- I clicked it only because of the title. But I stopped reading after the first paragraph, so I still don't know what does it mean to listen on a port, exactly because of the style, too bad for me. :)
But I strongly support your right to write any style you want, I'm avid supporter of authors' "right to burn".
Thanks! I don’t currently, but I tend to turn them around pretty quickly when I get an idea (this one was just an idea until the day before I published).
Exactly. After first sentence I immediatelly switched into "scan mode". I was'nt able to find the answer for the question in the title.
I was even not able to identify a section where the answer could be burried in.
"'No', our main character Tim, a computer science major from Ontario, the capital of Ottawa, Canada, mystifiedly said, shrugging his broad, well-toned shoulders."
If he was a good writer, maybe he could have made it work. But this reads like a "write what you did yesterday" assignment handed in by a 12 year old. If even.
It isn't interesting to list the generic shortcomings of a post. We want curious conversation here.
The moderation comments I post are for sure repetitive and tedious. The justification for them is not that they're interesting in their own right (they aren't! and they're even more tedious to write than to read). It's that without them, HN would be globally worse off. They're an out-of-band feedback channel forming one component of the system by which the forum regulates itself.
You can compare them to medicine which is toxic but which one takes anyway because the alternative is worse.
The style can work. In this case it largely doesn't, mostly because the writer insists in trying to be cute about "Tim". Tim is a sockpuppet for the author, like Socrates in Plato's works. After the first couple of paragraphs, there is no point pretending he is anything but that. Trying to flesh him out, over and over through the text, is a waste of everyone's time.
When I read that multiple processes can use the port it finally made sense to me how nginx is able to spawn so many workers. If they are listening to port 80 or 443 with SO_REUSEPORT, each worker would get a new client in a round robin fashion, effectively spreading the workload without depending on the main parent process to pipe the data to them (and back to the client), which is how I thought that it was working.
The funny thing about this submission was that when I read the title I somehow thought I was reading it on Stack Overflow and was expecting a noob asking this question and didn't know what kind of answers I was about to be confronted with and if I should get excited to see a really good one.
Then, while in the article, I liked how it started to raise so many issues, fanning out into these special cases which really made it a valid question to ask what it does mean to listen to a port, how complicated it can become.
Sadly the article does not go into any detail, like showing snippets of C code from the networking stack and explaining how it is really working, how each requirement (or kluge, for that matter) is being satisfied, but it's OK, it's an article which touches the subject in an enticing enough way while having an excuse to not deep-dive into the details by keeping it in the form of a dialog between two students.
I liked it.