Hacker News new | past | comments | ask | show | jobs | submit login
Falsehoods programmers believe about TCP (lwn.net)
290 points by todsacerdoti 50 days ago | hide | past | favorite | 223 comments



I find this "falsehoods programmers believe" format of making pointed claims that you intentionally don't clarify to be unhelpful and obnoxious


Yep, I work on low level networking software professionally and this post is largely meaningless dribble and is probably motivated by grandstanding.

It’s like an engineer who says “how does a screen show black” and then says “nope” to every response. It’s maybe a way to make people think, but beyond that the negativity and grandstanding of it is ultimately a turn off for many receivers which eventually either then has them bully others this way or deters them from the field, depending on how it affects them. There are far better teaching methods that work better for everyone and teach faster and result in higher accuracy and retention.


I thought you were probably exaggerating, but yes. I've never heard anyone make anything resembling any of those claims.

What I have said is something to the effect that if TCP isn't reliable over a given path, there's not a whole lot I can do about it as an application engineer short of making my own ad hoc, informally-specified, bug-ridden, slow implementation of half of TCP inside my own app, which I'm not going to do.

> 14. Weird networks that are not transparent to standard protocols are an aberration. I can safely ignore them.

I certainly can and will. If you wanna run an RFC 2549 network, I'm going to spend approximately 0 seconds making my app support it. If you want to do something weird, you make it work. I'm going to optimize for the other 99.99999% of customers.


The author probably doesn't understand the answers very well themselves.


Doesn't the author basically admit that when he prompts someone else to write "falsehoods programmers believe about TCP"? After all, if the author did have the understanding himself he could do it himself.

The reset of the writing just laments the pain of using products that make incorrect assumptions – continuing in same lamenting from the quoted segment he includes. It almost has nothing to do with TCP at all, so it is not clear where the parent comment here got the idea that it was trying to teach something about it.


Uhhh....how does the screen show black?


The screen knows what color it displays at all times. It knows this because it knows what it doesn't. By subtracting what it does from what it doesn’t, or what it doesn’t from what it does (whichever is greater), it obtains a difference, or deviation. The controller board uses deviations to generate corrective commands to drive the display from a state where it does not display black to a state where it does, and arriving at a state where it displays black, it now doesn't display anything.


For the uninitiated: The Missile Knows Where It Is[1]

[1]: https://youtu.be/bZe5J8SVCYQ


There is also version for those already initiated [1].

[1] https://www.youtube.com/watch?v=_LjN3UclYzU


I’ve just broken out in a rash. Thanks.


Each of the pixels is actually a little shining eye which watches your every move. When the pixel’s eyelid closes, that pixel turns black. That’s why they call it putting a display “to sleep.”


I like your explanation, but to be fair, it depends.

Some displays are implemented with dual-eyelid technology for the blackest of blacks. Naturally, like all genius engineering, we see this in nature: cats.


It depends afaict. OLED screens have a per-pixel light, and they turn off pixels to make black. LCDs have a single large backlight and pixels that the light shines through and they can change color (but not turn off) so in that case they turn as opaque as possible, but don't completely block the light.


There is also things like microled. Which means that there is bunch of small(bigger than multiple pixels) lights that turn on and off as needed.


And in a CRT the electron beam turns off when scanning over a pixel.


Black pixels in an OLED still reflect some light so their are not completely black either.


What, you've never heard of a "black light" before? They just turn on the black sub-pixels.

https://en.wikipedia.org/wiki/Blacklight


He said it, it is ultimately a turn off.


In case of a LCD, black pixel is turned ON to block backlight. It's clearly visible on monochrome LCD screens.

In case of e-paper, black pigment is attracted to the outer part of the screen.


I don't think you're taking into account the context or intended audience. It's a casual forum message posted in reply to someone else's message.

They have not written a "falsehoods programmers believe" article. They have proposed that one ought to be written and have given a starting point for what it might cover.

They offered their list to "get the ball rolling", confirming that they don't see it as a finished product.

They sent it to other readers of the same forum, who might be expected to have more knowledge of this topic, not to whoever runs across it on the front page of HN.


I believe the article that started it all is https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-... - crucially every entry is self-explanatory, which is a point that a lot of the subsequent “Falsehood…” list authors miss.


Every point is not self-explanatory, and some are clearly true (while the assumption is they must be false). For instance, it surely is true that any name will fit in under a Terabyte of text, if it can be encoded at all (and assuming the contrary is counter-productive). Claiming you should not assume any name can be spelled in Unicode is absurd as well. And, yes, it's perfectly fine to assume that if your system must have "real" (i.e. proved by any kind of document) names, you won't have to deal with Klingon names (even though it isn't a huge relief, honestly, since they still can have pretty much whatever format). For most systems, even more restrictive assumptions than that are totally fine.

You don't have to defend the "original" post just because it was patio11's. This idea was awful and stupid from the very beginning, and every new post of this "series" just repeats the offence.


> People’s names fit within a certain defined amount of space.

For any given fixed size that people use in practice, there is a name that does not fit. This is saying "use a variable-length field for names, because there are always edge-cases".

> Jullien was born in Sisteron, Alpes-de-Haute-Provence, and was baptised Louis George Maurice Adolphe Roche Albert Abel Antonio Alexandre Noë Jean Lucien Daniel Eugène Joseph-le-brun Joseph-Barême Thomas Thomas Thomas-Thomas Pierre Arbon Pierre-Maurel Barthélemi Artus Alphonse Bertrand Dieudonné Emanuel Josué Vincent Luc Michel Jules-de-la-plane Jules-Bazin Julio César Jullien. His father was Antonio Jullien, a violinist. The explanation of his unusual number of names is that when the time came for the baby to be baptised, his father had been invited to play at a concert given by the Sisteron Philharmonic Society, and considered it only polite to ask one of the members of the orchestra to be godfather: but since every member wished to be considered for the privilege, he was christened with the names of all thirty-six members of the society.

https://en.wikipedia.org/wiki/Louis-Antoine_Jullien


> Claiming you should not assume any name can be spelled in Unicode is absurd as well.

China (and Japan, and to a certain extent Korea and Viet Nam) exists? https://en.wikipedia.org/wiki/Naming_laws_in_China#Notable_c...

I'm pretty sure that patio11, having spent his life in Japan, would know that technology like SING glyphlets exists because of this exact issue.

(and before you answer "what are you talking about, it is in Unicode?", these characters are literally added after the relevant issue surfaced, and some characters like 𱅒 (U+31152) are recent additions that don't even render properly)


I think Prince proved the Unicode assertion. He had to distribute his own custom font so news outlets could continue to write about him.


No, he one proved that other people don't need to put up with your bs if you try to break conventions. Hence your use of ASCII "Prince" being entirely adequate.

Besides, AFAIK he only changed his stage name, not his legal name.


I thought he really did change his legal name, in an attempt to get out of a contract.


I wonder what he means by France having a “weird” naming system in common use. As far as I can tell, the traditional French naming system works exactly the same way as the traditional American one (except that it’s more common for French people to have several middle names rather than zero or one, but I don’t think that’s too rare in the US either).

Maybe he’s referring to the fact that some last names are two words (e.g. Marine Le Pen), but I don’t think that’s very common…

Anyway, it could be anything, so I wish he’d said!


Perhaps he is thinking of how marriage does not change your last name, but rather gives you an extra, optional last name. [1] The French ID card has two last name fields!

[1] https://fr.m.wikipedia.org/wiki/Nom_d%27usage_en_France


Err. It still does by default change your name (if you're a woman). But you can ask and keep your 'maiden' (ugh) name or have both. It gets a bit trickier for kids' family names...


> I don’t think that’s too rare in the US eithe

Anecdotal, but the only people I've met in the US with more than one middle name are people who originally came from another country.

Although, I wonder if maybe that is enforced by the fact tha legal forms and similar typically assume you only have first, last, and optionally a (single) middle name.


George Herbert Walker Bush comes to mind as a native son of the US with multiple middle names. He used H. W. in politics, but that still includes some whitespace and non-traditional characters.


Or George R. R. Martin.


So you are saying Donald John Trump is likely not a US born Citizen?


"more than one middle name" :)


Perhaps he means allcaps family name followed by normal case given name.


Even that could benefit from including a counterexample in almost every point.


I saw this one first, but seems patio11 was first:

https://infiniteundo.com/post/25326999628/falsehoods-program...


i thought this list was also self-explanatory. i didn't have a hard time thinking of counterexamples to any of the points, which is not true of patio11's article. but what's self-explanatory depends on your knowledge base. maybe i just know less about foreign cultures than i do about tcp


This. If these lists contained things that programmers actually believed and explained why they are false, they might actually be useful. It's hard to imagine an unsupported assertion that an ambiguous statement is "false" and yet its contradiction is also "false".


I agree. This isn't even a good one of those lists. It's more like "dubious pedantry to make me feel smart about my TCP knowledge".

1-4. Yes we know about the 2 generals problem. And yes we know what "reliable" means in this context. 5-6. This is just stupid. 7. Obviously not true. Nobody thinks this. 8-9. The reasons for and flaws of Nagle's algorithm are well known. 10. This isn't even true. Most of the time you don't need to care about it. That's the whole point of abstraction. You need to care about it if you are doing extensive performance optimisation, but usually you aren't. 11. Again untrue. You can think of TCP as a two way pipe. Again that's the whole point of abstraction. 12. Not sure exactly what they're trying to say here but again it's very well known that TCP and UDP are pretty much the only protocols that are likely to work on the internet. 13. Ditto. We all know why so many protocols are "over HTTPS", e.g. DoH. 14. This isn't a technical point. 15. Dunno what this is talking about but I'm guessing it's along the lines of "a byte is 8 bits", i.e. it is actually true in the modern world.


I’m pretty sure originals that defined this format did have examples and citations.

But I do agree that some of the later entries have felt a little lazy.


Right, I was about to comment that. One of the first ones I remember was this one, about addresses[1]; or this one, about names[2]. Both provide examples and information, which is the only thing making the whole article useful.

  [1]: https://www.mjt.me.uk/posts/falsehoods-programmers-believe-about-addresses/
  [2]: https://shinesolutions.com/2018/01/08/falsehoods-programmers-believe-about-names-with-examples/


I remember the address one. It was fantastic and I loved it. I wish they were all that helpful.


I think the original "names" and subsequent "addresses" were useful in that a conclusion(that programmers should embrace defeatism and refrain from parsing or evaluating or even trying to separate them into fields) can be drawn, and the lessons learned were slightly more specific than often realized...


Falsehoods falsehood-list makers believe.

1. That said items are falsehoods in the first place.

2. That said items are necessarily interesting or noteworthy.

3. That a list is necessarily the best format to present said items.

4. That they may speak for the involved parties beliefs.


I noticed that too and thought I was missing something. Some cool resources that are actually decent for network programming:

https://beej.us/guide/bgnet/ -- Covers what abstractions the OS provides for network programming and the guarantees that are possible.

https://www.madwizard.org/programming/tutorials/ - This is the very first ever good tutorial I read on socket programming. It's OG winsock. Introduces network programming from the most basic level. Aimed at C.

When you understand these guides you'll learn that how you structure your entire programs networking depends on whether you want to use blocking or non-blocking sockets. If you go with blocking you'll probably be using threads or processes. Otherwise you can't do any other work. With non-blocking it will be more about polling sockets and eventually you might end up with something resembling an event loop.

Until you come towards to the current approach to networking which is mostly async await -- an event loop works with non-blocking sockets, watches them for changes, and passes data from them to event handlers. There's a lot more that can be done on sockets to effect things like how data is flushed, how TCP errors are handled, and so on, but its a good start.


The following list contains only falsehoods:

1. You're wrong

2. Okay, you're right

3. Okay maybe you're right or wrong but certainly not both


Well perhaps your a photon and then you certainly are both.


Aha! You're null.


Or a false premise, perhaps (from which anything follows).


Previously: "falseoods programmers believe about X":

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...


I think these lists are often primarily intended as humorous, and perhaps a way to get you thinking about exceptions, not as a way to teach you more about the topic.


I agree, this article was uninsightful.


Yeah I stopped with 5 and 6 and will never give the slightest care what this person has to say ever again.


Falsehoods considered harmful.


You’re doing it wrong.

“Falsehoods programmers believe…” articles are designed to make you THINK about problematic assumptions. They are not like the 10 commandments and they are not decrees of absolute truth.


Now imagine how much time could have been saved globally if one person spent half an hour writing a short description of why each point is false instead of making hundreds (or thousands) of people spend hours thinking about and researching every one of them. You're probably left with more knowledge in the end if you're not spoon-fed by the author, but how many of us need really deep knowledge of the TCP inner workings?


Saying a thing and then saying the opposite, without elaborating, is not good at making you THINK. This list is doing it wrong.


Yep, the original "names" one was mostly written so negating each of the points gave you the exception you needed to handle. Even the cases written with both were done on a way it was obvious the negation didn't apply universally, so both worked.


I look at "Falsehoods programmers believe..." articles as a good source of test cases. If I'm parsing a date (don't do that), I'm going to look at "Falsehoods programmers believe about dates" to help build out my list of unit tests for that function. Same for names, street addresses and so on.


> remember, all of the following statements are false at least some of the time, but for some of these, perhaps not very often

> 5. There is a such thing as a TCP packet

> 6. There is no such thing as a TCP packet

I don't understand this at all. Either the concept of a TCP packet exists, or the concept does not exist. Even it's not being used in certain scenarios, I don't see how you can argue that "there's no such thing" any of the time. This might just be me misunderstanding whatever point they're trying to make, but I don't remember ever having such philosophical confusion from anything in any other "falsehoods programmers believe about..." article before.


Pedantically: TCP has segments, IP has packets, and Ethernet has frames. They are one-to-one in simple cases, but not always.

https://networkengineering.stackexchange.com/questions/50083...

In particular, fragmentation by intermediate routers means that the server and receiver may disagree about the frame and packet boundaries. TCP is expected to make a "reliable" pipe-like service out of whatever happens, and the application layer doesn't have (shouldn't need?) visibility into that process.


Falsehoods programmers believe: TCP/IP can be coherently mapped to the OSI model.


Falsehoods programmers believe: the OSI model



Falsehoods programmers believe: the OSI model is useless


Sure, it's misleading, needs a lot of "interpretation" doing a non-trivial amount of the lifting to make it map to anything in the real world, mismatches things that happen in the real world while leaving no room for other things that happen in the real world a lot, and will lead anyone who tries to use it to understand the real world deeply astray, but it isn't always wrong about absolutely everything so it has some non-zero "utility".

Fine. It's not wrong about absolutely everything all the time. It isn't bereft of all truth. It's just something that is of net negative value. I see no value in insisting on trying to "rescue" a net-negative value model of the world.

I suppose you could say ultimately I agree with you though. The OSI model isn't useless. It's worse than useless. You're better off trying to understand networking from basic first principles than through the lens it provides.


Analogies are rarely perfect, that's why they are analogies. The OSI model isn't intended to be perfect and yeah there are a lot of details that leak between layers, but is also expected, any non-trivial abstraction is always going to be leaky. That doesn't mean it useless or absent of value in discussion at appropriate levels.


It's useful. It's not controlling.


It's not useless, just bad at it's job.

I mean in practice it's so broken, imprecise and messy to a point it's often more misleading then helpful and IMHO should have been replaced in teaching with something better well over a decade ago.

And to be clear I'm not saying it's bad because some small implementation details don't fit, it's bad in it's job of being a high level abstraction where you ignore many implementation details.


I think I've heard the term "packet" used to refer to the general category of thing that Ethernet frames, IP packets, and TCP segments are all examples of.


Colloquially, yes, they're all just packets unless I'm writing an RFC for the IETF.

If I recall correctly, the formal term for the all-encompassing category is the "datagram".

It's like who-vs-whom. Prescriptivists will try to tilt that windmill forever, descriptivists observe that most people just don't care.


Those are all packets.


This one feels like the list-maker is struggling to communicate with me rather than telling me a surprising truth. I can't tell whether it's meant to mean "a TCP segment won't always fit 1:1 into an IP packet (but in the real world they mostly do)", "TCP has segments instead of packets (but everyone knows what you mean when you say TCP packet)", or something else entirely.


Yeah, that's basically the takeaway I had (and I failed to summarize as concisely, ironically enough). It seemed like there was some insight there, but I had absolutely no clue what it was.


What it really means is: Packets have well-defined boundaries between sufficiently-adjacent nodes. They are not guaranteed to keep those boundaries end-to-end over arbitrary middleware.


I guess it’s talking about how the TCP data stream is segmented into IP packets. From the IP point of view, there are packets; from the application point of view there is a data stream; but it’s more complicated than that. Applications have some control over when TCP’s PSH flag is set, roughly speaking, at the end of each write(); and that in turn affects segmentation because small pushed writes cause small packets. But if the sender can’t send straight away then buffered data doesn’t preserve write() boundaries and will be sent with large packets.


I think we're talking about different things.

TCP is a protocol that most certainly has packets ("segments").

You are taking about a SOCK_STREAM-like programming interface, which can be used to create TCP connections.


If what was originally a packet is fragged, the TCP headers may (likely) not be in all the frags. So if you're looking with wireshark and you think you can filter with TCP flags good for you, 90% of the world feels that way.

I work with DNS a lot and when a protocol which is datagram-oriented is translated to a stream-oriented medium and somebody wants to potentially handle multiple requests in that stream because "efficiency"... it's so important... they need a way to distinguish those embedded datagrams: "we should do it they way they did in HTTP, with a Content-Length: header, yea THAT's the ticket!" I'm sure that's what they were thinking.

Then along come the 90% and when they try to process DNS requests in TCP streams it's "what's this two bytes in front of the request? I dunno, but I just skip over it" and I suppose it works well enough, because with frags they'll drop the tail of the requests because "corruption" anyways, and who on earth sends multiple requests in a single packet, amiright?

You may think I'm kidding, but here's some really clever (really!) eBPF code: https://gist.github.com/oghie/b4e3accf1f87afcb939f884723e2b4...

If you want more on this topic: http://consulting.m3047.net/dubai-letters/dnstap-vs-pcap.htm...


> 6. There is no such thing as a TCP packet

Because the software abstraction is a stream of bytes; and it's up to the application to decide where the "packets" begin and end.

For example, I might write to a TCP socket: 100 bytes, 50 bytes, and then 125 bytes.

BUT, the receiver could get: A single event with 275 bytes. Or it could get an event with 75 bytes and then an event with 200 bytes. Or it could get 11 events of 25 bytes.

> 5. There is a such thing as a TCP packet

This one I struggle with. I think the author is talking about connection set up, acking, and connection teardown.


If you have a look at the underlying network traffic, you'll see IP packets carrying TCP data, ie. The protocol field in the IP packet header will be set to TCP; this could be assimilated to a TCP packet.


Well, maybe...


It's a typical problem of this lists they don't always point to the underlying issue they want to list.

5.,6. are likely about how all kinds of things infer with your packages mainly merging/splitting but also potentially messing with them in other ways (and in general the package->frame mapping). So if you want to build anything which relies on TCP packages being a thing it likely will not work, at the same time you can't always ignore package size either (e.g. for flushing byte streams, higher level protocol chunking etc.). Through as long as you don't create transport protocols write a network stack or fine tune your TCP stack or similar you likely don't have to care about it. I mean you also probably shouldn't use TCP directly but something more high level in most situations (e.g. HTTP, gRPC, etc.).


5 and 6 are mutually exclusive. They don't make sense logically. And most of the list was never explained at all.


> They don't make sense logically

In practice such situations can arise in one of two cases:

1. some non-sense creeped in

2. logic is applied to a self-contradictory set of axioms and definitions.

(1) is not very interesting, but (2) happens frequently enough because people often do not try to formalize their definitions and axioms. As a consequence they are using some vague concepts and their statements are true in some cases but not in others.

With all that said, I can propose the way how this logical non-sense could be right. (NB. I don't know if it applies to TCP, I'm just thinking generally, and just as an example to all that abstract words above) The notion of "existence" of the mistaken programmer can be wrong. If we accept their definition of existence, then TCP packets doesn't exist, but they exist in some other sense.


Yes. If one applies correctly the rules of logic on inconsistent axioms, the conclusions will be inconsistent. If one incorrectly applies logic to inconsistent axioms, the conclusions may or may not be consistent. It happens IRL sometimes; "being right for the wrong reasons". That being said, I suspect the game of the author is to play with leaky abstractions. TCP is a stream-oriented protocol, but is implemented on top of frames etc.


The point is that a lot of stuff in Networking (and Computer Engineering in general) is very context dependent, and that you cannot be extremely opinionated about this stuff.


No context was ever provided.


They are not mutually exclusive statements, because they don't exist in isolation: they are both potentially true and false depending on the context of discussion.


This seems like it's either a linguistic or philosophical question; either I don't interpret the words "such a thing" in a way that the author meant it, or I have a divergent different philosophical worldview on the concept of "existence" from the author's own view.

Either way, this stuck out to me because usually these type of lists have very simple, understandable statements that just happen to surprise some people by not being true; in this case, the statements themselves are confusing, so I can't really say for certain whether I believe them or not because I don't even know what truth they're asserting to try to negate it.


> these type of lists have very simple, understandable statements

I would classify these statements as koans, and further observe that one of the defining characteristics of these lists is that they are wry. With that in mind the couplet is evidently presented in cheek to illustrate the need for context, since this is the pathway to resolve the overt contradiction without getting stuck in a rigid thinking trap.


But they assert whether or not something exists, as an absolute statement. Maybe TCP packets don't exist in a particular situation, but there is still such a thing as a TCP packet in that case.


Yeah, I agree with you here.

I think the thing most related to that that I see people thinking is that send(2) & recv(2) calls translate 1:1 with packets send/recv. I.e., that they don't understand that the interface TCP exposes to applications is a byte stream. Which then results in things like thinking recv(2) will receive a complete "message" for some definition of message in the application protocol (i.e., the mistake belief that fragmentation won't happen).


If I had to guess, it would be an assumption that TCP was edge to edge with no translation in the middle.

My guess is that this is talking about systems in the middle of the network, changing (for example) their sizes by combining and splitting packets to fit through various transits.


It's about your abstraction level and the kinds of problem you are ignoring. It's true at the same time that you can't ignore the problems of stream communication nor the problems of package-based communication.


Perhaps it could be rescued by rephrasing them as "is always" versus "is never"?


This doesn't really change my reading of those statements; the issue to me is that "there is such a thing as X" sounds to me like an existential proposition, i.e. "X exists". The idea that an abstract descriptive concept only sometimes exists doesn't really make sense to me; it sounds like saying that addition only "sometimes" exists because there are equations that only use division and not addition.


It exists but more as a TCP stream, which is broadcast as packets...


Unicast. Unless we are talking about wifi... Which gets to very similar context based mess.


My only guess how this could make sense is if there is some ambiguity in the definition of a TCP packet.


It sounds like TCP is totally fucked and useless. It needs to be depreciated immediately and replaced with something better and more intuitive from a programming perspective. Maybe someone at Apple or Google is working on it lol.


I recall it blew my fiancée’s mind that I could unplug her ethernet cable, move it around an obstacle, plug it back in and all her connections were still alive. It’s designed to have bombs dropped on it.


Depends on OS settings these days. Lots of OSes want to help and detect link down and reset all your connections. Kind of a pain when you just want to move a cable.


Like Chrome’s oh so helpful ERR_NETWORK_CHANGED


Also known as ERR_FUCK_YOU. Yes, I know that I'm connected to a misbehaving wifi router. Just load the fucking page.


Loud


Like, I know?! Just reload the page already, I've told you twice.

It's rather irritating.


Drives me insane on Android with a VPN


.. on Linux. If you do that on Windows the MAC will detect the loss of link pulses, report the interface as down, and Windows will "helpfully" reset all your TCP connections.


I remember there was a (now rather obscure) patch floating around in the days of Win9x/2K that gave you the Linux behaviour, with an associated option to extend the timeout to very large values.

Later, Microsoft made it configurable:

https://learn.microsoft.com/en-us/troubleshoot/windows-serve...

Of course if the connection is active when that happens, the other end will probably not like the prolonged silence, and give up first.


That seems like way more sensible behaviour.


It's a tradeoff between robustness to transient errors and reporting errors quickly. "Most errors are transient" is a widely applicable rule of thumb. But both approaches have merit.


How so?


More immediate response to disconnection.

Rather than "IDK keep sending but I haven't heard ACK in a while"


When the disconnection is brief, there's no need to respond to it.

It's the conceptual equivalent of your phone "helpfully hanging up" when you change the phone from one ear to the other during a call.


It's the conceptual equivalent of your phone helpfully hanging up when you turn on airplane mode.


Turning on airplane mode explicitly signals "sever all communications" in a way that making a quick adjustement of an ethernet plug does not.


What happens in that case? I'm going to speculate:

1. Remote keeps sending stuff to your unplugged connection

2. You plug your ethernet cable back in

3. Your computer's TCP acknowledges the last sequence number it received for each new sequence it receives from remote

4. Remote sees duplicate ACKs for same sequence number, interprets it as packet loss and resends the stuff


With timeouts. You can’t unplug it for an hour and have this happen. But a few seconds is exactly what this is designed for. As another commenter pointed out, your OS could also try to be “helpful”.


Yeah, that's pretty much it.

If packets were sent while you were disconnected, they'll be gone, but if you're disconnected for only part of the burst, duplicate ACKing will trigger retransmits.

If you were gone for the whole burst, you'll get put right by timer based retransmits.

If you're gone for long enough, most peers will timeout on unacknowledged data (although that's not in the TCP RFC), and if there's no outstanding data, most peers eventually have some sort of periodic ping and timeout (tcp keep-alives is a reasonable fallback IMHO, if your application protocol doesn't have someything, although the default of IIRC 2 hours feels long in todays world of lots of NATs and much shorter timeouts).


It may be your local gateway. Seeing no packets from your host. Attempting to refresh your MAC address via ARP. Getting no response. Generating an ICMP message as a result.


Related: you can get at most once delivery or at least once delivery; you cannot get exactly once delivery. If I had a dollar for every junior who thought that a lack of exactly once delivery guarantees was a bug...


If you can get at-least-once delivery, why can you not build exactly-once on top of that?

[UPDATE] Apparently I need to be more explicit about this. My question is: if I can get at-least-once delivery, why can I not build an abstraction layer on the receiving node that provides the illusion of exactly-once delivery? It seems like it should be a simple matter of keeping a log of received messages, and discarding duplicates.


The principal difference between 'at most once' and 'at least once' is whether a sender re-tries when it is unsure if the recipient has received the message. If the recipient's ack never makes it back, then a sender cannot know whether they actually received the message or not (the two-generals problem).

So this hypothetical middleman will receive a packet, check that it's not a duplicate, and forward it to the recipient it's proxying for. How will it know that the recipient has actually received it? If the receiver doesn't ack the message in some way, which causes your abstraction to re-transmit the message again, then it exhibits 'at least once' behavior. If it the abstraction only ever forwards the message along once and doesn't care whether the recipient acknowledged it or not, then it exhibits 'at most once' behavior.

As a more concise answer - 'exactly once' delivery is impossible because you can't know if the recipient actually got the message. If you assume a perfect communication channel, then I agree the problem is trivial, but I challenge you to find such a channel! Even on the same machine, interprocess communication can fail in all sorts of fun ways.


> So this hypothetical middleman will receive a packet, check that it's not a duplicate, and forward it to the recipient it's proxying for. How will it know that the recipient has actually received it?

It seems like the answer is in the first part, the "check that it's not a duplicate".

Implement at-least-once but with a unique token to identify the request, and the receiver sends back acknowledgement with that token every time it receives the original message, but only hands it off for processing the first time. Stuff this behind library/API so it's hidden to the user and the application code doesn't have to handle it, and... isn't that it?


Yes, but the handoff can fail in the same way (it can't know if the thing it's handing off to actually got it). But the application can also just be resilient to that with idempotent operations and have the handoff be at-least-once.


> the handoff can fail in the same way

Everything in a digital system can fail, but by convention (and because it's not so far from the actual truth) some parts are assumed to be 100% reliable when modeling them. If you don't make this assumption, you can't guarantee anything.


> So this hypothetical middleman will receive a packet, check that it's not a duplicate, and forward it to the recipient it's proxying for.

That's not how I would implement exactly-once on top of at-lest-once. I would do it at the recipient, not at the intermediate nodes.

> 'exactly once' delivery is impossible because you can't know if the recipient actually got the message

But the recipient can know.


But the recipient is not one atomic thing - we're assuming perfect communication between the process/driver/hardware receiving the packets and doing the duplicate detection and the process which wants to receive the message exactly once.

There's still communication happening there, and it can still fail. Buffers fill, processes pause for arbitrary delays which exceed timeouts, etc. Your assumptions based on your model are correct, but your model doesn't include that communication.

But all models have some level of detail they care about, and assuming the computer always works is a perfectly valid model for plenty of cases. It's just not all of them. You'll be able to create real-world cases where this abstraction is faced with a choice of whether to retry or not, and at that moment it will be unable to deliver exactly once.


> we're assuming perfect communication between the process/driver/hardware receiving the packets and doing the duplicate detection and the process which wants to receive the message exactly once

My claim is not that you can provide exactly-once delivery unconditionally. My claim is that if you can provide at-least-once delivery then you can turn that into exactly-once delivery. The word "delivery" is not rigorously defined, but IMO any reasonable definition necessarily entails a certain level of reliability at the receiving node.


I agree with your claim, a recipient can cope with at-least-once delivery by being idempotent. You're right.

The meaningful distinction is that something on the recipient needs to be idempotent because the message might get received twice. The application can be oblivious to this, so long as you assume that channel to be perfect.

People on the Internet won't like you calling it 'exactly once delivery' because it's not exactly once - it's an idempotent at-least-once. Which is great! But the statement of the at-least/at-most problem is making a decision to re-try. There's no middle ground, I either have to retry or not. People won't like a claim that 'exactly once' delivery is possible, because it isn't, it's just moving the at-least-once-ness to somewhere else.


> People on the Internet won't like you calling it 'exactly once delivery' because it's not exactly once

That depends entirely on what you mean by "it".

Messages can get lost. So if you want guaranteed delivery, you sometimes have to re-transmit the same message, and so you might end up with the same message being delivered more than once. But it is trivial to put an abstraction layer on top of that to discard the duplicates and make it appear to the application as if every message is received exactly once.

The whole thing is a tempest in a teapot.


The problem is that sometimes your application intends to send messages[0] multiple times. If your "log-and-discard" system was implemented naively then each node could only ever send each possible message once and only once. Ever.

That would be like:

Alice: "Honey, where did you put the keys?"

Bob: "They're up on the counter."

(The next day...)

Alice: "Honey, where did you put the keys?"

Bob: (nothing, I already received this message, it could have echoed off the walls from yesterday)

What you need is for all sent messages to have unique IDs that will never repeat, and then log those. That's known as an idempotency token.

But even then, logging all those UUIDs forever is probably not a good idea for disk usage. At some point you'll have to trash old message logs and hope you don't have a rogue network router retransmitting six month old messages or something.

[0] Or the moral equivalent of messages, e.g. HTTP POST requests


> What you need is for all sent messages to have unique IDs that will never repeat, and then log those. That's known as an idempotency token.

So your problem is not really a problem because you yourself present the solution. The real problem is:

> But even then, logging all those UUIDs forever

But you don't need to log them all forever. Just make the UUIDs sequential, and all you need then is to keep track of the smallest id that has not yet been received. (You can be more efficient by storing more state, but it's not necessary. Remember, we're assuming at-least-once deliver here, so you can always force retransmission by not acknowledging receipt.)


You can get exactly once processing, but not exactly once delivery.

https://bravenewgeek.com/you-cannot-have-exactly-once-delive...


That seems like a distinction without a difference to me. Why should I care if the thing I get exactly one of is called "processing" or "delivery"?


Because you need to understand that your processing code is constrained by the fact that you can't get exactly-once delivery. You must write your processing code to handle it one way or another. There's some libraries that try to wrap the abstraction of processing exactly once around the code, but those libraries still impose constraints on the sort of code you can write. They can make it easier but they can't fully remove the need to think about how your processing code works. It isn't exactly the same.

This is why people like me insist it's important to understand that you can not have exactly-once delivery. There is no library that can make that just go away, they can only shuffle around exactly where the lumps under the carpet live, and if one programs with the mistaken idea that these libraries really do solve "exactly once" delivery, one will get into deep trouble. Possibly the "my architecture is fundamentally broken and can't be rescued" sort of trouble.


> Because you need to understand that your processing code is constrained by the fact that you can't get exactly-once delivery.

Why do I need to understand that? Why can I not put an abstraction layer that provides me with the illusion of exactly-once delivery?

> There is no library that can make that just go away

Well, this is the thing that I dispute. I believe that there is a library I can write to make it go away if I have at-least-once delivery. In fact, I claim that writing such a library is an elementary exercise. The TCP protocol is an existence proof. Where is the flaw in my reasoning?


TCP is at least once, not exactly once.

Here's another useful article: https://blog.bulloak.io/post/20200917-the-impossibility-of-e...

TCP does not solve the two generals problem. TCP gets around this limitation by requiring only one ACK.

https://en.wikipedia.org/wiki/Two_Generals%27_Problem#Engine...

https://www.scaler.com/topics/computer-network/two-generals-...


OK, but you have to do a little extrapolating here because the claim is not that you can do exactly-once under all circumstances. That is obviously false because you can't do exactly-once in a situation where all comms are down indefinitely. My claim is that if I have at-least-once then I can build exactly-once out of that.


This isn't about indefinite communication loss. Obviously no progress is possible in that case. The two generals' problem has nothing to do with a permanent failure.

I think there is a lot of literature out there if you're really interested in understanding and I'm happy to provide more links if you'd like.


What makes you think the 2GP is relevant here? The 2GP has to do with coordination and consensus, not exactly-once delivery.


The link I posted above explains the connection: https://blog.bulloak.io/post/20200917-the-impossibility-of-e...

Particularly:

> The sender cannot know if a message was delivered since transport is unreliable; thus, one or more acknowledgement messages are required. Moreover, the sender cannot distinguish between message delivery errors, acknowledgement delivery errors, or delays (either in processing or because or network unreliability).

> The recipient is forced to send the acknowledgement only after the message is either processed (or persisted for processing) because an acknowledgement before processing would not work: if the recipient exhibits a fault before processing that would cause the loss of the message.

> In the case that that acknowledgement is lost, the sender can’t know (due to lack of insight in the recipient’s state) whether the recipient failed before scheduling the message for processing (in essence losing the message) or if the recipient is just running a bit slow, or if the acknowledgement message was lost. Now, if the sender decides to re-deliver, then the recipient may end up receiving the message twice if the acknowledgement was dropped (for example). On the other hand, if the sender decides to not re-deliver, then the recipient may end up not processing the message at all if the issue was that the message wasn’t scheduled for processing.


This proof is flawed:

"Let’s assume that a protocol exists which guarantees that a recipient receives a message from the sender once and only once. Such a protocol could then solve the two generals problem! Representing the time of the attack as the message, the first general (the sender) would only need to adhere to the protocol for the second general (recipient) to have received the attack time exactly one time. However, since we know that this is not possible, we also know that exactly once is not possible."

The 2GP is not just the first general knowing that the second general received the message. The 2GP is the problem of achieving common knowledge between the two generals, i.e. it's not just that G1 needs to know that G2 got the message, it is that G2 needs to know that G1 knows that G2 got the message, and G1 needs to know that G2 knows that G1 knows that G2 got the message, and so on.

Exactly-once delivery is possible. The only thing that is not possible is for the sender to know when the message has been received so that no duplicates are sent. But exactly-once delivery is not only possible, it's trivial. All you need to do is discard duplicates at the receiver.


> All you need to do is discard duplicates at the receiver.

...yes, which would be exactly once _processing_ but not exactly once _delivery_.

Unless you're wanting to redefine "exactly once delivery" to mean "at least once delivery but I'm calling it exactly once because I have a strategy to cope with duplicate messages"


> exactly once _processing_ but not exactly once _delivery_

What exactly is the difference? What counts as "delivery"? How do you do "delivery" (on a computer) without doing at least some "processing"?


Delivery is communication between two actors. Processing is what one actor (the receiver) does with a message.

Communication is when two actors exchange a message. Communication is generally done over an unreliable medium because in practice there is no way to communicate without the potential of failure.

1. Exactly once communication between two actors over an unreliable medium is impossible. At the very least you have to account for the possibility of failure of the medium, so you might need to re-send messages.

2. At least once communication between two actors is possible -- just re-send a message until the receive acknowledges the message.

3. Because a message might be re-sent, the receiver must be able to cope with duplicate messages. This is what you're describing. This might be done by making message processing idempotent or tracking which messages you've seen before. In either case, you have achieve exactly once processing. That is, if a receiver is given the same message multiple times, only the first receive of the messages changes the state of the receiver.

---

> But exactly-once delivery is not only possible, it's trivial.

Considering that many in the field consider this problem to be impossible (or, at best, extremely difficult, e.g. https://www.confluent.io/blog/exactly-once-semantics-are-pos...), this should be a huge red flag to yourself that you're missing something. Everyone has blind spots and that's okay, but hopefully you understand that there's a pretty big mismatch here.

Alternatively, it's possible that this problem _really_ is trivial and you have some unique insight which means there's a great opportunity for you to write a paper or blog post.


> hopefully you understand that there's a pretty big mismatch here

Yep. But on the other hand, 1) I have a Ph.D. in CS, and 2) I have yet to see anyone in this thread actually produce a reference to a reliable source to back up the assertion that exactly-one delivery is impossible. Indeed, the one reference you provided has a headline "Exactly-Once Semantics Are Possible" so you are actually supporting my position here.

Finally, I will point out something that should be obvious but apparently isn't: "exchanging a message" between computers over a network is a metaphor. There is not anything that is actually exchanged, no material transferred from one computer to another. There is only information sent in the form of electrical signals which results in state changes in the receiving system, so there is no clean boundary between "communication" and "what the receiver does with a message". Receiving a message in the context of a computer network is necessarily "doing something" with that message. There is no other way to "receive" a message.


TCP implementations are an abstraction that work 99.99% of the time, but are still vulnerable to two generals when you look close. TCP is implemented in the kernel with a buffer, the kernel responds with ACKs before an application reads the data.

There is no guarantee that the application reads from that buffer (e.g. the process could crash), so the client on the other end believes that the application has received the message even though it hasn't.

The kernel is handling at-least-once delivery with the network boundary and turning it into at-most-once with the process boundary.


> still vulnerable to two generals

What makes you think the 2GP is relevant here? The 2GP has to do with coordination and consensus, not exactly-once delivery.

> TCP is implemented in the kernel with a buffer, the kernel responds with ACKs before an application reads the data.

True. Why do you think that matters?

> There is no guarantee that the application reads from that buffer

So? What does that have to do with exactly-once delivery? Even if the application does read the data, there's no guarantee that it does anything with it afterwards.

> The kernel is handling at-least-once delivery with the network boundary and turning it into at-most-once with the process boundary.

OK, if that's how you're defining your terms, I agree that you cannot have exactly-once delivery. But that makes it a vacuous observation. You can always get at-most-once delivery by disconnecting the network entirely. That provides a 100% guarantee that no message will be delivered more than once (because no message will ever be delivered at all). But that doesn't seem very useful.


> Why can I not put an abstraction layer that provides me with the illusion of exactly-once delivery?

You can do that. You can implement a video conferencing system ontop of TCP, and it will even work, technically. It will just have terrible performance characteristics that you'll never be able to fix. You might even call it fundamentally broken.


OK, I don't dispute that, but that is a very different claim than "you can't get exactly-once delivery." You can (if you have at-least-once delivery).


Read the two general's problem. It is proven that exactly once _delivery_ is a physical and mathematical impossibility.

Like you said, you can simulate it with exactly once _processing_. But to do that, you have to know to do that.

Others are saying "you can't divide by zero" and you are saying "yeah, but if I detect a zero and the do something different, then it is the same thing." No, knowing you have to do something is the very point of acknowledging you can't divide by zero.

Because you can't have exactly once delivery, you have to deal with it. One trick is duplicate checks or idempotent writes. This gives exactly once processing. This also takes additional overhead and why audio and video stream processing doesn't typically do the additional checks.

I have fixed many bugs written by people who believe the network is reliable. I even hired one when during the interview and we talked about this kind of issue that they realized why they were getting duplicated writes reading from sqs or sns at $existing_job. They were green but smart and she became a real asset to the team.


What makes you thin the 2GP is relevant here? 2GP has to do with coordination and consensus, not exactly-once-delivery.

> you can simulate it with exactly once _processing_

What exactly do you think is the difference between "simulating" exactly-once delivery and actually having exactly-once delivery? What do you think "delivery" means in the context of a computer network?


I don't know enough to answer this. I'm sure there is plenty of writing on this subject from people more qualified than me.


Because 'exactly once' delivery is arguably a misnomer, you usually really want 'at least once delivery with acks and idempotent processing on the other side'.

The difference is subtle but important in practice and specification.


> you usually really want 'at least once delivery with acks and idempotent processing on the other side'.

Why? I'm pretty sure I really want (the illusion of) exactly-once delivery, and it seems to me that I can implement that pretty easily given at-least-once delivery. Why would I not want that?

> The difference is subtle but important

Why?


You can absolutely abstract 99% of it out.

But not 100%. At some point, a counter move on the delivery has to be stored... -somewhere-.

And sure you -can- make it very very close to EOD and for some subsets you can totally do EOD, but you are, realistically, better off with ALOD+Ack once it makes its way into a system. There's always that 'moving the counter' problem.

The upshot is, things tend to get faster, easier to code review, and simpler to test.

Pragmatically speaking, I've found devs are better able to handle ALOD+ACK than "Exactly once but because reality you might get a message that's doubled because you couldn't persist the ack".

And I'll note I'm possibly extra pedantic about this because I've had a month and a half of dealing with the fallout of people trusting low-code salesmen alongside gartner reports leading to a 'you people thought this was exactly once and it was not' sort of problem.


> But not 100%.

Why not?

> At some point, a counter move on the delivery has to be stored... -somewhere-.

What is a "counter move on the delivery"?


Read the two general's problem. It is a mathematical impossibility. The counter is the delivery counter. Delivery, delivery + 1


What makes you think the 2GP is relevant here? The 2GP has to do with coordination and consensus, not exactly-once delivery.


You are correct


> I'm pretty sure I really want (the illusion of) exactly-once delivery

Do you know what idempotency is? This is exactly what he described.

Idempotency is important to prevent unwanted behaviour for duplicate actions. If you have "exactly-once", and accidentally execute the action twice that could cause problems.


> Do you know what idempotency is? This is exactly what he described.

Is it though? It seems like a false equivalency, even if the outcome is approximately the same?


> Do you know what idempotency is?

Yes.

> This is exactly what he described.

So? Idempotency and an exactly-once delivery abstraction are not the same thing.


Agreed. Idempotent _processing_, not delivery.


Uh, what exactly do you think is "agreed" here? My claim is that idempotent processing can produce exactly-once delivery, and so the original claim that you "cannot have exactly-once delivery" is false.


Change your words from "delivery" to "processing" and you in alignment with reality.

Idempotent processing is exactly once processing. This is true. Delivery is from the sender's point of view, not the recipient. How the recipient processes the information, idempotently or not, is not the concern of the sender.

"Cannot have exactly-once delivery" is true and is as true as not dividing by zero. Read about the two general's problem. It is, quite literally, impossible to have exactly once delivery. Like, your friend is in the other room, and you shout "someone is at the door" - how do you _know_ your friend heard you? If you shout again, you are attempting another delivery. What do you do if your friend doesn't respond? In exactly once delivery, you guarantee they heard you in the other room. In exactly once processing, you can shout a couple of times until you hear them acknowledge.

You may think that this is not material and at the end of the day, as long as one thing is processed, then who cares? Well, you have to understand how delivery can fail otherwise you will handle the failure incorrectly. Is it safe to try again? If I say "transfer money from me to you", and I don't hear back, is it safe to again say "transfer money from me to you" again? Will I be double charged?


> Change your words from "delivery" to "processing" and you in alignment with reality.

You are not the first to say this, but so far no one has been able to explain what the difference between "delivery" and "processing" is. How do you do "delivery" (on a computer) without also doing (at least some) "processing"?

> Delivery is from the sender's point of view, not the recipient.

I don't see what difference that makes. In fact, I don't see why a "point of view" should enter into it at all. Whether a message has been "delivered" or not (whatever that actually turns out to mean) is (it seems to me) a property of the system, independent of anyone's point of view.


I gave two concrete examples. What about each of those examples is not landing?

One of shouting at your friend. You want to make sure your friend knows someone is at the door. Two: you tell your computer to transfer money and you don't want to be doubled charged.

More in depth: you click send money on a computer. The computer connects to another computer and sends data to it over an unreliable network. Computer A sends data over the network to B, just like you shouting to your friend you think is in the other room. Data can be lost / your friend might not hear you. Usually, the other computer says "acknowledged, I got your message" - and that is how know that B is moving money or your friend is getting the door. If A never hears back, should A try again? If B gets two requests to move $25, should that be deduplicated or was there two actual requests and $50 should be moved? To know how to solve that, you have to first admit that you might get 0, 1, or multiple messages delivered to B when A wants to send 1 message.

Read the two general's problem. It is the gateway to distributed computing.


> I gave two concrete examples.

Neither of which was on point because they both ignored how exactly-once delivery can be done.

> Read the two general's problem.

The 2GP is not on point because it's about achieving consensus, not exactly-once delivery. Achieving consensus is indeed impossible with unreliable messaging, but that has nothing to do with exactly-once delivery.


You cannot while maintaining the half-duplex behavior of the current system.


Why not? (Please see the update on my OP before answering that.)


The mechanism you're describing already exists. TCP has sequence numbers. It can drop duplicate data.

The difference between "processing" and "delivery" relates to "network capacity." Process handling wastes capacity in favor of latency. Delivery handling increases latency in favor of capacity.

Systems which have "exactly once" delivery typically do so with "send/receive" and "release/delete" message pairs. You need additional round trips to actually accomplish this at the "delivery" layer.


> The mechanism you're describing already exists.

Yes, I know, which makes it all the more bizarre that people are claiming that this is impossible.

> Systems which have "exactly once" delivery typically do so...

Ah, so exactly-once delivery is possible after all?


If your goal is simply to look smart then absorbing the subtlety of what is said to you before you reply should be top of list.


What subtlety do you think I have failed to absorb?


So fun fact, you actually can get exactly once delivery out of your network, but your network has to be not Ethernet/IP/TCP to do it. Every single one of those layers is mis-designed to allow you to get exactly-once delivery of messages (TCP doesn't even have a concept of messages).

Your network won't have "exactly once" message transfer happening on it (it will internally be "at least once" for certain packets, but only small ones) and administering it will be very different than administering an Ethernet network, but network protocols absolutely can be designed to give exactly-once delivery to your software.

The real reason most people outside of HPC don't do this is that exactly-once at the network layer is not that useful for most web stuff. You're going to have a higher layer that will drop stuff and retry anyway, so you might as well push the problem up the stack.


Yup, I try to explain it with shouting a message to someone in a crowded room. You can yell at your boss "I fixed the bug", they can confirm it or ignore you, which is delivery at most once if you don't repeat the message. If you try to repeat the message until they confirm it, it is at least once delivery.

edit: Point is in confirming that message is received. If you don't receive the confirmation the message was delivered at most once.


This is a popular saying that is basically wrong.

You have very limited guarantees around an arbitrarily bad partition, but this is also a detectable condition. Lots of defective systems exist, but in general non-defective systems generally guarantee "exactly once delivery or detected failure"


That sounds like "at most once" to me


If you unplug the network you can't send messages, correct.

Other people upthread have already gone over how you can't separate delivery from message processing and how TCP's attempt to do so makes it defective (unless you layer a whole additional system on top of it rendering most of TCP's design irrelevant)

If you were trying to make a new, non-broken system on top of TCP or otherwise, allowing multiple delivery doesn't add any correctness/robustness benefits -- it just makes messages cheaper to send and receive. There is no "at most once or at least once" choice except in the pickwickian sense that if you don't require delivery or delivery confirmation you can save the effort of even trying.


Let me get some popcorn before reading these comments.


Has this author never heard of error correcting codes? The whole point of them is to assume there's lossiness and add bytes to allow correction (or at least detection) of tampered or missing bytes. That's why TCP (or maybe it's Ethernet?) frames include FEC bytes in their message format.

Additionally, I'm sure they're aware that HTTP over TLS has encrypted data frames, which would be unreceivable in a lot of cases if these situations arose a bunch. And considering how much of the modern Internet is built on this paradigm, I think that many of these points are rare and probably extremely pedantic.

This is coming from someone who agrees with much of the nuance implied (but not explained!) by the post.

All great technical writing (which I assume these clickbait articles are at least attempting to be) is written with mutual discovery and deeper understanding in mind, and if you leave no actual explanation in the post, you can't really achieve either of those.


> That's why TCP (or maybe it's Ethernet?) frames include FEC bytes in their message format.

Neither TCP nor Ethernet provide for forward error correction. Ethernet frames include a 32-bit CRC while TCP segments use the so called "internet checksum".


IEEE 802.3, the ethernet working group, defines FEC for many media types. While Ethernet frames do not themselves contain forward error correction, in many cases they are encoded with FEC before they hit the line. This is required by spec for most long low speed and all high speed links. You couldn’t get any frames through without it.


Cunningham's law at work!


Some flavors of optical ethernet do have FEC: https://www.cisco.com/c/en/us/products/collateral/interfaces...


See above :)


What part of the article do you think pertains to error correction codes or data checksumming? The first four points are true even if the transmission medium never changes bytes, just becomes unavailable at a given point.


> If the connection breaks while an ACK is outstanding, the sender will have no way of knowing whether the segment was received

The real question is, why this should be a problem that TCP must solve? TCP gives you a bidirectional waterflow-like pipe, and that's enough for you to create many useful applications. TCP never provided guarantee for correct delivery, that's your job.

For example, if a HTTP request is interrupted before the respond is received, the sender should assume the request never reach the server and try again with a new connection, while the server should mitigate duplicated requests (reject or return a successful code).

Well, maybe that's the point of the article, because many web pages gets confused if you send duplicated requests to them.


The server may or may not have seen the request, and the https://en.wikipedia.org/wiki/Two_Generals%27_Problem proves it impossible to know in every case (no matter how many acks, the last could be dropped). A request that alters state should be retried using the same idempotency key, and the server should try to ack with whether the requested work already happened.


I'll go out on a limb: inside datacenter on your own hardware, you can safely ignore low-level pedantry and mostly ignore “weird networks” and use TCP as two-way Unix pipe.

“Mostly” because you still care about bandwidth limits and packet RPS limits and latency of course.


I wouldn't, unless you've got a really solid understanding of your datacenter network and it's 100% good all the time. Which is unlikely, from my experience as a server person.

If you've got dirty optics between two switches, now you're getting packet loss and TCP rears its head. Hopefully it's not an issue now, but diagnosing microbursting[1] was lots of fun, and really wigs TCP out. I've also run into 'fabric congestion'. My true favorite though is when you've got 2x aggregation on servers, and 4x aggregation for top of rack switches to spine switches, so there's 8 paths in each direction between two servers in adjacent racks, and only one path (sometimes in only one direction) is only running at 99.9%. That's a real PITA to track down unless you have visibility into switching metrics.

[1] https://en.m.wikipedia.org/wiki/Micro-bursting_(networking)


Agreed.

Having done Akka.NET Remote/Cluster setups in prod that survived multiple 'new to the org' categories of DC Failures at their level of scale/capacity [0] there's a lot to account for if you want to keep everything happy and visible [1][2][3]

[0] - Cut fiber between DCs, Rack failures due to IO-ish type issues, bad switches... at least 2 out of 3.

[1] - The upshot was we were able to survive all of the scenarios in at worst a degraded state, Once or twice we needed a restart.

[2] - We also had enough metrics going on that we could detect DC/server outages about as quickly as whoever actually was monitoring the failing subsystem.

[3] - But here's the funny rub. An APM tool was the Achilles heel for both our Akka Links, as well as our SQLServer connections. Once they installed an 'agent' we more frequently had to do a 'full cycle' to clean things up after an outage, or even an MSSQL Server reboot. After I left the shop I got confirmation that yes, the APM module was the problem.


> We also had enough metrics going on that we could detect DC/server outages about as quickly as whoever actually was monitoring the failing subsystem.

Yeah, my Erlang clustering experience was that we (the customer) were the monitoring system for the DC/managed hosting provider. Although, by the time we left there, they would have outage notifications before we put in tickets.


I was always suspicious about self-hosted high availability solutions (typically just diagrams, not yet implemented) that included redundant switches.

Given how generally reliable switches are, I was inclined to believe that a misconfiguration or flaky network cable on one switch was more likely to cause a downtime (or significant degradation) than an outright switch failure, so adding another switch was doubling the chances of trouble and, as you note, making it harder to troubleshoot.


It kind of depends. You do get some weird stuff to debug, and more connections = more likely that one of them is broken.

Otoh, if you ever do any scheduled maintenance on your switches (which is likely if they're doing anything fancy), having properly setup redundancy means you can announce a likely brief loss of redundancy, rather than a likely brief full loss of connectivity. If you have the right knobs, you can gracefully fail out the switch under maintenance and everything goes smoothly. Of course, sometimes you reboot the redundant switch and it confuses the other one and servers lose connectivity anyway.


To some of the critics here: did you or did you not notice the “Somebody ought to write one of those […] Here, I'll even get the ball rolling” framing? A polished such article this is not claiming itself to be! I would go as far as saying the HN submission title is misleading as a result.


This reminds me of a very particular problem that we tried to solve when I worked at VKontakte. It was about instant messaging and flaky mobile data connections.

The problem: you're on a subway train and you send a message as it departs a station. The request does get to the server, but by the time the response arrives, the train is already in the tunnel and you don't have a signal any more. So the client thinks that the message failed to send, but it was, in fact, sent successfully. The client would retry when it's back online, and would send another copy of that message.

The solution was to send a client-generated "random ID" with each request. I much later learned that this is conventionally called an "idempotency token". This worked, except there was now another problem: you sometimes receive your own message over the long-polling thing before the response to the request that sent it. You don't know for sure whether it's the message you just sent, or something else sent by a different client on the same account, because you don't know the ID of your message yet. This was solved by me delaying the processing of outgoing messages on the client side until all outstanding messages are fully sent and their IDs are known.

Telegram solved this much more elegantly: when the client reconnects to the server, the server sends it all the responses that were not acknowledged during the previous connection. MTProto has its own acknowledgement mechanism in addition to TCP's.

So yeah, instant messaging seems trivial at the first glance, but it turns out that TCP is a leaky enough abstraction that you need to somehow plug those leaks at the application level.


I had to deal with the second problem in a file synchronization app. The solution was to propagate a “device id” through the request and poll/push, so the originating device could ignore changes that it originated.


Honest question: Why didn't you send the message's idempotency token back to the client? Then you're playing the familiar local database locking game.


You can't imagine how desperately I asked for that. Backend guys came up with all sorts of excuses to not do it. As far as I know, they did do it sometime after I quit.


> because you don't know the ID of your message yet

Wouldn't the client know the ID if they randomly generated it? (honest question)


I meant the permanent unique ID assigned by the server.


Ive learned that just about nothing is trivial when it comes to distributed systems


It really is astounding to me how so many still do not understand that tcp is not a function call, or behaviors like slow start and congestion avoidance.

Recently a new rate limiter for TCP went by that was so terribly, terribly broken, and I cannot help but imagine that most of the containers of the world suffer from Bufferbloat in general.


The rate limiter in question: https://github.com/cilium/cilium/issues/29083


Please note that it's not an article on lwn.net, it's a comment written by someone.


> There is a such thing as a TCP packet

In what way is that a falsehood?


So TCP has slow start, and exponential fall off and shit.

but you can get round that in a lot of cases by just having a load of TCP connections in parallel.

TCP is cheap and well optimised, especially if you are keeping a bunch of connections open. (opening can be expensive)

so if you have a high latency connection, or a bit of packet loss, and you want to reach line speed without having to figure out cornercases with UDP, just open up 100-1k TCP connections and multiplex them.

bish bash bosh, mostly line speed over a high latency line (mind you this was in the days of 100m-500m cross atlantic internet, you'll probably need more connections to saturate a 10gig line.)


Such hack is often not necessary.

Set larger kernel TCP send and receive buffers and enable BBR congestion control. Speed will usually be good also across high latency links, and no multiplexing logic needed. Especially if you control both sides of the connection.


> Set larger kernel TCP send and receive buffers and enable BBR congestion control

I mean yeah, but that requires having access to the kernel config. so for most people multiplexing TCP is a useful way to maximise a link, without having to fiddle with stuff that is a pain to deploy. (politically as well as logistically)

I deployed this "technique" before BBR was a thing. It worked well enough for what I needed it to do (move large images from London to California) It was pretty simple to engineer as well (mainly because I didn't have to make a fancy custom error detection/correction/rate limiting system over UDP )


1. A SYN will receive a SYN-ACK or a RST 2. A host from my machine is the same as from your machine 3. An IP from my machine is the same as from your machine


1. A SYN may receive a SYN-ACK, RST or nothing at all.


> Explainer for 1-4: https://en.wikipedia.org/wiki/Two_Generals%27_Problem. TL;DR: If the connection breaks while an ACK is outstanding, the sender will have no way of knowing whether the segment was received, and this turns out to be an insoluble problem no matter how much complexity you pile on top of it. You need something resembling Paxos or Raft to get a guarantee like that

The hashgraph algorithm is pretty sweet too and doesn't have the issue of a single write leader like Paxos and Raft. Basically multi-writers / leaderless

https://www.swirlds.com/downloads/SWIRLDS-TR-2016-01.pdf

But to be fair, I'm not certain that CAP theorem and partition tolerance really belong in a conversation about TCP anyway


This post is meaningless without clearly defining what reliable means.

Regarding ack not being received by sender when connection breaks, it's a weak and dishonest argument thinking it will strengthen their position, but completely ignoring the fact that TCP reliability is dependent on the simple and obvious fact that the connection exists!


> 11. This is all low-level pedantry. I can think of TCP like a two-way Unix pipe that goes over the network, and completely ignore how it is implemented.

I mean, that's true, insofar as pipes have incredibly weak guarantees too — after all, the other end of a pipe might be a program reading from/writing to a network socket, or other unreliable transport. Whenever you let your program be plugged into an arbitrary pipe, you have to expect all that same flakiness and then some.


> 11. This is all low-level pedantry

Yeah pretty much.

maybe don't write contradictory unexplained nonsense.


>"7. If we fail to connect to a well-known remote host, then we must be offline."

Now that is a very interesting one!

It's sort of related to the question:

"How much of the Internet is accessible from any given point (location, locality, etc.) at any given point of time?"

Which is sort of unknowable, at least, without attempting to connect with every possible connection point on the Internet, which (if it could be done) would still consist of a range of time, and every point in time following that point would bring changes, perhaps small relative to the whole -- but accruing over time -- more and more, as more time elapses...

Observation: That same (or possibly similar!) phenomena would seem to be at play with respect to the measurment (observation) of quantum systems, i.e., the more certain you are of position, the less certain you are of velocity, and vice-versa...

Well, the more you measure the connectivity to all points of the Internet at one point in time, the less certain you might be of the state of the entire system as more time elapses from that point in time...

But now, why?

Observation: Generally speaking, the larger a system is, the more degrees of freedom it has, in attempting to "lock down" (know by observation, be "certain" of) the entire state of that system at one point of time, the more the parts of the system with degrees of freedom (how many degrees of freedom does the entire Internet have?) will change/evolve/move/"be subject to change" as more time evolves the state of the system... in other words, if you can know position (instantaneous state) with certainty then you can't know velocity (where it's heading to and/or future state and/or that which predicts future state) with certainty!

Sort of like you can know the instantaneous state of the Stock Market and its history... but no one can exactly predict its future (it has many, many degrees of freedom, all of which are subject to change in various unpredictable and bizarre ways!)

Which brings us back to #7:

>"7. If we fail to connect to a well-known remote host, then we must be offline."

We might be offline... but then again, we might not be! (Ping, ICMP, UDP, Telnet and Gopher anyone?)

But then again, we might be!

The Internet's online/offline status (is it really off if it is off? Is it really on if its on?) -- is much like some modern relationships, that is, "It's complicated!" :-)

The Internet is a Black Box!

It's Schrodinger's Internet!

You know, "if a TCP packet travelling at 99.44% of the speed of light on a westbound train track meets a UDP packet travelling at 99.43% of the speed of light on an eastbound train track, then when do they meet?"

You know, "solve for x..."

You know, "assume that the speed of light is constant and that quantum effects are not present!" :-)



While I often do like "falsehoods <....> believe about <...>" format it doesn't always fit in well (and if placed alone without explanation often can at most help you to know where you have knowledge gaps but not which).

A common problem are points which aren't really falsehoods, but where people frequently take false conclusions from it.

E.g. if you ask if TCP is reliable, especially in a non CS paper context, the answer is yes. That is iff you take a reasonable definition of reliable (which doesn't expect literally impossible things) and a reasonable interpretation of mostly. And just listing it as a falsehood fails to point out that there are two potential issues with your understanding while making creating the risk of someone with expertise in that sub-field of IT potentially thinking TCP is quite unreliable when it isn't. I mean the most common usage of the word reliable is a gradient with its meaning in a yes/no question being a short form of "reliable _enough_". Furthermore for most use-cases the "unreliable" aspect of TCP isn't even the main relevant misunderstanding people can have with "TCP is mostly reliable" (through for some use case it is)

The main troublesome misinterpretation is what mostly means. I.e. if you would give it a regious definition it would be "if sampling typical devices used in typical situations across some target audience then for most target audiences (weighted by audience relevance) most of the sampled devices will in a sufficient large long term moving average be reliable enough" or something like that.

What that mainly means:

- even if it's mostly reliable there will be devices for which it is reliable unreliable and anything in between

- similar even if it's mostly reliable for a device that isn't necessary all the time

- nor do we do statements about the patterns when the mostly doesn't apply, i.e. for a device TCP might be mostly reliable except every Sunday 3am for 30s, would still be mostly

- there are use-cases where unreliability is much more common

- there are audiences for which unreliability is much more common

etc.

Similar for point 5,6 about TCP packages, they are definitively a thing and there is no falsehood there. The falsehood is in believing you can reliable control them, that your OS or some middle ware isn't messing with them (e.g. splitting/combining/rewriting). So in some situations it's best to pretend there are non, but in some other situations you have to care and this might differ for different parts of the same protocol. So point 5 and 6 make sense, but don't point in a helpful direction.

to be clear doesn't mean lists are bad, or this list being particular bad, but I which they had more references/details even if short and compact and more clearly separate things too




Consider applying for YC's W25 batch! Applications are open till Nov 12.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: