
Each socket should have two file descriptors - riobard
http://cr.yp.to/tcpip/twofd.html
======
pierrebai
A small compendium of errors...

For starter, a | b |c doesn't create two file descriptors. It creates four,
two per pipe. Then there is a problem with the central argument. For regular
files, and in fact all method of creating file descriptor, _except_ pipes,
opening in RW mode creates a single descriptor. The reason for pipe() special
behaior is two pronged:

1\. The raison d'être of pipes is for sequential communication between
producer/consumer processes. There is no other reason for their design. As
such, it made sense to break the convention of a single FD.

2\. At the time of their design, the 70s, memory footprint was a crucial part
of any design. Thus, sharing buffers between the producer and consumer FD was
primordial, and the best way to make it happen is to create the FD at the same
time.

The other problem is that the pipe analogy for TCP is wrong. Which client /
server protocol over TCP ever had single direction data transfer? Close to
100% of all TCP usages are between a single client and a single server,
communicating both ways.

The design for socket is practical for their intended purpose. Trying to shoe-
horn an artifical problem an a design will, unsuprinsingly, no yield proper
result.

~~~
tptacek
He said it creates two _pipes_ , and later that each pipe creates two
descriptors.

He never said there was no reason that sockets behaved differently than other
descriptors, only that the reasons didn't warrant the broken abstraction.

And finally, I think you missed the place where he claims the abstraction
breaks down. He's not complaining that you use "open" to get a file's
descriptor and "pipe" to get a pipe's two descriptors, any more than it's
weird to get a socket descriptor for "socket". He's saying that once you have
the descriptor, the single file descriptor for sockets forces the OS to
implement a socket-specific call simply to get a FIN sent properly.

The reality is that sockets could be represented not with integer descriptors
but with character arrays or structs and the Unix interface wouldn't be any
less incoherent than it is now with accepting descriptors, a select call that
works for sockets and not files, setsockopt, shutdown, recv, connected Unix
domain sockets, ioctls, and so on and so forth.

------
munin
this is something I hate about the "unix design philosophy". when you start
digging deeper, you realize that "everything is a file" is true, except for
network connections, and A/V devices, and input peripherals, and really that
the only things that actually are files, are files. everything else is
allllmost a file except in one little weird way, or, many big big ways...

~~~
nknighthb
That's not a problem of the philosophy, it's a problem of its implementation.
And that is caused by the myriad people and entities to have been responsible
for the chaotic evolution of the various parts that make up modern Unix
systems.

Design by effectively nobody can be at least as bad as design by committee.

As others pointed out, Plan 9 is much closer to a proper realization of the
everything is a file design philosophy. Unsurprising, since it was grown
entirely within Bell Labs under the supervision of the original Unix team.
It's unfortunate it hasn't reached a point of being ready for mass adoption.

~~~
duaneb
> As others pointed out, Plan 9 is much closer to a proper realization of the
> everything is a file design philosophy. Unsurprising, since it was grown
> entirely within Bell Labs under the supervision of the original Unix team.
> It's unfortunate it hasn't reached a point of being ready for mass adoption.

I don't think it's unfortunate, having used it for a month straight I'm not
convinced at all it's the right thing, and in fact, is exactly the opposite
direction from where I think we should be going (as far away from the
filesystem as possible).

~~~
nknighthb
Because you have a technical argument, or because of the poor user experience
that a research platform provides?

~~~
duaneb
My technical argument is that the filesystem is too complicated an abstraction
for questionable benefit. Yes, sockets are really cool as files, but that's
not any more useful outside of the unix shell.

I suspect that Plan 9 could be a good dev environment, but is not a great
deployment platform—it just doesn't offer enough better than developed *NIXes
to port code over.

------
tankenmate
I just don't see the problem that djb is highlighting. To me the crucial
mistake comes in this sentance; "When the generate-data program finishes, the
same fd is still open in the consume-data program, so the kernel has no idea
that it should send a FIN." generate-data and consume-data should _NEVER_
share a fd; the two pipes in the "same machine model" are two seperate sets to
fds (the pipe() returns both ends of the fd in one call). Likewise the TCP
model should use two separate (sets of)sockets. shutdown()'s real use is for
poorly implemented protocols where the server has no real time way of
initiating a control message to the client apart from "abort"ing the
connection; some protocols only allow the client to initiate sending a
message. Also note that one end of the connection sending a FIN doesn't
preclude the other end from sending more data.

~~~
drudru11
I'm going to take a stab at interpreting what this article is about.

First, the missing background:

djb likes unix.

the unix philosophy is to compose small programs together to solve problems.

djb's own programs illustrate this really well. They are all small, focused
tools. This allows each program to focus on their particular task or domain.

The primary method of composition in unix is the pipe in a shell. Each pipe
has two descriptors. One for read and one for write.

It is very easy to create a pipe and handle pipe IO.

The article:

At some point, djb wanted to have some programs live on the network. This
expands the composition beyond a single machine. If you just try to treat a
socket as a standard pipe, you encounter the problem he describes.

Any program utilizing a pipe requires two file descriptors. If someone built a
trivial 'netpipe', they could just 'dup' or 'dup2' the socket file descriptor
to make it look like a normal pipe. The problem with that is now the socket
won't close until both fds are closed. This means the remote end won't detect
EOF. This means the 'netpipe' program has to be very clever in order to detect
EOF and do a proper close on both so the remote can see the last bytes and
then EOF.

------
shabble
The original doesn't appear to be dated, but archive.org[1] has the same thing
going back to at least 2003.

[1]
[http://web.archive.org/web/20030805143958/http://cr.yp.to/tc...](http://web.archive.org/web/20030805143958/http://cr.yp.to/tcpip/twofd.html)

Edit: HTTP Last-Modified header looks like it might be right:

    
    
        Last-Modified: Tue, 10 Jun 2003 23:44:11 GMT

~~~
tptacek
I think this page may predate archive.org, and that it may have lived at a
different location on his site. I remember it as part of the ucspi
documentation, which (I think) just barely predates tinydns.

------
jingo
Hey, I got a better idea. Let's give each socket a "human-friendly" name.
Translating back and forth to the underlying numerical representation should
be easy enough.[1]

Heck, we could even create a centralized system for managing our new
namespace. OMG, maybe we could charge people money for the names? Yes! We're
rich!

And the result: Hundreds of millions of "parked" domains serving up cheap
advertising. Simply brilliant!

1\. See Hobbit's comments in netcat source code for a differing opinion.

I often wish that people like djb or Hobbit (=low tolerance for nonsense) had
designed the systems that we are now stuck with.

Though they are only applications, netcat and ucpsi have aged well and remain
a pleasure to use.

~~~
dsl
Nobody is forcing you to use DNS.

~~~
jingo
Ha. That is debatable. Define "force".

At the very least, I'd say there is strong coercion. If not in favor of using
names, then certainly in favor of deprecating the use of IP and port numbers
(e.g., for email). Lemme guess, now you'll say "No one is forcing you to use
email."

------
tomp
Hmmm... use two TCP sockets?

~~~
LukeShu
You've missed the point. It's not about "I can't do things with TCP". It's
that BSD's networking implementation, which modern Unix networking is based
on, broke the "everything is a file" philosophy of Unix. Instead of using the
existing file descriptor interface, they created a new socket interface, which
is logically 2 file descriptors. However, it's not actually 2 file
descriptors, so you now need a bunch of device-specific code.

~~~
eps
As others have said, it wasn't a valid point to begin with. A file open for
reading and writing is too "logically 2 file descriptors", yet it's a single
fd, just like a socket.

------
rumcajz
Absolutely true. Too late to fix now though.

~~~
angersock
That seems to be the general theme with a lot of network and systems
programming. :(

------
perfunctory
I don't quite understand the appeal of everything is a file philosophy.
Neither everything is an object for that matter. It makes the world look like
this:

[http://www.youtube.com/watch?v=HPeattKV74A](http://www.youtube.com/watch?v=HPeattKV74A)

------
nly
Now let's talk about how half the higher layer protocols in the world that use
TCP should probably be using a reliable datagram protocol...

~~~
EdiX
It doesn't matter, with the way IP works any reliable datagram protocol would
contain an implementation of 90% of TCP.

The more fragmented an IP packet gets along the way the less likely it is it
will reach destination, so you have to take into account path MTU size and
split your datagrams accordingly. You also want to send as many datagrams as
you have available in as few IP packets as possible and you want to do slow
start for the same reasons TCP does it.

Result: your datagrams need to become a stream of bytes to be handled
efficiently by any transport protocol sitting on top of IP.

------
binarymax
Someone please correct me if I am wrong, but is this something that Plan9 was
solving? Everything (including sockets) being treated as a file?

~~~
pfraze
(As I understand it) Plan9's file-system abstracted over network connections,
yeah. You'd bind files & folders to your view of the filesystem, and that
binding could cross network boundaries.

EDIT: theoh says it better
[https://news.ycombinator.com/item?id=6080324](https://news.ycombinator.com/item?id=6080324)

------
drudru11
tl;dr - this is about 'leaky abstractions'

[http://www.joelonsoftware.com/articles/LeakyAbstractions.htm...](http://www.joelonsoftware.com/articles/LeakyAbstractions.html)

this was written about by Joel in 2002.

This article probably dates before that, but is last modified circa 2003 (see
other comment on that)

------
beachstartup
netcat solves this.

~~~
sigil
netcat doesn't "solve" this unix design problem any better than the author's
own programs, tcpserver and tcpclient [0], solve it.

djb's argument here is that TCP sockets are more like pipes, with separate
read and write buffers, and separate read-side and write-side close
operations. This makes sense, but what about UDP sockets? What about
operations that apply to the socket as a whole, like bind(2), listen(2) or
ioctl(2)?

[0] [http://cr.yp.to/ucspi-tcp.html](http://cr.yp.to/ucspi-tcp.html)

~~~
jfb
This makes me long for a djb vs. Linus flamewar. Well, no, not really, because
that would be counter-productive, but man, it'd sure be epic.

~~~
aidenn0
Getting completely off topic now, but I would actually pay money to read the
spectate that. We just need to find something they disagree about that they
also both care enough about to rant at each other for.

