
FTP Must Die - werdnanoslen
http://mywiki.wooledge.org/FtpMustDie
======
evmar
You might be amused to browse Chrome's implementation of FTP, which has
separate modules for each of the different types of FTP servers you might
encounter, as each emits the "ls -l" output in a different way and that is (I
guess) the only way to get file sizes.

[http://git.chromium.org/gitweb/?p=chromium.git;a=tree;f=net/...](http://git.chromium.org/gitweb/?p=chromium.git;a=tree;f=net/ftp;h=a9ccad9efc957926f20f8825a4ffc95c24673729;hb=HEAD)
see all the "directory listing" files.

The unit tests are full of scary cases like

    
    
      // Tests for "ls -l" style listing in Russian locale (note the swapped  
      // parts order: the day of month is the first, before month).

~~~
toyg
That's actually the standard European order. Testing the Russian locale might
be a scary proposition, but certainly not for this specific reason.

~~~
sjwright
It's not just the standard European order, it's the standard order for nearly
everywhere on the planet except:

> MDY: Belize, USA, parts of Canada, Philippines, Saudi Arabia

> YMD: Japan, China, Iran, small bits of Europe

> DMY: Probably 3/4 of the planet's land surface

That said, YMD should make the most sense (and is most consistent with the
universally accepted HMS time format). I try to use YMD wherever I can.

~~~
reitzensteinm
YMD is logical, because it puts the most significant number on the left.

But both DMY and HMS are consistent in another way - the information is
ordered from most to least important for every day access, from left to right.

A vast majority of dates on advertising, tickets, timetables etc actually
leave off the year, unless it's ambiguous. Similarly, one might say something
like 'on the 4th', which implies 'the 4th of this month' (or the 4th of next
month, if applicable).

This is the reverse of HMS, where the most significant number (the hour) is
most important, and seconds are basically ignored day to day.

MDY, on the other hand, should be taken out the back and shot (IMO).

~~~
sjwright
[edited because I realized we're in violent agreement]

To me, MDY makes as much sense as writing pi as 14.3.159265

~~~
reitzensteinm
I might be missing something, but that's not at all what I was arguing!

MDY drives me nuts too.

~~~
robryan
It's more that different places use different versions that gets me. I have
wrote a bunch of importers for sales an nearly all of them list the date time
differently. Very few affiliate networks have used either a timestamp or the
standards based datetime.

------
sehugg
I don't think FTP must die so much as RSYNC MUST LIVE.

(Seriously, people forget how awesome rsync is. And it tunnels/compresses
nicely over SSH)

~~~
kree10
Also don't forget that rsync does not overwrite a remote file until it's
completely uploaded.

Compare to ftp, sftp and scp, which overwrite the file from the start,
possibly breaking your site during the transfer (or definitely breaking your
site if the transfer dies part-way through).

~~~
plq
> Also don't forget that rsync does not overwrite a remote file until it's
> completely uploaded.

which is really troublesome when you're transferring a big file and the target
doesn't have enough space for a second copy of that particular file. I, for
one, would prefer that "feature" to be optional.

~~~
kree10
Take a look at rsync's "--inplace" option. I think this does what you want.

------
omh
FTP is definitely outdated, and there are better uses in many circumstances.
But I find some of these criticisms a bit odd.

The client listening was largely solved by 'passive' mode, and just about
every server and client supports this now.

The firewall and NAT interaction is awkward, but most modern firewalls can
deal with this automatically (as long as there's no SSL involved)

And yes, the RFC is 20 years old. But so are many RFCs for long-established
veteran protocols that we use all over the net.

In many cases I'd be happy to see FTP replaced (all those anonymous FTP
servers may as well be HTTP now), but it's really not _that_ bad.

~~~
astrodust
FTP is as obsolete as telnet and has no place being alive today.

SSH and SCP provide a much more secure alternative. SFTP is a hack that
suffers from all the same problems.

~~~
MatthewPhillips
FTP is platform agnostic. As uncool as it may be it's the only realistic
choice for companies to share sensitive data.

~~~
roel_v
Please, name me one platform in common use that doesn't have an implementation
of ssh/scp.

~~~
MatthewPhillips
Java, C# which covers a good 80+ percent of enterprise software. Moving data
from one company to another for processing is big, big business. The type of
thing not discussed on HN because it's not relevant here, but it must happen
for those types of companies and FTP is really the only thing you can ask a
paying client to use without asking them to rely on unsupported/abandoned/old
third party libraries.

~~~
roel_v
-_- Java & C# are not platforms, they're programming languages. I'm making an effort here to understand you, but I don't see how a programming language can relate to the availability of file transfer tools.

~~~
MatthewPhillips
Programmatic access is essential in such an environment.

Here's a realistic scenario just to make it clearer. An email appending
company lands a big client. The client wants to obtain its customers' email
addresses on an ongoing basis. They'll send customer information to the email
append company who will then send them back the appended files.

Manually moving files is not an option on either side. What method other than
FTP do you suggest here?

~~~
roel_v
OK then, while I don't have much experience with Java and C#, 15 seconds of
googling showed me at least one broadly used and well supported, open source
library for SSH/SCP for each of these languages. Undoubtedly, there are others
that are not free but backed by a real company. This is aside from
'integrating' SCP in the way people 'integrate' FTP - by calling the command
line version.

So, whether we define 'platform' as 'OS' or 'programming language', my point
stands: name me one (actually used, not VMS) platform for which there is no
SCP available.

~~~
MatthewPhillips
The availability of SCP is irrelevant. What matters is support. I'm glad you
did your research but you came to the wrong conclusion; the conclusion you
should have come to is that FTP is much more tightly integrated. In C# it's
part of the standard library; in Java it's an Apache library everyone has.

I think you're overvaluing what is the "best" method of file transfer. When
you have to set up a file transfer process with a client, and tens or hundreds
of thousands of dollars are on the line, you are going to choose the path of
least resistance. That path is FTP. Clearly. If you are the much bigger
company you have a better chance of doing things the "right way" but that
rarely happens just because these are ultimately business decisions and
business doesn't care about implementation details unless they cost money.

------
pygorex
FTP is like religion: poorly designed at inception, implemented a thousand
different ways and accepted as normal only because it's ubiquitous.

~~~
sjwright
Except unlike religion, FTP began with a legitimate use-case scenario.

~~~
rplnt
Don't know about _all_ religions but some began with clear use-cases. To
retain (or even achieve) power over people. For example The Old Testament was
written/compiled for this very reason[1].

1\. <http://books.google.com/books?id=iS7rQwAACAAJ>

------
kree10
Text-only Google cache:
[http://webcache.googleusercontent.com/search?q=cache:6jeyroO...](http://webcache.googleusercontent.com/search?q=cache:6jeyroOZZnsJ:mywiki.wooledge.org/FtpMustDie&hl=en&gl=us&strip=1)

------
nviennot
There is something pretty cool that one can do with the FTP protocol: FXP
transfers to perform an inter-server file transfer:
<https://en.wikipedia.org/wiki/File_eXchange_Protocol>

------
SeoxyS
FTP is dead. I haven't used the FTP protocol in years[1]… that's what we have
SSH / SCP for.

[1]: Before you say that my anecdotal evidence does not a fact make, my point
is that there's nothing forcing us to use FTP today. Even the lousiest web
host supports SFTP, and none of my machines or VPSs run an FTP server. There's
no reason to proclaim that we need to kill FTP, because FTP is a non-issue in
today's world. Sure, you can still use it, the same way you can still use
telnet if you'd like. But practically nothing relies on it with no
alternative.

------
alecco
Browser HTTP file upload is still terrible for file transfer. One of the last
bastions of Flash, regretfully.

~~~
tptacek
_Browsers_ are terrible at file upload. The weakness here isn't really in the
protocol.

~~~
alecco
That's why I said browsers. But only recently with HTML5 XHR file uploads we
could have progress and status reported in-page (instead of poorly in the
status bar, if any). And it's still not good enough and definitely not widely
available.

~~~
axiak
Huh? As long as you have XHR and JS, you can do file upload progress bars.
Gmail has been doing it for years, and I've certainly had implementations
going back to 2007.

Now, if the upload is failed for whatever reason (i.e. the file is rejected
in-progress), then the upload failure mode is terrible unless you resort to
the usual iframe hackery (Connection Reset screen).

HTML5 XHR file upload is certainly a cleaner solution and something everybody
has wanted, but it's certainly not needed for progress and status reported in-
page without flash.

~~~
alecco
OK.

Small nitpick, Gmail used a Flash uploader for many years, at least until
2009:

[http://ajaxian.com/archives/multi-file-upload-in-the-
flickr-...](http://ajaxian.com/archives/multi-file-upload-in-the-flickr-and-
gmail-house)

------
joeyh
It's perhaps worth noting that HTTP has essentially the same unspecified
directory list problem as FTP. I've seen programs that screen scrape apache
etc directory lists (as displayed when there is no index.html). Oddly, gopher
was an intermediate protocol that got that right.

~~~
dspillett
HTTP wasn't really intended as a file/directory transfer protocol the same
though. The problems with FTP means that is fails at its primary function
(unless you apply copious work-arounds).

Many moons ago in my younger days I wrote an FTP based file/tree
synchronisation tool. I have since vowed to never touch FTP again.

------
cleaver
Does nobody use rsync? I see no mention in the comments here (and the original
site is down).

There's a few command line parameters to learn, but after that it is so
simple, efficient and reliable that there is no need for a GUI client such as
the various FTP FTP clients. I typically write scripts for specific purposes,
syncing only the files that have changed. Using my SSH key means I don't have
to type the password all the time.

------
moe
Sign me up, but where's the alternative?

WebDAV is a trainwreck. SFTP could be nice but the OpenSSH impl falls terribly
short as a FTPd replacement (the most useful implementation is ironically the
one in ProFTPd). Sendfile never went anywhere. Network filesystems don't cut
the FTP use-case either.

People don't use FTP because they like it. They use it for the lack of a
viable alternative.

~~~
peter_l_downs
cleaver and sehugg have both mentioned rsync, which I also think is a viable
alternative. If possible, I try to use scp - is there any reason neither of
these are viable alternatives?

The only reason I ever use ftp is because I'm forced to with my godaddy
hosting.

~~~
moe
For many use-cases rsync and scp are not adequate. For example neither can
provide a file-listing or interactively walk a tree, which is essential for a
wide range of push-based or fileserver-style applications.

~~~
jsight
rsync can provide file-listings:

jsight@jsight-ubuntudesktop:~$ rsync localhost:/home/jsight/hackernews/
drwxrwxr-x 4096 2012/01/29 20:02:48 . -rw-rw-r-- 0 2012/01/29 20:02:36 tmp1
-rw-rw-r-- 0 2012/01/29 20:02:38 tmp2 -rw-rw-r-- 0 2012/01/29 20:02:39 tmp3
-rw-rw-r-- 11 2012/01/29 20:02:45 tmp4 -rw-rw-r-- 11 2012/01/29 20:02:46 tmp5
-rw-rw-r-- 11 2012/01/29 20:02:48 tmp6

~~~
moe
Thanks! That was new to me.

Still, can it also serve as an interactive shell with mkdir, rename etc.? An
rsync-based ftp-clone would indeed be my favorite for a replacement.

~~~
sarnowski
Maybe you should go with ssh if you need a secure ftp replacement:
<http://news.ycombinator.com/item?id=3526713>

------
charliesome
> _The LIST command is Maximum Bullshit. And I'm not even touching the subject
> of opening a data connection to send the list itself. Did you know that the
> specification doesn't tell anything about WHAT should be sent as the list?_

Well it's a good thing that just about every FTP server in existence supports
the MLSD command then.

------
dredmorbius
Coral cache: <http://mywiki.wooledge.org.nyud.net/FtpMustDie>

------
toyg
So, what would it be a modern drop-in replacement for FTP?

Requirements:

* It should _only_ deal with filesystem operations (i.e. SSH is way too problematic).

* It should be firewall-, NAT-, proxy- and browser-friendly.

* It should have cross-platform servers and clients, fully operational via CLI as well as GUI.

* it should be _secure_ (i.e. no http, thanks; https, maybe.)

I'd say it's potential startup territory, except I cannot see any monetization
opportunity. Also, the usual XKCD on standards: <http://xkcd.com/927/>

~~~
soult
I don't see why you disqualify SSH here. If you want a system that _only_
deals with filesystem operations, I suggest NFS. It is neither firewall-,
NAT-, proxy- or browser-friendly, is not cross-platform (Windows support only
if you pay extra) and is neither encrypted nor authenticated. But hey, it
deals with filesystem operations only.

If, on the other hand, you want a system that meets your last 3 requirements,
just use SSH/SFTP.

~~~
toyg
SSH is a pain to secure properly, and the potential for mischief is huge.
There is no _sudo_ in FTP.

~~~
Hemospectrum
It's straightforward to permit SSH access but not shell logins.... _if_ you
can administer user and group settings. I guess stuff like scponly
(<http://freecode.com/projects/scponly>) is useless if you're running Windows
server-side.

------
thristian
It's been a long time since I've been involved with transferring files to and
from clients, but even five years or so ago I recall enabling IIS's WebDAV
mode in preference to handling customer-support calls about getting FTP
working. Of course, WebDAV is still a ball of horror, even if it's not as bad
as FTP, so please let's all stick to SFTP from now on?

~~~
eli
So you had XP boxes connecting via WebDAV? I think I'd take FTP over having to
support that.

~~~
thristian
Indeed. It worked pretty well, actually - in the standard Windows Explorer
'mount network drive' dialog, you could type an HTTP URL and get an ordinary-
looking Explorer window and drag files to and fro.

------
jbarham
Plan 9's 9P is a nice file protocol: <http://plan9.bell-
labs.com/sys/man/5/INDEX.html>.

A client implementation has been available for Linux since version 2.6
(<http://cm.bell-labs.com/wiki/plan9/v9fs/index.html>).

~~~
lubutu
Unfortunately 9P2000 tends to perform badly over high latency connections.
There was talk of writing a new version that would allow clients to group
certain messages, but nothing ever came of it.

Edit: I just found a very recent paper on _Improving the performance of Styx
based services over high latency links_ (Styx being the Inferno name for 9P):
<http://gsyc.es/tr-docs/RoSaC-2011-2.pdf>

~~~
jff
Ah, yes, that would be the work on Op, which IIRC can act as a sort of proxy,
meaning you shouldn't have to re-write any clients or servers. There is also
an experimental implementation of 9P "streams", which actually resemble a
passive FTP transfer--a separate TCP connection is negotiated via the 9P
connection and then used solely to transfer file data.

------
TwoBit
Time and time again I resort to FTP to transfer files between computers that
won't talk to each other via system-provided file systems. Mac<->Windows file
sharing is so unreliable that I don't even try to use the system any more and
just go straight to FTP. But yeah I hate the protocol itself.

~~~
wollw
Have you tried using SCP? I don't know if Windows comes with a client but a
quick search turned up a few options. It is my goto method for transferring
files these days but I admittedly don't use Windows.

~~~
ecaron
I'd love to know the transfer speed differences between SCP, SFTP and FTP -
because when I can view both ends of the wire then security is less of a
concern than transfer time.

~~~
astrodust
The overhead of encryption is marginal and the advantage of being able to
compress the stream is considerable.

scp and rsync, which is a fantastic companion, allows on-the-fly compression
of content to get truly impossible speeds over the wire.

~~~
zoobert
It is marginal if you want to transfer a file from time to time but if you are
in a business where you transfer thousands to millions of files everyday it
makes a big difference. FTP is still used because its interface is simple and
well-known: good old file-directory paradigm that everybody knows. It is also
used because it runs on every platform. I am in favour of having a new
protocol but it should: \- be faster than FTP (would have to use UDP instead
of TCP here). \- Firewall friendly. For that, you could use one port to do
everything and forget about the data connection (Is it feasible with UDP as
you would need to control the flow ?) \- Being FTP compliant from the
interface point of view. You should be able to replace your ftp client with
your fastTP protocol and everything would run. \- Add optional strong
security. Possibility to encrypt or not the data stream and encrypt the
password negotiation.

If we have this and it is marketed well then it could replace FTP.

------
sjwright
Surely the best answer is for the HTTP 2.0 working group to require the
updated protocol to include useful hooks for file-serving scenarios.

Critically, there's only a few missing things in my estimation, and they
revolve around resumable uploading and structured directory listings for GUI
clients.

~~~
pilif
WebDAV was invented as a HTTP based way for accessing remote files. In
general, support on the clientside is very good (built-in clients are in every
major OS).

What hinders wide deployment is the server side: the most widely known
implementation is mod_dav for apache and apache was never really made for a
common use-case of FTP which is people using it with their unix account
credentials for transmitting files.

If you have access to an OSX server, have a look at all the hoops they had to
jump through to allow WebDAV with mod_dav and still do that in the context of
the corresponding system user.

The other reason for FTP still being popular is legacy systems: over the
years, I interfaced so many ERP systems for our product and usually, the only
thing that customers can provide is good old FTP (or direct database access).

As this scenario doesn't involve unix accounts, I would love to use WebDAV for
all the reasons outlined in te article, but body supports it on their end,
despite it being around for 20 yars or so.

------
ew
The reason FTP is still so pervasive is that FTP clients use a similar
interface to the file browser on a computer. Your average person has no idea
that the command line even exists. Yes, FTP isn't a very good protocol, but
the interface is identical for SFTP.

------
hemancuso
We've got a writeup about how much fun it was implementing a FUSE based
filesystem that transports with FTP [ExpanDrive]

[http://blog.expandrive.com/2009/02/02/ftp-considered-
harmful...](http://blog.expandrive.com/2009/02/02/ftp-considered-harmful/)

------
jos3000
I agree. There is no excuse for using FTP when we have brilliant alternatives
like MegaUpload.

------
hcarvalhoalves
Who still uses FTP? I believe SFTP has been the standard for quite some time
already.

~~~
RexRollman
The only thing I use FTP for these days is downloading free operating systems
like OpenBSD and Arch Linux. For this kind of thing, I think FTP is fine.

------
jokull
Anyone remember Hotline? That was a great protocol/client/server for file
transfers.

------
hackermom
I think SFTP is an excellent alternative to FTP (as well as FTP over SSL/TLS).
It's a whole lot safer than FTP and it solves the archaic annoyance of data
ports. Sadly the OpenSSH solution is over the top awkward when it comes to
setting up chrooted access, but it works. Introducing something entirely new
at this point makes no sense - the situation isn't really as serious as some
people want to make it out to be.

~~~
sarnowski
Maybe you missed the OpenSSH 4.9 release?
<http://www.openssh.com/txt/release-4.9>

Since then, it is easy to setup a chroot'ed account which easily acts like an
ftp server. Have a look at ChrootDirectory and the internal-sftp subsystem.

~~~
hackermom
No, in fact I update every 6 months ever since I started using OpenBSD back in
1999. The problem isn't setting up OpenSSH to offer chrooted sftp access. My
gripe is that the sftp subsystem for no solid reason requires that a user
directory is root-owned in order to chroot even when the user account
experiences a forced sftp response (that is, denying shell access, making it
an sftp-only account).

~~~
throwaway64
the reason the directory must be root owned, is that the chroot directive is
also used for normal ssh sessions, where a user owned chroot directory can
mean that a user can break out.

~~~
hackermom
Yeah I'm aware of that part. I submitted a number of design suggestions, as
well as a patch, to the OBSD devs which disregarded the root ownership check
if the SFTP subsystem was called by a connecting client, but no one bothered
even discussing the topic. The whole /home/user/user/ directory nesting just
rubs me the wrong way.

------
kenrik
This should not be "FTP must die" but rather "someone needs to make a really
nice GUI for rsync"

I use rbsync to backup to S3 and it really works great but was a pain to setup
as it's command line only.

------
mjwalshe
Awww didums if the OP can't understand and use a simple protocol like FTP good
job he never had to work with grown up OSI standards - maybe they should stick
to working at mc Donalds.

And those mentioning rsync its not used for the same use cases as ftp.

ftp is usefull for quickly transfereing a few files between systems - rsysnc
is used for totaly diferent

