Hacker News new | comments | ask | show | jobs | submit login
Stop Using FTP (stevenf.com)
65 points by bdotdub on July 14, 2008 | hide | past | web | favorite | 65 comments

Note that "SFTP" has no single meaning. Depending on the client, it may mean "FTP encrypted with SSL" or "FTP with encryption extensions of one of a couple of types invented independently by FTP server authors over the years" or "SSH File Transfer Protocol" (which is actually the real meaning of the abbreviation, and was in use for several years before secure FTP began to be common).

Historically, people tried to convince folks to call it FTPS (for FTP over SSL, which is the most popular form of secured FTP), but that seems to have never taken off. Unfortunate, since SFTP means too many incompatible things.

It's stabilized somewhat in recent years, and usually clients will do the right thing--and the most common meaning of SFTP has come to be "FTP encrypted with SSL" (which is actually one of the weaker forms of security for FTP--but it's got some semblance of "standardization", and works in lots of clients and servers, so that makes it superior). And, of course, there is no RFC defining secure FTP, so it's a de facto "standard" which can lead to buggy implementations and a lot of finger pointing--server says client is wrong, client says server is wrong, user says "I give up" and uses FTP, instead.

I disagree. SFTP means SFTP. FTP over SSL is a different thing.

Since you haven't provided references, I will - look at Adobe Dreamweaver or other web based clients. SFTP is assumed to mean SFTP as implemented by an SSH server.

Traditional FTP uses two ports and it's a bitch to run via SSL - as a result, it's extremely rare.

PS. Moderated the article down for "This is going to sound a little weird at first considering what I do for a living, but I want you to stop using FTP."

It's 2008. You should have stopped using FTP in 1998.

You disagree with what? I thought I was quite clear that the appropriate meaning of SFTP is SSH File Transfer Protocol and has been for years. I certainly didn't think I was arguing that people ought to use it to mean anything else.

Many users (and a few clients and servers) do use the term incorrectly. I don't like it, and clearly you don't either...I don't see what you're disagreeing with.

And, of course FTP over SSL is not extremely rare...not even close to rare. All of the major FTP servers, and many FTP clients, support it quite easily, and many, many end users insist on it. ProFTPd, possibly the most popular FTP server, can be configured to use it with just a few configuration directives.

I'm not encouraging anyone to use the terms incorrectly or to use the FTPS protocol instead of SFTP. In fact, I've been recommending SSH FTP for years (one could probably dig up mailing lists posts that include me scolding people for using FTP from ten years ago).

I guess I've not been very clear. I apologize for misleading anyone into thinking I intended to praise the misuse of terms. I just wanted to point out that people use the terms incorrectly extremely often, and it's wise to be explicit about the protocol you mean.

You asked what I specifically disagreed with. Your original post:

> the most common meaning of SFTP has come to be "FTP encrypted with SSL"

I disagree with that. We both think people should use 'SFTP' to correctly mean as implemented by an SSH server, but you think the term has lost meaning. I am disagreeing with that.

OK. I'll buy that, and based on a search of the various terms, things seem to be improving. My personal experience is clouded by supporting our own products, which is possibly not an entirely representative sample. But, for what it's worth, we have about 1000 paying customers who are web developers...and we're still hearing misuse of the terms on a very regular basis. I'd like to think things are getting better.

Joe, while I've got your attention, somewhat off-topic, does webmin use an intermediary language that spans all config files, or does it directly parse for each format?

What do you use to make the parsers?

Independent parser for every config file. It's the only way to do it such that it allows you to edit config files outside of Webmin, using the command line, using other tools, etc. and have it respect those changes, along with comments and file order. I've used dozens of management products over the years that start from a database and push out the config files via templates, and other such "One True Vision" kind of thing, and it's just not worth it. I can see a few places where it makes sense...for example, I see a place in the world for tools like puppet. But I've never been in a situation where I was willing to accept a GUI or management automation tool as the only way to ever edit a configuration file (though, with Webmin, I almost never need to hit the command line, I always can).

Webmin is written in Perl, and it uses a few relatively simple libraries that Jamie has built over the years (Webmin is 11 years old this October) for slurping files into arrays, breaking them into key=values, etc. Thus each module has a lot of code devoted to parsing...some more than others. We've begun working on making this functionality accessible to other Perl code (and eventually other code in general) via APIs, but that's still coming along slowly...and some of it is unfortunately not abstract enough to be as useful as we'd like. For example, using the Postfix module and the Sendmail module doesn't necessarily have the same semantics--starting and stopping the service are almost always the same between services, but anything config file related is very specific and you have to have knowledge of the directives to do anything with them.

Sometimes the parsers use regexes, sometimes array munging with splice/unsplice/push/pop/etc. grep and map play a big role in Webmin code as well, as they make a lot of kinds of parsing a lot more concise. The code can be intimidating at first, as it's really big (~400kLOC), and because Webmin has never (intentionally) broken backward compatibility for module authors (in eleven years!) has a lot of pretty hairy legacy code. Newer modules and those that are more aggressively maintained (Dovecot, Virtualmin and its related modules, Sendmail, Postfix, Apache, etc.) are nicer to read than older ones.

Anyway, I don't think there is any one silver bullet to deal with configuration files on Linux. They're all pretty wildly different, and some have very bizarre quirks and complexity. Apache, for example, can have arbitrary directives and syntax added by modules, and anything that parses the configuration has to be able to at least understand it well enough to skip over those sections and directives.

Thanks for your thoughts - the various incompatible config file formats is something I've been pondering over for some time, and it's interesting to hear the thoughts of someone with experience.

I wonder about suggesting a common format capable of storing most config file structures as an RFC, then patching those apps to optionally use the format. If you could get Samba, BIND and Postfix the rest could follow...

It's a problem of momentum, and everybody having different ideas about what the right configuration file syntax should be.

One could make the case for something like "everybody use ini syntax", which is the Samba config file format, and the default configuration file format for Python's ConfigParser module (and most other languages have good parsers/writers including Perl). I think it has some concept of nested sections, but I'm not sure it could handle the complexity of BIND or Apache without difficulty. An Apache VirtualHost section can have many sections within, including non-unique sections (e.g. multiple Directory sections), and quite a lot of other tricky stuff, and I'm not able to think of an intuitive way to represent that in ini. I wrote an ad-injecting redirecter for Squid in the distant past that used ini format files...and I ended up having many files in a directory, because it needed nested data. My point is that there's a reason the Apache config file is as complex, and different from everything else, as it is...because it would be hard to convince simpler formats to express all of the information concisely.

Pretty much any config file format can be convinced to do just about anything, though--I'm often stunned at the whacked out crazy crap Jamie does with the Webmin configuration files, which are key=value files. He's got Perl code hiding in some of the values, for example--ternary operators and such--which gets evaluated into place as needed (this is much safer than it sounds). So, whatever format someone came up with would end up being extended as needed and that means you'd have to accomodate all of those extensions--but it still might be nicer for the folks building management tools to have at least a basic standard, even if the management tools don't necessarily know what all of the sections/directives mean.

That said, I suspect that if one could convince a standards body to get behind a "standard" config file format, you'd end up with some XML creature that no one would be happy with, not even those of us who have to parse the configuration files and would ostensibly be relieved to only have one file format for everything.

Momentum is also a problem. I don't foresee anyone on the Apache project, thinking, "Hey! Let's add a second config file parser so people can use SCFF (Standard Config File Format) so Apache configuration files will be the same as a couple of other projects!" It'd be more likely that one or two small projects would use it, and nothing much would change...we'd just have one more configuration file format to parse to support those couple of packages.

Jamie has a sort of funny, and deeply pragmatic, attitude about the whole thing: If it works, why break it? He hates it when anything changes, even to get simpler. New directives to old configuration files. Incompatible changes, even if they are simpler to parse. Etc. So, he'd probably be more angered than pleased to learn that everyone is going to change to some "universal" format.

Now that we're talking about it, I'll mention that Postfix has one of the best configuration systems I've ever seen. key=value main.cf, one-line arrays in master.cf, and space-delimited map files for all of the aliases, virtual domains, transport maps, etc. It takes a lot of files, but it's really easy to parse in just about any language, and I find it really easy to use as a human, as well. And, the kicker is that it's three file formats...so a single config file format wouldn't even work for the single Postfix service!

On that note, from a Linux system, what does `sftp` use on the command line? I had always assumed that SFTP was always and only SSH transfers, just like using `scp`. Is this wrong of me to think this way?

I had always assumed that SFTP was always and only SSH transfers, just like using `scp`. Is this wrong of me to think this way?

No, this is absolutely correct. But don't be surprised if someone misinterprets you.

I'm not sure I agree with you 100% on your police work there, Lou. SFTP is its own protocol; it runs over SSH, but is substantially more complicated than "scp". "sftp" is not just user interface, like scp is.

I took his comment to mean, "I've always understood SFTP to be the FTP protocol that runs over SSH", and so I replied that his understanding was correct.

I didn't consider him saying "like using scp" to mean he thinks "it is scp". Rather, I took it to mean, "it uses SSH to copy files, like scp".

But, I can see how one could interpret it that way. It was not my intent.

I'm staying the heck of out of the pedantry business in the future. This has been a ridiculous bundle of threads all spawned just because I had a bee in my bonnet about how the term SFTP is confusingly misused by lots of folks. I don't even use any of these "FTP" protocols for anything (I use rsync or SVN for website data, and scp for one-off file copies--I haven't used an FTP GUI since I owned an Amiga). I only ever touch FTP for testing problems our users report, and then always using lftp on the command line. I've got no dog in this race...

'man sftp'


Man, why can't people let me take 10 minutes of their life to explain trivialities for me to grasp in 60 seconds, instead of me being the one to waste 10 minutes looking it up? Are you all really that lazy and self-centered?

Because it would take about 15 seconds for you to find the answer by yourself by invoking the aforementionned command.

Seriously, here's the first DESCRIPTION paragraph of man sftp:

  sftp is an interactive file transfer program, similar to ftp(1), which performs all operations over an encrypted
  ssh(1) transport.  It may also use many features of ssh, such as public key authentication and compression.
  sftp connects and logs into the specified host, then enters an interactive command mode.

"... Man, why can't people let me take 10 minutes of their life to explain trivialities for me to grasp in 60 seconds ..."

If this is not an ironic comment: RTFM is good for you and it's not a good habit to encourage people to ask for "pre-digested" knowledge just because "you" see no value in looking something up.

Sarcasm is the name of the game. I was just so caught up in the article/comments that I momentarily forgot about manpages, usually my first place to consult.

Teach a man to fish...

"I had always assumed that SFTP was always and only SSH transfers, just like using `scp`."

Which brings up what I was wondering, what use cases do SFTP/FTPS/FSTP etc. satisfy that a combination of scp and rsync do not? Maybe better for GUI client and file browser interfaces?

Filezilla on windows assumes sftp to be SSH transfers. So I think you've got that backwards.

Filezilla on windows assumes sftp to be SSH transfers. So I think you've got that backwards.

What, exactly, do you think I have backwards? I just tried to explain how wildly diverse the meanings are, and how people use the terms. I don't care what your favorite FTP client means when it uses the term SFTP (bully for you, that you happen to use one that uses the term correctly)...I'm telling you that there is little agreement in the wild about what the term means, and many people use it incorrectly to refer to FTPS (and other kinds of secured FTP).

Just to add, here's what wikipedia says about the term:


Yes, it'd be lovely if everyone used the term correctly...but a large percentage of users do not.

Googling SFTP is actually very heartening--it takes quite a lot of searching to find anyone using the term incorrectly. This wasn't always the case (and the hearts and minds of the non-technical masses takes a lot longer). We still see at least 50% of our customers saying "SFTP" when they mean "FTPS" (and we don't support any of the proprietary FTP-with-encryption protocols).

I think part of our users confusion comes from the fact that the server that provides SFTP service is called sshd and the server that provides FTPS is called proftpd. Obviously, FTP is right in the name of one of them, so that's where all FTP comes from! System administration is hard.

It seems that there is no so much confusion. Maybe you've had bad luck and met a bunch of uninformed people, then you think that the misunderstanding is more common than it is. My experience is also that everybody I met knew what SFTP is.

You know, back in the day, there was an idea called "opportunistic encryption". How it worked was, you would put a FreeBSD box running S/WAN at the edge of your network, and if it detected a similar box at the other end, they would seamlessly encrypt your stream (and if not, just let it straight through). Now, if this idea had gotten the traction it deserved, we could all happily be using telnet, FTP, whatever, rather than everyone having to be aware of encryption, your SA would just sort it once, for everyone, and they'd keep on using familiar tools.

Opportunistic encryption is generally a good idea, but as it has no feedback method you can't really be certain if it is working or not. The seamlessness works both ways.

Right, but the current situation is silly. For example right now if I want to connect into work, I fire up the Cisco VPN client, then I ssh from my Mac to whatever Unix box I want to work on there. I'm not sure that "double encryption" really buys us anything here. Unless someone's actually in our datacentre tapping the wire coming out of the VPN host, in which case we've bigger problems...

It's not really for the double encryption. It has a big advantage in that you don't need to open SSH up to the internet, only to your VPN server. Even though you may have perfectly secure passwords, why expose SSH to the internet if you don't have to?

The VPN is a policy tool, not a security tool. There are no good security tools for managing SSH sessions; you might as well just not have a firewall if you allow it in.

SFTP is its own debacle. It's a handy alternative to "scp" when you don't want to open a second connection to grovel through "find" on the target system, but it's even less firewall-friendly than FTP: you can't enable users to grab files with SFTP without also enabling them to SSH to remote hosts.

Most enterprises don't allow outbound SSH through their firewalls, in part because SSH is its own slippery slope: if you're allowing SSH, you're allowing people to build VPN tunnels.

The real-world operational problem that FTP solves is upload. There is no download application that HTTP doesn't solve better than FTP. If Apache shipped with a mod_upload with a reasonable configuration system and a simple, workable web interface for specifying what to put where, FTP could be laid to rest.

I have seen enterprise systems where sftp access is granted to third parties by restricting the login shell severely to allow only sftp uploads. Is there a problem with that approach? SOX / PCI is making the 'secure upload problem' more pressing daily.

Yes, for instance: X11 and port forwarding.

The bigger problem is, it's just hard to know what else you're allowing once you let someone "only" access SFTP via SSH.

Most of the BigCo's I work with use "Secure File Transfer" software to solve this problem, which are just glorified upload CGI scripts under SSL. That's the best solution, I think.

I'd definitely leave age out of the argument that FTP is bad seeing as how plenty of people use old programming languages, old OSes, and old software for the simple fact that it "just works."

this is a matter of right tool for the job. for important stuff, everything should happen over a secure tunnel (ssl, ssh port tunneling or just using scp.) ftp is good for public file download/drop box where you want to easily support resume.

Almost all his arguments apply equally to HTTP.

Not really. FTP is an awful protocol, and as he mentions it doesn't really even have a spec. HTTP works quite well and has proven to be surprisingly flexible.

- Unless tunneled over a secure socket, FTP is 100% insecure.

This is true for HTTP.

- The spec defines no way of setting the modification dates/times of files.

The HTTP RFC defines the Last-Modified header.

- The spec defines no reliable way of determining the string encoding used for filenames.

The HTTP RFC defines several headers related to encoding.

- FTP was designed to be used interactively by a human sitting at a terminal, not by a GUI application working on the human's behalf. The spec doesn't even define the output format that should be used for directory listings.

This problem doesn't translate exactly to HTTP as it isn't used in the same manner. Directory listings are typically sent as HTML and a user clicks a link which all browsers are able to handle trivially. WebDAV adds features similar to FTP, but it is much more standardized.

- The spec defines no way of dealing with file metadata, such as Unix permissions, owners, and groups.

There are HTTP error codes which indicate lack of permission, but there are no RFC-defined headers for these things. I believe WebDAV does define a method to deal with them.

- FTP requires a minimum of two socket connections to transfer a file: the control connection, which is established first, and then data connections which are created and destroyed every time you transfer a single file, or request a directory listing.

HTTP has no such requirement. Keep-alive prevents sockets from being closed and the same connection can be used to get a directory listing and then transfer a file. There are no separate data and control connections.

- FTP is not friendly with firewalls. Because it constantly needs to establish new connections, this has led us to "passive mode" which might as well be black magic as far as most people are concerned.

This is FTP's biggest flaw and this problem absolutely does not exist with HTTP. Once it is encrypted FTP is impossible to handle in a reasonable manner with a firewall.

In active mode FTP creates one connection to a well-known port for the control channel (21). For the data connection the server attempts to establish a connection from a well-known port (20) to a high-numbered port on the client. This obviously doesn't work well with a firewall in front of the client.

Passive mode modifies this arrangement so that the data channel comes from the client to the server. The problem is the ports used. On both ends the port will be a high-numbered. This requires allowing connections from any high-numbered port (withing a large range) to any high-numbered port. Some firewalls understand the FTP protocol and are able to see the port that is to be used; the server gives the client this port with a command in the control connection. The firewall can then allow that specific connection. Once the control connection is encrypted this is impossible.

Now then. How do his arguments apply equally to HTTP?

"Almost all" might have been hyperbole, but "many" is certainly true.

On the other hand, HTTP is clearly far superior to FTP, especially if you include extensions like WebDAV and the power of custom headers.

I was just pointing out, insecure formats that don't define how to do listings or set mod times or deal with Unix permissions can be quite good. FTP sucks mostly because of the ports/multiple connection issue, as you pointed out.

Well, I'd gladly switch to sftp, but dumb-ass OSX and Windows don't understand it out of the box and most users are unaware of more powerful (and easier to use) alternative OS, which leaves us stuck with this commercialized mediocrity.

Why wouldn't we also stop using Apple Finder, FAT, IE6, non-UTF encodings and bunch of other legacy crap? Ohh-ho-ho... so many topics to blog about...

OS X does support sftp out of the box. Also, it can act as an sftp host if you enable Remote Login (ssh).

From the command line, yes, but not the GUI. When I do command-K in the Finder and enter a sftp://whatever I get this message:

URLs should begin with afp://, at://, file://, ftp://, http://, https://, nfs://, smb://, cifs://, or vnc://

It's easy enough to get a nice SFTP client though (I prefer Transmit. It even supports Amazon S3)

Ah, yes, that's right. It's easy to forget that not everyone is a developer.

Cyberduck (http://cyberduck.ch/) is also a good sftp/s3 client for OS X.

OS X doesn't even really support FTP through the Finder... It's read only.

A good substitute is MacFUSE and the SSH file system. Your server gets mounted as a disk (read and write!) and it's secure through SSH. If you want goodies like preferences there is a product called ExpanDrive that makes it pretty seamless.

I'd say that the reason why people aren't switching so readily is actually a function of the human condition moreso than any technical reason.

Looking at other industries, like health care, people are more content in spending effort/money on "miracle cures" than worrying about taking steps towards preventing these problems from happening.

The exact same thing happens in say, the Automotive industry - a large percentage of consumers dont take active measures towards ensuring their vehicle is in good running order, and are more content with dealing with the consequences.

The same thing seems to happen when talking about computer security. I dont think it's because people dont care about computer security, they're just more ready to accept the consequences rather than taking preventative steps.

It's a real damn shame too.

"people are more content in spending effort/money on 'miracle cures'"

... or businesses are just more interested in researching products that have obvious profitability. Let's say you spread the gospel of eating healthy foods. Where is your income coming from?

What? It's MUCH simplier than than: I can't send a Windows/OSX user an email with an SFTP version of this line:


It's that simple.

Why would you ever send ftp://anything when you could instead send http://anything?

Yep, that's a popular response I get from OSX people: if Apple can't do something, that's because "you don't need it".

If someone sent me an email with an ftp:// link I'd assume it was an old email and had been delayed in delivery by 10 years.

The only use for ftp is to make uploading stuff to public places easier than http. Apart from that it's useless and redundant like telnet.

Actually, OSX people can deal with ftp:// urls just fine. That does not make using them any less stupid though.


What's up with the sound? You just took a shit and can't reach your ass?

I dont disagree with you, in fact I wholeheartedly agree with you - I'm just saying that when it comes to pretty much anything, the majority of people (intelligent or not) just aren't particularly concerned with taking preventative measures.

So I felt like the article was essentially "Pissing into the wind"

For Windows, you can get WinSCP pretty easily. It supports SFTP and, obviously, SCP.

Good and simple program. I used to use WinFTP_LE but this one is where it's at for free Windows FTP tools.

I'm not sure what out of the box means. Good clients like Coda and YummyFTP support SFTP.

I'd like to use SFTP or scp for everything, but I've discovered that it's considerably slower than regular FTP. I wish that there was a way of encrypting the login but sending everything else via plain FTP for speed.

You can do that with WebDAV and Digest authentication. I don't actually know that it's comparable in speed, though.

I'd also question the speed argument...for large files, SSH can perform compression and make the transfer faster than FTP.

Thanks for the compression tip! My extremely informal testing produced the following results taken by copying a random binary file I had to another computer:

SSH: 2.0MB/s

SSH w/ compression: 2.5MB/s

FTP: 2.8MB/s

While SSH does not approach the speed of FTP it does come close, perhaps close enough to finally ditch FTP altogether in most situations.

It depends to a certain extent on what cipher you use. Try Blowfish, AES, etc for the kind of data you move and see which one is better. We aren't talking by much, mind, but every little helps.

sftp is not ideal; looks like the author is not aware that ssh (which is what sftp uses) is rather slow on WAN due to buffer issues:



I think using encryption limits the size of the files to about 2 GB in current versions. It is too little for DVD images.

Simply not true.

The openssh family of tools have had LFS support for many years. I've used scp several times to copy huge files...and our products have scp as a filesystem backup target, which means it deals with archives that are dozens or hundreds of GBs in size on a daily basis.

In what product? I have frequently transferred files larger than 2GB on SFTP as well as storing them with encryption on my computer and there is no 2 GB limit.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact