You might be amused to browse Chrome's implementation of FTP, which has separate modules for each of the different types of FTP servers you might encounter, as each emits the "ls -l" output in a different way and that is (I guess) the only way to get file sizes.
YMD is logical, because it puts the most significant number on the left.
But both DMY and HMS are consistent in another way - the information is ordered from most to least important for every day access, from left to right.
A vast majority of dates on advertising, tickets, timetables etc actually leave off the year, unless it's ambiguous. Similarly, one might say something like 'on the 4th', which implies 'the 4th of this month' (or the 4th of next month, if applicable).
This is the reverse of HMS, where the most significant number (the hour) is most important, and seconds are basically ignored day to day.
MDY, on the other hand, should be taken out the back and shot (IMO).
It's more that different places use different versions that gets me. I have wrote a bunch of importers for sales an nearly all of them list the date time differently. Very few affiliate networks have used either a timestamp or the standards based datetime.
That used to be the case in the UK, but I notice more and more how I'm the only one actually using that form in conversation these days, at least in my little corner of Northern England. Indigenous people have clearly moved on to DMY, i.e. "the 30th of January".
In my opinion YMD is the most useful because it is the most visually distinct from the other two. A date like January 2nd 2012, when arranged in either MDY or DMY format, is difficult to distinguish from February 1st. YMD on the other hand is obvious, independent of what the reader is used to.
MDY is syntactically correct English: January 30th, 2012. Or in older form (where it would have come from): "Dated: January the 30th, in the year of our Lord 2012." Even the US Constitution is dated this way. I strongly prefer this way because it makes the most sense in the English language.
For programmatic sorting YMD makes sense because is automatically sorts by Y, M, D, if compared as a string. (EG: on a filesystem.)
> the information is ordered from most to least important for every day access
I argue that MDY is ordered from most to least important for every day access. Year is easily implied so it is last so that leaves month first and then day. You'd be surprised how many Canadians, which is officially DMY, actually prefer MDY.
Here's a nice stimulus bill for the US economy: mandatory switch from MDY to DMY. That should keep "a few" programmers busy for years! I'm not joking, that's how you move the economy: it's the digital equivalent of building a new bridge.
(It'd be more forward-thinking to actually switch to YMD, but I can imagine the reactions at the idea of aligning with Iran and China.)
In my country YMD and DMY are used, and I don't see, which is more prevalent. And some people use "." some use "/" and some use "-" as separators. It doesn't seem to cause any problems. You just need to validate inputs (best - show example as the default value), and know which foramt you use.
But MDY is just ugly and should die. It's almost as strange, as German way of saying 123. One hundert, three, and twenty :) Who invented this.
Also don't forget that rsync does not overwrite a remote file until it's completely uploaded.
Compare to ftp, sftp and scp, which overwrite the file from the start, possibly breaking your site during the transfer (or definitely breaking your site if the transfer dies part-way through).
> Also don't forget that rsync does not overwrite a remote file until it's completely uploaded.
which is really troublesome when you're transferring a big file and the target doesn't have enough space for a second copy of that particular file. I, for one, would prefer that "feature" to be optional.
PASV mode came with its own security drama, and doesn't address the core issue with the (absolutely pointless) multiple-connection model of FTP.
And so, yes, firewalls and NATs "interact" with FTP --- because they all had to be hacked specifically to deal with the FTP protocol, which moots your rebuttal --- but in doing so they create additional security issues, like the NAT pinning stuff Samy Kamkar posted last year.
It shouldn't be necessary for middleboxes to hack in support for protocols by in-place editing TCP streams and dynamically changing filter/translation rules based on stuff that happens inside of connections. But, thanks to FTP, they do have to do that.
And for what? FTP is even on its own terms a terrible file transfer protocol! For instance, look at how it handles (or, commonly, doesn't handle) file listings.
FTP is an anachronism. It has no more reason to exist today than TFTP --- both were designed to make up for constraints in client software that simply no longer exist anywhere.
Has there been a piece of equipment shipped in the last 10 years that legitimately couldn't get a TCP running to bootstrap itself? That's why TFTP exists: to avoid the need for TCP.
Only tftp is widely supported by embedded ROMs, for netboot or for loading a new firmware for embedded devices with broken OS images.
It's not a matter of getting TCP running. You'd need a basic http transfer client, too. Not many network bootloaders or firmware loaders support anything other than tftp.
iPXE supports http. Get iPXE into embedded bootloaders for new firmware loading, and into network device ROMs for network booting, and then people can switch. Until then, it's tftp because that's what everyone supports.
While passive FTP solved the issue for the client, it totally breaks if the server is behind a NAT, so you are back to the firewall needing knowledge of the inner workings of the protocol.
having the two separate connections really is an anachronism which proves to be a big hassle or all parties.
And not even IPv6 will slve this one as a firewall still needs to know what port to let data through. And because the PORT command lists IP addresses, you can't even transparently run FTP over v6. It's one of te protocols where users need to have special protocol awareness beyond the length of the IP address.
One important firewall that can't deal with it: Amazon's NAT images for EC2. AFAICT, there's no way to run an ftp client behind Amazon's NAT unless you open all outbound ports, because the server is still the one that specifies which port the client should use for its PASV connection.
I think most firewalls solve this by actually rewriting the FTP packets on the fly (IIRC Cisco calls these "fixups"). That's seriously, seriously broken.
How do you mean, SFTP is a hack that suffers from all the same problems? SFTP is a new protocol, it's not the same as FTP. It doesn't concern itself with data port nonsense, and while it doesn't offer authentication or secure transfers in itself, there is - at least to my knowledge - no SFTP solution available that doesn't perform SFTP via an underlying mechanism for secure auth/transfer.
I work for a company that processes sensitive data, and almost all our interactions with clients and suppliers are over SFTP, which is more than platform-agnostic enough.
Java, C# which covers a good 80+ percent of enterprise software. Moving data from one company to another for processing is big, big business. The type of thing not discussed on HN because it's not relevant here, but it must happen for those types of companies and FTP is really the only thing you can ask a paying client to use without asking them to rely on unsupported/abandoned/old third party libraries.
-_- Java & C# are not platforms, they're programming languages. I'm making an effort here to understand you, but I don't see how a programming language can relate to the availability of file transfer tools.
Programmatic access is essential in such an environment.
Here's a realistic scenario just to make it clearer. An email appending company lands a big client. The client wants to obtain its customers' email addresses on an ongoing basis. They'll send customer information to the email append company who will then send them back the appended files.
Manually moving files is not an option on either side. What method other than FTP do you suggest here?
OK then, while I don't have much experience with Java and C#, 15 seconds of googling showed me at least one broadly used and well supported, open source library for SSH/SCP for each of these languages. Undoubtedly, there are others that are not free but backed by a real company. This is aside from 'integrating' SCP in the way people 'integrate' FTP - by calling the command line version.
So, whether we define 'platform' as 'OS' or 'programming language', my point stands: name me one (actually used, not VMS) platform for which there is no SCP available.
The availability of SCP is irrelevant. What matters is support. I'm glad you did your research but you came to the wrong conclusion; the conclusion you should have come to is that FTP is much more tightly integrated. In C# it's part of the standard library; in Java it's an Apache library everyone has.
I think you're overvaluing what is the "best" method of file transfer. When you have to set up a file transfer process with a client, and tens or hundreds of thousands of dollars are on the line, you are going to choose the path of least resistance. That path is FTP. Clearly. If you are the much bigger company you have a better chance of doing things the "right way" but that rarely happens just because these are ultimately business decisions and business doesn't care about implementation details unless they cost money.
Most programming environments have SSH/SCP libraries. If you want to maintain a simple upload/download model, SCP is the way to do it.
It's not entirely clear to me that an FTP-style upload/download model is best here, mainly because I haven't the faintest clue what an "email appending company" does.
Don't know about all religions but some began with clear use-cases. To retain (or even achieve) power over people. For example The Old Testament was written/compiled for this very reason[1].
Now the religion of SMTP.. I often wonder how the spam situation would look if we redid SMTP and required certified TLS. What's worse, we still use SMTP for real stuff every day, I get emails from Etrade, Fidelity and others that I simply don't want flying through cyberspace decrypted even if they don't contain real information.
The people who designed FTP were not stupid: they did the best they could operating under constraints that kids who have grown up with all the serious work of implementing the Internet already done for them cannot imagine. It's easy, with decades of hindsight, to claim you could do better. I bet you couldn't.
FTP is dead. I haven't used the FTP protocol in years[1]… that's what we have SSH / SCP for.
[1]: Before you say that my anecdotal evidence does not a fact make, my point is that there's nothing forcing us to use FTP today. Even the lousiest web host supports SFTP, and none of my machines or VPSs run an FTP server. There's no reason to proclaim that we need to kill FTP, because FTP is a non-issue in today's world. Sure, you can still use it, the same way you can still use telnet if you'd like. But practically nothing relies on it with no alternative.
That's why I said browsers. But only recently with HTML5 XHR file uploads we could have progress and status reported in-page (instead of poorly in the status bar, if any). And it's still not good enough and definitely not widely available.
But the choice isn't between "browsers doing FTP" and "browsers doing HTTP"; it's between special-purpose FTP software and (somewhat hypothetical, if you exclude Transmit) special-purpose HTTP software.
Huh? As long as you have XHR and JS, you can do file upload progress bars. Gmail has been doing it for years, and I've certainly had implementations going back to 2007.
Now, if the upload is failed for whatever reason (i.e. the file is rejected in-progress), then the upload failure mode is terrible unless you resort to the usual iframe hackery (Connection Reset screen).
HTML5 XHR file upload is certainly a cleaner solution and something everybody has wanted, but it's certainly not needed for progress and status reported in-page without flash.
XHR is awful for large file uploads. It doesn't support sending the file object (which presumably would be processed according to RFC1867) on Firefox, but it does support that on Chrome.
As a result, for cross-browser compatibility you have to upload via raw POST data and send your own file metadata headers. Not entirely a bad thing, but it's annoying because most HTTP servers don't support this. And because you cannot send the file object, you have to load the file into browser memory. No streaming options.
Thus, you have to make multiple request uploads that upload a chunk at a time to the server per request (spliced uploading), which is annoying and unnecessary.
To make matters worse, Chrome doesn't support XMLHttpRequest.sendAsBinary (WHY? you can easily implement this in javascript anyway), and a bunch of other random inconsistencies.
Uh, yeah, you needed flash. You can't read local files in javascript without the HTML5 file api. Reading the value of <input type='file'> tags is forbidden in javascript. Try it.
axiak is right, I've done this myself too. The server can read the file size during transfer and send this back to the client over XHR. It's quite messy to set up and most backend frameworks require obscure hacking (reflection in asp.net for example) to get it to work, but it certainly is possible.
Huh? You mean the same Flash that can't display a list of paths to local user files to the user for "security reasons" so the user can do batch uploads? I agree with the sibling comment, browser-based file upload sucks.
It's perhaps worth noting that HTTP has essentially the same unspecified directory list problem as FTP. I've seen programs that screen scrape apache etc directory lists (as displayed when there is no index.html). Oddly, gopher was an intermediate protocol that got that right.
HTTP wasn't really intended as a file/directory transfer protocol the same though. The problems with FTP means that is fails at its primary function (unless you apply copious work-arounds).
Many moons ago in my younger days I wrote an FTP based file/tree synchronisation tool. I have since vowed to never touch FTP again.
Does nobody use rsync? I see no mention in the comments here (and the original site is down).
There's a few command line parameters to learn, but after that it is so simple, efficient and reliable that there is no need for a GUI client such as the various FTP FTP clients. I typically write scripts for specific purposes, syncing only the files that have changed. Using my SSH key means I don't have to type the password all the time.
WebDAV is a trainwreck. SFTP could be nice but the OpenSSH impl falls terribly short as a FTPd replacement (the most useful implementation is ironically the one in ProFTPd). Sendfile never went anywhere. Network filesystems don't cut the FTP use-case either.
People don't use FTP because they like it. They use it for the lack of a viable alternative.
cleaver and sehugg have both mentioned rsync, which I also think is a viable alternative. If possible, I try to use scp - is there any reason neither of these are viable alternatives?
The only reason I ever use ftp is because I'm forced to with my godaddy hosting.
For many use-cases rsync and scp are not adequate. For example neither can provide a file-listing or interactively walk a tree, which is essential for a wide range of push-based or fileserver-style applications.
No way to chroot users to their home, generally finicky configuration (way too easy to give users shell-access by accident), no bandwidth throttling or other advanced ftp features, rather limited logging/auditing.
> The LIST command is Maximum Bullshit. And I'm not even touching the subject of opening a data connection to send the list itself. Did you know that the specification doesn't tell anything about WHAT should be sent as the list?
Well it's a good thing that just about every FTP server in existence supports the MLSD command then.
I don't see why you disqualify SSH here. If you want a system that only deals with filesystem operations, I suggest NFS. It is neither firewall-, NAT-, proxy- or browser-friendly, is not cross-platform (Windows support only if you pay extra) and is neither encrypted nor authenticated. But hey, it deals with filesystem operations only.
If, on the other hand, you want a system that meets your last 3 requirements, just use SSH/SFTP.
It's straightforward to permit SSH access but not shell logins.... if you can administer user and group settings. I guess stuff like scponly (http://freecode.com/projects/scponly) is useless if you're running Windows server-side.
Many recent OSes support it in some fashion natively, there is a cli client (cadaver), and it runs fine over ssl.
It is rather heavyweight and slow though, and doesn't offer quite the range of things as some ftp servers (ratios, etc). That is likely just due to the lack of popularity and the few options for servers.
SFTP (which you discounted) is probably a better option in many cases though.
9P does a lot of these. It's just not very fast over high-latency links. On the other hand, it's extremely simple, can be easily encrypted, and does only filesystem operations. Oh, and there are clients and servers for all the major platforms and libraries for a lot of popular languages.
I entertained the idea once of a file-manipulation protocol based on the Redis protocol (with possibly a bit of JSON mixed in for sending structured data like directory listings). The only downside is that the only way I know of to secure it is SSH tunneling...someone should come up with a way to do Redis over TLS.
It's been a long time since I've been involved with transferring files to and from clients, but even five years or so ago I recall enabling IIS's WebDAV mode in preference to handling customer-support calls about getting FTP working. Of course, WebDAV is still a ball of horror, even if it's not as bad as FTP, so please let's all stick to SFTP from now on?
Indeed. It worked pretty well, actually - in the standard Windows Explorer 'mount network drive' dialog, you could type an HTTP URL and get an ordinary-looking Explorer window and drag files to and fro.
Unfortunately 9P2000 tends to perform badly over high latency connections. There was talk of writing a new version that would allow clients to group certain messages, but nothing ever came of it.
Edit: I just found a very recent paper on Improving the performance of Styx based services over high latency links (Styx being the Inferno name for 9P): http://gsyc.es/tr-docs/RoSaC-2011-2.pdf
Ah, yes, that would be the work on Op, which IIRC can act as a sort of proxy, meaning you shouldn't have to re-write any clients or servers. There is also an experimental implementation of 9P "streams", which actually resemble a passive FTP transfer--a separate TCP connection is negotiated via the 9P connection and then used solely to transfer file data.
Time and time again I resort to FTP to transfer files between computers that won't talk to each other via system-provided file systems. Mac<->Windows file sharing is so unreliable that I don't even try to use the system any more and just go straight to FTP. But yeah I hate the protocol itself.
Have you tried using SCP? I don't know if Windows comes with a client but a quick search turned up a few options. It is my goto method for transferring files these days but I admittedly don't use Windows.
I'd love to know the transfer speed differences between SCP, SFTP and FTP - because when I can view both ends of the wire then security is less of a concern than transfer time.
you can use hpn-ssh (high performance) with aes128 (if your hardware has AESNI/padlock/etc) or rc4 (probably faster if your hardware lacks AESNI/padlock/etc) to eke a bit more speed over local connections on trusted lines. throw in compression and you can get good speeds.
rsync-to-rsyncd over a local link is probably faster though (requires more setup though).
You could probably just pipe tarballs through nc too. ;)
It is marginal if you want to transfer a file from time to time but if you are in a business where you transfer thousands to millions of files everyday it makes a big difference.
FTP is still used because its interface is simple and well-known: good old file-directory paradigm that everybody knows.
It is also used because it runs on every platform.
I am in favour of having a new protocol but it should:
- be faster than FTP (would have to use UDP instead of TCP here).
- Firewall friendly. For that, you could use one port to do everything and forget about the data connection (Is it feasible with UDP as you would need to control the flow ?)
- Being FTP compliant from the interface point of view. You should be able to replace your ftp client with your fastTP protocol and everything would run.
- Add optional strong security. Possibility to encrypt or not the data stream and encrypt the password negotiation.
If we have this and it is marketed well then it could replace FTP.
scp is very slow, but not sure how slow in comparison to the other two. Copying lots of data between servers on your own network, it's much faster to tar/netcat between the two than scp.
Surely the best answer is for the HTTP 2.0 working group to require the updated protocol to include useful hooks for file-serving scenarios.
Critically, there's only a few missing things in my estimation, and they revolve around resumable uploading and structured directory listings for GUI clients.
WebDAV was invented as a HTTP based way for accessing remote files. In general, support on the clientside is very good (built-in clients are in every major OS).
What hinders wide deployment is the server side: the most widely known implementation is mod_dav for apache and apache was never really made for a common use-case of FTP which is people using it with their unix account credentials for transmitting files.
If you have access to an OSX server, have a look at all the hoops they had to jump through to allow WebDAV with mod_dav and still do that in the context of the corresponding system user.
The other reason for FTP still being popular is legacy systems: over the years, I interfaced so many ERP systems for our product and usually, the only thing that customers can provide is good old FTP (or direct database access).
As this scenario doesn't involve unix accounts, I would love to use WebDAV for all the reasons outlined in te article, but body supports it on their end, despite it being around for 20 yars or so.
The reason FTP is still so pervasive is that FTP clients use a similar interface to the file browser on a computer. Your average person has no idea that the command line even exists. Yes, FTP isn't a very good protocol, but the interface is identical for SFTP.
The only thing I use FTP for these days is downloading free operating systems like OpenBSD and Arch Linux. For this kind of thing, I think FTP is fine.
I think SFTP is an excellent alternative to FTP (as well as FTP over SSL/TLS). It's a whole lot safer than FTP and it solves the archaic annoyance of data ports. Sadly the OpenSSH solution is over the top awkward when it comes to setting up chrooted access, but it works. Introducing something entirely new at this point makes no sense - the situation isn't really as serious as some people want to make it out to be.
> Sadly the OpenSSH solution is over the top awkward when it comes to setting up chrooted access, but it works.
With OpenSSH it's pretty easy to setup users in a chroot, with no shell access. The hard part is if you want logging (to audit what they do), then you need to create a log device in their chroot, which adds complication, and you need OpenSSH 5.2+.
sshd_config:
Subsystem sftp internal-sftp -l VERBOSE
Match Group sftponly
ChrootDirectory %h
ForceCommand internal-sftp -f AUTH -l VERBOSE
Since then, it is easy to setup a chroot'ed account which easily acts like an ftp server. Have a look at ChrootDirectory and the internal-sftp subsystem.
No, in fact I update every 6 months ever since I started using OpenBSD back in 1999. The problem isn't setting up OpenSSH to offer chrooted sftp access. My gripe is that the sftp subsystem for no solid reason requires that a user directory is root-owned in order to chroot even when the user account experiences a forced sftp response (that is, denying shell access, making it an sftp-only account).
the reason the directory must be root owned, is that the chroot directive is also used for normal ssh sessions, where a user owned chroot directory can mean that a user can break out.
Yeah I'm aware of that part. I submitted a number of design suggestions, as well as a patch, to the OBSD devs which disregarded the root ownership check if the SFTP subsystem was called by a connecting client, but no one bothered even discussing the topic. The whole /home/user/user/ directory nesting just rubs me the wrong way.
Awww didums if the OP can't understand and use a simple protocol like FTP good job he never had to work with grown up OSI standards - maybe they should stick to working at mc Donalds.
And those mentioning rsync its not used for the same use cases as ftp.
ftp is usefull for quickly transfereing a few files between systems - rsysnc is used for totaly diferent
http://git.chromium.org/gitweb/?p=chromium.git;a=tree;f=net/... see all the "directory listing" files.
The unit tests are full of scary cases like