* Wget's the interactive, end-user tool, and my go-to if I just need to download a file. For that purpose, its defaults are more sane, its command line usage is more straightforward, its documentation is better-organized, and it can continue incomplete downloads, which curl can't.
* Curl's the developer tool-- it's what I'd use if I were building a shell script that needed to download. The command line tool is more unix-y by default (outputs to stdout) and it's more flexible in terms of options. It's also present by default on more systems-- of note, OSX ships curl but not wget out of the box. Its backing library (libcurl) is also pretty nifty, but not really relevant to this comparison.
This doesn't really need to be an "emacs vs. vim" or "tabs vs. spaces"-type dichotomy: wget and curl do different things well and there's no reason why both shouldn't coexist in one's workflow.
Totally agree. I love curl for testing API request/responses manually. It's usually a huge part of navigating my way around a new API that doesn't have a client library available for whatever language I'm using at that time.
I also use it for weird requests that need special headers or authentication.
Wget is the first thing I turn to when I'm downloading anything remote from the command line or scraping some remote content for analysis.
Curl supports gzip and deflate. It would be great if support for sdch and br were added too. Brotli is in Firefox 44 and can be enabled in Chrome Canary with a flag. SDCH has been in Chrome for a while and is used on Google servers.
Use "-C -" to tell curl to automatically find out where/how to resume the transfer. It then uses the given output/input files to figure that out.
Also, I wouldn't consider that a stupid question. :)
#Edit. More info here: https://en.wikipedia.org/wiki/Byte_serving
It handles the basic case of fetching the remainder of an interrupted download, and can also support partial downloads e.g. for supporting a video stream with the user jumping to different places in the movie.
curl is a unix-way type program that interacts with standard out in a pretty predictable way.
Edit: nevermind, I just learned that wget can download a page's resources or even inline them (-k apparently), that's a bit of a way off from curl's purpose. Better keep them separate tools, although wget might benefit from using libcurl so they don't have to implement lots of stuff like https or http2.
$ aria2c http://yourlink.com/file.*
-x2 allows using 2 connections per host.
bropages is where it's at for that kind of stuff. curl is actually their usage example :)
Does it finally use filename from "Content-Disposition" without need for any switches?
Agreed, and even "emacs vs. vim" or "tabs vs. spaces" do not really need to be "emacs vs. vim" or "tabs vs. spaces"-type dichotomies.
wget -r -l5 -k -np -p https://docs.python.org/2/
I also prefer wget to `curl -O` for general file downloads, simply because wget will handle redirects by default, `curl -O` will not. Yes, I could remember yet another argument to curl... but why?
That said, I love curl (combined with `jq`) for playing with rest interfaces.
It's still early in its development, and would benefit from a tutorial and a few more features, but it's getting there.
echo "-L" >> ~/.curlrc
(you can also find archives for Python 3 and SciPy and NumPy)
For my money another tool that belongs in the toolbox is perl's WWW::Mechanize and its component WWW::Mechanize::Shell.
The general workflow I have followed was, "Download PDF if present. If documentation can be shown in single HTML page, download it, change all the anchors to point to my own filesystem". This changes everything in such a simple, useful, way.
I had a patch at one point to make pydoc style itself just like docs.p.o, but I'm not working much in Python these days. Maybe it's not even necessary any more, which would be cool.
Python's built in documentation tools are good, but the command I referenced also gives you the full language spec locally, as well as the FAQs and packaging docs.
It's not going to replace curl or wget usage, but it is a nicer interface in certain circumstances.
Whenever I go back to curl I feel the same minor aggravation I do when moving from homebrew back to apt (want to install? apt-get. Search? apt-cache search. List? dpkg. Remove? apt-get again, for some reason). It's just not a very user-friendly CLI and I'm constantly pulling up the curl man pages.
I think this is largely why aptitude exists; a single unified command for all common package tasks. Most of the Debian docs have been switched over to recommend its usage, but for some reason Ubuntu hasn't followed suit. That said, I just keep using the same old shell aliases I added in 2003 personally.
Also amazing: jq.
Putting it up on chocolatey might be a good idea. Not sure how feasible that is however.
Of course it's going to be miserable to use any command-line tool on Windows. It's Windows.
PS> $page = wget http://www.yahoo.com/
PS> $page.Images | sort width | select src, width, height | Export-Csv -Encoding utf8 images.CSV
Obviously if you work in a cross-platform environment, cygwin/mingw is still the only thing that will provide you some sort of consistency in your workflow on Windows machines.
The major annoyances were packages were limited, compiling anything was generally a disaster and file permissions between linux/windows are a mess. I happily used it everyday though.
I don't use Windows but I also dislike python dependencies if I can get away with just needing libc.
With msysgit (for git and msys) you can practically live in the command line on Windows and with python 2.7 the py things mostly just work. pywin32 helps for some things and if pip install doesn't work there's always the unofficial windows binaries:
I know it's not the same as installing a tool natively, but it lets me a) use some tools pretty much in the way I prefer to do so and b) check out stuff that I see on HN (and elsewhere) without a PITA install.
Installing was a one line deal if pip is installed (pip install --upgrade httpie), which in turn was a one line installation (wget https://bootstrap.pypa.io/get-pip.py -O - | python)
That's pretty close to one line.
Other than that, curl is always better.
 Aliasing `wget` to ~`curl -O` might be a good idea :)
$ man curl | wc -l
$ man wget | wc -l
$ curl --help | wc -l
$ wget --help | wc -l
$ curl --help | grep -- -- | wc -l
$ wget --help | grep -- -- | wc -l
You want the `wget --help` text over the man page, 99% of the time. The other 1%, you want the full info manual. The man page is an awful mix between the two; too dense for scanning through for the flag you need, but not containing the full information when you need specifics.
Except [at least] me.
I like man better because it's consistent. Some tools want --help, -help, -h, -H, -\? etc.
I like man better because I can search it.
I like man better because it gives me the details, not just a list.
Plus even if grep matched something you can't read the context without extra options.
man is much easier than doing all that, but the time you have your full search command you would have already gotten your info from man.
For context try grep -2, where 2 is the desired lines of context.
$ wget --help |& grep -2 base
-i, --input-file=FILE download URLs found in local or external FILE
-F, --force-html treat input file as HTML
-B, --base=URL resolves HTML input-file links (-i -F)
relative to URL
--config=FILE specify config file to use
existing files (overwriting them)
-c, --continue resume getting a partially-downloaded file
--start-pos=OFFSET start downloading from zero-based position OFFSET
--progress=TYPE select progress gauge type
--show-progress display the progress bar in any verbosity mode
But rather, doing all that seems easier than man to you?
Honestly, either is good. Grouping options is good if you don't know what you're looking for, and alphabetical is good if you do. The bad ones are like the help page for rsync - a ton of options, with no semantic ordering at all.
"[Wget's] ability to recover from a prematurely broken transfer and continue downloading has no counterpart in curl."
curl -C - -O filename url
I love both of these, but wish that curl was just like wget in that the default behavior was to download a file, as opposed to pipe it to stdout. (Yes, aliases can help, I know.)
curl http://api.example.com/json | jq '.["someKey"]' # etc., etc.
curl http://bogus.example.com/install | sudo bash
EDIT: Wow, surprised by the downvotes. I don't think I said anything controversial (y'know principle of least surprise and all), but maybe I was being a bit too opaque: wget, by virtue of being the first on the scene, built an expectation that $THING_THAT_GETS_URLS would result in a file without any other input/arguments. Curl, to this day, surprises me because I was around when wget was all you had.
Whoa... TIL something! I don't know if that's the official etymology, but that's a great mnemonic!
EDIT: ... and yes, I use both tools :).
If you post something, stand by it, do not worry about downvotes. I don't really like them, but nevertheless I'm proud when I get a downvote - it means I don't have a hive-mind mentality.
curl -o does not.
Or maybe there's a good reason for having two tools :)
curl is for everything else (love it when it comes to debugging some api)... Httpie is not bad too for debugging but most of them time I forget to use it.
- Supports splitting and parallelising downloads. Super handy if you're on a not-so-good internet connection.
- Supports bittorrent.
- Can act as a server and has a really nice XML/JSON RPC interface over HTTP or WebSocket (I have a Chrome plugin that integrates with this pretty nicely).
They're not super important features sure but I stick with it because it's typically the fastest tool and I hate waiting.
This means you can't securely download content using relatively recent (but not the newest) versions of wget (such as any in the Ubuntu 12.04 repos) from a server which uses SNI, unless the domain you're requesting happens to be the default for the server.
As an example, I found the file https://redbot.org/static/style.css only accessible with SNI. Try `wget https://redbot.org/static/style.css` vs. `curl -O https://redbot.org/static/style.css` on Ubuntu 12.04. Domain names which point to S3 buckets (and likely other CDNs) will have similar issues.
wget does that without any parameters. Curl requires me to remember and provide parameters for this obvious usecase.
So wget wins every time.
For example, `--mirror-url` was implemented. So, it is now possible to download from two sources concurrently.
For OS X users, you can get wget pretty easily with Homebrew¹. Just install it, then enter the following:
brew install wget --with-gpgme --with-iri --with-pcre
…well, those extra options aren’t strictly needed. Just what I used since I wanted wget compiled with support for those things (GnuPG Made Easy², Internationalized Resource Identifiers³, and Perl Compatible Regular Expressions⁴).
You can see all the compile-time options before installing wget by typing in:
brew info wget
¹ — http://brew.sh/
² — https://www.gnupg.org/related_software/gpgme/
³ — https://en.wikipedia.org/wiki/Internationalized_resource_ide...
⁴ — https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expres...
"Much more developer activity. While this can be debated, I consider three metrics here: mailing list activity, source code commit frequency and release frequency. Anyone following these two projects can see that the curl project has a lot higher pace in all these areas, and it has been so for 10+ years. Compare on openhub"
Under wget he has:
"GNU. Wget is part of the GNU project and all copyrights are assigned to FSF. The curl project is entirely stand-alone and independent with no organization parenting at all with almost all copyrights owned by Daniel."
Daniel seems pretty wrong here. Curl does not require copyright assignment to him to contribute, and so, really, 389 people own the copyright to curl if the openhub data he points to is correct :)
Even if you give it the benefit of the doubt, it's super unlikely that he owns "almost all", unless there really is not a lot of outside development activity (so this is pretty incongruous with the above statement).
(I'm just about to email him with some comments about this, i just found it interesting)
I did well in that course, granted it was an easy intro to programming one. ;)
I think this is oddly the major reason why wget is more popular. Saving 3 chars + not having to remember the specific curl flag seems to matter more than what we can think.
can wget do similar? I did not know it can or could however from my point of view if it cannot this is like comparing a philips head screwdriver to a powertool with 500pc set.
not sure how else to describe other than I do not think wget is capable of such functionality.
typically I use wget to download, and curl to troubleshoot http https.
however, if that is capable of keeping everything open I suppose thats a point for wget.
For example here's a link to download 7zip for windows from filehippo.com.
* Curl doesn't download it at all.
curl -O 'http://filehippo.com/download/file/bf0c7e39c244b0910cfcfaef2af45de88d8cae8cc0f55350074bf1664fbb698d/'
curl: Remote file name has no length!
2016-03-03 18:08:21 (75.9 KB/s) - ‘index.html’ saved [1371668/1371668]
03/03 18:08:45 [NOTICE] Download complete: /tmp/7z1514-x64.exe
No client can get this right, always. aria2c is not more reliable. It's just choosing to take
the filename from the redirect URL. It appears to be
the right thing to do in this case. But it would fail if the start URL was actually the one that had the right filename.
Hosts can use the Content-Disposition header if they want to make sure all (capable) clients get the right filename.
In saldl, I implemented `--filename-from-redirect` to handle your use-case. But It's not set by default.
- wget to download files (or entire sites even)
- curl to debug everything http/https
As there's no browser interaction in Telegram bot, the script just receives response back from Telegram server. This might help to kerp track of user state without a need of db?
They try wget, fail, and move on.
> Wget supports the Public Suffix List for handling cookie domains, curl does not.
This is outdated info. (lib)curl can be built with libpsl support since 7.46.0.
I'm actually considering using it for a large upcoming project but unfortunately there are some pretty significant bugs in their backlog. wget seems to be a bit more battle hardened.