
Command line tools for web developers - coderholic
http://www.coderholic.com/invaluable-command-line-tools-for-web-developers/
======
bretthopper
Ever wanted to know the progress of a long running operation? Copying a file,
importing a mysql db, etc. Pipe Viewer (pv) is what you need:
<http://www.ivarch.com/programs/pv.shtml>.

It lets you monitor the progress of any piped command and gives you time
elapsed, speed, time left, and a progress bar (wget style).

Copy a file:

    
    
      pv sourcefile > destfile
    

Import mysql db:

    
    
      pv dump.sql| mysql -uroot -p db
    
    

More tricks: <http://blog.urfix.com/9-tricks-pv-pipe-viewer/>

edit: To install on OS X just do

    
    
      brew install pv

~~~
brendano
Pipe Viewer dramatically increased my productivity for large scale data
processing. In particular, it lets you quickly know whether something will
take 5 minutes, or 2 hours, so you can plan accordingly. It's painful watching
people try to do this without pv.

------
NathanKP
For me the most invaluable command line tool that I have ever used is S3
Tools:

<http://s3tools.org/s3tools>

It provides a great suite of tools for interacting with S3, and is best used
on an EC2 instance you are connected to via SSH. It is also ridiculously fast,
much faster than trying to interact with S3 from a local FTP browser, or even
from Amazon's own S3 dashboard. For example on my computer using Amazon's own
web facing dashboard it will take about 30-45 minutes to make 14000 files on
S3 public, versus via the command line tool downloaded and running on one of
my EC2 instances it can make those files public within minutes.

I assume this is because it is local network traffic. Anyway, if you are ever
in a bind and need to move a bunch of files to S3, I highly recommend S3
Tools. It has saved me many times.

Along those lines wget is the most powerful command line tool I've ever used,
with so much capability. It is simply incredible when combined with S3 tools,
allowing you to easily grab gigabytes of images off of a personal server or
staging location and upload them to S3 very quickly.

And if you need to do more than move files around you can manage even more
aspects of AWS including EC2 instances from the command line using this
powerful command line tool:

<http://www.timkay.com/aws/>

Its great for writing automated bash scripts to manage your instances via cron
jobs.

~~~
epi0Bauqu
My main complaint with s3cmd is you can't easily throttle it, and on a busy
machine its fast speeds can cause issues.

~~~
spiffytech
Have you tried trickle[1]? It can limit the bandwidth of most programs I've
tried it on.

    
    
      trickle -u 10 -d 20 ncftp
    

This runs ncftp capped at 10 KB/s up, 20 KB/s down.

[1] <http://monkey.org/~marius/pages/?page=trickle>

~~~
epi0Bauqu
No -- but I will -- thx! I've tried nice, ionice, cpulimit and cstream.

------
Pewpewarrows
I prefer curl'ing icanhazip.com, and I make it a quick alias in my .bashrc:

    
    
        alias myip="curl icanhazip.com"
    

Apachebench (ab) is a decent alternative to siege, and knowing how to use
tcpdump and netcat comes in handy for debugging. Other than that, my new
favorite command line tool over the past several months has to be vagrant,
which lets you script and streamline VM creation and builds from the command
line. If I need to completely reproduce my production environment on a test
box, it's my utility of choice.

~~~
RyanKearney
That only returned my IPv6 address so I can't get my IPv4 address with that.

~~~
madars
Try ipv4.icanhazip.com and ipv6.icanhazip.com .

------
genieyclo
My most used cli tool outside of the default nuts and bolts is dtrx, the best
and easiest file extractor for *nix. No more fiddling with flags or looking
up, handles issues with putting lots of things in different directories or the
wrong permissions on files. It has saved me a ton of time over the years.

<http://brettcsmith.org/2007/dtrx/>

~~~
divtxt
Wow! Just wow! Bookmarked! A mere upvote is not enough for this! Thank you!!!

Questions:

\- Is there a site pointing out amazing command line tools like this? (their
existence as opposed to usage examples like at
<http://www.commandlinefu.com/>)

\- Is there a site listing the OS X equivalents for Linux command line tools?
(e.g. what's the OS X equivalent for dos2unix / flip?)

(edit: merge 2 comments into 1)

~~~
genieyclo
Yep, check out <http://onethingwell.org> which is exactly that. Software (incl
cli) that do one thing well, as according to the UNIX philosophy. It's nicely
tagged as well, so you can click the 'osx' tag and get tools just for OS X and
so forth.

I actually found dtrx on hacker news though a few years ago, but it's also on
onethingwell I believe.

~~~
BasDirks
Amazing, cheers.

------
dexen
lftp is a practical SFTP, FTPS and FTP transfer program, including automatic
upload/download resumption and synchronization (mirror) mode. Good for both
interactive use and scripting.

~~~
nikcub
lftp can also continue downloads and run multiple concurrent download threads,
acting as a download accelerator.

for eg.

    
    
      lftp :~> pget -c -n 10 http://warez.com/large_file.mp4
    

will start 10 concurrent download processes (n) and continue (c). lftp is full
of hidden gems.

------
perlgeek
curl -I and wget -S are particularly helpful when debugging redirects.

Sometimes I migrate URL schemes, and set up permanent redirects in my
.htaccess files. Testing them in a browser is a real pain, because browsers
cache the redirect (which is the point of having a permanent redirect), so
even if you change the .htaccess, you still get the old response. And pressing
the refresh button is no help, because that reloads the destination page, not
the source of the redirect.

That's when a non-caching command line client saves your day.

------
cultureulterior
<http://freshmeat.net/projects/apachetop/>

I find it amazingly useful.

------
flexterra
screen is another tool I use a lot, specially when connecting to remote
servers over ssh.

~~~
reinhardt
screen is one of those things that's been on my todo list for way too long,
along with its alternatives(?), tmux and byobu. Anyone that uses/used all
three and can offer a comparison?

~~~
ez77
Consider the more lightweight dtach and dvtm. Following a well-known
tradition, they each do one thing only, and do it well ;-).

~~~
qwertyboy
dtach is very tempting. Is there a nice way to keep all the scrollback
history?

------
HerraBRE
It's been mentioned on Hacker News a few times before, but my project PageKite
(and showoff.io and localtunnel) is designed to help out with quick web demos
and collaboration.

Getting started is now two lines:

    
    
      curl http://pagekite.net/pk/pagekite-0.4.py >pagekite.py
    
      python pagekite.py 80 SOMENAME.pagekite.me
    

... answer a few questions and whatever is running on port 80 will be visible
as <https://SOMENAME.pagekite.me/> within moments, almost no matter what kind
of network connection you have. :-)

There are also .deb and .rpm packages available for heavier users.

------
nobuff
Oh-my-zsh with git and git flow plugins work well so far.
<https://github.com/robbyrussell/oh-my-zsh>

------
DevX101
Are there any graphical interfaces for load testing, in lieu of siege?

~~~
pbogdan
You might want to have a look at JMeter (<http://jakarta.apache.org/jmeter/>).

------
hasenj
`curl ifconfig.me` is pretty awesome

But I didn't find anything else to be interesting.

~~~
ams6110
ngrep was news to me. I think I've heard of it once before, recently, but
didn't really realize what it was. Having not tried it, it sounds like a
really nice option when something like Wireshark or Ethereal would be overkill
or just too much effort to bother with.

------
tecoholic
Coderholic.com is Timing out, Anyone else having issues?

~~~
brs
Perhaps everyone is trying out siege.

------
ryduh
Note: ngrep can also be installed using brew on OS X.

~~~
LiveTheDream
Also available via MacPorts.

------
marcusramberg
This was worth it for me for the reminder about siege alone, as 'ab' seems to
have some serious issues in Lion.

------
vld
also useful:

wget post-file.it --post-file /path/to/file

------
sfoguy
iMacros firefox addon for web automation via command line (when curl is not
enough):
[http://wiki.imacros.net/iMacros_for_Firefox#Command_Line_Sup...](http://wiki.imacros.net/iMacros_for_Firefox#Command_Line_Support)

------
gluecode
siege is a very useful tool for benchmarking. I use it a lot.

~~~
ez77
How come sending concurrent requests, as explained in the article, doesn't
flag you as a DoS offender?

~~~
mnutt
I was surprised that he ran siege against www.google.co.uk without some kind
of "don't do this" disclaimer to new users. Running it against other people's
websites is pretty poor form.

~~~
rorrr
Google has rate limiting protection from a single IP. If you make too many
requests, they send you to a captcha page.

