A few spy tools for your operating system (other than strace)

[Audiophilip]

http://jvns.ca/blog/2015/04/06/a-few-spy-tools-for-your-operating-system-other-than-strace/

[taliesinb]

That's indeed a useful list. lsof alone covers a lot of ground!

Mac OS X has a pretty nice set of DTrace scripts built in: http://dtrace.org/blogs/brendan/2011/10/10/top-10-dtrace-scr...

The ones I use the most:

* iosnoop: see all disk I/O. especially useful to find disk-chatty/poll-y apps.

* execsnoop: see new processes being spawned.

* opensnoop: see file opens. especially useful for failed file opens that break an app.

* dtruss: see all system calls. get access to the entire OS interaction history of an process (or app).

* errinfo: trace failing system calls. where there is smoke...

* iotop -- who is using disk

There is just a crazy, crazy list of things available, built-in:

  man -k dtrace

Edit: wow HN formatting sucks so bad. I wish I could make that list more readable, but apparently I can't.

[WaxProlix → taliesinb]

You'll take your low-overhead-and-proud design and you'll like it. (that said, your post is very readable)

[13 → taliesinb]

Have a Unicode bullet point, on the house.

•

[fafner → taliesinb]

Those scripts and more for Linux (3.2 or newer) using perf_events/ftrace:

https://github.com/brendangregg/perf-tools

[sigjuice → taliesinb]

IMHO, dtruss output is rather useless because hardly any of the arguments are human-readable.

[taliesinb → sigjuice]

That's true, but for getting a timeline of what happened, it's just the right tool.

[vacri → taliesinb]

    - Four spaces at the start of a line makes 'preformat',
      useful for code or lists. 
      - You'll want to manually wrap long lines, though.

[taliesinb → vacri]

Two spaces are actually enough.

But I don't want my descriptions to look like code!

Ideal for this example would be a table where the first column is formatted as code, and the second column as text.

[_ytji]

Article mentions lsof but doesn't go too deeply into it. lsof has an absolute _plethora_ of options and has been pretty indispensable to me. i.e. lsof -i (much akin to netstat), -u <user> (all files open by a particular user), etc etc. Check the man page :)

[oimaz → _ytji]

I often use `lsof -p <pid>` to find the log file location of a given process :)

[bizarref00l → oimaz]

lsof -c <command name> eg: httpd

[mpercy]

If you haven't seen Brendan Gregg's Linux performance tools page & slides, you should check it out. He explores a wide variety of great stuff, especially perf and ftrace (as mentioned in the article) as well as a few dozen others that he describes in lesser detail: http://www.brendangregg.com/linuxperf.html

[realusername]

I will just mention Sysdig for the ones who don't know about it already (http://www.sysdig.org/). It's really the swiss knife of monitoring tools, you can have any kind of information you can think of. If you have not already installed it, give it a try !

[oimaz]

can't believe no body talks about ss (socket statistics) - http://www.cyberciti.biz/tips/linux-investigate-sockets-netw.... It crazy fast compared to netstat

[glandium → oimaz]

And contains more information, too. Before ss, it was close to impossible to know what the other end of a unix socket was for a given connection (except wild guesses from the ids in /proc/pid/fd).

[kazinator]

pmap: dump the memory mappings for a process, such as shared libs.

    $ pmap -x $$
    13609:   -bash
    Address   Kbytes     RSS   Dirty Mode   Mapping
    00110000       0      20       0 r-x--  libnss_compat-2.15.so
    00117000       0       0       0 r----  libnss_compat-2.15.so
    00118000       0       0       0 rw---  libnss_compat-2.15.so
    00119000       0      20       0 r-x--  libnsl-2.15.so
    [ ... snip ... ]

[rikkus]

vmstat [1] is also a very useful command. It shows various essential statistics in a very compact one line display and optionally repeats this every [interval]. I often start something running while vmstat is giving me stats in another window every few seconds.

[1] https://www.linode.com/docs/uptime/monitoring/use-vmstat-to-...

[rpcope1]

Julia mentions atop (which is good), I think htop is also worth noting here as an improvement on top.

[rodgerd]

One minor correction: atop doesn't need to run as root, it does disable a couple of functions if it doesn't.

[atombender]

pidstat! It's like top, except it prints a continuous log of everything that used CPU (or I/O with -d, etc.) since the last sample. It's the only too, I know that can show per-process I/O activity. Requires root access, unlike top.

[franole]

Nethogs [1] groups bandwidth usage by process. Very handy. [1] http://nethogs.sourceforge.net/

[notfoss]

For viewing active network connections along with traffic stats, give iftop a try.

[feld]

procstat, systat, sockstat, gstat, dtrace, truss on FreeBSD

I know there's more I'm not thinking of...

[galapago]

Don't forget powertop