This is pretty cool because back in the dot com days I got paid huge amounts of money to do what they did in the first 60 seconds. I started at $1000/day and the phone rang off the hook. Talked to a friend and he said "double your rates" and I said "WTF? I'm not worth $2000/day, that's crazy". He said "double your rates". So I did, phone still rang like crazy. I got up to $4000/day and had 5 days/week work. Went to $8000/day and got about 3-4 more days and then it dried up.
And that is how I made payroll for BitKeeper in the early days.
I think it's cool that this stuff is far more common knowledge these days. All I was doing was figuring out if it was CPU, network, disks, file system, vm system, memory. 99% of the time I knew what the problem was in a few minutes, all the real effort was tracking down what program (or programs) were causing it. Funny enough, most people didn't care what cause was, they just wanted to know what to buy to make it not an issue. Crazy amounts of money being thrown at Sun equipment.
that's a pretty awesome summary. to that i would add:
* `iftop` that allows me to quickly check which network streams are hogging the machine (this is a little like sar -n DEV 1 but much more detailed!)
* `tcpdump -c 100 -vv`- poor man's alternative to iftop or systat if they're not available locally
* tail those logfiles, with systemd it's even easier with journald as all logfiles can be checked at once: `journalctl -xf -p notice | ccze -m ansi` (ccze colors the output, purely optional)
* probably more i am forgetting now
the great takeaways in the article for me are:
* pidstat: great to have an average, beats top for sure
* back to basics: hit dmesg, vmstat and iostat first, as you learned in the beginning! :)
And `glances` is pretty great for big-picture stuff, all-in-one. It's not as precise on CPU/memory as htop, but you get network bandwidth, disk I/O, disk usage and more.
problem with (h)top is that it doesn't catch short-lived processes, and it displays only the most basics stats. For more detailed perf analysis you will likely be better with something like atop
htop is useful, but i find it doesn't add enough on top of top to justify changing my hand-wired reflexes to add that "h". it's often not installed everywhere either, so the top habit remains.
I think Netflix may have just rolled out a blog redesign. (The original link worked for me at first, but now redirect me to media.netflix.com.) Too bad they broke old links though. Hopefully that's temporary.
The real gem for me in that article (particularly since I already knew about those linux commands) was the USE method [1] and then consequently finding the TSA method [2] on the same linked site.
If you want more from Brendan Gregg on analyzing performance, be sure to check out his book, Systems Performance: http://smile.amazon.com/dp/0133390098
Is there any way to analyze memory bandwidth usage?
I've had a couple of programs over the years that I wondered if we were just hitting memory bandwidth limitations but I couldn't find any way to prove that, or even particularly gather evidence.
It will show up like CPU usage - the hyperthread that is waiting for memory will appear busy executing.
perf can record hardware counter events - use "perf list" to see the list - including those that count frontend and backend stalls (stalls in the instruction decoding and execution stages respectively). A high stall ratio / low instructions per cycle throughput is an indicator that you're running into memory bandwidth limitations.
Yes, PMCs (CPU performance monitoring counters; also known by many other terms, such as PMU counters, PICs, CPCs, etc). In the past I've written tools that print usage of memory busses (really, CPU interconnect ports) via the PMCs.
They are also a PITA to work with.
As another person said, it shows up as CPU utilization. So I check that along with IPC (instructions per cycle), and if IPC is low (what "low" is depends, but say, < 1.0), then that's a good clue you're blocked on memory.
... but of course, I want actual throughput (usage), bandwidth (maximum), and utilization (ratio), which is more digging with the PMCs.
I've identified a memory bandwidth issue in the past by keeping an eye on truss/strace output and "counting" mem operations.
After that, I compiled and ran a tiny executable called Stream[1] and got the numbers I needed in order to explain why one machine was twice as slow as another.
Using truss/strace to figure out memory bandwidth issues sounds pretty unreliable. I would not have guessed there was much correlation between memory syscalls that truss/strace can observe (mmap/munmap, brk), and the CPU load/stores that consume memory bandwidth.
> Pidstat is a little like top’s per-process summary, but prints a rolling summary instead of clearing the screen. This can be useful for watching patterns over time, and also recording what you saw (copy-n-paste) into a record of your investigation.
top has a batch mode: "top -b" will print all processes periodically to the terminal, without starting the curses-style UI, providing the same "rolling summary".
"pidstat 1" prints active processes, making it much less noisy that top -b, and therefore highlights malevolent processes. I think the top equivalant would be "top -b -i", though that has more output and is a bit noisier.
> Don’t miss this step! dmesg is always worth checking.
Best advice given in the whole article. Many times I go to check something that "can't figure out what is wrong." dmseg | tail and then the swearing begins on their end.
Can you imagine a world where the Windows System Event Viewer was this useful?
It's amazing how much time one useful error message can save.
Also, I'm a little disappointed that the author didn't drill down into what the actual problem was in his example. That java process is of course suspicious, but that might just be the video server and expected behavior.
dmesg and being able to run most things in a terminal to check what its doing (grumble, skype, grumble) is likely what has drawn many to Linux in the first place.
And a likely source of misgivings regarding recent developments in the ecosystem...
One improvement (which may or may not be related to systemd, but is in newer kernels) is that dmesg has timestamps enabled by default which makes it much easier and `dmesg -H` is really nice as well.
Has been available for years to print out human-readable timestamps. Check to see if dmesg is an alias to that on your system. Also, -IIRC- dmesg has printed system-uptime timestamps with every line since at least the early 2000's.
One caveat: the times printed by dmesg -T will be incorrect if the system has suspended to RAM or disk.
You sound like you doubt the readings. This is netflix, I assume they have some pretty hefty kit backing up their service.
For a massive server the CPU reading might not be unusual. Maybe it has 32+ CPU cores and a multi-threaded java app is spinning most of them.
Also remember that on a heavily loaded system the task that is reading CPU use is itself competeing for time. Timing issues and other monitoring vagueries can make such readings noticably imprecise (though for CPU time usually in the downwards direction by missing tasks that started+worked+ended between readings).
The 233GB of memory seems high but there are possible explanations for this. A server with 32+ cores is not unlikely to have a lot of RAM too, the motherboard in my home server supports up to 128GB so perhaps that large java process genuinely does use more then 200. Also all that memory might not really be in use: it could have been allocated but never accessed so it isn't yet holding pages in RAM or swap for all of it.
The 233GB is VIRT, meaning virtual memory. It need not all be backed by physical memory. For example, if you mmap a file, and then access only a small portion.
The RES column shows how much memory is resident (eg, currently backed by physical memory), and that is a much more reasonable 12GB.
I'm guessing that the java process is their video server. It's probably having to DRM and bandwidth regulate hundreds of HD streams. Unfortunately, this might mean that all of the stuff he talked about in here didn't actually help him solve the problem because none of it had any visibility inside of their Java VM.
I'm wondering if there's a way to automate this kind of analysis to give you condensed, interesting points of a system performance status. À la powertop for power management.
Agreed, considering they've built Vector (mentioned by bgregg in the second sentence - https://github.com/Netflix/vector) on top of Performance Co-Pilot. While PCP doesn't yet have all the wrappers to mimic each sysstat output in a fully compat manner. The underlying mechanisms to remotely fetch that data (using the tools Vector is already built on), is already there.
I had one of my EC2 instances (Ubuntu 14.04) lock up on Monday morning. I use it to run a Ruby on Rails app and to do a handful of ETL jobs. The website wasn't able to load and I wasn't able to SSH into the box. So, I went into the AWS console and rebooted the instance. This has happened 3 times in the last 6 months. It looks like the CPU spikes up to almost 100% around the same time when the instance locks up.
I'd love to be able to identify the root cause of the instance locking up. It seems like this article is more about the commands you'd run to assess the health of an active/working EC2 instance and not one that you're unable to SSH into. Any idea on how to identify the problem with my EC2 instance?
I can recommend atop[1] for that. It runs every ten minutes by default and writes lots of information to /var/log/atop/atop_YYYYMMDD. With that you can examine what happend before the crash, just open a file with atop -r /var/log/atop/atop_YYYYMMDD.
One thing we do at $work is collecting each server's static with https://collectd.org/ and often the graphs are pretty revealing. Sometimes it's the disk activity that kills the server, sometimes it's the memory usage, or number of processes, or a disk running full.
For things like databases, you can gather more specific information as well, for example number of queries and connections, failed transactions and the likes.
>and stolen time (by other guests, or with Xen, the guest's own isolated driver domain).
This isn't necessarily true. If there is any sort of credit scheduler interaction ( http://wiki.xen.org/wiki/Credit_Scheduler ) resulting in the CPU being throttled, it will show as steal. Steal actually just means the CPU was not in a runnable state, which can be caused by multiple things, but predominately throttling by the CPU scheduler.
This is yet another awesome contribution from Netflix.
Just curious though, anyone know of a similar type of writeup or post regarding windows?
I have an MSSQL box I want to kill with fire - but I would like to be able to measure its perf with as much insight as I might on a comparable linux box.
What is the best way to measure disk IO perf on a windows box, specifically?
Drag in the relevant graphs into your current view, and understand how the yellow bar aggregates data. MSSQL probably has event providers - you might want to look those up and add them to wprui when you generate your ETL.
Do you use a application monitoring solution in production? An APM records all such data and you can drill down (starting from pretty graphs down to call stacks and long lasting SQL snips).
And that is how I made payroll for BitKeeper in the early days.
I think it's cool that this stuff is far more common knowledge these days. All I was doing was figuring out if it was CPU, network, disks, file system, vm system, memory. 99% of the time I knew what the problem was in a few minutes, all the real effort was tracking down what program (or programs) were causing it. Funny enough, most people didn't care what cause was, they just wanted to know what to buy to make it not an issue. Crazy amounts of money being thrown at Sun equipment.