

Performance Tuning Linux Instances on EC2 - r4um
http://www.brendangregg.com/blog/2015-03-03/performance-tuning-linux-instances-on-ec2.html

======
simula67
Brendan Gregg seems to have worked extensively on extracting maximum
performance on all major UNIX kernels ( FreeBSD, Linux, Solaris ) out in the
real world. If he writes an essay around performance of these kernels, I feel
it will be very close to the true state of the affairs.

------
falcolas
Ironically, a swappiness of 0 may be detrimental to the health of software
running on a VM, as it severely limits what may be swapped out of memory. This
means that the OOM killer ends up being more aggressive against the processes
on your system.

A better value is one of "2" \- this allows the Kernel to swap out data in
response to memory pressure, without being overly aggressive about it.

~~~
jdub
EC2 instances are typically configured without swap at all.

------
mortenlarsen
Slideshare is very annoying, when you are trying to get the PDF.

First it tells you that you must sign-up sign-in with Linked-In or Facebook.
Then after finding a non Linked-In or Facebook sign-up, I need to enter my
phone-number to get a link as SMS.

 _SIGH_

------
deathanatos
Note: there is some justification in the slides; the article itself just lists
the "recommendations" outright.

(from the article)

> net.ipv4.tcp_tw_reuse = 1

(from the man page)

> Allow to reuse TIME_WAIT sockets for new connections when it is safe from
> protocol viewpoint. It should not be changed without advice/request of
> technical experts.

Why? Are sockets in TIME_WAIT a problem somehow?

> net.ipv4.ip_local_port_range = 10240 65535

Again, why? My understanding is that this controls the range of ports that the
kernel selects from for new sockets; e.g., if you make a TCP connection to
google.com on port 443, on _your side_ the connection is <your ip> : <a port
from that range>; the default range is [32768, 61000], and this is per
destination IP. (You can have two connections to two separate IPs with the
same local port.) The default range is nearly 30k ports wide. Are you opening
>30k connections to a single host?

> In the talk I described these tunables as our medicine cabinet, and to
> "consider these best before 2015".

Does that not mean that these are expired now? (This article was written
today, though?)

~~~
brendangregg
The problem is opening >30k connections to a single host during a TIME_WAIT
period, 60 seconds. 500 connections per second. For backend servers, eg, an
application server talking to a database, 500 connections per second is easy
(although it's preferable if they can keep-alive).

I liked Vincent Bernat's post about TIME_WAIT:
[http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-
li...](http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html)

~~~
bboreham
Could we say it a bit stronger than that? Opening and closing connections from
the same source to the same service is wasteful since you need multiple extra
round-trips and it resets all the TCP dynamic tuning like CWND.

I thought your talk was great; one minor niggle: you said that the result of
too many sockets in TIME_WAIT would be dropped packets; it should refuse to
open the new connection if no slots are available.

~~~
brendangregg
Yes, it's really wasteful to have unnecessary connect()/accept() calls, plus
handshaking and buffer allocation, and TCP dynamic tuning, etc.

And you're right, thanks, TIME_WAIT full should just error on the Linux
client. I was thinking of a different kernel which has bugs in this area, and
ends up dropping SYNs...

------
hassy
I don't seem to see a mention of the file descriptor limit. As recently as
last year the default on Ubuntu was 2048 iirc, which is laughable given that
it was 2014.

I guess it's not strictly performance related, but should definitely be one of
the first parameters to tune.

    
    
        # in in /etc/sysctl.conf:
        fs.file-max = 100000
    
        # then:
        sudo sysctl -p
    
        # in /etc/security/limits.conf
    
        * soft nofile 100000
        * hard nofile 100000
    
        # then:
        ulimit -n 100000
    

(tweak the exact number as required)

------
jorlow
I saw Brendan's talk live at re:Invent. Highly recommended.

