
The ~200 Line Linux Kernel Patch That Does Wonders - pietrofmaggi
http://www.phoronix.com/scan.php?page=article&item=linux_2637_video
======
jws
I don't think I've managed to piece all this together, perhaps someone can
fill in the blanks.

• The patch automatically creates a task group for each TTY.

• The patch automatically assigns each new process to the task group for its
controlling TTY.

• In the case where there are large (>cores) numbers of cpu bound jobs, the
latency of interactive jobs is vastly improved.

I think the piece I'm missing is the behavior of the scheduler. Does it now
make its decisions based on task group cpu consumption instead of process? I
saw options to that effect back around 2.6.25.

Why is this an improvement over just nicing the "make -j64" into the basement
and letting the interactive jobs have their way as needed? (Likely
possibilities are that it is automatic, or maybe there is something about disk
IO scheduling happening from the task groups as well.)

~~~
tbrownaw
> I think the piece I'm missing is the behavior of the scheduler. Does it now
> make its decisions based on task group cpu consumption instead of process? I
> saw options to that effect back around 2.6.25.

Yes, and yes. Previously you could set things like this up explicitly, this
makes it (optionally) automatic.

> Why is this an improvement over just nicing the "make -j64" into the
> basement

That gives you... I think 10% less CPU weight per level, so you can get down
to 13% of base. So your 64 processes will still weigh about 8x one base
process.

This lets you consider everything spawned from your terminal as one group, and
everything from your X session as another, so your compile processes
collectively (no matter how many you have) weigh as much as your GUI processes
collectively weigh.

~~~
nitrogen
So does "tty" or "terminal" in this context also refer to pseudoterminals, so
that each xterm/konsole/gnome-terminal instance gets its own scheduling group
as well?

~~~
tbrownaw
I would be very surprised if it didn't, this should be the same as what shows
under the "TTY" column from "ps -f".

------
bobf
I don't follow kernel development extremely closely, but it fascinates me that
people are _still_ actively working on the kernel's scheduler and achieving a
"huge improvement" like this.

~~~
maximilian
I remember reading something about an anesthesiologist who got into hacking
the scheduler targeting desktop use cases. He generally said that the
scheduler gets a lot more attention from people who care more about server
work loads and such. He had his own custom kernel patches that people used to
get directly from him that weren't in the mainstream kernel -- This was before
the era of multi-core, but people said his scheduler had better responsiveness
than the default one.

~~~
slewis
Con Kolivas. He's also mentioned in the article.

<http://en.wikipedia.org/wiki/Con_Kolivas>

------
ludwigvan
It makes me wonder whether it is a sign that desktop responsiveness has been
neglected by the kernel devs which possibly prioritize server issues. I had
read a Google engineer suggesting Canonical should hire decent kernel
developers : " P.S. Next thing for Ubuntu to learn --- how to pay their
engineers well enough, and how to give them enough time to work on upstream
issues, that once they gain that experience on Ubuntu's dime and become well
known in the open source community, they don't end jumping ship to companies
like Red Hat or Google. :-)"

<http://news.ycombinator.com/item?id=1321029>

~~~
krakensden
Canonical has some decent kernel developers, just... not enough, especially
for their install base and the amount of work they do.

~~~
reinhardt
At least they're aware of it; just yesterday they posted this:
<http://webapps.ubuntu.com/employment/canonical_KD%20PG3/>

------
silentbicycle
Here's the actual patch ([http://marc.info/?l=linux-
kernel&m=128978361700898&w...](http://marc.info/?l=linux-
kernel&m=128978361700898&w=2)), with a bit of a summary of what it does.

I don't have enough context to fully follow it, but it sounds like it sets up
a better hueristic for grouping related processes into task groups in the
scheduler.

------
gxti
So what's the downside? You almost never get optimizations like this for free.
The post hints that this is also good for server workloads, but what suffers?
Realtime would, but realtime usually involves a different scheduler anyway.

~~~
leif
There isn't one. It's an existing option in the kernel, you can configure
cgroups that way already, but most people don't do this so the feature is
wasted. All this patch does is roughly approximate a decent-looking cgroup
configuration by splitting processes by tty automatically.

~~~
astrange
Of course there will be a regression if you change the scheduler policy. ck
tried something similar to this and mplayer performance suffered with it
(though I don't remember the details). It also broke gnome-startup because it
assumed some specific schedule ordering, though this patch is more limited so
it might not.

~~~
leif
When your application breaks because of _scheduler ordering_ , You're Doing It
Wrong.

~~~
astrange
Yeah, well, kernels need to support existing programs…

By the way, your post is the single most obvious statement I've read this
year. You got upvoted just because you capitalized some words?

~~~
leif
I got upvoted because I'm right. Kernels don't need to support horribly
designed programs just because they exist, just like they don't need to
support horribly designed programs that don't exist yet. Kernels support an
interface and that's it. If you write code that abuses the interface, get
ready to become a regression, and that'll be your own fault.

(TBH I have no idea why I got upvoted, it wasn't that insightful, but I stick
by what I said)

(EDIT: I'm talking about gnome-startup. That's a stupid regression that never
should've happened. The mplayer performance bug is totally understandable if
you're mucking with the scheduler. What we really need is for someone
(distros?) to pick up cgroups and provide a nice UI for it, some sane but
nondestructive defaults, etc. Until then, this is a nice patch that keeps
badly behaving programs from dragging down the entire system. At the very
least, we mostly get user separation in multi-user environments.)

------
pietrofmaggi
It's not down for me... but here's the text:

In recent weeks and months there has been quite a bit of work towards
improving the responsiveness of the Linux desktop with some very significant
milestones building up recently and new patches continuing to come. This work
is greatly improving the experience of the Linux desktop when the computer is
withstanding a great deal of CPU load and memory strain. Fortunately, the
exciting improvements are far from over. There is a new patch that has not yet
been merged but has undergone a few revisions over the past several weeks and
it is quite small -- just over 200 lines of code -- but it does wonders for
the Linux desktop.

The patch being talked about is designed to automatically create task groups
per TTY in an effort to improve the desktop interactivity under system strain.
Mike Galbraith wrote the patch, which is currently in its third version in
recent weeks, after Linus Torvalds inspired this idea. In its third form
(patch), this patch only adds 224 lines of code to the kernel's scheduler
while stripping away nine lines of code, thus only 233 lines of code are in
play.

Tests done by Mike show the maximum latency dropping by over ten times and the
average latency of the desktop by about 60 times. Linus Torvalds has already
heavily praised (in an email) this miracle patch.

Yeah. And I have to say that I'm (very happily) surprised by just how small
that patch really ends up being, and how it's not intrusive or ugly either.

I'm also very happy with just what it does to interactive performance.
Admittedly, my "testcase" is really trivial (reading email in a web-browser,
scrolling around a bit, while doing a "make -j64" on the kernel at the same
time), but it's a test-case that is very relevant for me. And it is a _huge_
improvement.

It's an improvement for things like smooth scrolling around, but what I found
more interesting was how it seems to really make web pages load a lot faster.
Maybe it shouldn't have been surprising, but I always associated that with
network performance. But there's clearly enough of a CPU load when loading a
new web page that if you have a load average of 50+ at the same time, you
_will_ be starved for CPU in the loading process, and probably won't get all
the http requests out quickly enough.

So I think this is firmly one of those "real improvement" patches. Good job.
Group scheduling goes from "useful for some specific server loads" to "that's
a killer feature".

Linus

Initially a Phoronix reader tipped us off this morning of this latest patch.
"Please check this out, my desktop will never be the same again, it makes a
_lot_ of difference for desktop usage (all things smooth, scrolling etc.)...It
feels as good as Con Kolivas's patches."

Not only is this patch producing great results for Linus, Andre Goddard (the
Phoronix reader reporting the latest version), and other early testers, but we
are finding this patch to be a miracle too. While in the midst of some major
OpenBenchmarking.org "Iveland" development work, I took a few minutes to
record two videos that demonstrate the benefits solely of the "sched:
automated per tty task groups" patch. The results are very dramatic. UPDATE:
There's also now a lot more positive feedback pouring in on this patch within
our forums with more users now trying it out.

This patch has been working out extremely great on all of the test systems I
tried it out on so far from quad-core AMD Phenom CPUs systems to Intel Atom
netbooks. For the two videos I recorded them off a system running Ubuntu 10.10
(x86_64) with an Intel Core i7 970 "Gulftown" processor that boasts six
physical cores plus Hyper Threading to provide the Linux operating system with
twelve total threads.

The Linux kernel was built from source using the Linus 2.6 Git tree as of 15
November, which is nearing a Linux 2.6.37-rc2 state. The only change made from
the latest Linux kernel Git code was applying Mike Galbraith's scheduler
patch. This patch allows the automated per TTY task grouping to be done
dynamically on the kernel in real-time by writing either 0 or 1 to
/proc/sys/kernel/sched_autogroup_enabled or passing "noautogroup" as a
parameter when booting the kernel. Changing the sched_autogroup_enabled value
was the only system difference between the two video recordings.

Both videos show the Core i7 970 system running the GNOME desktop while
playing back the Ogg 1080p version of the open Big Buck Bunny movie, glxgears,
two Mozilla Firefox browser windows open to Phoronix and the Phoronix Test
Suite web-sites, two terminal windows open, the GNOME System Monitor, and the
Nautilus file manager. These videos just show how these different applications
respond under the load exhibited by compiling the latest Linux kernel using
make -j64 so that there are 64 parallel make jobs that are completely
utilizing the Intel processor.

~~~
riffraff
OT: is "make -j64" overkill unless you have dozens of cores or am I missing
something?

~~~
fragmede
You're right - but that was the point. The patch was trying to fix problems
with the process scheduler, and "-j64" is going to make lots of processes that
want to do work and need scheduling.

~~~
riffraff
thanks, but then the "that is my tipical workload" thingy does not hold, as
you rarely have >60 cpu bound processes running at the same time. Well, flash
player in chrome notwithstanding ;)

~~~
SkyMarshal
It probably approximates Linus's typical workload, which I imagine involves
constant compiling and testing while compiling. He's probably still CPU bound.

~~~
juiceandjuice
make?

------
hippich
I am newbie when it comes to compiling kernel. Is it a pain to do with stock
ubuntu 10.10?

Sometimes I run something heavy on my laptop and desktop freezes annoy me. If
this patch will allow me to get around it - I would be glad to try it out.

Anyone having url of some niuce tutorial to compile new kernel for ubuntu
10.10?

~~~
krschultz
It isn't hard to do, but a painless way to learn is to install a ubuntu 10.10
instance in a virtualbox and try it all in the box. If you screw up, who
cares. After you have been through the process once it won't be intimidating
to do it for your real OS.

~~~
eru
Yes, it's indeed quite easy to compile your own kernel, and virtualbox is a
good idea. (For compiling you won't actually need the virtualbox, but for
booting from it without worries that you broke something, virtualbox sure
comes in handy.)

I use "sudo make menuconfig", which gives you a text-based menu, there may
also be a graphical version. I would not recommend "sudo make config", as that
only gives you a long list of questions to answer.

Anyway, the trick is to read all the documention, and stick with safe choices,
if you do not know what you are doing.

~~~
sp332

      # sudo make xconfig
    

should give you a point-and-click interface to the same menu. I haven't used
it in years, but it was pretty clunky back then. It's just nicer to poke
around in than the text-mode menu.

~~~
jrockway
You don't need to be root to compile a Tk application and run it! You also
don't need to be root to compile the kernel.

You only need root to copy vmlinux to /boot and copy the modules to /lib.

------
cookiecaper
Anything from Phoronix should be taken with a grain of salt. This looks legit
since it has a message from Linus praising the patch, but there have been
several similar stories out of Phoronix that turn out to be hoaxes or
misunderstandings.

That said, such a patch would be pretty rad.

------
zemanel
i whish something similar could be ported to BSD/Darwin, OSX. I have a MBP 6,2
(i5) with 4GB mem/5400 rpm disk and it's quite easy to hog it down, to almost
unbearable sometimes.

~~~
rufo
What sort of tasks?

FWIW, I had a similar configuration to yours (just an older MBP) and
installing an SSD helped _immensely_. I can hit 200% CPU load and not even
realize it until the fans kick in…

~~~
zemanel
I know the subject is pretty much Apple's and oranges but i'm currently
running: \- Terminal with 2 tabs \- firefox with 2 tabs \- chrome with about 9
\- gaim and skype \- iTunes streaming soma.fm \- Netbeans with an opened
project \- jEdit \- Colloquy \- Postgres instance

and as soon as i booted Windows XP in VMware, well, took me a while to be able
to reply to this post (after the vm settled).

I also that you might be saying "d'oh" but i've had Gentoo running on this
metal and a "similar" environment AND compiling stuff with -j4 doesn't freeze
away my UI.

My user experience with "OSX" is that it's, way more prone to unresponsiveness
due to load, but hey, who cares :P _clicks Time Machine_

~~~
rufo
Your typical workload sounds almost exactly the same to mine - right down to
VMware temporarily killing OS X performance.

Not that Anonymous Guy On HN is worth much, but get an SSD - it'll be the best
upgrade you've ever purchased.

(Or maybe I just needed to sidegrade to Linux. :)

~~~
gte910h
Do you have a suggestion on a good and large SSD? And what did you install it
into?

~~~
rufo
The Intel SSDs are good; they top out at 160GB and they're not as fast as some
of the newer drives - but their track record is nearly flawless. It's what I
have, but I ordered one the minute it was posted on Newegg last year. Some of
the newer Sandforce-based drives are supposed to be good, though you'll want
to pick one from a reliable manufacturer and with a stable firmware. If you
have a Mac, OWC[1] is a good choice, as I believe they have firmware that
garbage-collects HFS+, which helps to keep the SSD as fast as possible. I also
think OCZ is good, but check reviews to make sure people aren't having too
many problems.

You can also get a brackets that replaces your optical drive, and allows you
to fit a 2.5" HDD. I have one in my 17" non-unibody MBP, and it's really the
best of both worlds - I keep OS X, my working files and apps on the SSD, along
with my main Windows XP web testing VM. My iTunes library, media, and games
stay on the HDD, along with a Boot Camped copy of Windows 7 (though it's a
pain to get the installer to run without an internal optical drive). I keep a
cheap Samsung bus-powered DVD burner in my bag, but in reality I rarely need
it. I think OWC sells a bracket for Macs, but if you can figure out exactly
which bracket you need, a site called newmodeus.com sells them for almost
every laptop ever made for considerably less.

I really do believe my SSD is the best upgrade I've ever spent money on; no
computer I use from now on will be without one. It's not so much that the
computer is faster; it's more the feeling that the computer does not grind to
a halt or slow down, no matter what's going on. (I may have compared my
computer to the Terminator amongst friends once or twice… it just doesn't slow
down.)

1: macsales.com

------
BoppreH
9 upvotes and it's already down. Anybody has a mirror? This seems pretty
useful.

~~~
sandGorgon
anybody know a ubuntu ppa for this ?

~~~
SkyMarshal
Do they put kernel patches in PPAs?

~~~
ra
No, but you can make a deb of a patched kernel and compile that for PPA
distribution.

It wouldn't be very difficult to make, I would expect to see one in the next
24 hours or so.

Be wary of getting your kernel from a PPA though - consider it experimental.

~~~
SkyMarshal
Thanks, I've only used PPAs for apps and such, didn't realize people put
kernels up too. Though like you say, probably would only use that for a
virtual machine instance, will wait till the next kernel patch for my base
workstation.

------
teoruiz
Just for the brave ou there: to build a custom kernel for Ubuntu, based on the
actual Ubuntu kernel image, you have to follow these instructions:

<https://help.ubuntu.com/community/Kernel/Compile>

Apply the patch before compiling and there you go.

------
yason
Makes me wonder that don't they have kernel APIs for process schedulers and
I/O schedulers by now? The scheduler tweaks have been going on for ages.

Instead of compiling a single new kernel module (or downloading it prebuilt
from an apt repo or a PPA) and kicking it in with modprobe, we now need to
obtain the sources for the whole kernel, apply the patches, configure, build,
and deploy. Sure Debian/Ubuntu has that partially automatized but it's still a
pain.

At least I'll wait for stock 2.6.38 on Ubuntu and cross my fingers they put
this patch in.

~~~
rcoder
Pluggable schedulers have been proposed, implemented, and shot down by Linus
several times in the past. IANAKH _, but Linus's argument seems to basically
boil down to this: for a monolithic kernel, delegating something as central as
task scheduling to pluggable modules is a pretty big hit in terms of latency
and complexity vs. just putting the best, most tightly-tuned scheduler you can
smack dab in the heart of the beast.

_ : I Am Not A Kernel Hacker

------
sliverstorm
This is funny and cool at the same time- back when I still used Linux
regularly, it was because it was way smoother than windows under load. I don't
know if it regressed since then and was fixed, or just got better, but either
is awesome!

------
slowpoison
So, is this really going to help, if I don't have tons of busy processes (a'la
"make -j64") running?

~~~
nitrogen
It can help prevent background processes (like disk indexing or manpage index
updating) from interfering with interactive processes.

------
slaxor
i am impressed, i have never expected such dramatic improvement on my desktop

