

Live patching for Linux 3.20 - meskio
http://lkml.iu.edu/hypermail/linux/kernel/1502.1/00753.html

======
poims
And it was officially merged into Linux this evening:

[https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....](https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=1d9c5d79e6e4385aea6f69c23ba543717434ed70)

~~~
AlyssaRowan
I am very happy about this. It's been years in coming!

We might finally get live updates on distros. (No thanks to Oracle, of
course.)

Wonder if we can also do it that well in userspace? (How does systemd behave
with patching, actually?)

~~~
lemonade
The original KSplice project was actually fully open source, funded by a Dutch
charity called NLnet that funds lots of interesting projects like that. I've
been running it ever since.

After this was all done the team got acquihired by Oracle. I was actually
amazed that the team was allowed to keep the service up for some non-Oracle
distro's.

But very happy to see a broader adoption of this kind of technology, it is
essential for all these unmanned systems out there in the cloud that they can
be patched whilst running.

~~~
desdiv
Any reason why no one forked KSplice when the original team went to Oracle?

~~~
AlyssaRowan
Yes: KSplice had software patents - Oracle bought them. And _everyone_ knows
what Oracle is like with software patents: aggressive!

I'm not clear what they actually cover, and can't look them up right now, but
I'd thought they were specific on how KSplice in particular operates, both
applying hotpatches and analysing the source to create them. I don't know
whether they'd apply to anything else, or whether there is prior art, but
they're an obvious landmine to be aware of and to avoid. So a simple fork
wouldn't do unless it'd change the way it actually worked. A fresh approach
was needed, and we seem to have two fresh approaches here.

I'm trusting they've been avoided here. They probably have, as this is much
more general? The _concept_ of hot patches are of course fine, people have
been doing that for decades, and you can't patent concepts.

The lesson here: please don't patent stuff jn your open-source software, in
case you wake up one day and got acquihired by Evil™.

~~~
AlyssaRowan
OK, I've now looked up the Ksplice patents that I know of. (I may not have
found them all, but I think I probably have?) Here be dragons! (Those who are
ordered not to read patents: Don't click on the links in this post.)

Of course the time they were granted (to Oracle, after Ksplice were bought)
the applications became nigh-impenetrable patentese that _really need_ a US-
qualified patent attorney to interpret, so I'm absolutely not going to try and
I'm just going to post what I found here.

Application:
[https://www.google.co.uk/patents/US20100269105](https://www.google.co.uk/patents/US20100269105)
became patent
[https://www.google.co.uk/patents/US8612951](https://www.google.co.uk/patents/US8612951)
(B2) "Method of determining which computer program functions are changed by an
arbitrary source code modification". (They've also cited a patent for a…
coffeepot. OK, I'm pretty sure that bit's a typo. <g>)

Application:
[https://www.google.co.uk/patents/US20100083224](https://www.google.co.uk/patents/US20100083224)
seems to have become patent
[https://www.google.co.uk/patents/US8261247](https://www.google.co.uk/patents/US8261247)
(B2) "Method of modifying code of a running computer program based on symbol
values discovered from comparison of running code to corresponding object
code".

Application:
[https://www.google.co.uk/patents/US20100269106](https://www.google.co.uk/patents/US20100269106)
does not seem to have been granted directly, but then there's patent
[https://www.google.co.uk/patents/US8607208](https://www.google.co.uk/patents/US8607208)
"System and methods for object code hot updates" which I think is a
continuation-in-part of it and oh I've gone cross-eyed, get a professional.

------
matteotom
Here's some more discussion from LWN:
[https://lwn.net/Articles/597407/](https://lwn.net/Articles/597407/)

(Note: from May 2014)

------
SEJeff
And for those who prefer the official lkml.org link:

[https://lkml.org/lkml/2015/2/9/534](https://lkml.org/lkml/2015/2/9/534)

~~~
sigjuice
lkml.org explicitly states that they are unofficial.

From [https://lkml.org](https://lkml.org) , "In case you haven't read the
titlebar of your web browser's window: this site is the (unofficial) Linux
Kernel Mailing List archive."

~~~
agumonkey
Firefox Nightly doesn't display that neither in the tab name nor the Xwindow
title. I'd never have seen this ...

~~~
vlad003
That's because the <title> element only contains "LKML: " on the thread, and
"LKML.ORG - the Linux Kernel Mailing List Archive" on the main page.

Maybe they forgot to keep the "unofficial" in the title?

------
bjackman
This is cool as fuck, I didn't know about kpatch or kGraft.

Does anyone know if any other OSs have live kernel patching?

~~~
feld
FreeBSD has not gone quite that far, but there was a PoC for loading new
kernels without rebooting.

[https://www.bsdcan.org/2012/schedule/events/325.en.html](https://www.bsdcan.org/2012/schedule/events/325.en.html)

~~~
SEJeff
Yup Linux has had a similar feature, kexec, for several years.

~~~
feld
Everyone I talked to about it since has said "nobody seems to care enough to
bother completing it"

It is kind of a niche feature, really.

~~~
SEJeff
Linux has this create feature of being able to include the operating system in
the initrd. Put this with a nice PXE infrastructure where you pxe the OS and
download the initrd with the OS in it. Then you simply kexec to upgrade or
downgrade the entire operating system in ~30 seconds. I used to manage a
production environment that worked exactly this way for several thousand
nodes.

Not sure I'd call it niche, but yes, somewhat specialized.

------
resc1440
And lkml discussion of the follow-up patches:
[http://lkml.iu.edu/hypermail/linux/kernel/1502.1/00694.html](http://lkml.iu.edu/hypermail/linux/kernel/1502.1/00694.html)

------
mkonecny
Does anyone have a simple example of how this would work? I can't wrap my head
around code evolving during runtime for any arbitrary binary change.

~~~
DSMan195276
I took a quick look at the accepted patch. while I can't guarantee I know
what's actually going on, my understanding is that patching individual
functions works by sticking the replacement functions code somewhere new in
memory, getting a pointer to it, and then over-writing the code in the old
function to jump to the new one. (Kinda like short-circuiting the old function
- all the old code still calls the old function location, but that location
simply says 'jump to this new location over here').

It looks like, however, because kernel modules seem to be in elf format (Don't
quote me on that, just going from the code), elf format includes a 'relocation
table', which is basically a table that says "this function is located here,
and this next function is located here, and ..." for every function in the
module. Ignoring why that is actually there, they can take advantage of the
relocation table and replace a functions location with the location of the
replacement function, effectively overwriting the old one. Even if it's still
in memory (I can't tell if it gets removed or not) the code will never be
called again.

From there, the discussion mostly seems to be around how to 'stop' the kernel
enough to be able to replace the function without resulting in a mess because
something was trying to use that function at the same time that you replaced
it.

~~~
donavanm
Basically yes. Check out the kprobes docs for a nice description of how these
frameworks work
,[https://www.kernel.org/doc/Documentation/kprobes.txt](https://www.kernel.org/doc/Documentation/kprobes.txt).
Being able to intercept (and mangle) kernel function calls is awesome. With
uprobes the same techniques work in userland as well.

------
peterwwillis
An interesting case of the Cathedral causing siloed duplication, and the
Bazaar creating a better solution without duplication.

~~~
rockdoe
Ignoring the fact that both SuSE and Red Hat are paying the involved engineers
their paychecks in order to get the feature in their silo first.

