
Regression in Linux 4.14 – using bcache can destroy the file system - panny
https://bugs.gentoo.org/638206
======
Jach
I really love Gentoo's mask system. I upgraded to 4.12.12 not too long ago, it
is the latest stable: [https://packages.gentoo.org/packages/sys-kernel/gentoo-
sourc...](https://packages.gentoo.org/packages/sys-kernel/gentoo-sources) You
have to explicitly decide to use masked versions, at your own peril. (Of
course, many Gentoo users love the bleeding edge, and I too have a number of
things I manually unmasked... But the kernel? Stable only please.)

~~~
craftyguy
You will probably want to stick on a LTS kernel... unless you like unpatched
kernel vulnerabilities.

~~~
Jach
LTS kernels don't magically patch themselves, so if you can patch, why not be
on the latest patched stable? If you're really concerned about kernel security
you're already using grsecurity/PaX patches (which have neutered many
otherwise 0-days) or OpenBSD anyway.

~~~
bandrami
Well, I mean, because of stuff like this, right?

It's always a question of "pick your poison": Debian stable users actually
missed out on heartbleed completely because Lenny's OpenSSL was too old to
have it.

~~~
Jach
Gentoo didn't mark 4.14 as stable (it has its own procedures for stability
despite the kernel folks treating 14 as the new LTS following 9), so only
users who immediately upgraded to it (manually specifying they wanted it)
would be affected by this.

I really think gentoo has the best packaging story of Linux distros, except
perhaps 'newer' stuff like nixos/guix. Rolling releases (my gentoo box has a
continuity since 2009 with nothing ever resembling a dist-upgrade of other
distros that you're forced to do to get the latest and greatest), generally
gets the latest versions of stuff and their patches first or among the first
if you want to live on the bleeding edge, it's easy to find overlay packages
for things not in the main tree, and even create your own 'packages' since
everything is source-based, USE flags for customization, and anyway it has a
good stable-only option too.

~~~
vthriller
And unlike in other rolling release distros (or at least unlike in ArchLinux;
[citation needed]), emerge is smart enough to keep older versions of .so files
so you're less likely to get unusable system/programs due to partial upgrades:
[0]. I really cannot use Arch on personal computers simply because sometimes I
need to keep various packages at older versions for various reasons (bugs,
coredumps still in TODO, sensitive processes still running, whatever), so I
inevitably end up choosing between bar+libfoo.so.X, baz+libfoo.so.Y, and
rebuilding everything from scratch. (Edit: and I also cannot install older
version of baz 'cause it's already gone from the mirrors.)

[0] [https://wiki.gentoo.org/wiki/Preserved-
rebuild](https://wiki.gentoo.org/wiki/Preserved-rebuild)

~~~
Jach
Yeah. Plus things like quickpkg (make a backup of your older installed package
whose version may no longer be in the tree), demerge (record states of
installed packages and configs over time), slots (conflicting pkg versions
like gtk2 and gtk3 being installed at the same time) all make for more
niceness.

It's definitely possible to go overboard and get into trouble. But there's a
lot to help. And with gentoo I was able to weather the gnome3/systemd madness
and keep that stuff off my system despite having to keep around old gnome2 and
udev packages until mate, eudev, etc. were ready. I don't know too much about
Arch except the pain stories friends have relayed as they eventually gave up
and switched to something else. Back when Arch decided to make Python 3 the
default python well before it was ready someone on IRC quipped "<dash> well
that confirms my impression that arch was invented by a bunch of guys who
thought gentoo was too stable and easy to use".

~~~
bandrami
I haven't used Gentoo in about 10 years, but maybe I should give it a look
again. I remember seeing it allows musl as a system C library, which is cool.
I think mostly I just prefer the pace of Slackware though.

------
jnwatson
In case anyone is wondering, Bcache allows one to use an SSD as a read/write
cache (in writeback mode) or read cache (writethrough or writearound) for
another blockdevice (generally a rotating HDD or array).

~~~
SteveNuts
Is it possible to use bcache in front of an NFS volume?

~~~
khc
No but you can already use FS-cache?

~~~
ptman
fs-cache / cachefilesd seems to still be buggy. was it even deprecated?

~~~
khc
Not that I know of. You can also try
[https://github.com/kahing/catfs](https://github.com/kahing/catfs) which I
wrote recently but I don't promise it's not buggy :-)

------
pcunite
So, and single line of code change? This is why the language choice is not the
biggest issue here. Keeping all the necessary state in a programmer's head is
the real issue. Using <insert fav lang>Rust</> would not have helped.

 _bio- >bi_partno = bio_src->bi_partno;_

[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=62530ed8b1d)

~~~
khuey
A language with copy constructors could have prevented this.

~~~
bonzini
A language with copy constructors doesn't prevent you from putting a bug in
them.

C++ or Rust might have put a 0 in the uninitialized member, but the bug would
have remained.

~~~
masklinn
> A language with copy constructors doesn't prevent you from putting a bug in
> them.

If you don't override it, it will copy everything properly.

> Rust might have put a 0 in the uninitialized member

Rust does not do that. If a member is not initialised, it's uninitialised and
in this case the structure will be rejected. Here even if you could not just
have derived Clone (struct bio looks non-trivial) your hand-rolled
implementation _would_ have complained that you had not properly initialised
the member.

~~~
bonzini
> If you don't override it, it will copy everything properly.

Sure, so does memcpy or bare struct assignment in C. The problem is that
trivial copy constructors are relatively rare.

> Here even if you could not just have derived Clone (struct bio looks non-
> trivial) your hand-rolled implementation would have complained that you had
> not properly initialised the member.

Right, but this is more similar to an assignment operator. DerefMut in Rust I
think wouldn't have caught the bug.

~~~
masklinn
> Right, but this is more similar to an assignment operator.

Only if you don't zoom out. If you do, this is alloc/init/fill which in C++ or
Rust would use a regular copy and RVO.

------
xwvvvvwx
Curious if anyone knows how the kernel is tested?

~~~
simcop2387
As best as they can, and by a lot of people running release candidates on test
systems. Unfortunately there are basically an infinite (not really, just
incredibly impossibly large) number of configurations of software and hardware
that can lead to a bug like this not triggering so it's not always possible to
catch this ahead of time.

~~~
xwvvvvwx
How often does a bug of this severity slip through the net?

~~~
katastic
The only one I can find is in 2012 there was a RAID bug that damaged metadata
required for RAID access, under specific/rare conditions.

[http://www.h-online.com/open/news/item/Bug-in-Linux-
kernel-c...](http://www.h-online.com/open/news/item/Bug-in-Linux-kernel-can-
damage-RAID-arrays-1621164.html)

But if anyone is running critical data, they shouldn't be using a bleeding-
edge kernel in the first damn place. It certainly "sucks" for the users that
found it, but again, that's the risk you play.

And, this bug seems to have been found... a mere week after articles even
reported the release of Kernel 4.14:

[https://bugs.gentoo.org/638206](https://bugs.gentoo.org/638206)

[http://www.omgubuntu.co.uk/2017/11/linux-kernel-4-14-lts-
fea...](http://www.omgubuntu.co.uk/2017/11/linux-kernel-4-14-lts-features)

Which is a pretty amazing turn-around time for "crowd-sourced" discovery of a
bug. And according to the bug tracker, it looks like they _patched it the same
day_.

So as bad as this is (and I feel for those who lost data, I really do), the
solution is simply to not run bleeding-edge kernels on release day. The very
fact they have a minor version number 4.14.1 (.1) implies things get released
that need fixed, and if you have critical data you should be more patient.

~~~
raller
> bleeding-edge kernel kernel

> risk you play

uhm you realize that the kernel was already released upstream right?

~~~
striking
Yeah, and which distros are shipping it?

Using the newest upstream kernel is something you opt into, on non-production
hardware. This isn't something that would impact a regular user.

~~~
rdc12
How many people would be using bcache in production fullstop, I'm guessing not
many.

------
rodgerd
Reading the bug it appears it may affect more filesystems than merely bcache.
Nasty.

~~~
xwvvvvwx
pretty impressive response time though.

------
pkaye
How come none of the static analyzers that seems to be run on the Linux source
code not catch this?

~~~
caf
Because the field wasn't formally uninitialised - it was initialised to zero
by bio_init(). It was instead a logic error, which are not as amenable to
static analysis.

------
nategri
Slightly OT but I'm happy to see breaking news from a gentoo.org domain. I
long ago decided installing Gentoo was a young person's pursuit, but I'll
always have fond feelings for it.

~~~
dguaraglia
You can always try Arch, it's like the more-adult version of Gentoo. Still a
lot of work though.

~~~
mohaine
Agreed but Arch isn't much work after the initial install. Well worth it for
the rolling updates.

Arch mostly just works. The big issue I run into is that the upgrades are
_almost_ flawless, which bites you once a year or so when you least expect it.

~~~
nerflad
It becomes slightly more work if you depend on AUR packages (especially if you
end up maintaining the packages you depend on!), but overall I think it's a
great daily-driver system.

------
baobrien
I lost data to this one after building a 4.14 kernel.

~~~
harry8
Be positive. You learned an important lesson about using very recently
released system software without having backups. You're a better engineer for
it. Apply the lesson elsewhere and it's probably significantly more valuable
than the lost data.

"Experience is what you get when you didn't get what you want."

------
X86BSD
I... WOW. This one is epic. I don’t even... wow.

~~~
snvzz
On a Linux .0 release, it's not that uncommon for fs corruption bugs. I
remember quite a few of these.

Best to only upgrade Linux when the next release is already out (e.g. 4.15 out
to consider 4.14). Or just use something else.

------
mtgx
Linus worried so much about the new security features that he missed this one?

~~~
__david__
Here is the commit that introduces the bug:
[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=74d46992e0d9)

Can _you_ spot the error?

Here is the commit that fixes the bug:
[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=62530ed8b1d)

~~~
lomnakkus
Why is this a single commit? I would expect renaming like that to be done in
_completely_ separate commits and then subsequent commits to do any fixups.

Is it just a merge commit? I imagine it's probably unfair to blame Linus for
this (assuming it's a huge merge), but it would also weird for you to point to
a _merge_ commit as evidence of the opposite.

Color me confused.

~~~
kelnos
The "GitHub model" is very different from the original kernel-dev model of
using git.

GitHub encourages you to make a bunch of smaller commits and then roll them
all up into one pull request -- so the unit of change is the PR, and not so
much the single commit.

In kernel-dev land, the unit of change is the commit. Sometimes larger changes
will be broken up into a series of smaller commits, but in general you tend to
see single-commit changes (which often come from a series of commits someone
has done and then squashed down to one).

~~~
bonzini
No, the kernel definitely encourages submitting patch series instead of large
changes. That's why git has the "git am" (apply mbox) command.

Patches that are submitted by external contributors are never squashed by
maintainers (except to occasionally fix a bug that was discovered after the
submission).

