
Systemd Forward Secure Sealing of System Logs Makes Little Sense - louwrentius
http://louwrentius.com/systemd-forward-secure-sealing-of-system-logs-makes-little-sense.html
======
UnoriginalGuy
This article is predicated on the assumption that an attacker goes from zero
access to full root within 10 seconds. While there are vectors which have
allowed such rapid escalation, they're less common now than historically
thanks to Linux's security looking more like an onion rather than just a
binary state (root or user).

Also the author picks apart a quote, but if the assumptions in the previous
paragraph is taken at face value (that an attack can have sup-10 second root
with automated attacks on the logs), then even remote logging isn't a secure
vector as they often aren't sent instantly so an attacker could intercept or
alter them.

However I suggest that such a scenario is uncommon. Most attackers have to
start outside and fight their way in which will generate plenty of logs for
either external logging or FSS.

> It seems to me that systemd now also wants to play the role as some crude
> intrusion detection system.

FSS isn't a "intrusion detection system." It is an attempt to make logs
immutable, nothing more. The nice thing about FSS is that it can be coupled
with an actual IDS and the IDS can look for attempts at altering the FSS.

> If the systemd author would just have implemented sending log events to an
> external log server, that would have been way more useful security-wise, I
> think.

It isn't a binary choice. They can implement FSS and remote logging. Also it
is open source so you yourself can contribute external logging.

~~~
killnine
>they're less common now

Source?

>arent sent instantly so an attacker could intercept or alter them

intercept where?

>Most attackers ... generate plenty of logs

Source?

>They can implement FSS and remote logging.

Point is, thus far they have not, and their first choice was not what the
author views as more secure.

~~~
UnoriginalGuy
> Source?

Experience. I've set up SELinux, AppArmor, etc. For example back near 2000 a
lot of Apache servers I worked on were running as root, now few are, and
better still many have limited permissions even as their user/group.

The same can be said for many Linux services today. And better still the few
that HAVE to run as root often come shipped with some level of permission
controls to limit exposure.

I can link you to a random book on Linux access controls and security if you
want an actual third party source. Or be a total dick and link to a man-page.

> intercept where?

It depends which system you're using to send logs remotely (and how it is
configured). Sometimes the filesystem, sometimes IPC, etc.

e.g.
[http://www.rsyslog.com/doc/queues.html](http://www.rsyslog.com/doc/queues.html)

>> Most attackers have to start outside and fight their way in which will
generate plenty of logs for either external logging or FSS.

> Source?

Source on, what? That attackers who break into systems generate logs? You
REALLY need a source on that, REALLY? Come on now.

> Point is, thus far they have not, and their first choice was not what the
> author views as more secure.

Alright, and then criticise them for that. If the article had been about
systemd's lack of remote logging that would have been one thing, but it was
complaining that FSS was bad because systemd lacked remote logging. FSS
existing is irrelevant.

Also, again, open source. Go implement shit.

------
teddyh
This starts off looking like a reasonable criticism of systemd, or rather that
systemd claims to do more that it (or any alternative) actually can guarantee.
It quotes the author of systemd to prove that the claim is made, but it does
not give a link or say from where the quote was taken (so we don’t know the
context or age of it, i.e. we don’t know it is still relevant).

But then it veers off: “ _This whole observation is not original, in the
comments of the systemd author 's blogpost, the same argument is made by
Andrew Wyatt (two years earlier). The response from the systemd author was to
block him._” No links are provided, so we can’t know what any of them actually
said or did.

This smells like deliberate character assassination. Especially the fact that
no references are provided seems fishy to me.

~~~
DSMan195276
I found the comments from Andrew Wyatt - They're on the same G+ page linked in
the article actually (Though it would be nice if the article pointed this
out). Here's a full copy of the relevant comments:

Andrew Wyatt:

> Sigh.

Lennart Poettering:

> +Andrew Wyatt Sigh?

Andrew Wyatt:

> +Lennart Poettering I'm not going to waste my time explaining why.﻿

Lennart Poettering:

> +Andrew Wyatt cool, then what you doing is not helpful, and just noise, and
> hence waste of time, and hence why do you post this at all? You are
> contradicting yourself there...﻿

Andrew Wyatt:

> +Lennart Poettering I forgot that this wasn't a free internet, forgive me
> for sighing in public about something that doesn't really solve the problem
> that it aims to solve.

> One of the principle reasons to ship logs is not just to protect and
> preserve them, but to remove the control of the logs from the administrators
> of the systems - as sometimes the attack happens from the inside.﻿

Lennart Poettering:

> +Andrew Wyatt Cool, I'll translate this for everybody else: +Andrew Wyatt
> has no clue what he is talking of, but is full of negativity, hence tries to
> spread FUD, and be as unspecific as possible, making wild claims he makes up
> of thin air, to just make this negativity stick.﻿

Lennart Poettering:

> +Andrew Wyatt I never claimed that FSS tries to solve the problem of attacks
> from the inside. That is a fundamentally hard problem, since it is entirely
> against the protection model of Unix: the admin is in power, and this gives
> him the power to stay in power. If you want to fix that problem then an
> external log server won't help you at all. You need to change Unix from the
> ground up, and not just that as the untrusted admin problem is primarily a
> social problem that you can only do so much about solving technically.﻿

Andrew Wyatt:

> +Lennart Poettering it is actually a pretty easy problem to solve, I should
> know as someone who has solved it. Log syslogd events including starts and
> stops, and ship every log message to a server or cluster of servers that
> administrators don't control. A good forensics auditor would know who was on
> the server when the syslog kill event occured, and if the nodes are
> configured right - at least some of the steps executed prior to.﻿

Lennart Poettering:

> In case anybody wonders, I have blocked +Andrew Wyatt now. If people get
> personal I am more than happy to remove their comments, and ultimately block
> them.﻿

~~~
angersock
So, in one fell swoop, Poettring demonstrates both his own immaturity as well
as a practical example of not having an external log server--had that been the
case, the comments couldnt've been deleted by a "disgruntled admin"
(Poettering). :)

Why the hell would you trust a software community with this guy? He doesn't
seem reasonable at all.

~~~
DSMan195276
Granted, Andrew Wyatt was acting immature as well, but his last response was
decently reasonable and seemed like a fair point. I could have understood
ignoring him, but the fact that Lennart was clearly provoking him isn't what
I'd want to see from a project leader. That's simply not a way to treat people
who seem to have legitimate gripes. I could understanding calling him out for
his 'Sigh' comment, but I don't understand claiming he has no idea what he's
talking about and is just spreading FUD and 'making wild claims up out of thin
air' after he explains his point. That comment was simply _completely_
unnecessary and just Lennart being angry.

If Andrew did keep acting like he was at the beginning and Lennart answered
all his questions/complaints, then I could understand either blocking him or
discontinuing the conversation. But the fact that Lennart either never
responded or removed all the responses after that is disconcerting, and
honestly just annoying. If he had just answered it and left the responses up
then most complaints would be gone and this article wouldn't even exist.

------
vezzy-fnord
An interesting thing to note is that systemd's FSS feature is based on a FSPRG
[1] (forward-secure pseudorandom number generator) contributed by Lennart's
brother Bertram Poettering, based on the latter's research paper entitled
_Practical Secure Logging: Seekable Sequential Key Generators_ [2].

[1]
[http://cgit.freedesktop.org/systemd/systemd/tree/src/journal...](http://cgit.freedesktop.org/systemd/systemd/tree/src/journal/fsprg.c)

[2]
[https://eprint.iacr.org/2013/397.pdf](https://eprint.iacr.org/2013/397.pdf)

~~~
angersock
Oh, goody, so it's homebrew crypto as well?

------
steventhedev
I think the systemd complaints really boil down to two very real problems:

1\. The authors of systemd aren't dealing with criticism well. They've allowed
it to turn into a PR nightmare. Agree or disagree with them on specific
issues, but they've allowed this to grow out of hand. There are nearly daily
popular posts on HN, Reddit, etc. Most of these are bikeshed arguments, but if
you have so many bikesheds that it becomes a daily issue on news aggregators,
then you have yourself a culture problem.

2\. Perhaps as a result of the above, there haven't been any limitations of
the scope of systemd. This means that there is a lot of feature creep as stuff
gets added in, and criticism gets swept away without considering it.

Unfortunately, I don't see an easy way to fix the culture problem. But it
needs to be addressed.

EDIT: formatting

~~~
click170
Posts like yours are one of the first steps. At this point we need to address
the culture problem and the people contributing to it instead of focusing on
the distracting fallout.

I've tried to keep an open mind about all things systemd but after finally
reading some of the exchanges people have with the systemd folks, I'm
surprised systemd isn't more widely rejected as a viable alternative simply
because of the attitude of some of the people currently maintaining it. You
can't be an butthole to folks trying to offer valid criticisms and expect
those same people take your project seriously. That just does not inspire
confidence, it inspires skepticism and distrust.

I wonder if it's simply a critical mass problem, if once enough devs get
peeved they will fork systemd and lead it the way they see fit sans any
caustic contributions or contributors.

~~~
Cyther606
It has been forked [1].

> uselessd (the useless daemon, or the daemon that uses less... depending on
> your viewpoint) is a project to reduce systemd to a base initd, process
> supervisor and transactional dependency system, while minimizing
> intrusiveness and isolationism. Basically, it’s systemd with the superfluous
> stuff cut out, a (relatively) coherent idea of what it wants to be, support
> for non-glibc platforms and an approach that aims to minimize complicated
> design

He could use better marketing, but I think removing the kitchen sink aspect of
systemd and keeping the service file syntax is a step in the right direction.

[1] [http://uselessd.darknedgy.net](http://uselessd.darknedgy.net)

------
ghshephard
No security professional will be bothered to critique "FSS" because anybody
worth their salt know that any log _ON_ a server is, by definition, not
trustworthy. They've known this for at least 20 years (if not longer), and
everyone uses syslogd or some variant thereof.

So, in that sense, there is no controversy here, and this feature is, for the
most part, completely useless.

In the comments below, Andrew Wyatt is absolutely correct and this, "Lennart
Poettering" individual doesn't know what they are talking about.

------
jimrandomh
This is a neat feature, which I didn't know systemd had, and which I recall
having wanted in other contexts. The criticism about the 15-minute interval is
valid, but that's an implementation detail which, from the sound of it, can be
fixed if someone puts in the effort. So while the author may have intended
this as a pure criticism, it didn't really come off that way.

------
the_mitsuhiko
So the argument is: it's not good enough, so we need to throw it away?

~~~
click170
No, I think the author's point was that people shouldn't misrepresent FSS as
equivalent to logging to a remote server. They can compliment each other, but
they are not the same, and thinking they are can lead to serious security
snafus.

~~~
louwrentius
Exactly.

------
xorcist
What I find troublesome with journald is that they ignored everything that was
done before in the log signing space. There has been some research done, even
an RFC published. They implemented their own solution incompatible with
everything else. Any questions about their design was met with, at best,
indifference.

Since then rsyslog gained a log signing solution, sadly incompatible, but
which at least to my eyes look a bit more conservative in design. We don't
really have any second opinions on journald. That would be fine if it was just
a hobby project, but I find it problematic to include by default in stable
enterprise distributions.

------
erhardm
Systemd is starting to show the signs of feature creep which IMHO is good.
Failing early means the ship can be turned around if there's enough will, or
at least (newer) replacements of systemd(in case it goes really off the rails)
will learn from others' experience.

~~~
UnoriginalGuy
This was added in 2012.

~~~
erhardm
You mean the article or the feature? The article is dated by 22/11/2014\. The
feature may have been added a few months/years ago, but the point still stands
about systemd showing signs of feature creep. Note also the DNS resolver
issue[0]

[0] - [http://seclists.org/oss-sec/2014/q4/592](http://seclists.org/oss-
sec/2014/q4/592)

~~~
the_mitsuhiko
The feature was added in 2012 or something, whenever the log support came.

The DNS issue affects not just systemd but any stub resolver.

~~~
pdkl95
The DNS issue is worth in systemd's resolver because it is _NOT_ merely a
"stub resolver" \- it implements a cache. From the same thread:

"The DNS specification does not require rewriting of upstream responses to
filter out parts for which the queried server is not authoritative. This means
that a downstream caching resolver will tend to poison its cache if it adds
data from such responses that are not directly in response to the QNAME."

[http://seclists.org/oss-sec/2014/q4/602](http://seclists.org/oss-
sec/2014/q4/602)

Adding entries (including additional records, which are common in DNS) without
validation is just _asking_ for a DNS poisoning attack.

------
KaiserPro
Log signing is nice to have, but ultimately not that reliable. Systemd has
quite a broad attack surface __, and if the attacker has root, you 're fucked
anyway. It might be a useful tool for poking local sysadmins.

 __compromising systemd, forcing it to re-sign the logs (assuming that
practicable.)

Potentially you can have an alert that checks the signature, which might be
useful. However you can do something similar now, but comparing log files
(local vs remote) although its not that easy or elegant.

I've yet to use systemd in anger. I'm sure when we update all 25k machines to
it next year we'll become intimately familiar with all its pains.

~~~
Tuna-Fish
> compromising systemd, forcing it to re-sign the logs (assuming that
> practicable.)

This is completely unfeasible. The point of FSS is that you sign the logs,
create a new key using your old one, and then permanently throw away the old
key. Once this is done, no matter what control you have on the system, you
cannot re-sign the logs without the tampering being obvious.

