The problem is not so much that selinux is too complicated (it is as complicated as it needs to be), but that we all run software we don't understand.
The whole IT ecosystem has become a hail mary. Even admins usually have no idea what a certain program actually wants to do. If the admin knows how to install the app so that it actually runs, you call them a good admin.
From a security point of view, an application is like a nuclear power plant. It's good if it works as planned, but if something blows up it endangers your whole enterprise.
The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl. That way the radiation hopefully stays in, but history has shown that it really doesn't. Also, the wheel of history has just turned one more iteration and now admins just view the sarcophagus as something you deploy as you previously deployed the app. Who is responsible that it is air tight? Well, uh, nobody, really.
You can't even blame the applications for that. Let's say you want to build a good, secure application. How do you know what files your application will try to open? What syscalls it wants to call? Library and framework functions tend to not document that properly.
Obscure files like /etc/localtime, /etc/resolv.conf, /etc/ld.so.conf, /dev/zero ... how can you expect devs to build well documented and well sandboxable applications if they don't know which files their library functions will open?
You may have heard of /etc/resolv.conf ... but have you heard of /etc/gai.conf? /etc/nsswitch.conf? /etc/host.conf? Wouldn't it be great if the man page of getaddrinfo mentioned those (mine only mentions gai.conf)
> The problem is not so much that selinux is too complicated (it is as complicated as it needs to be)
I disagree.
> but that we all run software we don't understand.
I fully agree.
My disagreement lies in the fact that you've described the problem, but are proposing that some software (SELinux) that fails to solve the problem is somehow good.
SELinux might be a perfect tool in an ideal utopia where everyone understands all the software they run, but that isn't the real world, and having a tool that works in a theoretical world isn't particularly useful.
Either SELinux is applicable to the real world, or it's not useful and not really fit for purpose.
This isn't a "blame game" - it's not about figuring out why things are bad (no-one understands their software components), nor blaming SELinux (it hasn't caused these problems, it's just failing to mitigate them). It's about figuring out how to improve that situation. Does SELinux do that effectively?
>My disagreement lies in the fact that you've described the problem, but are proposing that some software (SELinux) that fails to solve the problem is somehow good.
My opinion is that there's a cultural and policy problem and you're simply not going to solve it with technology. I don't think SE Linux is bad, it just isn't the answer here. One (clearly not the only) main reasons people just run a bunch of software they don't understand is due to expectations in productivity. In an idealistic world with caution built in, we are willing to sacrifice some productivity for security, stability, understanding, etc. I think many engineers like these ideas because they like to work with and build solid systems they understand very well--otherwise they wouldn't be practicing any form of engineering.
What we instead have is a world of engineering with no sort of guard rails. Unlike Civil Engineering, there's no licensing, no sorts of inspections, no oversight, liability is completepy waived in a 300 page agreement people just click through, etc. Even when things go horribly wrong (e.g. Equifax) the repercussions are limited. We see abuse in other engineering fields where actual life and death are factors and they're still just amortized in a cost analysis.
Ultimately though engineers simply don't lead the world (and arguably,
maybe they shouldn't) , people with wealth lead the world (arguably, they shouldn't either) because it underpins pretty much all other systems anymore. As such, productivity is always prioritized over safety and security. This issue isn't limited to software, it happens everywhere almost ubiquitously at this point. Software is one of those areas where most people just care less, so you can get away with more and more of this. The issue is that I believe we're building a house of cards that will be difficult to revert to a reasonably stable state once it finally collapses.
> otherwise they wouldn't be practicing any form of engineering.
This is incorrect. The dirty little secret here is that software engineering is an extremely lucrative career, and a ton of people have gotten into the field for that reason. I'm not saying they're wrong to do so, but I have never met a wizard who didn't love the craft. I have met tons of people faking-it-till-they-make-it who are there for the RSUs and free food. The latter tend to practice resume-driven development, and tend to have very little idea how their code actually works.
> ... and tend to have very little idea how their code actually works.
Or they know how their code works, but do not care. These are the min-max'ers who will MVP their way on every conceivable angle to grab the checkmark from the project management office and run off to get the next checkmark as quickly as they can to rack up enough for the next promotion. They look awesome on paper to metrics-staring management. They are accelerated bit rot to any organization's codebase. And to rein them in, you have to challenge them in such excruciating detail that it starts to appear like bikeshedding to outsiders.
Except the difference from bikeshedding is the code they leave behind is utter crap to maintain, extend, operationalize, debug, instrument, monitor and only tested under the happy path with maybe the handful of sad paths you dragged them kicking and screaming through code reviews to implement. They know they can out-wait you and the impatient business users in these reviews, and will miserly implement only what you force upon them in these painful, political battles.
The only way you can out them is within an organization where operations teams have the power to write out in detail the challenges with their code after taking on operations duties for awhile, and return operations to the developer with literal pager duty attached until the gaps are addressed.
> Unlike Civil Engineering, there's no licensing, no sorts of inspections, no oversight, liability is completepy waived in a 300 page agreement people just click through, etc.
This! I have come to the same conclusion. If you want to construct a building you have to comply with multiple standards and industry regulations that serve as limitations and safeguards that follow your project from the initial idea up to (sometimes) the point of destruction of the constructed object.
In our industry even with the standards that we have they are oftentimes treated as guidelines not rules. Of course there are exceptions but those are just a byproduct(or proof) of the rules themselves.
>If you want to construct a building you have to comply with multiple standards and industry regulations that serve as limitations and safeguards that follow your project [...]
Also, don't forget that the laws of physics also weigh heavily on the minds of most engineering disciplines. Stuff has density, strength, thermal and kinetic properties that reflect omnipresent constraints on most decisions.
SELinux is a perfect tool in its niche — where that niche is "secure-installation system integrators." It's not a tool for regular people; it's for e.g. defense subcontractors who want to certify their virtual appliance as meeting certain standards demanded by certain customers; where meeting those standards requires them to actually go through their virtual appliance from top to bottom and prove the whole thing secure and sandboxed.
It's not a tool for application developers (who at most can supply SELinux templates to start with); nor is it a tool for sysadmins (who really don't have the time or the incentives to deal with it.) It's a tool to be wielded specifically by people occupying a role that, in most organizations, doesn't exist.
SELinux is applicable in the real world, but you must choose if you want it to be Fast (use globs of rules written by unknown people who don’t know your requirements), Cheap (start from zero and develop policies specifically for your app), or Good (do both while learning a substantial amount about SELinux).
If you’re using SELinux rules off the shelf for a handful of Fedora boxes for personal use, maybe that use case doesn’t work. But is that more “real world” than an institution like the NSA trying to completely lock down its production systems?
I think what people are missing (and perhaps the author is at fault for not calling this out more explicitly), is that the complaint is effectively: SELinux is more difficult to configure than it needs to be / should reasonably be.
Perhaps due to unnecessary internal architectural complexity, the lack of a good configuration interface, and/or bad config docs (which of these is the biggest culprit I'm not sure, but they all seem to have some level of truth to them).
The gp posited that it's necessarily complex - subjective, but I don't buy it. The author provides a good selection of examples that seem straightforwardly non-sensical in their operation: why doesn't audit2allow integrate properly with the overall SELinux architecture instead of requiring separate knowledge of semodule for application. Why doesn't the system keep track of label deprecation as system dependencies are changed & updated? These seem like pretty obvious out-of-the-box basics you would design in, if SELinux had been "designed", but the answer to these questions seems likely to be that it's more of a bundle of un-integrated tools and loose-text configs/blobs without well-thought-through centrally architected database of labels, maps and APIs.
One could even take this blogpost as a constructive set of ideas to improve SELinux, but the scale of the issues do seem overwhelming to change on such a large project, and based on the comments here on HN there seems little will to do so. To the point that perhaps a separate / new approach would be easier.
Honestly, the bigger issue is that most SWEs just aren’t very good. It’s extremely telling that when you spend time in tech forums most people dread system design questions as the harder side of interviewing for senior level SWE roles…
System design, though, is the actual point of SW ENGINEERING. That’s the part that is responsible for creating a foundation of quality to build on.
The other side is that sysadmins have largely become DevOps or SRE roles, which means most folks left as a sysadmin are those that couldn’t hack it in the other roles. In the end, as we grow the number of people in tech, the number of software and systems, and the complexity we experience a regression to the mean across the board.
It’s always been a pet peeve of mine that I can count on my hand the number of SWEs in my ~20yr career that had similar understanding of the system they worked with as their SRE counterparts… but this should be table stakes.
> It’s extremely telling that when you spend time in tech forums most people dread system design questions as the harder side of interviewing for senior level SWE roles…
Yes. It's absurd that anyone that can talk about this stuff at a basic level is seen as some sort of ultra-skilled professional. See also: the relatively small amount of chatter about this stuff in the blogosphere, where there's a million tutorials that amount to "here's how you run create-react-app" and about 3 that talk about even moderate system design topics.
> Honestly, the bigger issue is that most SWEs just aren’t very good. It’s extremely telling that when you spend time in tech forums most people dread system design questions as the harder side of interviewing for senior level SWE roles…
I think this is a bit unfair, the software industry is so big now there are lots of different sub-sectors. Big Tech companies have huge scale problems that are kind of unique. Most developers work on products that only have a few hundred users at a time.
Your 20 years experience is a bit telling. I'm the same age, back then SWEs usually had full access to production environments. Now everything is segregated and devs can't experiment in prod and end up getting detached.
Maybe we're just in different circles, or perhaps have different definitions of system design, but that's not been my experience. Usually I see developers being very excited about system design and quite good at understanding system design questions - nothing's more fun than architecting a new system and thinking about stuff like scalability.
Where people tend to fail is when they have to implement these systems, and they've planned for all these high level building blocks but then actually have to configure them. Redis, RabbitMQ, GraphQL, Docker, NGINX, the list goes on - there's so much to configure and you really don't truly know how they work unless you dive into the source code, so you inevitably get bitten by subtle bugs and errors where your expectations of how something should work based on the docs are violated by how it actually works.
Should a SWE have a deep level understanding of all this stuff? I'm not sure. Usually you can still hack it into a working system after a little bit of firefighting, and that's likely more time efficient than understanding source code or rolling your own infrastructure. But I definitely think there's room for more comprehensive and transparent documentation to bridge the gap between the source code and barebones docs.
Agree completely. I've never known a software engineer who didn't love designing new systems on paper and starting the initial development. It's when it comes time to stabilize and turn a system into a usable product minds quickly wander to designing the next system that fixes the flaws of the last system.
>But I definitely think there's room for more comprehensive and transparent documentation to bridge the gap between the source code and barebones docs.
This is usually provided by technical books, which are available for a much wider range of systems than you'd think.
> It’s extremely telling that when you spend time in tech forums most people dread system design questions as the harder side of interviewing for senior level SWE roles…
> System design, though, is the actual point of SW ENGINEERING.
I mean, I would say I'm pretty good at engineering and system design (10 YoE, senior/principal for most of them). But I dread the system design interview questions because (in my experience) they bear almost no relation to the system design experience I have and almost no relation to the actual requirements of the job. Things like "design an ETL flow" or "build a distributed wikipedia downloader" are just not in my area of experience. (I work mostly in the lower layers of single-host operating systems and file systems.)
In the real world, if I was tasked with "build a distrbuted wikipedia downloader" I wouldn't just start randomly guessing.
I'd get input, feedback, discuss, narrow down options.
Yet for some reason, that's an actual interview question. Start randomly guessing based on your knowledge of something, in a way that you'd never do in a real scenario.
Edit:
I feel like I should do an interview like this. Refuse to explain how to build it, and instead explain the detailed process on how I would successfully build and complete the project. lol
I totally agree that system design questions can be silly sometimes. When I do system design interview questions for senior engineers, I ask a really broad and complex questions and let the candidate drive us towards the system. The point of the interview is evaluating if the candidate can complete those soft skills in a reasonable way. I try to answer basic questions such as these:
Did the candidate just go for design without asking questions?
What questions does the candidate ask?
What areas are they asking questions about?
Can they take critical feedback on their design well?
There is a little bit of discussion on technology choice, but typically if they can justify an out of the norm choice it is fine.
That makes sense. I guess I've delivered incredibly complex systems in a tremendous number of diverse use cases, and yet I have zero confidence I could pass a systems design interview.
The few times I've had them, they've focused on how much knowledge of a particular choice I have.
When I build systems, it's incredibly common I don't know much about the topic. Especially in innovative spaces where you're literally the first team ever building a solution at this scale.
Take for example critical feedback, I can't imagine having to judge someone on that in an hour phone call.
The best teams argue quite a bit. That's healthy and reasonable. The question is if they accept, and respond to the argument, with positives and negatives, and how basically how they dance.
I don't know how to put into words what that dance is like, but you know it when you feel it. Maybe that's what you're looking for, a feeling of what that dance is like, but I'm glad I'm not making that determination on a quick basis.
Just random thoughts, thankfully I get all my employment through glowing referrals, and try to avoid interviewing anywhere without them so I can skip all of this questionable stuff.
I think most devs see that the jobs are asking for overloaded responsibilities from multiple other jobs and they're trying to live up to that. It also became the norm to see job postings which would have been 3-4 distinct positions a decade back.
True, but even in the 90's and early 2000's, before cloud took off, most developers weren't sysadmins, and they didn't want to be. That wasn't their specialty. The "Unix wizard" developer was always a rare breed.
> System design, though, is the actual point of SW ENGINEERING
Software Engineering is about problem solving using software and hardware (we'll just say computers for simplicity). Whatever solution system you propose has inherent constraints and tradeoffs. That's the Engineering. System design is a catchall phrase for a subjective problem scale.
Speaking personally, the reason I dread system design questions is because it's asking me to give an off-the-cuff answer to something I would prefer to take weeks to consider and research.
> The problem is not so much that selinux is too complicated (it is as complicated as it needs to be), but that we all run software we don't understand.
There are many many problems.
One of the biggest problem with SELinux is that it is trying to graft Mandatory Access Controls on a userland that is not designed for it.
Unix, frankly, is not designed for security. It is designed to get work done by writing a bunch of little buggy C programs that you string together in novel ways.
Security is something that was grafted on it. And it shows.
How many decades of security vulnerabilities have occurred because of a shared /tmp space? 40 years?
How do you graft access controls on a system designed with no access controls and potentially billions of combinations of programs, paths, and various other resources without breaking anything?
The answer is: You don't. You can't.
Were as you have a system designed for security, like Android, and literally hundreds of millions of fully SELinux-enabled fully locked down user-facing Linux devices are out there being used by people who haven't the faintest clue what "audit2allow" is and wouldn't understand it if you tried to explain it to them.
So it's less of an issue of "we all use software that we don't understand". SELinux is complicated enough that you can devote your life to trying to understand it and still fail to craft good rules for other people.
It's more of an issue of "Linux userland follows the basic Unix design from the 1970s which is kinda shit if your goal is security".
It is just bad design. Pure and simple. That is all there really is.
However there is a way out.
The way out is to give each process and each user their own little itty bitty special Unix environment where they can do whatever they want. And then you use strong divisions to keep them mostly unaware of each other. Use a default deny policy and only poke holes in it when required.
My android phone came without a calculator app. I recently realized this in a setting where I needed a calculator. I commented out loud about the lack of calculator and got some responses from the group I was in at the time.
"Just download one from the app store," I was told. "But be prepared to grant it network access, file access, contacts access, camera access, and email privileges."
The alternative, which is Linux, is to grant your calculator app the permissions to read and write to the same resources that your browser uses to store the password for your bank.
Well "grant" is too strong of a word. "By default and there is nothing you can do about it unless you are exceptionally skilled" is more accurate.
Also your calculator app can read your sudo password as you type it, which you do a dozen times a day to carry out complex and security sensitive tasks such as "Connect to printer to print out mom's recipe for brownies" and "restart bluetooth because it's buggy and you want to listen to spotify on your wireless headphones".
Just yesterday, Microsoft found a few[0]. There's no shortage of these, but more important is the haphazard way fixes are backported to longterm (e.g. [1]). This reached the point that Google's security advice is 'always follow the latest kernel'[2], except most users and distros simply cannot afford it, so they are stuck with a vulnerable system.
Linux is not unique here. In the longterm, all the typical desktop OSs will need significant structural changes far beyond 'chase vulnerabilities and patch everything all the time'.
Nothing says that Google couldn't prioritize applications that need the least permissions, or make it easy to filter by permissions, or even have an acceptance policy that said "A calculator app can't look into your contacts".
The system could favor the user in this regard, Google just doesn't want it to.
Single page of instructions from google apps has me download a tarball, add a hello world command, and Presto! I have a webapp on my computer! Then change this one line and PRESTO! My webapp is in the cloud!!
No telling how, or with what components, or what the dependencies or security implications are, or how to see the logging for the app. Just a single paragraph of instructions and you can be the next internet startup IPO!
'Hail Mary' is the best way to describe the situation I've yet heard.
I fully agree with your last point. In my experience, the all-versions-packaged approach of containers actually leads to a worse patch state than a simple Ubuntu or Debian installation with unattended-upgrades.
I've tried to explain to our Devops guys that they need to automate their patching so that a new container is fully patched before deployment. They had no idea what I was talking about. I said, "you have containers that run for months, right? And they aren't modified at all after deployment, right? So how do you address new vulnerabilities? How often are you patching your base image?"
How would you accomplish this exactly? Your use case may vary, but in my mind dependencies shouldn’t just be upgraded by some automated system without some sort of feedback mechanism (like tests).
If you’re going to run tests against the new dependencies, then why not just shift that whole activity to the development process itself (not the deployment system)?
When dependencies are sufficiently different from the existing prod dependencies, you can release a new version, instead of expecting ops/devops to take care of your dependencies for you.
I'm actually agnostic as to how it's achieved. I just think that the idea of running a container for three months without patching the underlying OS is nuts. Generally our guys only redeploy a container when there's been an update, so that OS is pretty stale compared to what I'm running on traditional VMs. It really was more of a reflection on how DevOps can become too focused on application development and deployment, as opposed to deploying holistic systems. We have coders writing apps that feed into our CI/CD system, we have admins who manage kubernetes/Swarm, and we have traditional sysadmins managing VMs. All with differing priorities and focuses.
I kinda stopped worrying too much about it because the "OS" (really userspace) that lives in a container isn't in a position to be exploited like the underlying VM OS is. Nothing in the container is privileged and there are no security boundaries that exist within a container. You should care exactly as much about container image updates as you do gem/npm/pip updates and I guarantee you have months old deps pinned in a lockfile somewhere.
What people hear is "Oh my god you're still running Ubuntu 14.04?!?" but what is actually happening is all the defenses and security boundaries are being enforced by a fully patched RHEL 7 instance with SELinux, the completely unprivileged application code just happens to have an old version of libz bundled with it. So for sure the application is in a position to be compromised depending on how old the libs and the vulns that might exist but that's the developer's concern exactly like their NPM deps.
So an application runs in a container that has a vulnerable OS. This has network access, DB access, NFS or SMB access. All valuable stuff. And then the container's OS gets breached due to the vulnerability. Sure, the VM hosting the Docker containers is secure, but that doesn't really matter if the breached container had access to PII or other valuable data.
This idea that people have about containers being 'the "OS (really userspace)' is fundamentally wrong. It's the equivalent of a VM, just (hopefully) stripped down to the bare essentials. There's no magic that protects it. The Docker host sure doesn't protect it, and in fact the Docker host can be vulnerable to exploitation from its containers, just as a vSphere host (or any other hypervisor) can be exploited if its VMs are insecure.
This can't happen. There is no "OS" actually running. It's a bunch of binaries and files sitting in a tar archive that the process running can access and link against. What are you going to breach? All the networking, firewalls, kernel, remote access like SSH, and privilege management is the hosts. The only process running in the container is the application.
What you mean is "what if the application running in the container is compromised?" For example what if there's a RCE vuln in the version of libcurl that the app links against? Absolutely! Now you've got a huge problem but this is exactly the same as if there was an RCE in leftPad. In container world that's now the developer's problem instead of yours. Everything in the container is vendored, just pretend it was all statically linked. It's not like an VM where ops patches it.
> It's the equivalent of a VM, just (hopefully) stripped down to the bare essentials
No, no 1000 times no. It's in all ways the equivalent of a process (or tree of processes) running on the host machine. From a security perspective a container is no different than any other binary running on the host. You have it exactly right, there is exactly zero magic and no inherent security.
You should lose exactly as much sleep over outdated container images as you lose over outdated rubygems in your developers' Bundle.lock. Whether that means a lot or a little depends on the company and how security critical the application is.
Thank you for this clear explanation without any attacks! I hate to think how many people I've misinformed about this at work. I think what caused my confusion is how our devops team has been using Docker. They've basically treated it as a vm environment that doesn't require approval from my team. And some of their practices just seem to cause me stress that I should just free myself from.
Would it blow your mind to hear that some devops people actually run more than one application in a container? And that one of those apps might just be sshd? That's one of the many that I know about. Why they wouldn't just run docker exec?
I'm just so sick of the container security shills; in particular I'm tired of people selling debian advisories with zero value add on enterprise contract.
If you're using an upstream image from a distro, pull that down, update it, push it back to a registry, then use that as your base image. Do that every day in a CI/CD (or just cron) system, such that you've got logs, auditing and fires off alerts if something breaks.
Add update steps to the builds of your final image containers too at the start, to catch any delta that may have happened during last base image patch.
There are also container scanning tools but they can produce some false positives sometimes but useful nonetheless.
You still run tests as normal against the new images before promoting, just like normal.
Ransomware incidents are rampant and PII breaches are quite common. Remember Equifax, who lost the PII of basically all adult US citizens?
Sure, there are extremely sophisticated, targeted hacks. But those are the minority. There are also hacks where hacker groups accidentally (!) shut down an oil pipeline. The reason the internet hasn't completely burned down is three-fold:
- We accepted breaches as something "that just happens" and that "nothing is 100% secure" - which is true, yes, but applied very broadly.
- Our operating systems are generally well secured and most exploits are very situational.
- The hacker groups (at least most of them) don't put in too much effort either, since they make a very good living of the lowest common denominator.
I work for a stodgy company with over $4B under management. In a highly regulated industry where you have to publicly announce breaches. Sure it may not matter to a quick startup where someone is re-inventing Uber for gokarts, or other small industries, but when you have to satisfy auditors and regulators across different jurisdictions, you can't take a laissez faire attitude towards patching and compliance.
And considering how many breaches occur on a weekly basis compromising customer PII etc, its foolhardy to say the Internet hasn't exploded yet. It's been in a continual burn for decades.
Unless there's a vulnerability in something internet facing, you can go a very long time without updating anything. So, yes, it doesn't matter... except when it does. I've logged into production systems with multi year uptimes, with updates that haven't been applied in just as long.
Many orgs have a if it's not broke, don't fix it policy. Applying needless updates are just as likely to screw something up.
Many orgs get breached (or have no way to detect if they've been breached).
It depends on both your industry and your risk tolerance. We generally apply every security patch that comes along for RHEL. Application patching is different, it depends on the severity and what the vendor will support. If we can't patch, we try to apply compensating controls to mitigate the risk.
And the pride we all used to have with long uptimes is really a poor badge of courage. It shows a misplaced trust in either your security, or the threat model you face.
OS security patches (at least for RHEL) pose little risk of breaking applications. In the last 15 years, I've never once had to roll back a security patch from RH.
Security patches can be applied even to the kernel without restarting anything though, so having a high uptime is not mutually exclusive with staying up to date. Of course, you do have to pay extra for live patching.
I would argue that the fundamental problem is that the companies selling software "engineering" products do not actually take any responsibility that the product they have engineered works as intended. (see: https://www.snopes.com/fact-check/car-balk/ )
And of course, the main reason they do not take the responsibility is that the customers won't pay for it.
It is kind of interesting. We are very good at making bridges that do not fail unexpectedly, even if there is unlimited amount of unknown failure modes when working with physical materials. And on the software side, well, managing to come up with a fizzbuzz without failure modes is used to screen people on interviews. What would be similar for bridge engineers? Here is a hammer, nail and two pieces of wood. Can you make the two pieces of wood stick together?
The engineer building the bridge is liable. Hence, you get engineers with qualifications (instead of a random person with X years in Y), and those qualifications certify they know what they are doing.
Which in my opinion brings up the certification argument, but I, as I’m sure others, have found that most certifications mean nothing in regards to whether someone can actually do the job. Is the key component missing in hiring in our industry a vastly more rigorous/extensive/difficult testing and certification protocol?
I'm an engineer in physical disciplines, and I don't think it's a certification issue. It's more that "engineering" in the software world is conflated with craftsmanship. The word "engineering" as used in relation to software is basically interchangeable with "technical work". Let me hasten to add, this is not intended to be pejorative or haughty, rather an observation.
Whether it's bridges, aircraft, or other complex engineered systems outside of software, there is a fundamental commitment to correctness, as assessed against physical principles such as conservation laws and both high-level and low-level verification criteria. Some software orgs (I think mostly in safety-critical systems) apply this kind of rigor, but most software development does not. Just look at the pushback on this site for development systems that increase rigor at the expense of "development velocity", such as memory-safe languages, powerful type systems, formal methods, etc. No civil engineer is going to be OK "shipping" their product without stress analyses, material property knockdowns, and the like.
My suspicion is that the root of this lies in an inability to systematically reason about software the same way we can about physical systems today. The tools and principles for correct-by-construction software are more primitive and harder to use than their equivalents in the physical realm. Also, lack of rigor in software still gets you a lot of powerful software and society seems remarkably tolerant of shitty software outcomes.
All software doesn't need to be engineered, the same way no one engineers a hand-crafted piece of furniture. But our software infrastructure ought to be, and it surely is not today. It will be awesome when software engineering starts living up to its billing, as I hope it one day does.
I guess, that person is just being sarcastic. However besides certification, qualification etc I think huge factor is cost of IT project is variable. And even worst failures are mainly delays, extra budget or moving work to different vendor/product. So companies can save money without endangering lives.
A bridge, even a fancy modern one, is pretty simple compared with the software running on a 90s-era computer, much less a modern computer. Fizzbuzz is probably about equivalent to building a simple engine out of a box of parts.
The problem is that we have built all our software on Unix. The Unix security model is based on an attack model where users have to be protected from each other on a terminal server. Code is implicitly trusted and exploits were an unknown unknown.
That security model is almost completely useless now. Terminal servers are an extreme edge case. Services implement their own security model between users. Special “Users” for services is just a hack that tries to use an inadequate system as much as possible.
The Unix security model is deeply baked in everywhere and it’s nearly impossible to tack on a security model fitting to todays requirements afterwards.
I agree with your points about the Unix threat model being designed around protecting users from each other, but for completeness I'd like to point out that while modern Linux/BSD systems ship with Discretionary Access Control (DAC) by default, SELinux implements a Mandatory Access Control (MAC) system which is much more fine grained. SELinux is not limited to the traditional Unix security model.
Short simplified example highlighting DAC/MAC differences: DAC asks "is user Alice allowed to read Bob's files?" while MAC asks "is the SMTP server (subject user/role) process allowed to read private keys (object type) of the HTTP server (object user/role)?"
And if you're really motivated (read: want to have fun diagnosing unexpected file permission issues), you can associate files and processes with different security categories and levels (MCS/MLS), implementing horizontal and vertical separation. For example, "is this software update service allowed to read confidential files owned by the accounting database or keys used to encrypt classified information?"
There are other MAC systems besides SELinux as well.
Is Windows much better with its VMS legacy kernel? It has a complex permission model with ACLs and everything but that trades inadequate permissions for something like the SELinux problem. There are too many knobs. (For this discussion leave aside the backward compatibility baggage which is another issue.)
The only approach that I see as viable without rebuilding the entire compute universe is VMs, either the web browser (JS and WASM sandboxes) approach or the qubes type approach.
You have traded an absolutely enormous attack surface of hundreds of syscalls and a whole system full of files and apps for a much narrower attack surface involving the CPU and whatever the hypervisor exposes which can just be virtio-style pseudo-devices. Besides if the CPU is vulnerable than regular apps can probably exploit it too.
Also I didn't say the best approach would necessarily be hardware VMs, though that is one option and is valuable to let you just plug and play existing applications unmodified. I personally think we need to get away from shipping all software as raw hardware-tied binaries in favor of something like WASM. Java and the .NET CLR had the right idea decades ago, but these were insufficiently versatile and too tied to just one language or platform instead of being a general purpose VM bytecode. WASM is heading in the right direction but doesn't seem to quite be there yet.
Only apps that actually need to use e.g. special CPU features or run tightly optimized ASM code should ship with native binaries or native modules/libraries for the high-performance parts, and that should be something you have to approve.
I hear a lot of, "Let's solve isolation with VMs," without taking into consideration that if you put two fully-patched kernels side-by-side, very few people (i.e., only those sitting on 0-days) have any idea how to exploit the workload running directly on the fully-patched kernel, while everyone knows how to exploit the (Intel) host when they have root/CAP_SYS_ADMIN in the VM and the host has HT enabled.
VMs are far from magic security sauce, and especially if you need HT on Intel hosts, you actively want to avoid using VMs as a "security boundary." You're far better off with the attack surface of a fully-patched OS image from a reputable vendor.
The problem I see with wasm as the sole basis for a secure app platform is that then a web browser can't be just another app on that platform, unless the performance of the browser's JS engine is hobbled or perhaps wasm gets some kind of JIT compilation support. So maybe the web browser engine needs to be the platform, as in Chromium OS, although even Chromium OS has now compromised that original purity with support for Android and desktop Linux VMs.
Using VMs like this means the VM itself becomes hard-core security-critical but if apps have stupid bugs their damage is quite limited.
It also means you can mitigate things like Spectre or RowHammer by just tweaking how the VM JIT compiles bytecode into machine code. You don't have to update apps or even the kernel per se (though the latter would be a good idea).
I agree with you, but if we look past the hack of dynamically creating new users per process, isn’t the security model “sufficient”? E.g. I believe android’s security is quite elegant with the different user per-process model, communicating only through IPC and with heavy use of SELinux (which is much more useful in this model).
Some of the problem is that historically we've built systems badly engineered for security.
Take for instance something like xscreensaver. Something in there needs to be setuid so that it can verify your password to unlock the screen. That something is fortunately a dedicated binary, and not every single screensaver, but still, that's bad. Writing that executable is a delicate thing. Get one of them wrong, and it's a glaring hole.
And /usr/libexec/xscreensaver/xscreensaver-auth is quite the thing. It links to a whole bunch of graphical libraries and talks to X11 to read your password, so that thing has a huge potential attack surface. Far more than seems comfortable.
What we should have instead is some sort of authentication service. A program confined by SELinux to only interact with a socket and the password database, speaking a simple but well designed protocol.
With that in place we'd have much simpler security. Policies get simpler because random stuff doesn't link to PAM anymore, and so doesn't end up touching a whole bunch of critical security related files. There's one thing on the system that deals with that, and it ends up with a small, understandable policy.
And the policy for the users is now expressed in terms of "Do we want this program to be able to validate user passwords?", which is also far simpler.
With that kind of redesign over time we could have much smaller and more understandable policies, and a more secure system as a result.
I get your overall point, but your example is not a good one: on Linux, at least, xscreensaver does not have a component that needs to be setuid root.
xscreensaver-auth uses PAM to authenticate, which (typically) runs a small program called unix_chkpwd, which is setgid shadow (not setuid root), as /etc/shadow is owned by root:shadow and is readable by group.
xscreensaver is one of the very few screen lockers that is reasonably secure, because the bit that locks the screen is pretty minimal and only links with a few fairly-low level X11 libraries, not a full-scale GUI toolkit.
> What we should have instead is some sort of authentication service. A program confined by SELinux to only interact with a socket and the password database, speaking a simple but well designed protocol.
That sounds pretty overengineered. All you need is a small binary that is setgid shadow, that can take username/password on stdin, and exit 0 if auth succeeds, 1 if auth fails. But we already have that: unix_chkpwd.
Oops. Yeah, serves me right for writing from memory, pretty sure it was setuid at some point in the past.
> That sounds pretty overengineered. All you need is a small binary that is setgid shadow, that can take username/password on stdin, and exit 0 if auth succeeds, 1 if auth fails. But we already have that: unix_chkpwd.
That's probably too limited actually. Because you can have a whole bunch of stuff in PAM: LDAP, OTP, Yubikeys, and all kinds of other fancy modules. Doesn't seem that unix_chkpwd handles any of that.
Also, I still think it has to be a network service, because the unix fork model offers lots of avenues for attack. The parent process gets to mess with a whole bunch of things before exec'ing anything, and that list keeps getting longer as the kernel adds features. It's not possible for the child to reliably defend itself against anything the parent might do.
Thus I think for security, a network socket is the best way as it doesn't allow the client process to manipulate the environment of the security service.
> That's probably too limited actually. Because you can have a whole bunch of stuff in PAM: LDAP, OTP, Yubikeys, and all kinds of other fancy modules. Doesn't seem that unix_chkpwd handles any of that.
Yes, you're right. unix_chkpwd doesn't handle any of that and, in fact, was never intended to handle any of that -- and that's the entire point!
The entire design of PAM is, well, to have separate "modules" that are "pluggable" depending on how you need to handle "authentication" -- LDAP, OTP, Yubikeys, etc.
That is, the pam_unix module (which uses unix_chkpwd) is used when you enter in your (local user account's) password. If you're using something else -- LDAP or NIS or whatever -- for user accounts (i.e., your "passwd" database) there are separate (PAM) modules for that!
> Also, I still think it has to be a network service, ...
No, it really doesn't and, besides, there are other alternatives that would be much better to use instead of a network socket (such as a local UNIX socket, for one).
I really try not to make such remarks here on HN, but in this case it does seem that you have a fundamental misunderstanding of just how this stuff works (which is almost certainly why you're comment has been so downvoted).
Well I have been playing around with Wayland + Sway the last month. I like it but there are some caveats. For example I end up modularly picking the tools I want to use with it. So by default I think there is swaylock and then there is waylock but a few times I got back to my seat and the locking application had crashed, exposing my session to my co-workers. That ain't the desired effect. I resorted to physlock which swaps to tty and allows to disable things like echoing of kernel messages and sysrq. It has yet to crash. Also, all of this relies on PAM and according to the OpenBSD devs PAM is a mess.
What's the difference between a service that you send a string to and get a string back and a binary that you execute with a string argument and it prints a string argument back? I quite like the latter, as you're free from keeping state and thus have a smaller attack surface or potential to leak resources. Of course one difference is the execution environment, with a systemd service you can have it set up exactly as it should be so no changes to LD_LIBRARY_PATH &c can poke holes. I wonder if its socket activation feature can be used like a sort of CGI server - for each connection on a socket, run the binary in a controlled environment and connect its stdin/out to the socket, with stderr going to a log file.
You said it yourself. LD_LIBRARY_PATH, and a myriad other knobs.
Think of all the stuff you can do: mess with filehandles, signal handlers, chroot, resource limits, seccomp, capabilities, program arguments... and more appear over time. You can't defend yourself from things that didn't exist at the time the code was written.
Yeah but that means we need to have a way of securely running binaries, not move a simple program to a service. Something like taking a snapshot of the system at various stages during init and have the binary start in a known context. This would also help a lot for desktop users - many programs need to start in a specific environment and having the option to configure per-user per-program launch environment would help a lot.
> The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl. […] Who is responsible that it is air tight? Well, uh, nobody, really.
The people who designed the container/sarcophagus system. If it's not secure don't sell it as secure: see the difference between Linux containers and FreeBSD jails or Solaris zones.
> You can't even blame the applications for that. Let's say you want to build a good, secure application. How do you know what files your application will try to open? What syscalls it wants to call? Library and framework functions tend to not document that properly.
As a developer / software "engineer" it is your responsibility to know how the components you choose work and how much you can rely on them. When a structural engineer selects certain types of steel or concrete to be used in the construction of a bridge, it is their responsibility to know the characteristics of the raw materials.
When you ship a software product, enable SELinux or AppArmor and audit what it does: then ship an SELinux/AppArmor profiles.
See also OpenBSD's pledge(2) framework where the developer embeds system calls in their (C) code to promise to only access certain OS features:
> it is your responsibility to know how the components you choose work and how much you can rely on them
Unfortunately that is not always your choice and arguing against it most commonly costs you more than you like. My experience is quite often timelines get underestimated massively with software so you end up doing what you can with the resources given. If you pull this into the bridge metafore the bridge still stands but can collapse any time and as long as it doesn't collapse during rush hour and can be rebuilt before the next morning it's all fine.
A door with a broken lock still provides value. An online PDF converter on a website without SSL still provides value. I'm not saying security doesn't matter or that it shouldn't be a priority, but things don't need to be perfect to be useful. It makes sense to focus more on functionality than security in many areas. I host a bunch of web apps behind a firewall that aren't secure enough to expose to the public Internet. Even commercial ones.
Unfortunately, that's exactly the problem. If it didn't provide value, no one would use it and its lack of security would be a moot point.
The PDF converter is a great example. It's functional, people use it - to transfer potentially highly sensitive documents over an insecure connection. At that point, it provides value in the same way that a Nigerian prince provides banking services.
For you, no risk seems acceptable when it comes to security. That is generally not the case for most people, who will accept some level of risk (ie an insecure program) if it provides value to them. That's the whole point of risk management.
Pardon me for being pithy. We've all heard the adage about perfect security being a logical impossibility. I will say instead that if a tool doesn't have security proportionate to its risk factor integrated in its design, it doesn't work.
There are lots of decision points. Some people are deciding what to sell; others are deciding what to purchase. An organization that de-prioritizes security in what they produce could be punished by organizations that prioritize security in what they consume. In this way, decisions are shared to some extent. Consumers might be misled over the short-term, but at some point they become complicit.
If a datacenter continues to purchase a model of generator after five high-profile failures of that model, they can argue all they want that it is the only model on the market that meets their needs. But clearly uptime is not their biggest consideration.
> When you ship a software product, enable SELinux or AppArmor and audit what it does: then ship an SELinux/AppArmor profiles.
It's not just sysamins -- developers don't understand SELinux, either.
As a dev, I find least-privilege sandboxing (e.g., look at how sshd is structured) and capabilities (e.g. FreeBSD capsicum) somewhat non-intuitive, but at least tractable to understand. SELinux is a blackhole.
>> The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl. […] Who is responsible that it is air tight? Well, uh, nobody, really.
> The people who designed the container/sarcophagus system.
Original comment misses two nuances with the gripe.
1. Container creators and software creators are not the same people. So they can (and should!) have different goals and priorities. For containers, security and reliability. For software, make it work.
2. There is a nuanced, different interface between software-system and container-system, created by encapsulating a system-like environment inside the container. Much of what software does can be encapsulated. Which allows a more controlled (and much smaller) subset to exit the container.
> 1. Container creators and software creators are not the same people. So they can (and should!) have different goals and priorities. For containers, security and reliability. For software, make it work.
If the container system designers did their job properly it won't matter what the software does inside of it, as breakouts shouldn't be possible.
It has been a few years since I looked, but last time I checked there hasn't been a vulnerability in FreeBSD's jails code that allowed someone to break out. The closest to it was an issue with devfs that allowed 'tunnelling' out, but not with the actual Jails code itself:
One side has a huge footprint to secure (software-system), because it's every way software could ever need to interact with a local system.
The other can expose a much smaller one (container-system), because it only needs to include things one would have gone off-local-machine for (i.e. networking).
So from a security boundary, containers are basically a machine-internal firewall between programs and the host.
It's effectively impossible to ever provide security at the program-system interface, due to the surface area. (I.e. the SELinux problem)
The very concept of having a container (that internally simulates a system) creates fundamentally different opportunities that allow both (programs that work) and (security).
Whinging about whether or not current containers do a good or bad job of it is a less interesting, short term quibble.
Ideally all apps & libraries would ship their own selinux policies with a common framework for combining them & base layer for privilege sets (eg, "let me open any file the current user can open" or whatever). If that was the case then your concern wouldn't be an issue. You'd just say "I use libc sockets" and you'd inherit whatever file path permissions are necessary for that to work, as defined & exported by the libc in question.
But that's not a thing. So distros are attempting to add it later themselves, which is a disaster.
> Wouldn't it be great if the man page of getaddrinfo mentioned those (mine only mentions gai.conf)
That's a huge part of the problem. Was looking for a monograph to pass on my successor that wasn't so familiar with linux - modern linux - as in systemd, docker, nssswitch, pam_homed, network-manager is penetrable but more often than not I'm just looking at the c source in github and I am in my 30ies, starting in my teens running linux nonstop and I'm still throwing up my hands every other day...
Add time constraints and pressure to deliver (don't waste your time understanding this, just do xyz) and here we are - on the other hand it's not okay that you have to devote years of trail and error to get a comprehensive undestanding of the system.
I guess a nuclear power plant has at least a training plan and complete reference book - that still needs to be read and grokked and trained upon but modern linux is often just undestandable by reading the source if you hit a problem - for some projects like things in the freedesktop ecosystem and partly systemd even that is kind of difficult because i.e. this bug https://github.com/systemd/systemd/issues/19118 explains the problem - there is no overview documentation, no clear way to look up what's happening, not even a good way to introspect and it's several components that interact with each other fail subtly. I've chosen this one because I've also hit it, not because I want to blame systemd, which is not so bad there are much worse things out there but it's part of a trend to introduce complexity - not sure what is necessary complexity and what is unnecessary - went to uni in compsci without ever someone slapping out of the tar pit (http://curtclifton.net/papers/MoseleyMarks06a.pdf) in my face and maybe that's part of the problem.
Exactly. ‘ps ax’ on a fresh OpenBSD install gives you about 10-12 processes, each of whose purpose is either obvious or easy to discover.
Try doing the same thing on macOS or Ubuntu. Last time I looked carefully at the former I discovered, for example, daemons for classrooms (??) It’s a free-for-all.
I would like to emphatically point out that: no, you can't. You can't use strace to get this information.
Let's say you want to use seccomp to whitelist allowed syscalls.
Your code opens a file, uses stat to get the file size, then mallocs that many bytes and reads the file contents into the buffer.
Trivial program, right?
glibc will turn open into openat, stat into statx, malloc can become either nothing, or sbrk, or mmap, and maybe also munmap. If you hit an error, it will also call write to output an error message to stderr.
dietlibc will do open, stat, mmap.
This is also libc version dependent!
But can you at least use strace to know which files will be opened? No, not even that! Because the code may open some files only under certain circumstances. For example, localtime will open /etc/localtime -- but then it will cache the result for a while. /etc/localtime may be a symlink. If you construct a container you would also need the thing it points to.
What if you use a malloc that has explicit hugepage support?
Also note that seccomp and landlock are very much different from pledge and unveil. I am particularly appalled by landlock as compared to unveil. Go look up the landlock API if you don't believe me, and compare it to the unveil man page. If you thought the hoops seccomp makes you jump through are ridiculous, you haven't seen anything yet.
OpenBSD is not all gold either. pledge is not transitive. If your process pledges something but is allowed to execve something else, the process you exec is not bound by your pledge.
There is MUCH room for improvement all around.
However, the persisting idea that you can use a "training mode" or observe via strace to construct a whitelist IS WRONG and dangerously so. Unless you have 100% test coverage (and if you had, why do you still need a sandbox then?) you will miss error handling and optional code paths that you didn't enable in the configuration, and handling code for circumstances you didn't trigger and didn't foresee.
> Unless you have 100% test coverage (and if you had, why do you still need a sandbox then?)
100% test coverage does not guarantee absence of vulnerabilities in any way.
> you will miss error handling and optional code paths that you didn't enable in the configuration, and handling code for circumstances you didn't trigger and didn't foresee.
Having sandboxed many things, this is plain false.
Normal applications and daemons have no reason to call reboot() or stime() and so on in any obscure code path.
If they do, the sandbox should stop them and that's a feature and not a bug.
Their point about test coverage is that you don't know what syscalls will be made, or with what arguments, without running the program with enough coverage to find out.
This is probably the most well known issue with sandboxing - maintaining the sandbox. It's especially hard with seccomp, because you could upgrade your distro, or a dependency, and suddenly you're making a different system call.
>Also note that seccomp and landlock are very much different from pledge and unveil.
No, this is incorrect. They fundamentally do the same things. You're trying to say they're different because the API is different but that's completely missing the point. If you really prefer the API of them then you can go and use one of the emulations of pledge and unveil that have been built on top of seccomp and landock. They work because there isn't really anything special going on there.
Docker actually comes with a seccomp and an apparmor config that bans many things by default. while it is absolutely not noticeable unless you are trying something like docker in docker. There are so many syscall that shouldn't even be relevant to normal programs.
It's actually a bit more than that. The syscall docker allowed isn't really fixed. It is affected by what linux capabilities the container had granted. Like: if you whitelist the container about CAP_SYS_PTRACE, you probably also want ptrace(2) to be whitelisted. Instead of a all or nothing/your program will still break even cap added model.
Then decompile the binary, bust out gdb, and do some forensics. Or realize you're about to embark on a multi-decade mission Learn It All(TM) which equates to maybe learning the quirks of enough niches of the programming community to maybe not be surprised by what you find... Eh.. 40% of the time.
OpenBSD doesn't support syscalls from alternative libcs, so you reliably get consistent syscall behavior.
But I believe GP is mostly talking about pledge(2), a pretty easy way to implement common sets of seccomp-like restrictions, and unveil(2), an easy way to limit path visibility. These are OpenBSD security features that Linux does not have direct equivalents of.
> The problem is not so much that selinux is too complicated (it is as complicated as it needs to be), but that we all run software we don't understand.
I don't think these statements are meaningfully different. "too complicated" implies "...for humans to manage". Maybe that's sort of your point?
> You can't even blame the applications for that. Let's say you want to build a good, secure application. How do you know what files your application will try to open? What syscalls it wants to call? Library and framework functions tend to not document that properly.
Agreed. It's too burdensome for software developers to understand exactly what syscalls their program needs to make and the security implications of permitting those syscalls. It also doesn't help that Linux naming conventions and concepts are very counterintuitive (yes, dear Kernel Hacker, I'm sure they're very intuitive to you, but we lowly mortals struggle).
And unfortunately the SELinux policies are tightly coupled to the application such that you can't make SELinux policies the purview of a dedicated sysadmin expert and leave the appdev to the development teams. They have to collaborate which is swimming against the current of Conway's Law or else you make SELinux policies the responsibility of appdev and suffer their lack of expertise.
We had similar problems with operations in general, but containers largely solved this problem by allowing sysadmins to focus on a platform while developers focus on developing and operating the application. We need something similar for security. This is probably a rephrasing of your "sarcophagus" point?
> Obscure files like /etc/localtime, /etc/resolv.conf, /etc/ld.so.conf, /dev/zero ... how can you expect devs to build well documented and well sandboxable applications if they don't know which files their library functions will open?
Who the fuck invented that convention that fine-grained permissions must be file-based? It's insane. No, no developer will anticipate that he needs to read /etc/nsswitch.conf. No developer should. A developer should anticipate that the software needs permission to connect on network hosts.
As much as it is too granular, file based permissions aren't fine-grained enough. Asking to connect on random hosts is absurdly wide, most programs only need to connect in to a few of them, or connect to an user-supplied host (that can be a permission by itself).
Anyway, yes, the manpages should interlink better. What is a different issue.
> The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl
Reminder: containers are not meant to be security tools.
Fine-grained sandboxing (e.g. seccomp) is. And it can be layered upon OS-level VMs.
Additionally, bundling tons of stuff together with an application (like docker but also flatpak do) is not good for security. Same for static linking. They all increase the workload of updating vulnerable dependencies.
By multiplying the number of potentially-vulnerable copies of a library by the number of ways to detect the presence of said library and by the number of ways to upgrade said libraries when found.
Plus, it makes updating an application way more difficult: you need:
- the distribution to recompile and test all binaries using a vulnerable dependency [x]
- users to download and update all affected binaries
compared to:
- the distribution patching a single package
- users updating that package
Imagine if OpenSSL was statically linked: even the smallest fix would require users to update half of the OS. And that would discourage both maintainer and users from doing regular updates.
[x] And this is assuming that thousands of automated rebuilds are possible. In reality many statically linked languages encourage locking the versions of build dependencies, making it impossible to do just one backport of a patch.
I've been recently thinking along similar lines about libraries. In web dev, it's common practice when you have a problem to first look for a library that solves it for you.
I get the "don't reinvent the wheel" sentiment but, I think we take it too far sometimes. I've been looking at the source code for some dependencies at work lately, and many of them actually don't hold up to our own code quality standards. Kind of subjective, yes, but many of our dependencies would probably fail code review if actually reviewed by us.
Then when there is a bug in a dependency, nobody actually understands how it works and the code is often tucked away and not easily changed.
when it comes to JS libraries I'm usually a lot more interested in how good their testing is. Code quality can be very subjective in some areas. But if the dang library doesn't have a lot of good testing then I can't really trust any updates from it :/
Coming from marine and aerospace contracting, I'm always shocked by the cavalier attitude The Rest Of The Software World has about pulling in third party dependencies. "I have a problem to solve. Let's Google for: [problem] [language]. Aha! This library on GitHub has 12 bajillion stars! Pull that one down, build it, and YOLO!" No in-depth look into what other things the dependency does, how it increases the attack surface, what user data it gobbles up, what license it uses, what is the update cadence and how often do we need to pull from upstream, what is the contingency plan if it ends up not being suitable, nothing. Just git push and close that JIRA ticket!
Inadvertant bugs are the last thing I worry about in the javascript (node) ecosystem.
I'm more concerned with the fact that running anything requires transitive trust to hundreds or thousands of projects in node_modules, any of which can run scripts while being installed.
How many people out of a thousand would you trust running commands at your terminal?
Used to be, you could update it manually. Then NetworkManager (or whatever its successor is) came along and would change it back on you. Oh well, you could still edit it for a quick test.
Then I set up a Wireguard tunnel. And it changed /etc/resolv.conf. Oh, but an entry is missing, I'll just add that and test. Nope, read-only. But I'm root! What gives?
Turns out the wg-quick script mounts a file on top of /etc/resolv.conf! I didn't even know that was possible until I saw it. Nobody messes with wg-quick's resolv.conf, and that's final! Until some other tool ups the ante and gets code to undo that.
I have, in moments of desperation, `chattr +i /etc/resolv.conf` before. I understand the rationale behind managing that with a more robust service, but in my experience that's much more prone to the black-box effect.
>>The problem is not so much that selinux is too complicated (it is as complicated as it needs to be), but that we all run software we don't understand
There is truth to this, and it nails the fundamental asymmetry of the bad guys vs good guys in the security war. To program selinux you need to understand the software you run at the syscall level. and potentially have a deep understanding of its file usage, particularly if it’s doing ipc.
In general, I think that is a good goal. More understanding is more understanding and that is Good. In practice? I equate it to the problem of writing secure and robust code in C, I don’t know how good you have to be todo it and I basically assume that anyone who says they do is full of shit. I have contributed to the Linux kernel, I have decades of UNIX and specifically Linux experience as a software engineer, and I am still surprised when I fire up strace from time to time. You look at something like the recent Dirty Pipe bug, and I have a difficult time accepting that many people can fully grasp it all. The cost of a fairly simple system interface is all the subtlety and edge cases.
I view the container movement as a way of managing interactions. It’s really hard to manage dependencies in such a heterogenous systems. Containers simplify it all, at the cost of owning ones full dep tree. But with a good release process, one can keep on top of that and put less effort into the combinatorial growth when interacting with all the other dependencies other programs require.
containerization is less about security these days, and is being used to work around the broken package/dependency management situation on linux which is caused by the wild west of libraries that make little effort to assure any kind of forward/backwards compatibility. Given the fact that distro package maintainers cant maintain 30x different versions of any given package to satisfy all the applications stuck on a particular library version, means that larger applications which can't be constantly churning their code bases to replace foo(x,y) with foo_bar(y,x) are stuck shipping their own pinned version of a library.
So, I don't think anyone seriously thought that containers provided significantly more security than simply running an application and trusting the kernel syscall and filesystem permissions were bulletproof. More software layers are unlikely to improve security.
It's the forward march of progress. As things get more complex they also get more fragile and inscrutable, and harder to solve problems for.
Containers are a reaction to the flaws in the system. Fixing the root cause of those flaws is hard, because they're systemic. We would need to re-design everything to fix those flaws. Containers are a stop-gap measure that makes things easier to deal with.
The thing that I've learned about systems is they are all crap, but humans constantly munge them to keep them working. We don't end up with the best systems, we end up with systems easiest to munge. Every once in a while, a spasm of collective organization redesigns the system, and then we keep going with that one. It's evolution. So however crap our current system is, eventually it will change, probably for the better. It just takes a really long time.
This seems like something that WebAssembly (misnomer) is set to resolve, to some degree. Sandboxing capabilities is easy because all of the wasm module imports must be provided by the process that initializes the module, so you can view imports as a kind of list of "requests for capabilities".
I think that's leaning even more into the parents point.
We make a sandbox (in this case a very inefficient one) -- we assume the sandbox is the only way to deploy software, but mostly it's not because there will be times you need more than the sandbox or the software doesn't work, and then you're running "privileged" WASM routines.
The sandbox itself might even require some kind of leaky behaviour, such as accessing files or shared memory.
The wheel keeps turning. Some new sandbox will come along.
WASM will probably be always slower than native but call it very inefficient is unjustified imho considering the compiler needs to be fast enough to run in browser from what I have seen its around ~1.5x slower than native code with there still being many low hanging fruits to be optimized.
1.5x slower for a weak sandbox that's already hardware compromised (spectre) with no plans to ever be fixed, along with a questionable portability story (cool the instructions itself are portable, but there's no APIs. Which are, and always have been, the actual portability issue) does not exactly sound compelling. It makes sense on the web where the browser is already providing API portability & permission sandboxing, and all that's necessary is actually just instruction portability and protection against eg memory corruption (actual security sandboxing still being done via process isolation), but beyond that? Like desktop apps? Pretty much entirely useless.
And it doesn't address any of the issues here anyway. As soon as you bind libc to it (or any other system library), which isn't exactly unlikely, you're right back into the problem of not knowing what the selinux policy should be. And you still want selinux policies since WASM is only a memory sandbox, not a permission sandbox.
prefer not to get into a debate about the efficiencies of x vs y but sufficed to say: Computers are only getting negligibly faster over the last 10 years and our software has gotten overall quite considerably slower.
A runtime overhead that slows everything down to 1.5x, even with low hanging fruits is only going to accelerate this issue.
From a consumer perspective: We all act as if everyone has 8-16G of ram, because that's what we're used to, but the reality is that the majority of people have 2-4G of ram, even these days. That's not counting the anemic CPUs that are often inside awful cooling solutions.
From a server perspective: we outsource our Ops to cloud providers and pay a significant premium for computational speed, which means things like runtime overheads have direct costs.
The reason I called it inefficient is because it's not adding anything we don't have, it's just "another layer" with a large runtime overhead.
That talk makes the point that replacing the swiss cheese hardware security models with a software one can shift your performance. You run 1.5x slower, but you also run 1.5x faster by avoiding all the hardware checks. On the whole, you can theoretically run about the same speed, but with better security.
Even better, if WASM takes over the world, you can bake parts of it into hardware and remove a lot of other bottlenecks.
Why would a completely new compilation target be needed for that when “old” sandboxes can work just fine by hijacking syscalls? Sure, memory safety is somewhat improved by Wasm, but valgrind and the like are effectively giving you the same thing.
Wasm addresses memory and execution isolation from the rest of the system, however anything running in it maintains it's exploitable logic internally. If the application needs to read/write files, and someone exploits that logic, wasm won't help. The only thing that could is limiting the filesystem access in the first place and building secure software to begin with.
> "The problem is not so much that selinux is too complicated (it is as complicated as it needs to be)..."
Completely disagree, if the target users are advising each other to disable it... then the tool is definitely more complicated than it needs to be.
Using tools that don't hide any complexity are very painful to use. It feels like the creator doesn't care about the user and put no thought into the display of information or workflows of the tool.
Solving a complex problem with a perceived complex tool is relatively easy... don't hide any complexity.
Solving a complex problem with a perceived simple tool is difficult... hiding complexity at the right time, revealing functionality it based on the user's intentions and experience is not easy but greatly appreciated.
if the target users are advising each other to disable it... then the tool is definitely more complicated than it needs to be.
This does not follow. I you ask a 5-year to repaint the Sistine Chapel and the result is somehow less than stellar, are you similarly going to blame the paintbrushes used? There exists no situation where the task at hand is too complicated for the user?
A better version of your example would be a 5-year-old that cannot open the packaging to get to the pieces of the paintbrush that need to be assembled, in order to paint his first stroke... that's the (arguably) unneeded complexity of SELinux
Uhh isn't this precisely what the whole snap/flatpak movement is trying to fix?
I can talk about snaps, but at least they are offering a specific permission model which is usable for users to understand. i.e. Via the interfaces, I can know software x accesses my password-manager, home directory, camera etc... I can disconnect access to the given permission and have it enforced via the kernel and apparmor.
The applications themselves bundle only libraries that themselves are sandboxed/snapped.
>> It doesn’t help that no one wants to be told: “stop what you’re doing, spend a week to learn this other thing, make a tiny change, and then return to your real job”. They’re looking for solutions — preferably quick ones — and not a week’s worth of study.
If you're a system administrator running SeLinux why dont you already know SeLinux? It's not like some obscure command, it's a big part of the infrastructure you're running.
Indeed, that's possible and a good idea - as long as you're not talking about user shells [1] running as confined processes. It turns out that having an unconfined root shell is actually not a bad idea unless you're interested in extreme levels of security. My experience is that confining user shells is a significant hassle.
Running daemon processes etc. as confined root on the other hand is a good idea, but if you're going to the trouble of confining a root process you might as well just run it as a normal user instead if that's possible.
[1] Meaning shells used by a real person attached to ssh.
As an experienced RHEL admin, a few years ago I probably would have said this is very bad advice in any professional context, and you should spend the time to learn it because it will save you one day.
Now, I think my advice would be: Put everything in a container, and learn how to run Docker or Podman (or k8s) in a secure way (ie no root containers, be very careful with volume mounts, etc). Yes, they aren’t as mature as SELinux, but containers aim to provide many of the same benefits that SELinux does (and even more) except in a way that’s much easier to manage. Even better is that these container runtimes often come with SELinux and AppArmor policies out of the box on good Linux distros.
I agree that containers are easier to manage than SELinux, but still not easy enough. Linux provides many low level primitives for restricting applications and SELinux, Apparmor, Docker, Flatpak, and systemd all provide high level abstractions for those. But IMHO none of them really finds a sweet spot between flexibility and usability. `systemd-analyze security` for example lists 80 (!) different settings, even though some of them are very high level such as `ProtectSystem`.
Containers have made the conceptual shift from allow/deny to isolate/share. Somehow this feels better even though it is effecitvely the same.
I am still waiting for an abstraction that uses all the low level features and wraps them in a high-level interface that puts usability front and center.
I am not sure if this is even possible though because many applications are not built with sandboxing in mind. Adding another file somewhere on the system that needs to be accessed is not considered a breaking change by most. So maybe we need a more fundamental shift.
> I am still waiting for an abstraction that uses all the low level features and wraps them in a high-level interface that puts usability front and center.
Bubblewrap? FlatPak's sandboxing features are built on it already.
In my opinion, if an application requires more access than a docker container gives by default, then that application should probably just run in a VM. If the application needs more access because it needs to manage or control some hardware, then it should be tailored to the O/S and have a small core service that runs naked under systemd or whatever. If fancy management of that core service is needed, it can expose a port that an application can talk to, that's safely isolated in a docker container.
Maybe that's just daydreaming about a perfect world, but I agree that it should be easy to reason about the access levels.
What about an application like vim? It should be able to access any file I pass that is explicitly opened, but not much more. That is hard to express with current tooling.
You shouldn't run an application like vim in Docker. Unless your server is running Vim as some sort of service, in which case it should only open files in a volume you bound to it.
Vim is exactly the kind of application you would run in a virtual machine.
You could do it with a file picker API that opened an out-of-process file picker. I'm pretty sure this is how WinRT works. Would be tricky for a purely command line workflow, but very doable with a GUI.
Even file picker APIs are severely lacking with regards to any multi-file filetypes, where opening one file means having to subsequently access further additional files, too (but which files exactly cannot necessarily defined in advance because it depends on the contents of the initially opened file).
As a lightweight alternative to Docker-based (or any container-based) solutions: Try firejail. You can set up a directory that will be the "home" of the sandboxed application you're running, then you can do something like `firejail --private="${HOME}/my_firefox_jail" firefox`. There are built-in profiles for many applications already, and you can customize them (by adding `.local` files, not editing the existing `.profile` files). See the following link for details.
Convenience matter. Without it being automatically on everywhere, it doesn’t protect from much. Sure it can be good for the occasional random software you trust the least, but how many exploitable bugs were found in any of your completely trustable tools?
Also, afaik firejail runs as suid, making any possible escape much more serious.
> For a server, the process exposed to the outside world runs as an unprivileged user (unbound or nobody). The process is started by a separate process running as root (as explained by @Ferroin above). The starting process is never exposed to outside.
> The same is true for Firejail. By the time the unprivileged server process starts, Firejail is already sleeping.
And I think Docker has a similar problem as mentioned in the "warning" section in [2]:
> Warning: Anyone added to the docker group is root equivalent because they can use the docker run --privileged command to start containers with root privileges. For more information see [3] and [4].
I'd generally agree with this, but for production use (where it'll be CRI-O or ContainerD in all likelihood) it's worth noting that using containers does increase your attack surface.
We've had container breakout CVEs in the Kernel+ContainerD+CRI-O this year, and the ContainerD/CRI-O ones were not complex at all, so you do need to stay on top of patching as well as the other good practices mentioned.
I was about to argue with you until I read to your last point. Totally agree one of the great things about containers is indeed they integrate very well with various Mandatory Access Control implementations SELinux included and don't generally require specialized policies for each app.
AppArmor is great. It can do much of what SELinux does, but it uses plain configuration files as opposed to incredibly obscure and clunky filesystem-level hacks.
AppArmour assumes everything is allowed unless it is explicitly denied by policy, SELinux assumes everything is banned unless it is allowed by policy. This makes SELinux fundamentally more secure because unlike apparmour you cant forget to deny some access that leaves you wide open, instead you forget to allow things and your app breaks. It also makes SELinux more complicated to implement because you have to allow everything you need or your app breaks whereas apparmour lets you deny the things you're worried about and ignore everything else.
SELinux also supports interesting things like applying security levels and contexts to data, so you can have data that is only accessible to appropriatly cleared users in one department but not to people with similar clearances in other departments. Very few people use this stuff though.
Selinux uses extended attributes to store its labels which is a pretty standard way of associating metadata with objects, and is supported by most standard unix commands (via the -Z flag) so I'm not sure what you mean by "filesystem hacks". Fundamentally they are both implemented as security modules in the kernel and do pretty much the same thing, which is best depends on whether the added security of SELinux is worth it to you.
> AppArmour assumes everything is allowed unless it is explicitly denied by policy
No, AppArmor uses a whitelist approach as well for its profiles. "Deny" rules are used in exceptional cases.
> SELinux also supports interesting things like applying security levels and contexts to data, so you can have data that is only accessible to appropriatly cleared users in one department but not to people with similar clearances in other departments.
AppArmor has subprofiles and variables that can be used for this approach.
I was daemonizing a uvicorn (Python) process using systemd and could not get the Python script to write its log file anywhere. Not the current directory owned by the user, not elsewhere, and no errors showing up in the system logs.
After a few hours banging my head against the wall I discovered systemd's ReadWritePaths. Once I set that, it would log to any directory I specified.
It's the only service I've created where this has been necessary, and others write logs, so I have no idea why this one caused errors. If I did something wrong, let me know how this should be handled.
Most systemd services log to stdout and let systemd put it in the journal, or they log to syslog and let systemd put it in the journal. Most don’t write their own log files any more. I really recommend going that route because it is nice to be able to search the journal, especially if you provide extra metadata for each log message.
It sounds like you were using the DynamicUser=, ProtectSystem=, or ProtectHome= settings which can greatly limit where a service is allowed to read from and write to and thus require you to open specific directories back up. Instead of using ReadWritePaths=, you might prefer to set LogsDirectory=.
> It sounds like you were using the DynamicUser=, ProtectSystem=, or ProtectHome= settings which can greatly limit where a service is allowed to read from and write to
Thank you! I have DynamicUser set - I can't think why I added it here, perhaps no reason at all - and that is the difference to previous systemd services I've written. I did not know it had such an effect on behavior.
It is a nice security measure; it means that no other service can accidentally read or write to the same files. An older form of the security measure was to create new system users like “ftp” and “mail” so that the ftp daemon couldn’t be tricked into overwriting someone’s mail queue and so on. The downside is that this takes a certain amount of effort on the part of the system administrators and distros to arrange; the bookkeeping becomes a significant source of manual effort.
The DynamicUser= setting tells systemd to invent a new user on the spot, with an id that is not in use by any other service. You can add as many such services as you like¹ without having to do any extra work to keep /etc/passwd updated.
¹ as many as you like as long is it isn’t more than several thousand, unless you recompile systemd with larger limits.
How does this work when you want to log different things? Like if I had a web server, I wouldn't want the server log and access log to go to the same place. Sure, I guess I could send one to stdout and the other to stderr, but what if I have three things?
Maybe systemd/journald supports this use case by forcing you to adopt some sort of structured log format that you emit to stdout, but I don't think I want to have to conform to whatever journald wants me to do, unless it's standardized and well-supported among various logging tools.
Try `journalctl -o json` to see all the fields that get logged. Applications can add more.
So it very much means that a webserver logging to journald can already provide it all with fields for the vhost, status, URL, referer, and whatnot, and you can just search the log by that and don't need to do your own parsing.
Fair, but still annoying. The simple solution, that always works, is to write separate streams to separate files. At least journald supports JSON, and not some proprietary format, though I imagine the particular fields it supports might not be supported by other logging systems.
I'm just tired of all this added complexity, when I don't really see much of the benefit.
The point is that you can include arbitrary metadata of your choosing, in addition to the standard fields that journald keeps track of for you. For example, if you include the id of the logged–in user with every message, then when they complain about a bug you can query the journal to find exactly what they did without any noise from all your other users. And you don’t even have to pay a SaaS company for the privilege.
> I'm just tired of all this added complexity, when I don't really see much of the benefit.
There is a lot less for each application to implement on its own, such as log levels, timestamp format, log rotation or even just the paths where logs should go. Instead those details can be configured by the system administrator using a uniform interface across all applications.
Container approach is not always feasible, for example when you need to stay on bare metal. There are Linux applications where selinux definitely gets in your way and there are legitimate reasons to turn it off completely.
edit: I like how I'm getting down-voted for sharing factual information. No-one in high frequency trading employs containers because it doesn't help soft-realtime, low-latency goals. There are other performance-sensitive domains where containers aren't an option. Sometimes I feel like this place is turning into Reddit.
What cases can linux containers not handle? Containers can access GPUs, /dev/kvm, block devices... I'm having trouble thinking of anything they can't do. After all, they're just processes in a glorified chroot, not that different from processes on the linux host.
Ultra-low latency applications on Linux are definitely one domain were containers are a big no-no for performance reasons. In my field (HFT) you won't find any serious performance-sensitive deployments using containers. So that's one example.
A container is just namespaces and cgroups. You can still have your mellanox or solarflare zero-copy network stack for a containerised process.
That is, a containerised user space process can directly access the hardware, bypassing the kernel but isolated from other processes.
There are plenty of footguns for sure though but my experience working in HFT has been everyone is pretty elite skill-wise so sharp edges are not an issue in practice.
Not sure about need but I do remember it being a pain in the ass to run anything that expects to be managing docker inside docker. Things like self hosted gitlab ci where its trying to start up containers itself.
> ...anything that expects to be managing docker inside docker.
Now that's an interesting problem to have!
If you trust those tools and don't have untrusted users or untrusted code, you can sometimes just mount /var/run/docker.sock and use the VM/VPS/server's Docker directly. It is actually the approach that was used by excellent tools like Portainer, though it's also really risky as well.
Alternatively, you can try to just run Docker in Docker (DinD), which is a bit more tricky and the opinions there are split about whether to do it and when to do it: https://github.com/jpetazzo/dind
Of course, someone might also jump in and suggest that Docker is architecturally problematic (i don't care much, just want my containers to run, then again; i don't deal with untrusted code or any sort of multitenancy) and you should use Podman or another set of technologies, which is interesting advice but would necessitate other approaches.
In short, like with most technologies: Docker and OCI container in general get more messy as your requirements become more advanced. For the problems that they do solve easily (app packaging), they are pretty good, though!
One project in this space that looked quite promising to me is sysbox[0]. I've used them once for a gitlab runner set-up similar to what is described in their blog[1].
It's currently working great and I have not had any major crashes/incidents for at least the past 8 months.
Mounting the docker socket makes it trivial to escape to the host since you can just launch a privileged container or mount arbitrary files. Plus the daemon generally runs as root.
At that point you may want to just run it on the host, but basically you'd better trust whatever it is.
I've had quite some problems setting up DPDK and other device drivers inside of containers. It looks like it should be easy, yes, but then reality hits you in the face.
And when everything is in docker you can use SELinux again: A docker container is not allowed to access the host system or access the filesystem of other containers. That‘s the default. SELinux protection without any weird problems.
As a completely new admin, could you elaborate on "be very careful with volume mounts"?
In the k8s env I run, the filesystem is backed by ceph, which creates persistent volumes that the different pods can claim. Is it just "be careful not to mount docker.sock"?
With IllumOS I can agree on zones, I can partially agree on lpar on AiX, I can accept GNU/Linux limits with cgroups. Enlarging the attack surface to follow the current wannabe Gurgle mania... No. Like no in the recent past to full-stack virtualization on x86 to enrich VMWare, buy big desktops sold as powerful servers just to play matryoshka doll for someone else business...
Of course, it's hard to satisfy hw "subdivision" vs sw subdivision, in the past we have tried the hw path with blades for the small side, infiniband and co for the large side and both proved to be not very scalable but the issue is in actual OS design and can only be solved changed that: An OS mush be a single application, with internal namespacing, networking. We have seen it in various past OSes. It's about time to make them again accepting that the IBM model is good only for IBM, witch means for selling stuff to people having no choice but buy or ignoring something better can be done.
I'm guessing part of the problem infiniband switches get expensive AF when you're going for 32x100 or rack-level anything. And you're still bound to up to 200Gbit per link (and the cpu load that entails) or lower bandwidth bonded links, why is not as interesting as QPI.
Aside from purely guessing I'm interested in scalability pains war stories with infiniband too :-) as I'm looking into a 400GbE-everywhere datacenter build...
It's not a problem with infiniband but a problem of sw mutating needs vs actual hw assets: matching is hard. Hw cost money so anyone try to be moderately tight, looking at life expectancy and supposed sw changes in that timeframe, but hw and sw have different lifecycle. At a certain point in time sw infra change while hw is still the same, at another point in time available/current iron change but software on it remain the same.
In classic times that's just a matter of balancing both, carefully designing both, but in modern time developers just "produce code", operation is pushed aside, communications are constrained often by the policies and the result it's a mess => companies choose "the cloud" witch means outsource the issue to third parties. Those third parties count on the fact that even if anything change in the end a computer is always a computer so they can live on a changing world because if single companies have irregularly changing needs on average the rate of change is similar. Also on scale redundancy needs lower the costs and that's why we switch from classic big iron to desktop crap packed in racks. Now we start seeing more and more the limit of such model, and no solution is here.
Big iron is long lost, now way to expensive not much in absolute terms but in demand terms: try to sell a Power system, the customer look at specs and say "hu?! a couple of PCs from nearby grocery store are equally powerful!". Such iron modularity coming from classic mainframes (infiniband in the end is crylink changed a bit by SGI and others) is considered too expensive, on the other side of the spectrum small-board ARM super-blades tentative still results in fails.
Until we accept that hw and sw can't really be two separate world and both must be open to being able to interoperate on scale we will not find any really scalable solution.
Sw side it's not different: these days is the containers mania days because after many have suffered burn with full-stack virtualization on x86 they look for something with less overhead and equal design: hw-sw decoupled, possibility of run an infra you do not know buy pre-made images/containers from third parties who maintain them without needing to know your infra. The result again is an enormous amount of layers and giant attack surface no one really know how extended it is. Since coms are needed we punch holes here and there, and on the other side "safety features" arrive to compensate, making more complicatedness layers on complicatedness instead of admitting the basic issue: in sw terms our OSes are badly designed for current needs, they are a relic form a long lost past where they aren't even good but just cheaper alternatives to something better.
If we ever admit those errors probably we will be able to design modular iron and modern OSes marginalizing those issue, but so far no one want such a nightmarish adventure...
I would say it's not necessarily easier to manage, but it's much easier to understand, because containers create natural borders between the different software running on the machine.
Docker actually comes with an apparmor config that disallows you to interact with any files outside of the container. It's not a so strong guarantee compares to inode based approach (selinix), but is still much better than don't have one.
And if you enables uid mapping, your permission in the host system is just as high as nobody if there isn't a kernel exploit happened somewhere.
But the fact it is nobody is also problematic, because dir mount will be almost alwyas read-only now.
It comes with both an apparmor profile and an SELinux profile. What gets used depends on what's on the system... and I guess you need to actually set the flag on dockerd to enable SELinux.
I've always preferred apparmor. SELinux has always seemed radically more complex for very little benefit, unless you have a tightly constrained OS (like Android, where the VM does most of the work and every app has the same sort of security policy) or a team of admins working full time to maintain it (again, like Android).
Apparmor is weak in all the same ways SELinux is weak, at least in terms of the ways that actually matter - that is to say, a kernel exploit is the simplest way out of either. But anyone can write an apparmor profile for any program in an hour, and if you actually know wtf you're doing you can build very strong profiles and trivially test and maintain them.
SELinux is "good" in that if you are building a system like Android, great, consumers get very tight policies for free, and SELinux is ultimately more expressive. But I think 99% of companies looking at an LSM MAC should probably roll apparmor.
I think my main problems with AppArmor are the feature set isn't coherent across all distros that use it and it isn't integrated too well with audit tools. OpenSUSE ships distro patches that are a major hack to make ausearch properly show AA denials, while others need to look at dmesg and remember to disable printk ratelimit. The supported feature set is fun, the kernel mainline vs Ubuntu vs SUSE feature sets differ a lot. Ubuntu supports fine grained socket mediation (as in specific ops) & dbus. OpenSUSE supports some of the socket mediation, but as life has demonstrated some rules that work on Ubuntu are incorrectly downgraded and for instance all unix socket access gets blocked (though thanks to the AppArmor folks this got fixed now). Arch which I'm considering to be close to mainline has no dbus and no fine socket mediation. In theory one can use the same policy file on all systems, but the way it gets downgraded is not always predictable. At least the features are somewhat discoverable though /sys/kernel/security/apparmor tree, but I don't know what happens when say the kernel reports dbus mediation, but dbus-daemon was built without it. Then there are also some missing features, I think the abstract socket mediation is still not there. Lastly, the parser is super slow and memory hungry, especially when throwing some AARE heavy profiles at it. It happens that Ubuntu Core makes heavy use of apparmor profiles to lock down snaps on embedded Linux devices, and all the timings data that we collect show calling out to apparmor parser taking the largest chunk of the snap installation time.
FWIW mediation of dbus & network is already supported by SELinux in all implementations I've tried, but then SELinux has a separate set of warts. Eg. trying to use the same source policy and building it on hosts with different tooling/refpolicy. Using `optional_policy` doesn't cut it if types are not known at all, and as debugged with good folks from #selinux, the only way to work around this was using `ifndef` and passing defines to m4 interpreter.
IMHO most RedHat systems tend to be overengineered. Even the config file split is overengineered in RedHat compared to many other distros. And documentation, although existing, tends not to be detailed enough to understand how to deal with the systems. You need trial and error.
Well, unnecessary complexity full of YAGNI is exactly what the E in RHEL stands for. Red Hat products need to be complex enough so that somebody will pay for support. :)
> There’s nowhere on the system where you can view the policies and look up why something might or might not work.
I always thought I had to be missing something with SELinux because this is what it seemed like to me and that can't be right. My impression is that the documentation for SELinux is extensive in all of the areas that aren't affecting you but it's really hard to nail down exactly what the policies are, what labels are available, which labels you should be using, and generally how everything interacts. Is there a SELinux Wiki somewhere that I've missed that has a simple breakdown of each and every possible label with interactions? Some tool I can use to generate said list?
Which says that things (i.e. processes) with the httpd_t label are allowed to perform getattr, ioctl, lock, open, read, and search operations on directories with the httpd_t label.
ps -Z and ls -Z will then list the labels of processes, and the labels of files and directories. If your processes label does not have an allow rule for your file label for the correct action you are denied.
The only trick really is that when it reports a label as say "system_u:system_r:httpd_t:s0" thats 4 different colon separated fields and the only one you really care about is the type field "httpd_t", because thats what the rules are defined for (unless you're the NSA or doing Very Interesting things (the other fields are user, role, and security level, but the default Red Hat etc selinux policies dont use them))
There is no wiki with definitive list of labels because they are just strings and the policy writer can call them what they like, even the _t suffix is just a convention to denote the type.
I have to say, for sysadmin work I find this view sorta overblown.
My experience is with RHEL and derivatives, not Fedora, and for the most part those are very straightforward and while selling does rear it's head every once in a while, it's usually not that problematic to work around.
When you have something failing that you need to fix and isn't an upstream problem (i.e. your own app or some third party vendor), you set the system to permissive mode and let it run with the app long enough that you think you've hit all the use cases, and then you use ausearch and audit2allow to generate a module for it. And bi, while the selinux audit log entries (which ausearch shows) aren't super clear, you can learn to interpret the gist of them, and piping them to audit2why and audit2allow gives more context.
The real "trick" in the above is to make sure you run the system in permissive mode for a while. Otherwise you'll usually only see the first deny in a code path which the app will then fail on, and you won't see subsequent ones. Permissive mode means nothing is actually denied, but all thing that would be denials are logged as such.
For things that ship with the OS repos for RHEL and derivatives, my experience is that very rarely are there problems or things that are not obvious to fix (running errors through audit2why will tell you when there's a boolean to toggle to allow what you want, such as outbound tcp connections from the http context that is causing some PHP app to malfunction).
In my experience creating and applying new policies with audit2allow is almost always the wrong thing to do. Most SELinux issues are either a SELinux boolean that needs enabling, or SELinux file type that needs changing.
The few times where you do need to create new policies then audit2allow is useful but still requires you to review and understand the changes it's going to make before applying them.
> In my experience creating and applying new policies with audit2allow is almost always the wrong thing to do. Most SELinux issues are either a SELinux boolean that needs enabling, or SELinux file type that needs changing.
I think that's very dependent on what you're doing, and how supported the thing your doing is by the OS. We write a lot of custom software. We also end up installing a lot of applications that didn't ship with good (or any) rulesets, or that we packaged into RPMs ourselves. Using audit2allow (after first checking audit2why and booleans) has become standard enough in those cases to generate a ruleset that makes sense (either for immediate installation or adding to a package we're building) that my experience seems to be somewhat different than yours. Or maybe we just run a lot more servers and while the percentages are the same the total numbers look higher, I don't know.
There's also the cases where we basically threw our hands up and said screw it, run in permissive mode indefinitely, because the alternative it too annoying and error prone (I'm looking at you, Jira).
> The few times where you do need to create new policies then audit2allow is useful but still requires you to review and understand the changes it's going to make before applying them.
Yep, and I noted that I don't think the audit logs and autdit2allow output aren't actually all that hard to understand if you take the time to look.
Exactly this. I’ve run SELinux on for various workloads for over a decade and have barely ever had to touch audit2allow. And what I have done is codified in ansible so you can just copy and adjust for new cases.
> and then you use ausearch and audit2allow to generate a module for it. And bi, while the selinux audit log entries (which ausearch shows) aren't super clear,
That still sounds pretty complicated for most people. I mean, I know a lot of otherwise capable people who think NoScript is too much to manage and that's something I've heard repeated online many times. That's just clicks, and only when you actually need to allow something. I would think that if even the people who own websites where they write entire articles about linux find something over-complicated and poorly documented most computer users wont stand a chance.
This is a common misconception. No normal person ever becomes a sysadmin. In the cases where you think that has happened, they invariably fly their freak flag high soon after. ;)
The three paragraphs starting with "In the best-case scenario" address exactly what you're talking about. And in my brief (hobbyist) usage of CentOS 7 with trying to run some self-written software, what TFA says was exactly my experience.
As it turned out, the hardening provided by systemd (private tmp, restricted syscalls, etc) was sufficient, and easier to understand, for my use case.
Sure, but it also sounds like your usage isn't really sysadmin level, which is what I prefaced everything I said with to put it into that context.
I don't expect random users to run with selinux on and deal with problems for all the stuff they might do. I do think it's probably a good idea for system administrators, and if those sysadmins have problems getting some custom in-house software running through selinux, that seems like a good place for them to focus some learning to rectify that issue.
You can also write a stub policy which is permissive for all actions, with logging turned on. This means you can run just the one application in permissive mode, get a sense of its behavior, and then lock it down.
Like others here, I had a similar experience. I setup a simple minecraft server on a SELinux secured OS. So far, so good. I wanted to setup a systemd service to startup and shutdown the minecraft server.
After ~1 hour of work later, I came to the conclusion that I was going to disable SELinux. Another hour later, I disabled SELinux.
Much as the article mentions, there didn't seem to be much good help, especially w.r.t. learning what the incantations meant and how to use them properly outside of a narrow path. Similarly, I did not have any decent way of introspecting what was going on in there. The error messages were of the "you must google this to even have a remote chance of figuring out what it means at all."
I understand SELinux is probably designed for enterprise or organizational specialists and not for normies to touch. It just seemed a bit too extreme towards that end.
You hit the main issue with selinux: most documentation is aimed at policy developers and not end users. Your main issue was that you were trying to adapt the reference policy without knowing its inner workings.
I was in a similar situation. It always takes me way to long to find out that SELinux is responsible for a seemingly impossible to understand error behavior, until I give up, ask a colleague, he mentions SELinux and then it's suddenly obvious. So my conclusion is that the fail-behaviour is just not good. I would prefer my process to be killed with a nice clear error message somewhere. I think it all about giving the user the right information instead of failing silently.
Apparmor is -much- easier to write policies for; even more so if you use one of the ubuntu based distros that have decent rules already set up for most apps you'd use.
the linux fiefdoms have a serious UX problem. SELinux being a prime example. As the article articulates, no wonder why people just turn it off. If your subsystems are not consistent, discoverable, palpable, and most important logical your setting yourself up for lousy adoption. And just "reading the docs" does not solve this problem. Your subsystem does not get to consume my professional time slice.
The reason docker became the de facto entry point into containerization in yesteryear is because if you were dealing with 'containers' you were dealing with the 'docker' cli entry point. Everything you did with linux containers in the (mainstream) came from 'docker' and you can '--help' to your hearts content -or- google as much as you required with others that had the same shared experience with 'docker'. We've moved on in recent years but its important to remember the power of a well described, but imperfect interface.
SELinux has none of this mindshare. What is my canonical entrypoint to SELinux on any particular distro? There is none. I have to specifically know to install support packages for 'audit2allow' or 'audit2why' to do any reasonable troubleshooting on why a processes wont start. Why? Because any raw logs are so chocked with implementation details as a administrator I cannot make a real-world decision on what is broken on the system. Sysadmins do not start every day thinking about SELinux and memorizing its maze of tools and procedures. Something is starting to smell here...
For SELinux I need to know about, and sometimes explicitly install, half a dozen cli tools to administer selinux. Most of which don't follow any particular naming convention or entry point. I now need to learn a completely new markup for policy AND compile them AND install them using other esoteric tools . I need to explicitly refresh system state after making any changes, and return to my blunt 'audit2why' what-is-this tool to figure out if I did anything right.
The principles of SELinux are fine. The UX of SELinux in terms of getting shit done day to day is not.
when you acknowledge this, then continuing your advice with
setenforce 0
is a spectacularly bad idea. You can make individual domains permissive using
semanage permissive -a ${context}
and then run the failing test using only that permissive domain.
Of course, the story becomes much more complicated if the failing test requires type transitions -- but blanket advising people to put an entire PROD system in permissive mode is not a good idea.
SELinux has a horrible misfeature called dontaudit, that lets policies using it deny actions without any evidence being logged anywhere. Because of the existence of this, the only reliable way to know if a problem is being caused by SELinux is to temporarily disable it and see if the problem goes away.
After years of setting permissive on my personal machine I wanted to do the right thing and set it up properly. I watched the presentations and all seemed so reasonable. It's there to help you, audit2allow will help you out even further if you need to allow something. Look at how detailed those error messages are explaining the problem and the solutions.
I thought I was going crazy that things weren't working and there were no denials in the log. After much pulling of hair I finally found out about dontaudit, I felt so cheated, it wasn't even playing fair at that point.
For anyone else hitting this, you can use semodule to disable dontaudit in step 4 [1]
Not only that, but the official policies rely on dontaudit rules because there are known policy violations that are in fact harmless (like processes scanning entire directories, or attempting to open a file in various locations). So disabling dontaudit means your logs will get cluttered with non-related error messages that were up to then silently ignored.
I see a few obvious areas of improvement, all of which are "implementation details" and don't undermine the core principles of SELinux.
> The SELinux denial audit log messages are too vague. You’re told that a label was denied reading from another label. Okay, what do those labels mean? Which programs? Which files, sockets, or whatever?
Improve logging in the SELinux system. Clearly it was able to map the program/files to labels, why aren't those programs/files provided in the error message?
> There’s nowhere on the system where you can view the policies and look up why something might or might not work.
Improve SELinux's policy analysis tools. As it is, it appears `seinfo` and `sesearch` provide too much detail and are impossible for non-experts to use. Are there alternatives that any sysadmin can use?
> A package’s SELinux policy is — most often — not part of the package but is installed as part of a larger meta package containing tens of thousands of policies (e.g., the Fedora Linux project’s selinux-policy mega-package).
Policy should be included with the software package (or have a clear, computer-resolvable link to the package the policy applies to).
Seems to me that SELinux is an afterthought in most distributions, which I can understand in volunteer/community-driven distributions, less in Enterprise ones. How does RHEL do it?
--
However, the fact that these problems remain decades after the introduction of SELinux (2000!), indicate a lack of will by the Enterprise Linux/Security industry. Considering its origins at NSA, which I'm sure is willing to train its sysadmins and not bothered by its obtuseness, I'm not surprised.
Tangentially, SELinux vs AppArmor vs Tomoyo demonstrates a fundamental challenge of the Linux kernel development model, where the kernel is developed independent of userspace, but promises never to break userspace.
This leads to everlasting APIs that can never get obsoleted, and on which "New and Improved" API replacements accrete, and then users get confused about which ones to use, and support effort gets divided.
I blame Redhat squarely for not mandating a document file detailing these systems, these contexts, these users.
Redhat has co-opted and has become an IBM of consulting firm: use obscured notation of an error code as a corporate revenue flow.
This is not a knock against SELinux nor against the inventor (NSA) thereof.
SELinux is a great privilege boundary setter at an extreme granulated level. SELinux is superior over many other system audit approaches in all area but one: that one is ease of use.
Another fault is that the Buy-ins of SELinux was placed upon the group of distro/package maintainers instead of upon the developer.
Another aspect on the ease of use is that There are no auto generation of SELinux policy given one executable file. No tools have attempted to adequately scan the code or object files for such its needed audit policy file. But this mapping of code to resource is often a skimped step of software development.
Unlike the development of apps for a GUI/WindowsManager environment, SELinux is NOT mandated by its own library to say “thou shall use all of me” or you don’t get go to heaven (or be seen on the computer).
> Red Hat Enterprise Linux (RHEL) has some of the most accessible documentation on creating custom policies.
Red Hat is the Pinterest of Google searches for Linux problems. You’ll often find a page where someone is describing you problem but the resolution is locked behind a subscription login.
Set SELinux to permissive for your personal systems at your own relative risk.
Never disable it, you'll end up in a situation where nothing is labeled and relevant policy adjustments probably don't exist. Making re-enabling it later a nightmare.
I'd argue it's particularly advantageous for desktops where untrusted software is a more common occurrence... and other controls (eg: network isolation) aren't as robust.
For business use, it's probably worth some investment. Any serious compliance program will probably want an explanation for why you aren't using it/equivalent.
It's not some amorphous thing, it can be learned and handled. I used to routinely disable it but I haven't in years
There are clearly defined objects and areas of responsibility. It's just buried under piles of documentation
semanage and all of the other tools are indeed cryptic, but you can generally get by with two steps:
- Identifying triggered hits
- restore contexts / update the policy as appropriate for this use case
I won't say it's easy, an employer of mine paid for two weeks of training that focused a lot specifically on this.
A lot of it I can't even articulate that well, I've just developed a set of patterns
It's definitely a bit of a problem that it takes as much research to command as it does... but I don't believe it's so grand.
The distribution provided policies offer some sane defaults, smaller shops like this are less likely to be writing in-house software breaking the mold.
When they do run into a problem though it's bigger, because unless their people have been doing enterprise-y things, they don't really get the necessary exposure
It's been years since I was responsible for administering Linux Servers, but half the issues I ran into were generally fixed with `restorecon`.
I used to be that "never disable SELinux" person, but I've since become a bit more pragmatic. But I still think you should at least spend a bit of time trying to get it to work before defaulting to `setenforce 0`.
I'm a Detection and Response Engineer now, Analyst for several years prior, and I've seen several incidents that would have been stopped by SELinux. Almost all of them were web shells or web app based RCEs. E.g. https://nvd.nist.gov/vuln/detail/cve-2019-3396
Restoring the defined contexts is the appropriate method -- the policy modifications to make $service work should be well defined at the time of integration, never a surprise.
If something is suddenly breaking because it needs to break a new policy, that's a problem with the processes being followed or the software; not SELinux.
After using CentOS for years, I have come to establish a debugging rule: If things just don't make sense, check if you're fighting SELinux policy. It's just one of those things that you beat your head on the desk after crashing into multiple brick walls that eventually works its way into your debugging process.
I guess it just shows how effective SELinux is if it is preventing the admin from doing something. /s
> Yes, it has a learning curve, but it's worth it.
That depends wholly on the circumstances in which those tradeoffs must be considered - additional security vs ease of use.
For people who don't work in banks and whatnot, the choice is probably rather simple to make, they'll just prefer to do whatever will let their code and software run as they want (be it permissive mode or disabling it).
In those circumstances, SELinux can be little more than a nuisance and a roadblock, something that they didn't ask for and don't want or care for.
When security is paramount, however, things are a bit different.
> That depends wholly on the circumstances in which those tradeoffs must be considered - additional security vs ease of use.
Of course, horses for courses. That's always implied, and I always keep that in mind.
However, if there's a case which needs that level of security, telling "Nah, just disable it, we'll do containers instead" is wrong at best, very dangerous at worst.
Also, I find having a background knowledge about these technologies beneficial. Having more tools under one's belt is never a disadvantage. It'll give you abilities in the best case, and provide you perspective in the worst.
> However, if there's a case which needs that level of security, telling "Nah, just disable it, we'll do containers instead" is wrong at best, very dangerous at worst.
Agreed, though partially. Those two technologies don't have to be mutually exclusive, though they sometimes are considered to be (e.g. working around the "system limitations" by using containers, when both could work in unison).
There are cases where you'll want containers AND SELinux/AppArmor. There are cases where you'll want full blown VMs. And even in regards to containers, there are multiple classes of technologies to consider: LXC/LXD, jails (BSD), containerd/Podman, Firecracker/gVisor and so on. And also approaches, like rootless containers, user remapping etc.
My point largely is that people will try to "get things done", everything that impedes them from doing that be damned (in their eyes)!
> Agreed, though partially. Those two technologies don't have to be mutually exclusive.
No, of course. I only objected using containers instead of SELinux / AppArmor. I don't consider containers as "Security Layer" per se. The situation is just a side effect in my eyes, and if you need proper security, you need proper hardening. Solely and blindly trusting secondary effects for security is wrong and dangerous.
It is _entirely_ unmanageable for personal usage. The official documentation that the OP found and calls some of the most accessible doesn't teach you how to write a custom policy.
Let me just assure everyone that it's not broken by design. Many of us use SElinux to harden systems and services on a regular.
But that said, it's not user friendly either. It's definitely a skill I list on my CV. Either learn it and feel better about it, or set it to permissive mode and live with that.
I agree with mindwok, SElinux defaults from distro + containers work just fine. Rarely any policy issues, and you get the isolation of containers on top of that. So I wouldn't set SElinux into permissive mode just to use containers, I'd use both. Which I currently am doing on my container host at home for example.
Edit: Today's press[1] have an excellent example of a vulnerability that should be stopped by SElinux default policies. Look at NetworkManager/dispatcher.d as an example, semange fcontext -l | grep 'NetworkManager/dispatcher' and you'll see it only has the exec context on that one directory.
So even if you could exploit a directory traversal and place malicious scripts in other places, it would not be allowed to execute them.
The question is: what good is a security system that tells me to "just allow whatever the problem is", once a problem comes up? Without being able to see the specifics of a request and blindly following the whitelisting I could end up allowing some hacked module access to critical files without knowing. Opening the system to exactly the dangers I'm seeking to stop.
Operating system security was important back before VMs became cheap. At this point you are really wasting time if you invest significant effort in host security. It's the new network security.
Separate concerns into VMs and network connections separated by strong authentication, authorization and encryption, and practice least privilege. It doesn't matter if your OS or containers are made of security swiss cheese as long as compromising one thing doesn't lead to compromising another. For containers you need Firecracker VMs; there is no other strong isolation for Linux containers, everything else is pretty easy to pop.
I’ve heard several high-level security engineers take this stance in organizations with stringent security requirements. There may be some truth here when prioritizing work, but I have yet to see strong authn/authz prevent red teams from wrecking havoc.
It’s too complex. We like to think of systems as being fairly constrained, but in practice I am not convinced that is true. Once you start adding other business functions to your environment, like observability, you can introduce non-obvious gaps in your policies.
Would having OS security completely prevent an attacker from wrecking havoc? Probably not. But it would make it much more difficult. It can also protect you from container/VM escapes.
Are SELinux policies complex? Yes, but they are constrained to a host’s system and are yet another layer of defense that can be used to secure software.
I’ve seen again and again how an SELinux policy can prevent an exploit from causing harm. It can also be an effective and quick way to mitigate risks from a new 0-day being posted while software teams work on patching.
I am fairly convinced, or bullish, that making SELinux easier to configure is a solvable problem. The biggest hurdle to it is this puzzling movement within security to avoid OS-level controls and instead focus on network-level controls. It puzzles me why a well-resourced org thinks you can only invest in one or the other.
This article is a prime example of why one should think about the content of a page before accepting it's premise.
I would argue that no solution in and of itself is complete.
I hope someday we can get away from the idea that containerization is the complete solution.
I have been a system admin for over 20 years, and it seems we still have not been able to get the concepts of defense in depth, and specifically many layers of defense on each level to soak in.
I see multiple people espousing the belief (in the comments on this article at least) that containerization has solved this issue and SELinux is no longer needed.
I can think of a certain chaos goose (Ian Coldwater) who has been providing plenty of reasons why it's bad to trust containers alone for years.
In my personal experience SELinux has stopped attacks in their tracks when a 0 day hit, until a patch from upstream became available and is an invaluable tool for blue team in general.
It is not the most intuitive of tools, but most truly powerful tools are not intuitive, that seems to be a common trade off.
SELinux policy can (and does, by default, on rhel-like systems) enforce the security guarantees the container frameworks provide.
This is very much an optimal scenario for SELinux when you have some macro-level policy restriction and you want to enforce it.
SELinux is just not very well matched for the usual scenario of trying to graft a security policy on software and a system admin that is completely unaware of it.
This means people get frustrated when an Apache server can’t read some files in some folder arbitrarily set in the config, because there never was any policy of the Apache server only being able to read files with the right security context, that was never part of the HTTPD sever documentation, it’s a default system policy called selinux-policy-targeted that originates with Fedora.
Now if you run the same HTTP server in a container, you have first class concept of a volume, and the container framework is aware of that and can label files with the right context as needed; in fact with Podman it will even use MLS to prevent cross-container contamination.
So IMO containers are not in any way incompatible with SELinux, they are in fact a great abstraction match and they work great together. There’s been many container escapes that were prevented by SELinux policy.
How can I trust some opaque policy I cannot see? Not just trust that it is well intentioned not evil, (Oracle Linux?), but that it does what it says. And does what I need and that the opaque developers have correctly envisioned what I need to do.
Security by obscurity is worse than none at all. Security by buying an "Enterprise" product is worse than that.
What SELinux says is "real MAC security is super complicated, but don't worry we've taken care of all that for you, but you can't understand it or change anything". What I say is "easy peasy; if you access the machine you're root. Make sure that either it doesn't happen or it doesn't matter".
That is not true. SELinux is not hard to understand nor change. Most of the issues you might encounter are simply solved by changing a label in a file or directory. It is just not straightforward to debug and understand if you don't know it is there. I think most people who complain about it are multi distros/OS users and just only realize after losing precious time that an issue is selinux related.
really? I don't mean understand how to apply a new label. I mean understand what the policies are and how they work, be able to create new ones that apply to you, and verify that the ones given to you by the distro are correct for your use. You're saying this is not hard to understand: https://github.com/SELinuxProject/refpolicy/blob/master/poli... ?
Otherwise you are blindly applying some black box.
But what am I, as the sysadmin, supposed to label things? Even if I know that I probably need to apply some labels, how do I find out what the policies are? What labels can interact with this program? You can turn to Google and find some incantations in a blog or forum post from ten years ago, but you can‘t easily see which labels a program is allowed to interact with or not. (If you even know to ask that question.)
Fedora comes with manpages. So for instance, say you want to place http pages outside of /var/www. Fedora comes with the selinux-policy-doc package, which gives you the httpd_selinux manpage:
It's a tad dense, but in the Sharing Files you can find a section explaining what context is used and for what. You can also look at the existing context, eg:
$ ls -laZ /var/www
total 4
drwxr-xr-x. 4 root root system_u:object_r:httpd_sys_content_t:s0 33 mar 17 17:32 .
drwxr-xr-x. 21 root root system_u:object_r:var_t:s0 4096 abr 18 10:57 ..
drwxr-xr-x. 2 root root system_u:object_r:httpd_sys_script_exec_t:s0 6 mar 17 17:32 cgi-bin
drwxr-xr-x. 2 root root system_u:object_r:httpd_sys_content_t:s0 6 mar 17 17:32 html
That shows the list of what labels are allowed to do what to which labels, so run ps -Z to get the label of your process and grep the sesearch output for it, and that will tell you what labels it's allowed to interact with and in what ways.
selinux is generally pretty straightforward, but it has a reputation for being impenetrable which puts people of trying to learn it.
Because plenty of unscrupulous ISPs and governments log unencrypted web traffic. Some ISPs use that information for advertising, or inject custom javascript into unencrypted web pages their customers visit for who knows what reason.
HTTPS is free and stops most of these tricks from working. You can still sniff the domain name via DNS or SNI, but HTTPS blocks attackers from seeing the URL you're visiting, or seeing the traffic the server sends back. It also stops people from doing MITM attacks on the connection.
Its 2022. HTTPS is fast, it works great and its free. Use it everywhere.
Also technology wise its not 2022 everywhere, if you know what i mean.
Many poorer nations suffer from inferior infrastructure, which could mean slower loading / processing times for https based webpages.
I remember reading an article along these lines a few years ago when arguing against using https absolutely everywhere.
Nope. SSL was pretty okay performance-wise 15 years ago.
So no, we don't know what you mean. 'Poorer' infrastructure excuse is void.
If we cared about performance we'd get rid of almost all javascript, not https.
> Because plenty of unscrupulous ISPs and governments log unencrypted web traffic.
Ok. Not relevant here. I don't care if my ISP knows I'm learning SELinux. And they would know that even with HTTPS.
> Some ISPs use that information for advertising, or inject custom javascript into unencrypted web pages
Sounds like an ISP problem. Buy service from a better one. No need to impose SSL on the website for this.
> HTTPS is free
Absolutely not. It takes time and effort to set up. And then maintain forever. Plus you're introducing an unnecessary dependency on multiple third parties.
Thats not always true, thanks to awful middleware boxes. Especially inside corporate networks.
Also TLS is required for http2 - which is much faster than HTTP1.1 in almost every case.
> It takes time and effort to set up. And then maintain forever.
People level the exact same accusations at SELinux. Some would say SELinux doesn't add a lot of security, and its a pain to set up and maintain.
HTTPS is much easier to set up than SELinux thanks to letsencrypt and certbot. And it adds real security for your users. Its weird to promote SELinux but not bother with TLS.
But all these complaints are moot in this case anyway because this website is hosted by github pages[1]. Github will do all the heavy lifting to make HTTPS work. The owner just needs to set DNS up:
Oh yeah and you know what I was saying about http2 being faster than http1.1? Github enables http2 automatically on github pages. But only, y'know, if you set your site up correctly.
[1] At least according to the invalid cert warnings I'm seeing
It is a problem to solve over the wire, because we should assume at this point that bytes over the internet are monitored. Both in the east (Great Firewall of China) and the west (NSA + Snowden). I don't want my ISP or my government knowing which websites I'm visiting, what apps I open on my phone or, really anything else about me without a court order. Insecure-by-default is a bad default.
To protect users, traffic over the internet should always be encrypted. Even if you don't care, as a user I care. And thankfully the people in charge of our software, protocols and ecosystem are slowly forcing the issue on our behalf:
- Unencrypted http will eventually be deprecated from web browsers. (Newer browser features are already banned in insecure contexts).
- iOS forbids applications from making unencrypted HTTP connections.
- The IETF more or less requires new protocols to be secure.
We'll see more of this over time. Its a good thing. Get on board.
Lots of things are free. Doesn't mean you need to do them. The burden of justification lies on the people advocating for adding complexity and dependency to a simple system.
Someone in the middle could teach you a variant of the material in hopes of creating more insecure servers out there. Or perhaps link to an "official companion app|pdf|exe" on the page.
Longtime SA, constantly ribbed or outright insulted for my absolute hatred of SELinux. Yes, I just turn it off first thing. Well, used to when stuck with RHEL environments.
If you want people to adopt and really use something at large, you need to make it simple, explain when and why it's useful, and document it well. SELinux doesn't do any of that particularly well.
And now that most things are effectively running on a vm inside a vm, the use case seems dwindling even further.
Security can't be made simple. At the core what SELinux does is simply enforcing a policy. A list of "Should X do Y to Z?".
This is something that can't be implemented with just file permissions. What SELinux does is sandboxing every process it controls.
For instance, my cupsd is running as root. It's accepting connections on port 631.
If it started accepting connections on port 22, that's probably a bad thing. If it started messing around with /etc/passwd and /etc/shadow, that's probably not good either. It probably also has no business with my GPG keys, or /dev/sda.
But under the Unix permission model, if it runs as root, there's nothing you can do about it. It's root, it gets the entire system. If it has a hole, that's that.
SELinux first applies labels to stuff, then defines what actions can be done on what. So under SELinux there's a rule that says "A process with the label "cupsd_t can listen on a port labeled ipp_port_t". That's what lets it listen there, and absent a rule that allows it to listen on the ssh port, it's denied even if running as root.
So with a well implemented policy, if somebody exploits your cupsd, they can waste your paper. But despite it running as root they can't run a shell, can't open ports, can't mess with your password database, and can't install its own kernel driver. With the right rules, the attacker can find themselves at a dead end, where there's nothing useful they can do.
You can to some extent do this kind of thing with containers, but it really amounts to doing the same kind of work in the end to a large extent. If your container's security is at all useful you'll have to tell the system what should the cupsd container be able to do, in a very precise manner, and not just "eh, it can have root access to everything". At that point you've got another SELinux-like system to deal with.
That said, it seems like the more basic use cases can be accomplished more simply. For example, using capabilities and proper groups instead of running things as root has been the recommendation for a while now. In that scenario, cupsd would never be able to do much to the system if compromised in the first place.
A proper iptables/nftables default deny all incoming is pretty simple to setup, perhaps less so to maintain as services are added. I find the separation of concerns and maintenance a lot easier to reason about, ymmv.
I agree that SELinux is largely unusable in the real-world (especially for custom apps). I've had better experience with path based MACs like tomoyo and apparmor. Firejail is also great for end users who want to safely do online banking and surf the web at random on the same machine.
IMPO, this sort of isolation is the future of endpoint security. Linux has seccomp (to filter syscalls), landlock (to limit filesystem access), and other ways that devs can build these restrictions into their source code (no external MAC needed). I've lost count of the major and minor MACs that Linux offers now. OpenBSD has pledge and unveil that basically allow the same sort of thing.
I would not consider docker/podman as equivalent. They are great for bundling and running apps, but they are not doing mandatory access control. So be careful if you consider docker as a replacement for SELinux... it's not the same thing at all.
Out of all the tools you mentioned, pledge and unveil are the most pleasant to use from a developer and operator's perspective. I'm hoping something something similar will arrive in Linux without it becoming xkcd 927.
The capability to do something like that already exists in Linux, just nobody bothers to implement it at the application level because pledge and unveil are actually just more terrible hacks and are only really suitable for packages that are built into the system. In real syadmin-land, nobody wants to recompile applications just to change some security settings.
Furthermore any of these things that are bolting more ACL or ACL-like restrictions onto Unix permissions are bound to eventually end up with the same problems as SELinux.
Honestly I thought this was going to be about the problems of fully utilizing SELinux. There are bits of SELinux which are just missing - like a shared network policy server, or the various concepts of transiting SELinux contexts across things like network links to provide network-wide security.
SELinux is one of those things that makes total sense to me, but the implementation just leaves so much to be desired - i.e. basically no practical usages of "users" in an SELinux context even though the concept is there.
I say this, because it's very hard to write the policy yourself at the beginning. So let the system run for a week and then use the tools to build the policy for you.
I wish it's easier to sit down and write it, but it's not atm.
How does that work? If you run selinux without a policy, you also get no type transitions and no file labels. That means that after a week, you only have a list of things your system does, with every action coupled to the init_t domain (or possibly even kernel_t), and file access targeted at unlabeled_t (unmanaged_t?).
At the bare minimum, you need a base policy that already contains separate type contexts for each of your services, file contexts to apply to the service daemon binaries, and type transitions so that each service gets run in its proper context. If your daemons run in the wrong context, then immortalizing that in policy with e.g. audit2allow only deepens the swamp you're in.
I don't think there's any practical way to build an selinux policy from scratch by just running your system and recording it, even if the tools allow you to do that. You can only start from your OS' reference policy or indeed write the initial base policy by hand.
Seatbelts in cars are also constantly getting in my way. Why shouldn't I just cut them off?!
I think the problem is, that people don't understand SELinux. And containers are not a solution, because I can exploit a vulnerability and escape the container. But if I have SELinux running, then me escaping a container would not cause any harm.
There is no good tutorial for SELinux and most people don't invest the time to understand it.
For anyone curious or learning about SELinux, I highly recommend the SELinux Coloring Book [1] as an overview of core concepts. It doesn't go into any of the commands or mechanics of interacting with SELinux, but it has some simple and clear examples of objects, labels, and enforcement types.
I've always found that the concepts are not the difficult part of SELinux. Instead it's the complexity of the configuration and tooling which is always a headache.
> I've always found that the concepts are not the difficult part of SELinux.
There’s tons of good information on this part.
> Instead it's the complexity of the configuration and tooling which is always a headache.
There’s practically nothing on this. `man selinux` lists some of the tools (last updated in 2005) in “see also” section. On Fedora, half the referenced man pages are missing, though. You’ll need to read a book (or a book’s worth of man pages) to get anywhere.
The gist of the problems is that the policies are not transparent (probably due to security reasons?) to the user/admin, did I get that right? And that there's a difference in enforcement between user policies and policies provided by the distribution?
All this sounds to me like a good idea that got lost in the implementation.
If you have no way to look up which policies are in effect and what is labeled and how, something that even Windows 10 gets right, then yeah. SELinux is a toxic mess.
> The gist of the problems is that the policies are not transparent
Yes.
> (probably due to security reasons?) to the user/admin, did I get that right?
No, that isn’t the reason. They’re open source. The policies are, simply put, compiled binary blobs from a comprehensive set of allow-rules and label-path definitions. The complexity of analyzing and difficulty in overriding these complex rulesets is the problem. There are tools for interacting with the system, but good luck figuring out what they’re even called. All the tools are optimized for the ruleset developers and not the sysadmins that have to play by the rules (and are never told what the rules are).
Answering the question, “What do I need to label files that should be read by program X?” is waaay to hard. You’re expected to put your files in certain locations and then some things work out-of-the-box. However, there’s no documentation on where you’re supposed to put the files.
Sure, /var/www is where you put your website files by convention. Apache can read from that directory. But what if you put it in /var/web? You can relabel the directory, but it’s not at all apperent why everything grinds to a halt when you do. The /var/www dir is owned and readable by the apache user, but Apache still says complains it can’t read it. That’s the SELinux MAC in action blocking you from doing something totally normal just because you didn’t follow the strict rulesets (that no one told you about).
> There are tools for interacting with the system, but good luck figuring out what they’re even called.
They are mentionned in the selinux documentation of your distro.
> Answering the question, “What do I need to label files that should be read by program X?” is waaay to hard. You’re expected to put your files in certain locations and then some things work out-of-the-box. However, there’s no documentation on where you’re supposed to put the files.
The default directories are usually mentionned in the manuals of those programs. If you are using an selinux based distro, you know that if you don't use those default dirs you will have to label your custom dir. This is not hard to understand nor to do.
They make it so that you usually don't have too. You run into selinux issues usually only when you:
- run some server stuff
- try to serve a non default directory
And 99.99% of the time you will usually understand the issue from the log file and just label the correct files/directories. All this without having to change a policy. Policy creation/modification is pretty much the job of a package maintainer only.
I have been using Fedora daily for something like 7 years and have never had to write/modify a policy. The article originally linked is about some guy who do tests and run beta distros on prod and fiddle with his system by blindly copy/pasting stuff he sees in forums without really trying to understand what he does, then complain about his distro breaking on update. I don't think you can ever render your distro unbootable unless you do some very stupid things. And Fedora do not steer you to do those stupid things.
There are just so many policies it is not practical to list them all if you don't need them.
Only complain I would say is that sealert is not installed by default so if you want to have desktop notifications about an selinux issue you need to install it manually. I'd prefer it to be just a toggle in settings defaulting to no to decide to show or not show those notifications.
> Now, most sysadmins are probably not aware of SELinux nor its policy set until they can’t achieve something because it violates the policies.
First, many applications come with SELinux policies. For the ones that don't, users will need to create a pipeline. When I've written custom apps I usually have a "soak test" pipeline whereby the application runs through all of its commands and configuration and we capture denials, produce a policy with audit2allow and then review the syscalls to ensure they match what we'd expect from the application.
If you're logging into a box or receiving an OS image that has SELinux turned on and you don't know how to query, that bit is just a lack of knowledge. It's a very simple command and there's a finite number of modes.
The wider problem is that enterprises beyond FAANG have a very tough time perceiving systems as a product or systems engineers as anything beyond advanced troubleshooters. That's why you get these huge OS bakeries that ship out a cookie cutter system that everyone must use, or else. This is all usually motivated by compliance.
On top of that, while at enterprises I've noticed that they'll teach all the courses in the world about programming and databases yet there's conspicuously no classes to hone your skills as a systems engineer. The pipeline I mentioned earlier isn't magic, I came up with the idea because I understand the workflow of SELinux and I know enough about syscalls to interrogate my own code. It's kind of like the people who like to read machine code and compare it to their application but for a systems engineer. There should be classes for the kernel and userland alike. Point being, because enterprises commonly view the system as a checkbox rather than a product, or worse produce an undocumented cookie cutter product with "best practices" it harms the profession of systems engineering.
Coincidentally, this is also why I mainly use the title "Software Engineer" rather than "Systems Engineer" with big corps.
I came to the same conclusion, but in my case the problem was that there were no audit messages for one particular denial. After a week or two of investigation I finally found the solution. But, I decided that I simply couldn't trust selinux any more, so I turned it off.
Oh, What I meant to say is. The author is correct, selinux is too hard. I too am incompetent and unable to understand the many examples that demonstrate solving selinux issues.
What we really need is a shittier solution that lets good programs work while stopping bad people from doing bad things by using psychic powers and having a deep understanding of software that is yet to be written.
Author is correct that SELinux is not trivial and takes work to figure out.
Author is full of it in that they're obviously not aware of the tooling and information available. Eg:
All of this is wrong:
> Fedora Linux. There’s nowhere on the system where you can view the policies and look up why something might or might not work.
You can install the policy's source and look at it all you want. There are also tools to examine the current policy.
> The SELinux denial audit log messages are too vague. You’re told that a label was denied reading from another label. Okay, what do those labels mean? Which programs? Which files, sockets, or whatever?
You get that in the logs. What you get is an inode number, which isn't very user friendly, but it's very much there. SELinux has tooling to turn the log messages into something nicer to work with.
> So, you run the audit2allow command as instructed, and end up with some policy blob files.
> God only knows what changes the blobs do; you can’t be expected to, nor are you given enough information to evaluate them.
This is also wrong, along with the blob you get a text file.
TL;DR: The author has a minimal point, but clearly hasn't read the docs. Help and tooling is available and not that obscure.
Basically linux is moving from a hobbyist operating system that a single person can reasonably manage in their spare time to an enterprise ish complex monstruosity that requires training and a time investment. Probably because the main Linux sponsors have found that it's the only way to make money with it.
SELinux... systemd... who knows what's coming next.
Let's not forget ipv6, although that's not a linux specific problem.
Edit: went through the discussions and it appears that all defenders of SELinux have used it in a paid professional context where it's justified to have a tight security polities. Hobbyists can't afford that time investment.
> Basically linux is moving from a hobbyist operating system that a single person can reasonably manage in their spare time to an enterprise ish complex monstruosity that requires training and a time investment
This is less a Linux problem, and more an Internet problem. The reason we have more security vulnerabilities now is not because software suddenly sucks and everyone is incompetent (rather, that is a universal constant): it is because software is valuable, and the internet is huge. That is why we need ipv6, but more importantly that is why we need tools that provide security in depth, which is a great deal of the complexity in Linux today. (For example, there are heaps of vulnerabilities that sound terrible, but we know their effects will be mitigated by a container, or by SELinux). Yes, as a hobbyist, one could probably grab a copy of Debian 2.2 Potato and host a website from it. Running a modern stack sounds like an investment? Keeping that afloat will need a dedicated team.
As a Desktop environment, major Linux distros have never been as well polished as they currently are. It's gotten to a point that regular people can use them without much hassle.
From a sysadmin perspective,one always needed to invest time in Linux to be proficient in it, I don't see how that has changed greatly.
Eh, what? SElinux is completely manageble and usable. I've been happily living with it for a long time. Now when I have to live with apparmor (which is exactly unmanageble) I really miss SElinux and really don't feel any secure without it.
Another wave of excuses in the manner of "i don't wanna learn anything".
I have ran selinux in production on rhel and centos for years, had very few problems. All packages that are maintained by red hat comes with selinux policies included, over time stuff barely changes, just some security fixes applied.
In case you have trouble, just install (setroubleshoot-server) and run sealert -a /var/log/audit/audit.log and it'll inform you whats needs to be done, i.e set a sebool or generate a policy. Solves 95% of the problems.
90% of the places I've worked, it's either been turned off or blindly hammered with audit2allow until SELinux shuts up.
Mostly because the "admins" are too lazy to do things the right way (non-standard file paths, ports, etc) or create/change the actual required context/labels. I'll admit that like anything it can be cumbersome at first and frustrating to learn, but it is helpful.
Off-topic, but the polite “disable your adblocker” message unobtrusively displayed at the end of the article is a nice way to do it. Lets me know I can support the site that way without hindering the reading experience.
We've been implementing CIS hardening standards across our heterogeneous fleet of VMs for the last two years (Thank the maker for Ansible). When it came to evaluating the controls we would implement for L1, my team and our Security team negotiated which controls were appropriate for our environment. The first that I whacked was SELinux, and the Security guys couldn't understand why. In a greenfield environment where I had the time to run it in permissive mode and hopefully catch all the odd permissions it would require, maybe. A small maybe. But our environment has been around for decades, with apps that have transitioned across multiple teams, each time losing institutional knowledge.
Trying to implement SELinux on theses servers would be hugely disruptive, for minimal gain. Once I was able to explain the risk/reward calculation to the Security guys, they understood. Besides, we have so many compensating controls, I have no worries about expanding the attack surface slightly.
This genuinely surprised me - I much prefer SELinux to a number of other access control systems out there (looking at you, Active Directory).
It sort of does the work for you, as long as you 'train' it across a broad enough range of daily tasks. Of course, its not always appropriate to play it that way, but for personal use I've had no issues.
"It doesn’t help that no one wants to be told: “stop what you’re doing, spend a week to learn this other thing, make a tiny change, and then return to your real job”. They’re looking for solutions — preferably quick ones — and not a week’s worth of study."
I've never had a job that would allow me to take a week to solve a production issue.
Fundamental assumptions in the SELinux model (or in my understanding of it)
* Every executable has only one role in the system.
* That role does not change often
* Every executable can be trusted, does what it says it does, and never gets confused
* Fixed access control lists, security tags, are adequate in the 21st century.
I've started writing a library for dropping privs (as if that should even be necessary), but the devil is in the details, so I've not pushed the code yet.
SELinux works well on Android because the OS is delivered by a single entity and unmodifiable, and is maybe fine on a desktop, but on a custom server it seems to me that using VMs or containers as a security boundary is far better since it's trivial to understand and in the case of VMs more secure.
I used to be really grumpy about SELinux until I learned about audit2why and audit2how. With those two utilities SELinux is just as easy if not easier to configure than any systemd process.
Every job is hard when you are using the wrong tools. Don't change your oil with pliers, and don't setenforce 0.
I've advocated for years that each program should be ran as its own user and group, isolated from the rest of the system.
So let's say you run Firefox, is should run as firefox_username user and group.
This way any exploit is ran with the user/group permissions, and unexploited it has no read write access outside its own user and group.
Selinux is a beast but there are tools to help with determining which settings need to be applied.
If you desire a truly locked down system, where you have a person or can afford a person dedicated to its security it's a good tool. Otherwise just turn it off.
How likely is it your server is going to be actively attacked? Usually very unlikely. We don't count the random and always present background brute force attempts, they're just background noise.
The problem of selinux getting in the way of being able to boot/login has been going on for years. Tossing selinux=0 on the grub/kernel command line anytime the machine fails to boot is common for most people running rawhide. Which itself is a problem because relabeling can take a _long_ time if the system in question has a large filesystem.
Selinux has stopped a number of active exploits, but OTOH, its probably stopped 100x as many perfectly functional systems from working.
So more selective enforcement is probably the right path, and one that systemd seems to be going down as it attempts to secure services it manages, but right now welding the door shut to stop the bad guys seems like the solution we are stuck with.
The other thing is that not all sysadmins are security engineers and you essentially have to be one to be able to manage SELinux well. You really have to understand the security implications of incorporating policy X for application Y.
Minimally, Redhat needs to come up with some better documentation around managing it along with better tools for managing policy. Like say, exporting the existing policy set before each upgrade or policy change and import them back? I'm thinking sort of what is done when a package is updated and you are prompted if you want to keep or replace your existing configs.
Does anyone have any examples of where SELinux saved the day? I appreciate that some of the time you'd never know, but I am curious if anyone has any real life examples where it stopped an attacker.
For me, nothing too solid but it makes me sleep better.
Not so much an attacker but bad actor(s)...
It has saved us from having people with expected (limited) access to systems from spawning services listening on the network that shouldn't be there
Granted, we have a default drop policy in the firewall, so it shouldn't really matter in a sense. I like the approach of not having the process rather than preventing access to it
The attacks could be worse. For example, some users SSH access was compromised and the attacker wants to use it -- opening a reverse tunnel on localhost out of the environment over their SSH connection... bypassing the usual perimeter for egress.
SELinux should provide some nice control over this that would be independent of service
I used to switch SELinux to permissive with the intention of checking and correcting the policies later, only to completely forget about it when other issues get priority.
From my relatively basic understanding of SELinux, it seems like has a lot of powerful mechanisms for enforcing security policy, but a lackluster interface for actually showing violations or creating robust policy.
Luckily, I think there’s a lot of community work coming up to make these policies easier to write and more robust.
I have to say that I strongly disagree with the arguments made and the advice. Disabling SELinux is a terrible idea, it will increase the attack surface of your system by magnitudes. I'll be addressing arguments made (paraphrased, not direct quotes).
A) "SELinux documentation is too complicated and geared towards policy makers, not end users"
This is because the documentation's target demographic are system administrators using the tools to write policies. This is like complaining that a carpenter handbook aimed at professionals is too complicated for end users. You want to be looking at either the documentation for the software (Docker for example tells you how to get around permission errors with mount, or setting the appropriate bool for httpd servers) or reading the error message that tells you exactly what to do (run audit2why, file a bug report so the distro maintainers can take a look at it).
B) "You should disable SELinux"
Would you disable your home security system because it's too complicated to figure out? If there is a genuine bug in the policy, then it should be reported, not disable the entire thing.
C) "Start from a clean slate and write your own policies"
The policies are written by those who are intimately familiar with SELinux and how it operates within the context of the distro. Again as said earlier, if there's an issue, report it upstream. Trying to write your own policies, especially if you're unfamiliar with SELinux, will result in an insecure application/system. Most errors are easily solved by google or running the appropriate command.
D) "It's too complicated for end users; use Docker instead"
I see this argument a lot, but it makes no sense within the context of IT. The end user and system admin are two different groups of people. Why does it matter if it's too complicated for end users when the audience are other IT professionals? Docker also is NOT a replacement for SELinux. Docker is secured BY seccomp profiles, by itself it is actually significantly more insecure than people think. This shows a fundamental misunderstanding of the threat model of containers and what benefits SELinux provides.
> After the update, I was locked out as the login screen no longer could read some files needed to authenticate my login.
My last experience with SElinux (around 10 years ago) I got this exact same issue with the login manager while trying out Fedora... and after fiddling around and working around it, another issue, and another - it must be turned off, it's clearly not suitable for general purpose usage and shouldn't be enabled by default (and isn't on most distros).
I have built ad-tech systems which get large amounts of traffic and continuous hacking attempts/DDOS attacks.
In 10 years, the servers have never been hacked. I went to the trouble of enabling SELinux and locking things down. Not sure if it was all needed, but I can't say I am unhappy I did it. After setting things up, it hasn't been a problem.
SELinux, and all of the other bolt on security model extensions, all have a parallel system problem. Fundamentally it's about adding a security definition layer that just doesn't exist in the traditional UNIX privileges model; and it isn't even the typical extension of ACLs on objects. Instead it's a whole new set of tools or patch extensions to existing tools (which may as well therefore also be custom tools) that is completely opaque and therefore impossible for anything to interact with or provide good error messages for when it does inhibit interactions.
What I really dislike about SELinux and similar tools is the pivot in management. Inevitably since they're bolt-ons they also add a second policy area, no longer on the file or device but instead in some other database of inscrutable configuration files. A second and totally alien location to check when anything isn't working.
Since security properties are a global aspect of a system there is something to say for having them sit separate from the rest of the system. On the other hand that means every change (potentially) has to be made twice.
It's good if you want to have a committee check every change but it slows changes down accordingly.
where the reference to an object and the access to the object are tied together. On paper it's a great model but it hasn't seen a lot of real life use.
Mobile applications require various permissions which can be enforced via SELinux.
SELinux policy can also be written to mitigate many vulnerabilities and vulnerability classes/vectors. Over 40% [1] of Android users may no longer be receiving important security updates, potentially putting them at risk of malware, data loss and cyber attacks.
Seems to make sense to have it enabled on Android based on the above.
Another reasons it is turned on, alongside seccomp, is to make sure app developers don't mistake Android for Linux, something that Termux guys aren't very found of.
> Mobile applications require various permissions which can be enforced via SELinux.
What do you mean by this? SELinux can only deny things that would otherwise be allowed; it can never allow required permissions that would otherwise be denied.
Android's systems are developed alongside those policies, though. Things like lighttpd don't ship selinux policies, and bolting it on later is fragile.
I did my RHCE bootcamp back when RHEL 5 was just released and the bootcamp course materials were not yet updated for it. The instructor worked around this pretty well. One of the earlier topics was SELinux, where he stated we would cover it for the course, but it's not on the certification exam and it's not needed past that part of the course, so we would disable it once we were done with the lesson. It's been well over a decade so I don't remember if this was day 1 or 2 of a 5 day boot camp.
Disabling SELinux and implementing more manageable controls has been a default for me for quite a long time, even before my certification. My preference is for apparmor, to be honest.
The difference between SELinux policies and the systemd security settings for locking down a service is about integration and where they are maintained.
The SELinux policies are not integrated well with the applications and normally the apps don't think about shipping their own SELinux policy tweaks. With systemd security settings this is much different because the app normally provides the systemd service file and the security settings are part of it and maintained by the app developers.
Edit: With AppArmor it's similar and the app devs normally don't think about providing an AppArmor profile but I think it's at least a bit easier than with SELinux policies.
As a Fedora user for the last 5 years I have to admit... number of times SELinux prevented an malicious access: 0, number of times SELinux bricked the Fedora desktop: 3. Changes are often released, with selinux-policy catching up a week after.
It's interesting because the author lists some very concrete problems in the way SELinux is implemented in modern distros, but their conclusion - instead of fixing these problems - is to disable SELinux completely. Will it really help the sysadmin in the long term? What we need to have is concrete feedback to the SELinux team listing the problems and suggesting the ways to solve them. I believe making the logs more verbose is probably the easiest to implement; working on high-quality and accessible documentation is another; documenting the particular policies implemented by the vendor of a particular distro is a must.
Writing policies by hand is nearly impossible, error prone and that is the exact problem we are trying to solve - to make SELinux and AppArmor easy for K8s workloads and now host based workloads.
I can't say my apparmor experiences have been great. SELinux and Apparmor are just two heads of the same problematic coin. As other comments on here point out, it's easier to have one sane policy for docker and containerize everything else.
How the hell do I dockerize firefox or vbox? I get it for server workloads but I was talking from a desktop perspective. Even if that is possible seems like even more stuff to manage and troubleshoot.
Also, you are not supposed to ise containers as a security boundary??
I gave up on selinux early on when I learned about dontaudit rules. It's nice that that amount of control is present, but don't waste my time. Disabling selinux is the first thing I do.
I just can't get over the fact that it comes from the NSA. Sure, it might be more secure, when properly configured, than without it. But it's the fucking NSA. The same agency that backdoored Dual_EC_DRBG. They're ostensibly an agency that is supposed to protect us from malicious digital threats, but they have always sacrificed that mission any time they have had the chance. I will never trust them, and I say that as a family member of someone in the NSA.
As someone who is relatively new to the Redhat ecosystem, what exactly is the purpose of SELinux? If you only run trusted applications why do you need it?
Trusted applications can be full of vulnerabilities that allow them to become malicious. SELinux and such can limit the capabilities of such trusted applications to do only what they were designed to do and not execute reverse shells or run Crypto mining software etc..
I tried SELinux perhaps too early and found that, unless I was right on the happy path, I simply couldn't get anything to run. I recall a feeling of victory after finally being able to multihost some websites. However, I am not a great sysadmin and was pressed into service at the time.
It feels good to know that I was not the only one.
One trick that makes SELinux on RHEL much less of a bother is to always use the system-provided default paths for data and logs, and just mount additional volumes for data at these paths.
The vast majority of SELinux trouble I've seen stems from people customizing things needlessly and then they need to adjust the default policies.
Another common problem is not separating the OS / application binaries, configuration / data and log directories, which is just bad hygiene even without SELinux. I've seen many installs where people have just dropped their stuff in /home/ec2-user and then the whole server breaks at some point because / gets filled by a sudden burst of log entries.
I don't think I agree. I've been running things with SELinux enabled for more than a decade, haven't really had problems.
The issue with SELinux is that it's security, and security is a complex issue that requires understanding what you're doing and why. It's not a problem domain that's friendly to the approach of poking at random until somehow things start working.
It's not really different from filesystem permissions actually. Why not 'chmod 777 /etc/shadow' to get yourself out of a problem? That question can only be properly answered by knowing what /etc/shadow is for, and why it's a bad thing for everyone to read and write it.
SELinux just forces you to have to understand the system in terms of what operations a program should be able to do on what. Should Apache serve your ssh private key files? Should a random tool be able to listen on port 8080? There's really no instant answers to questions like that. You have to understand what Apache is, what ssh keys are, and who is supposed to access them.
If you have a compromise, you may need to do some forensics and figure out what happened, and that will require answering such questions anyway, at a most inconvenient time. Where did they get in? How did they escalate privileges? How did they get their hands on your data? Is there still a backdoor somewhere on the system? Why is that process listening on that port?
And by the way, audit2allow produces text files, which it proceeds to automatically compile for you. Right next to the binary blob there's a text file with the policy, you can read that.
Ok, let's look into this in more depth.
> At the heart of the problem is that the SELinux policies themselves are sort of magical. The policies have probably been provided by the maintainers of your Linux distribution, e.g., Fedora Linux. There’s nowhere on the system where you can view the policies and look up why something might or might not work.
Yeah there is, you can install the policy's source and look at it all you want. It's complex, granted, but it's very much there.
There's also tooling to examine the current one, like sesearch.
> The SELinux denial audit log messages are too vague. You’re told that a label was denied reading from another label. Okay, what do those labels mean? Which programs? Which files, sockets, or whatever?
You do have that information. It's specified as an inode number for performance reasons, which isn't terribly user friendly. Which is why there's tooling to turn those obscure log entries into something much more readable, such as the mentioned sealert. It's even mentioned in the logs.
> So, you run the audit2allow command as instructed, and end up with some policy blob files.
> God only knows what changes the blobs do; you can’t be expected to, nor are you given enough information to evaluate them.
Wrong, the information is right there in text. audit2allow compiled it into a binary, true. But right next to it you'll find the source, in the .te file. And for most cases, it's a short, quite understandable one.
> Then what do you do? You don’t know what the module did, where it was installed, or how to remove it.
That seems like a problem of not reading the docs. I mean, there's the semodule command for that.
The blog author's problem is that they're trying their best to do the absolutely minimum possible. Which in a way I understand, but doesn't make them informed on the subject matter. It's like writing code by randomly changing stuff without understanding until the compiler finally shuts up. Yeah, sometimes it gets your assignment a bit closer to done, but most of the time the result is crap because randomly messing with stuff isn't a replacement for understanding.
Most of what they complain about have tooling, documentation and solutions. They're just not using them.
> Most of what they complain about have tooling, documentation and solutions. They're just not using them.
OP's problem is that they do not want to learn. In general - if you see a person complaining about SELinux - it's because they have no interest in learning.
Given android uses this so much I don't get systemd getting dragged into the rant, this screams of being reluctant to follow the way the community is going
SELinux and Secure Boot are largely orthogonal. Secure Boot can be used to establish a chain of trust to guarantee that you are running software that is not tampered (using asymmetric cryptography). For example, Secure Boot can verify that the bootloader is an unmodified blob from a trusted vendor. In turn, the bootloader can verify the kernel. The kernel can verify modules/extensions. In principle, you could extend this chain to ordinary programs.
You can use SELinux without Secure Boot and Secure Boot without SELinux. Secure Boot can make SELinux more effective, because you can use it to verify that the kernel is not tampered with, so you know that it is properly enforcing the SELinux policies. You could also verify cryptographically whether a policy is not tampered with, but I am not sure if that is currently supported.
The comparison between SELinux and SIP is a bit more complicated. SELinux is a security framework that allows you to write all kinds of policies, whereas SIP is a very specific set of defenses.
On of the main features of SIP is that it makes certain system paths unwritable for every process that doesn't have the right entitlements, including processes with PID 0 (root), unless they have the the necessary entitlements [1]. Since SELinux is a generic security framework, you can implement something similar to SIP's path protection using SELinux. For example, system paths could be labeled with a special label and writes can be restricted to processes with a particular SELinux context. More concretely, you could use SELinux to forbid that any process, including root processes, can write to a system path like /usr/bin. Then you could allow the security context in which DNF/RPM (the Fedora package manager) run to modify system paths like /usr/bin [2].
[1] SIP does more stuff, like disallowing unsigned kernel extensions, preventing code injection, etc.
[2] Russell Coker has set up an SSH machine for which he gives away the root password. But you can't do much with it, because the SELinux policy prevents you: https://www.coker.com.au/selinux/play.html
Security-Enhanced Linux (SELinux) is a security architecture for Linux systems that allows administrators to have more control over who can access the system. It was originally developed by the NSA.
Secure Boot and SIP are about boot integrity preventing malware early on in the boot process [1]
How practical is just turning off systemd today? I hate it too, but hasn't Red Hat basically strong-armed so many important Linux components into adding a hard-dep on it that your only way of avoiding it is to use a distro purpose-built to (e.g., Devuan)?
Systemd is only really required if you want to run gnome or kde easily. It's not an issue otherwise. Probably more trouble than it's worth to replace it in a distro that uses it by default though!
I don't think as a user you actually can remove systemd at all. Everything is set up to work with it. Removing systemd would be closer to forking the distro and maintaining it yourself. If you wanted to avoid systemd you would have to pick a distro which doesn't use it.
I have to complement your choice of FreeBSD and jails. If I didn’t have a Linux stack, I’d run FreeBSD and jails, too.
I wrote my own kernel modules that do exactly what I need in my stack and use case. No need for complicated SELinux, AppArmor or other solutions. I always roll my own solutions from the ground up (usually in a couple of all-nighters).
I needed around 5 years to slowly move away from Linux, open/-solaris/omnios/smartos was also in my mind, but...well you probably know the sad story behind it.
But FreeBSD really kicks, from firewall to database all the way down, everything FreeBSD ;)
Well illumos is just the kernel and userland (bit like BSD without ports/packets), Omnios is a really clean "server-distribution" with illumos at it's core but well same problem, not enough eye's and hand's (aka packets, documentation etc). I decided, that FreeBSD is the "more worry-free" solution for me...and still happy.
The whole IT ecosystem has become a hail mary. Even admins usually have no idea what a certain program actually wants to do. If the admin knows how to install the app so that it actually runs, you call them a good admin.
From a security point of view, an application is like a nuclear power plant. It's good if it works as planned, but if something blows up it endangers your whole enterprise.
The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl. That way the radiation hopefully stays in, but history has shown that it really doesn't. Also, the wheel of history has just turned one more iteration and now admins just view the sarcophagus as something you deploy as you previously deployed the app. Who is responsible that it is air tight? Well, uh, nobody, really.
You can't even blame the applications for that. Let's say you want to build a good, secure application. How do you know what files your application will try to open? What syscalls it wants to call? Library and framework functions tend to not document that properly.
Obscure files like /etc/localtime, /etc/resolv.conf, /etc/ld.so.conf, /dev/zero ... how can you expect devs to build well documented and well sandboxable applications if they don't know which files their library functions will open?
You may have heard of /etc/resolv.conf ... but have you heard of /etc/gai.conf? /etc/nsswitch.conf? /etc/host.conf? Wouldn't it be great if the man page of getaddrinfo mentioned those (mine only mentions gai.conf)