I've attended a local CNCF meetup where the chainguard folks presented Wolfi and their related tools to create container images and SBOMs.
I was already skeptical of the product, having heard of it before. Unfortunately, attending that talk confirmed to me that they've just re-invented a severely limited version of Nix powered by yaml files that can output SBOMs in a standard format.
Their software repository only has around 500 packages vs the 80k+ in nixpkgs. When I read "Alpine Linux", I was immediately reminded of its use of the musl libc which has endless DNS resolution issues[1][2] in and outside of Kubernetes, however, thankfully, it seems like they realized bundling musl was a bad idea and ship glibc instead.
I'll stick to Nix and nixpkgs to build my reproducible container images with full dependency graphs back to the source code.
I opened that apkindex file and it had duplicate entries for a ton of packages with different versions, taking a look at https://github.com/wolfi-dev/os I only see about 840 yaml files which I assume define the packages. I don't think claiming to have 10k packages when only 10% of them are actually different pieces of software is a good claim to make. Nixpkgs would have millions of packages if we added up every single unique package from every revision.
The real number is probably somewhere in the middle - one yaml file can define many packages - see the gcc or clang or argocd ones for examples of that.
Even so, I did a quick search on repology and Nix derivations with multiple outputs (the nix lingo analogous to the subpackages you mentioned) are counted as a single package. For example, bash has 5 outputs but only counts for 1 package in the 85k figure, so I think comparing 900 packages to 85k is a valid comparison.
Anyway, this is all besides the point I was trying to make which is that I don't see why I should use _yet another_ software distribution that has 1% of the amount of packages found in a mature distribution that already has frequent automatic updates and bleeding-edge software revisions.
We needed Wolfi to be able to create minimal (distroless if you like) container images based on glibc with 0 vulnerabilities. Turns out a lot of other people are interested in Wolfi for various reasons, and we're more than happy to work with them.
Can anyone help me understand musl libc and DNS issues? Note that I’m most interested in this “DNS over TCP” issue, since the other case I’ve heard of is for custom DNS setup—not for resolving host names in a default configuration.
My reading indicates that DNS resolution simply might not work in certain cases. This seems like a huge problem, yet Alpine Linux is widely deployed and I think Zig uses musl libc as well. In fact , every fully static binary I’ve seen (except for Go?) relies on musl.
For what it’s worth, I’ve seen DNS errors on Alpine (specifically I was getting EAGAIN), but I assumed this was unrelated to musl (and am still unsure). In general, I feel like I see a lot more transient networking errors on Alpine, and I wonder if this is related.
Edit: I also didn’t think DNS would be in libc, so I’ve got a lot to learn…
To summarize the issue: DNS is done optimistically over UDP because it's faster, but this doesn't work when DNS responses are large because of the design of UDP. TCP should be used as a fallback mechanism when responses are large. This is uncommon normally, but increasingly DNS responses are large in special scenarios; for instance when you're querying an internal DNS for service discovery (read: k8s or nomad deployments, most commonly).
Musl's maintainer interpreted the spec for a libc's resolver to not require TCP fallback (source: https://twitter.com/RichFelker/status/994629795551031296?lan...), so for a long time Musl simply didn't support this feature, justifying it as better UX because of the more predictable performance.
I don't agree with the maintainer on this interpretation, but I am glad the feature was added and the issue is no longer a concern as an otherwise very happy Alpine user!
As mentioned correctly in [2], DNS issues with musl stem from the fact that it follows the DNS specs strictly and exposed bugs in certain DNS servers.
For an HTTP server replying with 200 OK instead of an 404 Not Found upon ENOENT, the situation is clear that this is not a client-side bug. The same should be assumed of DNS/NXDOMAIN.
As much as I agree with your reasoning, I have experienced this kind of breakage first-hand due to cloudflare and DNSSEC and will continue using DNS resolving code that doesn't cause me pain and suffering (anything that isn't musl) because making this work with musl is quite literally out of my control.
My personal 'musl broke it' story comes from resolving domains from Cloudflare that use DNSSEC in a K8 cluster. Basically this:
- K8 sets container to use `ndot:5`, causing the search list to be used
- Musl walks that search list looking for domain
- Cloudflare does not set the NXDOMAIN flag on a DNSSEC domain but does include an NSEC record (if you query with the dnssec flag).
- Musl takes this 'NOERROR' reply and returns an EAI_NODATA.
Is Cloudflare wrong? I don't know, maybe. They say some things about the standards[0] and that it's technically 'right'. I don't see why they couldn't change the behaviour for queries without the dnssec flag, but I digress.
The issue is that every other libc I tested will continue searching and actually resolve the domain. Musl is the odd one out, and _only_ in the case where the search list ends up with domain using Cloudflare and dnssec.
Even if Musl is 'right' here, when it disagrees with major implementations and a major DNS nameserver does it really matter?
If you hadn't made that HN post I would probably still be scratching my head as to why I have intermittent DNS resolution issues in my personal Kubernetes cluster. I've been having them for years only for some specific software (my mediaserver stack, packaged by linuxserver.io). Since this isn't mission critical stuff I just shrugged it off and dealt with it. Eventually I read your post and it all clicked. I can't find any maintained distributions for that software that don't use alpine/musl sadly so I added[1] dnsConfigs to all of my deployments using musl to force ndots to 1 and accept the fact that all of my cluster name resolution for those deployments will have to be fully qualified. It's a really frustrating situation and made me realize how badly musl plagues the container ecosystem due to its use in Alpine Linux and alpine's popularity because of the small images it produces. Alpine Linux and musl are on my shitlist for life.
musl doesn't implement TCP DNS at all. When it gets a respond with the truncation bit set, it just assumes the completed records in the truncated response were sufficient, the best records even, and proceeds as if the lookup succeeded. It's hard to take seriously claims that it's strictly implementing the protocol.
I would love to see non-trivial examples of using the nix toolchain to build images with multiple OS, architectures, SBOMs. As someone unfamiliar with the nix ecosystem, it seems like a tough ask for contributors to require nix knowledge rather than just changing out my existing base image.
I'm not sure what you mean by "non-trivial" but here's a simple discord bot I wrote in python, that I distribute as an OCI image and that is built with Nix for both x86_64 and aarch64 linux via GitHub actions: https://github.com/starcraft66/attention-attention
There is no SBOM because I didn't bother publishing one but the way Nix builds derivations, you basically get the SBOM for free. You could use a tool like sbomnix[1] to trivially generate an SPDX-format SBOM from the nix derivation that builds the container image.
Edit: Since you mention swapping out base images, I think there is a misconception about how building images with Nix works. There is no such thing as a "Base" image, nix builds images from the Dockerfile equivalent of "scratch". You would ditch the Dockerfile completely and use only Nix to build the image.
Not sure an OS created to meet the specific need of VC funded startup, created exclusively by employees of that company should be called a "community Linux".
Totally understand the concern. We've tried to draw a very clear line between Wolfi (the project), and Chainguard Images (our product). They're in different GitHub organizations, have different maintainers, and are documented differently.
We (Chainguard) still represent most of the maintainers to Wolfi, but there are external folks too. We're going to need help scaling Wolfi to get it to where we want it to be, and that will require a real community.
We do fully intend for wolfi to be a community project, but it will take some time. We do say on the home page:
>What are the plans for long-term Wolfi governance?
We intend for Wolfi to be a community-driven project, which means over time it will have multi-vendor governance and maintainers. For now we're focused on building the project and community, and will revisit this in several months when a community has formed.
We use Alpine Linux in all our docker builds for AWS ECS / Fargate. I noticed the FAQ says this:
> Wolfi is a Linux undistro designed from the ground up to support newer computing paradigms such as containers. Although Wolfi has a few similar design principles as Alpine (such as using apk), it is a different distribution that is focused on supply chain security. Unlike Alpine, Wolfi does not currently build its own Linux kernel, instead relying on the host environment (e.g. a container runtime) to provide one.
Is it possible to expand on this more? Why should I, a technical expert, use this over an already established distribution (Alpine)? What are good talking points I can bring to management / architects that would sell them on a switch?
Can you expand on what “supply chain security” actually means?
In this case, supply chain security mostly means compliance. Alpine and other distros already do a great job at most aspects of supply chain security, but we make package/image signatures and SBOMs very easy to get in Wolfi.
We're also more flexible on packaging extra software in Wolfi than other distros are, which has an effect on your overall supply chain security posture. We're aiming to package the entire Cloud Native landscape (to start!), so you can simply "apk add" anything you need, without having to drop back to "curl | bash" in your Dockerfile.
By making it easier for devs to get the software they need "the right way", they'll have a reduced need to work around the already existing secure distribution mechanisms built into their distros.
If you're already on Alpine and like it, that's great! If there are some packages you can't find in Alpine, or musl vs. glibc ever causes you issues, or you need SBOMs for some reason, give Wolfi a try!
Alpine is out of the picture for us because the guy that works on their security tracker just doesn't care, and responds half a year after filing an issue. The tracker itself is broken for over a year and the response was to basically rebuild our own package index and host our own security tracker.
So I would not say that Alpine has security as a high priority, even though in theory there are the secfixesdb.
Redhat Enterprise, Debian and Ubuntu are used because they provide an OVAL feed that are easily integrated with zero development overhead. So if compliance is your focus, I'd heavily recommend generating an OVAL feed when you're regenerating the secfixes json files.
Source: Am building a cross-linux-distro vulnerability database and I am scraping _all_ linux security trackers. [1]
For one, you refer to me as "a guy." I am, in fact, not a guy.
Secondly, the issue you opened was against an internal project used by the security team to do continuous CVE triage of the distribution. It is not meant to be used for public security data. We are tired of security vendors scraping our internal tool, as generating those reports in real time is very expensive on resources.
I'm sorry I did not ask you beforehand about your preferred pronouns. If you want people to use the correct pronoun and it's such a big deal for you, add it to your profiles.
Regarding the rest: don't call it a security focussed distro if you don't even care about providing the data for vulnerabilities.
If you can't see the constructive criticism I tried to provide in the issues (and pull requests) that I filed then I made a good choice avoiding your distribution ecosystem from now on.
Maybe @dang wants to chime in here to prevent more chan-level escalations.
We do provide the data for vulnerabilities, at secdb.alpinelinux.org.
The security tracker is a tool for the security team to remediate vulnerabilities.
The data provided by it is not particularly useful nor intended for consumption by people other than the security team and alpine package maintainers: it generates reports for possible CVEs to review and possibly mitigate in the package collection. The presence of data in the tracker that is not present in the secdb (either as an ACK or NAK) is just an indication that there is a vulnerability to investigate, not that anything has been confirmed or denied. Really, the data is not relevant as a product for end users to consume.
The secdb outlines what package versions fix what CVEs, and what CVEs have been formally NAKed. Speculative data from a distribution-wide vulnerability scanning tool is not useful data to be making security-related decisions with.
I’ve worked in a few industries that were crazy about compliance. Do you have any certifications for different compliances? Ie FedRamp, IL4/5/6, HIIPA, etc.
I don’t think any other container images are strictly compliant with these standards - they’re not certified, but the technology stack (ECS, EKS, etc) is certified. Either way, if you can get this certified with different compliance platforms that would be a big selling point for B2B customers.
We're working on FIPS builds and will self-attest to the NIST SSDF stuff later this fall, but there aren't too many other requirements that directly apply to our images.
Wasn't really sure where this was going, but the background to this approach adds some details - namely what they mean by distro-less and how they pull it off.
Seems to have a confusing lack of "getting started" documentation. On the plus side, I now know what the world's smallest octopus is, so the important stuff is covered!
From my side I'd love to see clear comparison to distroless[1] images, the best info I could find was this Github issue comment[2], although I would expect to find it in https://github.com/wolfi-dev#faq. Or maybe my google-fu failed me :(
I was (one of) the original creators of the Google Distroless project. The main difference is that Original Distroless uses Debian as the upstream. This is great in many ways, but makes it so stuff outside of Debian is hard to package in.
Wolfi is its own upstream - packages are sourced and built directly from source, without another Linux distro.
The rest of the differences stem from here - we get more control over what we package and how, and are able to more quickly add new versions of packages, or roll out CVE patches.
Do you see any significant changes in comparison to the "gcr.io/distroless/static" image, which is commonly used in Go ecosystem? Thanks for answer btw, really appreciate that.
For static, not really just because there's so little in it. For apps that need CGO there's a benefit as more dependencies are required. CGO apps are broken right now IIUC in distroless/glibc-dynamic because of the Debian glibc bump.
I like the idea of having an SBOM as a standard, baked-in feature from the outset. That kind of thing is becoming critical in software supply chain management.
Thanks! By building the SBOMs as part of the build process we can ensure full coverage, vs. the other approaches that rely on "guessing" the contents after. We wrote a bit more about this approach here: https://www.chainguard.dev/unchained/make-sboms-not-guessbom...
If nothing else, it's at least useful to ensure you are able to comply with all the open source license terms. Now how many copies of the MIT and BSD licenses do I need to include with my software? ;)
99% of Wolfi is the package repository. We've compiled all the software to run an OS from source against glibc and made it available as APKs (apart from the kernel). The major use case for this is composing container images using tools like apko.
When you run Linux containers, you're using the kernel from the host, but everything else is provided in the container i.e. file system, utilities etc. So you can effectively run Debian or Alpine on top of a Arch kernel by just doing "docker run debian" etc. And now you can also do "docker run cgr.dev/chainguard/wolfi-base".
Debian does a great job! But here's one example where their packaging system makes container workloads hard. Debian, like many other distros has a strict "one version of every package" rule, meaning that Debian only ships one version of common things like programming languages or webservers.
If you "apt-get install nodejs", you can only have Node.js v18, in the very very recently released Debian bookworm. If your devs need Node.js v20, or even v22 which will both be LTS releases during the bookworm release cycle, you're out of luck.
You can go install those outside of the Debian package manager, but then you're on your own for vuln assessment and security fixes. Worse - many scanners don't actually even "see" packages that are installed outside of package managers, so you may not know that these exist or are installed.
We designed Wolfi to be more flexible around package versions. This is a very very hard problem to solve in general, but since we're focused on immutable containers, we can skip a lot of the complexity around conflicts, upgrade/downgrade scenarios, and cross-version compatibility.
Yep, but workarounds come with a cost. We're trying to package basically everything, so you don't need to pick between "up to date software" and "software from a trusted distro".
It's going to be hard to scale, but we're going to at least try! We have a lot of ideas on how to make this work that I'm excited to try out.
We can always come back to the discussion of statically linked vs dynamically. If your build system is strong an can adapt to these changes thats another option.
And the fun part there is that this image is effectively a workaround - node.js 20 is installed via curl | bash. That means CVE scanners and SCA tools can miss node itself in that image.
As silly as it sounds, try running "snyk container test --print-deps" on that image, and look around for Node.
This approach works fine, but means that you might not be able to rely on most container security scanners to let you know when there's an issue.
Sounds to me like CVE scanners aren't doing a great job if they can't pick up a nodejs installation from the official nodejs image distribution. Just looking at package manager metadata effectively won't give you the full picture.
Before reading the linked page, I thought this was about a Linux distribution optimized to run both local but containerized applications and remote/cloud web applications, kind of a ChromeOS like distribution with the same focus on simplicity and security, but with an addition local execution approach based on containers, and less tied to Google.
So, this is not at all this, but now I wonder if a distribution like the one I imagined exists?
I just realized your question may have implied a desktop os, whereas Bottlerocket, Flatcar, and likely the others in this specific thread are server-side. I don't have much experience with trying to solve that problem on the desktop except for the horror-show that is snap
They could've just improved Alpine's supply chain (or built on top of it) rather than making a new distro :( But they're a company, so pushing out new products faster is more important than the community.
We work with much of the Alpine community and some of the maintainers. The glibc choice instead of musl alone means that a separate toolchain and distro is required, but there are also a lot of other distinctions.
We still work with them upstream on apk and apk-tools, and would be happy to collaborate on more tooling.
We have contributed the majority of our improvements in such a way that other APK distributions, including Alpine, can make use of them, and the original plan was to just leverage Alpine for all of this. Indeed, we even sponsored the refactoring of musl's DNS implementation that allows it to support DNS over TCP now.
But we heard overwhelming demand for glibc support from customers. So we built a companion distribution for customers who needed to run GNU/Linux workloads, but still wanted to leverage the advantages of Alpine.
While I do appreciate the great work than Dan and team do, I feel like posting this headline, at this time, is click-baiting and playing off the recent Red Hat shake up to try and grab attention.
The fun part of HN is that you don't get to pick your turn on the front page. This was a surprise to me too, but I'm guessing it's up here for the same reasons you are.
Is this the first and only submission about Wolfi?
It may just enjoy traction because of the recent news. Also, posting and upvoting timely and relevant links is something that makes HN valuable, among other things.
Ah, they changed RHEL Source visibility from Public to subscribers only to make it harder for Rocky Linux, et al to provide strong compatibility. Lame.
Think of a circle with a fine split in it. At one end there's using a distro. You go around the circle to distroless, and on the other end of the circle, close to using a distro, but not using a distro, is un-distro.
It was meant to be a tongue-in-cheek way of pointing out that a it's a Linux distro without the Linux kernel, so it's really an "unLinux distro", or an (un)distro for short.
You don't really have a choice - containers always use the host kernel. So packaging the full kernel itself is a waste of space at best, confusing to CVE scanners, and something you just don't need to bother with if you work with containers only.
Wolfi is for running inside the container instead of on a VM or bare metal host. They're complementary - you'd run something like Clear Linux to boot up into a container host, then run this inside the containers.
I was already skeptical of the product, having heard of it before. Unfortunately, attending that talk confirmed to me that they've just re-invented a severely limited version of Nix powered by yaml files that can output SBOMs in a standard format.
Their software repository only has around 500 packages vs the 80k+ in nixpkgs. When I read "Alpine Linux", I was immediately reminded of its use of the musl libc which has endless DNS resolution issues[1][2] in and outside of Kubernetes, however, thankfully, it seems like they realized bundling musl was a bad idea and ship glibc instead.
I'll stick to Nix and nixpkgs to build my reproducible container images with full dependency graphs back to the source code.
1: https://martinheinz.dev/blog/92
2: https://news.ycombinator.com/item?id=35058094