Hi all, Docker founder here. It appears that the explanation in the pull request is not very clear. Sorry about that. We didn't expect the PR to be the first thing people read: there is a new website at mobyproject.org. Unfortunately Github is struggling under the load, and I can't update the PR text to clarify.
Here is an updated version with some clarifications:
<<<
Docker is transitioning all of its open source collaborations to the Moby project going forward.
During the transition, all open source activity should continue as usual.
IMPORTANT NOTE: if you are a Docker user, this does not change anything. Docker CE continues to exist as a downstream open-source product, with exactly the same interface, packages and release cycle. This change is about the upstream development of the components of Docker, and making that process more open and modular. Moby is not a replacement for Docker: it's a framework to help system engineers build platforms like Docker out of many components. We use Moby to build Docker, but you can use it to build specialized systems other than Docker.
- splitting up the engine into more open components
- removing the docker UI, SDK etc to keep them in the Docker org
- clarifying that the project is not limited to the engine, but to the assembly of all the individual components of the Docker platform
- open-source new tools & components which we currently use to assemble the Docker product, but could benefit the community
- defining an open, community-centric governance inspired by the Fedora project (a very successful example of balancing the needs of the community with the constraints of the primary corporate sponsor)
>>>
EDIT: I'm happy to answer any follow-up questions here, if it helps clarify.
Do you know that keeping your users in a constant state of confusion is not the way to convince them that you're building a stable and secure product?
Also, isn't it kind of ironic that you built your company on OSS and then invited a well known destroyer of OSS onto your main stage?
Plus, I want my DockerCon money back. What exactly did you announce besides multi-stage builds at the general sessions that is actually benefiting Docker users? I don't care about your plumbing and constant rebranding, other than finding it very discouraging, I want to know what you're actually doing with your product this year. I guess nothing.
> Do you know that keeping your users in a constant state of confusion is not the way to
> convince them that you're building a stable and secure product
This.
I was a very early proponent of docker, and have been using it continuously since the very early versions. I read the release notes and most weeks I read the weekly email newsletter, and I still don't really have a clue where Docker is going, what its current status is (as far as how the various pieces fit together, etc) or what half the new features even do.
The docker team's communication is really poor, and their documentation seems even worse, unless I'm missing some oracle somewhere that fills in all the details that the actual documentation leaves out...
if you don't currently use any of these new features, and can't work out what they do, perhaps you really don't need them. Therefore, ignoring it and continuing on is the best cause of action. It's not docker team's responsibility to educate you!
Well, sure, you can make that assertion, but when you have someone that is a) already pretty familiar with the technology, b) continuously keeping track of the product for years, and c) goes out of their way to try to educate themselves... and that person still feels kind of lost... sure, you could put all the blame on them, but that seems a bit disingenuous.
(Or you could make the assertion that I'm just an idiot, but my career suggests otherwise.)
> Plus, I want my DockerCon money back. What exactly did you announce besides multi-stage builds at the general sessions that is actually benefiting Docker users? I don't care about your plumbing and constant rebranding, other than finding it very discouraging, I want to know what you're actually doing with your product this year. I guess nothing.
I was annoyed at the fact that I paid $150 extra for a workshop, "Docker for Publishers" that while useful in itself, I could have done in large part for free as a Hands On Lab: "You'll then deploy both a certified Docker image, as well as a certified Docker plugin."
Hey throwawaydc2017 sorry to hear that. I was one of the presenters, was there anything that you wanted to have covered that wasn't? If you have time could you email store-feedback@docker.com with more info?
> Also, isn't it kind of ironic that you built your company on OSS and then invited a well known destroyer of OSS onto your main stage?
I think I missed this; who are you referring to? I scrolled through the list and didn't recognize anyone that I would consider a "destroyer of OSS" so I'm not sure if I missed something, if you're just exaggerating or a little bit of both.
Illumos is a fork of the OpenSolaris code base. We still call the file system ZFS, and all the operating systems (including Illumos) that use ZFS are working together on the OpenZFS project.
I think he means Microsoft. There was a lot of emphasis on playing nice with windows in the conference. Good for developers all around. We were joking that it might be setting up the stage for an acquisition. I am personally hoping Docker follows the path of Redhat.
Playing nice with Windows has long been a priority. I saw the symptoms of this when I was following Docker GitHub ~2014-2015. In the area I cared about, Joyent compatibility, solid Linux/Unix approaches were significantly delayed or worse debated out of existance, because the concepts were incompatible with Windows. When you try to be all things, the result is something no one is satisfied with.
Guest speakers for the last (second) general keynote were:
-Guest speaker #1: Docker and Microsoft demo by Mark Russinovich, CTO Microsoft Azure
-Guest speaker #2: Docker at ADP by Keith Fulton, CTO, ADP
And just somehow I sense that the parent poster isn't objecting to the CTO of ADP. /s
(What makes the objection doubly silly is that Mark Russinovich isn't the usual corporate drone one might expect. He's the guy who did the Sysinternals tools and several editions of the "Windows Internals" reference on the architecture of the Windows OS.)
I am primarily talking about the Oracle/Docker partnership. Oracle is well known for acquiring open source projects and stripping them down to skeleton crews to remove competition, pushing the developers who built the OSS onto forks. Microsoft has improved its image, it's still got plenty of skeletons in its closet though. Remember the "Get the facts campaign"?
The fact that they’re constantly brought on stage to be paraded as a symbol of enterprise maturity and stability just tells me that the product is not mature or stable enough to speak for itself, and that the original OSS community really has no say in the direction of Docker while enterprises are sold the ship and wheel for their rubber stamp approvals.
> (What makes the objection doubly silly is that Mark Russinovich isn't the usual corporate drone one might expect. He's the guy who did the Sysinternals tools and several editions of the "Windows Internals" reference on the architecture of the Windows OS.)
From Sysinternals to the Azure CTO... what a weird transition. That guy has had an interesting career.
And most entertainingly those Sysinternals were open source originally and were immediately close sourced after he was hired during fierce negotiations by Microshaft and many angry at him for thumbing them publicly, especially for the NT SKU registry change debacle.
(Interestingly citation needed; people still circulate the code on BitTorrent of very old copies.)
Ironically a MSer told me he was famous for his infamous condition he be a Senior Engineer with an unorthodox, exceptional contractual or understood conditions: no painful managing other MS people. This only changed when made the CTO of Azure.
No idea if that's rumor's true. He's a personal hero and I'm yet to read his novel about cyber war (yes, you heard me).
Also their refusal to allow a feature to let a company change the default registry. Many enterprises need their own registry (and really want it to be the default) for a multitude of reasons... but they always just go "but we want everyone to have the same experience so we won't add this feature", even though there's been patches offered.
I recently went through a Docker Pluralsight course that made extensive use of this removed feature. I can scarcely imagine the politics behind this but it makes things much more confusing and the third party open source tool people recommend to replace it is difficult to use and doesn't provide the same information.
Part of the reason for removal (beyond just being difficult to maintain) is it is tied to the (now old) method of image storage where images were all just a tree of layers and any layer is considered an image even if you can't actually run that layer.
Today images are flat. You have a proper thing that is an image and and image is composed of layers. Some images may share layers (so they only need to be downloaded/stored once) but there is no hierarchy. You don't see layers anymore except by inspect an image (and even then only the content hash).
There are quite some reasons for this to be the preferred method.
I agree with most of this; all the change has not instilled confidence in the product. I'd use a product with zero styling and a squiggled line for a logo if it meant it wouldn't pull me away from other things while failing after an update.
If you're not paying Docker for Docker you're not a user they care about confusing. By building a large community and Google footprint for "Docker" you've served your purpose to the company. Now you can go and play in the corner with your Moby. Just don't call it "Docker" anymore because they own that trademark.
But they're certainly doing a lot with the product this year. It starts with cashing in the open source goodwill and rerouting the eyeballs to their enterprise offerings.
Together with Solomon and a core group of maintainers, we drafted some words to clarify the relationship between Moby and Docker at the Moby Summit here at DockerCon. We're updating the website at mobyproject.org to include this text as well.
Docker is, and will remain, a open source product that lets you build, ship and run containers. It is staying exactly the same from a user's perspective. Users can download Docker from the docker.com website.
Work has been ongoing to break Docker into modular components for some time, with runc and containerd as examples. Containerd for example has been donated to the CNCF. We are now completing this work with the goal being that the monolithic docker repo eventually ceases to exist, instead being assembled from a set of components.
Moby is a project which provides a "Lego set" of dozens of components, the framework for assembling them into custom container-based systems, and a place for all container enthusiasts to experiment and exchange ideas.
Docker the product will be assembled from components that are packaged by the Moby project.
The Moby project provides a command-line tool called moby which assembles components. Currently it assembles bootable OS images, but soon it will also be used by Docker for assembling Docker out of components, many of which will be independent projects.
As the Docker Engine is split up into multiple components the Moby project will also be the home for those components until a more appropriate location is found.
Tell us more about trademark and branding guidelines. What do you allow a downstream implemented by others to be called without requiring trademark fees ex by parties similar to amazon (public cloud) or ovh (data center) or redhat/fedora (software vendor) and even nvidia (cuda docker-like) .
For example fedora and firefox has strict trademark policy eventhough they are opensource.
For the Docker trademark, we will adopt guidelines similar to other open-source products, such as Ubuntu or Red Hat Enterprise Linux.
For Moby, there is no need for a strict trademark policy because Moby is a "meta-platform". In other words downstreams don't run Moby: they build their own specialized platform with moby. So they don't need to worry about the Moby trademark anymore than they need to worry about the trademark of gcc or make. Anyone who uses Moby to build their platform will be more than welcome to say "built with Moby".
- Who will be able to use the Docker trademark? If someone built a fork of Docker CE from Moby, can they still call it Docker? What about distribution packagers?
- Right now a single binary ('docker') is used to build images and runs clusters (among other things). As the monolith is broken down, will the new binaries still be named/branded 'docker'? Will there be rules about packaging only some of them? Will Docker CE/EE still have a single binary for all purposes?
> If someone built a fork of Docker CE from Moby, can they still call it Docker?
No.
> What about distribution packagers?
We're still working this out, with the guidance of a few distro packagers. There will most likely be a change of some sort, but don't want to rush anything until we are certain that it won't break users.
> Right now a single binary ('docker') is used to build images and runs clusters (among other things). As the monolith is broken down, will the new binaries still be named/branded 'docker'?
Yes. The same 'docker' binary will continue to exist. We are going to spin it out into its own repository under the docker github org. It will only act as a client, and we will encourage the development of a healthy ecosystem of alternative clients.
> Will there be rules about packaging only some of them?
Basically only Docker can create something called "Docker". Everything else will be regular open-source project rules: build, patch, package any way you want.
> Will Docker CE/EE still have a single binary for all purposes?
It depends on the target platform. Docker for Linux is a deb or rpm package with several binaries. Docker for AWS is a Cloudformation template. Docker for Mac is a Mac application with a very complex array of components built in. etc.
Could you explain the reasoning behind the need for a new name?
I ask this because would it not have been a better choice to name the other releases differently.
For example:
Docker (docker/docker) = open source development.
Docker CE (docker/docker-community) = free product release
Docker EE (docker/docker-enterprise) = commercial product release based on Docker CE (private).
The components that the monolith is being broken into could just live beside the other repos in the docker organization:
We're separating the upstream open-source project, and the downstream open-source product under 2 different names, because a lot of people in the open-source community asked us to.
It's a common concern in the free software community that having an open-source project too tightly coupled to a single company's product is a bad thing. Moving the upstream open-source project into Moby, and keeping the downstream open-source product into Docker, solves that problem.
I for one think it's fantastic that you were open to modeling the project in this way. It seems to me that there is a lot of hostility thrown at Docker in this thread because people assumed the worst based on a comment in a pull request. This is a good thing for Docker and a good thing for the Docker community and containerization in general.
> We're separating the upstream open-source project, and the downstream open-source product under 2 different names
I think a lot of the confusion comes from the downstream product split between Docker CE and Docker EE.
Many people are waiting for the other shoe to drop, because the point of releasing a separate binary version for 'CE' and 'EE' versions is very vague if the products are identical, with the extra EE functionality not really being necessary since it can be added via external services like DDC or plugins for LDAP authentication.
Is Docker EE also open-source?
Breaking out core Docker into components sourced in an external project like Moby makes it easier for you to replace the OSS components of Docker EE one piece at a time with your own private, closed-source versions where the paid development is focused.
So you get Docker CE which pulls all components entirely from Moby, then you get Docker EE which starts off identical. Then at some point you fork an individual Moby component, add some enterprise feature, and Docker EE now pulls from this. Docker EE is now a variant that's "mostly" open source with that one closed component. This component is focused and optimized for enterprise for a bit, then the same thing is done with a different component.
Rinse, repeat. "Docker EE" drifts away from its open source roots and over time loses resemblance to the original project while still taking advantage of the brand that's been built and maintained by thousands of contributors. Essentially, it's a way to co-opt an open-source product.
That's the source of my initial misgiving, anyway.
I get it, you're a business and you want to make money. Just stop being coy about Docker EE and Docker CE continuing to be identical except for the 'support and certification' mentioned in the Docker EE announcement.
So moby now uses containerd, which I guess is not related to systemd even though systemd also has its own container system. Are there any components to moby that are OSS and a part of the linux stack or is this whole endeavor to break away from linux dependencies?
Correct. Moby uses containerd because Docker uses containerd.
> which I guess is not related to systemd
You guessed correctly.
> even though systemd also has its own container system
Correct I believe systemd uses a tool called nspawn to isolate processes in containers. We don't use systemd/nspawn, but I think it would be a cool customization to build a Moby assembly that uses systemd.
> Are there any components to moby that are OSS
All Moby components are open-source.
> and a part of the linux stack
All Moby components run great on Linux.
> or is this whole endeavor to break away from linux dependencies?
Moby works great with Linux and will continue to. In fact you can build a complete custom Linux system with Moby, thanks to LinuxKit. (But you can also target an existing Linux system, LinuxKit is optional).
>> or is this whole endeavor to break away from linux dependencies?
> Moby works great with Linux and will continue to. In fact you can build a complete custom Linux system with Moby, thanks to LinuxKit. (But you can also target an existing Linux system, LinuxKit is optional).
I think the question was more along the lines of "Are these changes being made to add support for other Operating Systems?" and less "Are you abandoning Linux?"
I don't think anyone was questioning your commitment to Linux, though I kind of am now because of how defensive that non-answer was.
Somehow I feel that with this announcement and his (defensive) answers to simple questions that call for clarification, that the interest in rkt just went up some.
where's the Rocket Hub? Seriously how long is Docker Hub expected to float up there? Some are good quality, all the versions of tomcat for example. How long can such quality continue to exist?
That would be https://quay.io/ , but also the internet. Since rkt (or appc discovery rather) just relies on DNS/URL hierarchy to refer to images. Any web server can be a "registry".
I'm one of the people working in the OCI community. Discovery/distribution is something that I care alot about personally and the whole "any web server can be a registry" idea is definitely where I want OCI to go with this. As someone who helps develop a distribution (openSUSE / SUSE Linux Enterprise) my opinion is that the current state of distribution really needs to be improved.
I also recently talked to some CoreOS devs at Dockercon and have started considering extending rkt to better support OCI runtimes (and images though images are "supported" at the moment). Exciting times.
There's https://quay.io but it doesn't really function as a marketplace for official images (except the ones from CoreOS).
I wish the market of container images had more competition. Right now there's Docker Hub/Quay (expensive, lots of features) and Amazon ECR/Google Container Registry (cheap, few features).
> I think the question was more along the lines of "Are these changes being made to add support for other Operating Systems?" and less "Are you abandoning Linux?"
I see. Looks like I had misunderstood.
Yes, being able to target more different platforms is one of the reason for the change.
Docker the company looking into other host OSs would make sense to me for two reasons:
- RedHat eating Docker's lunch, and dominating Linux development
- the somewhat questionable practice of using Docker as a GPL-circumvention device (eg. commercial images pulling a Debian/Ubuntu userland on first load), though I'm unsure about the legal implications
Right, I was just assuming that the reason to build alternatives to these existing linux utilities would be to target other host OS like Windows or Mac. Thanks for the explanation.
If Moby depends on LinuxKit, why the LinuxKit's README ask you to build the Moby tool as first thing, and all the examples imply using moby? Shouldn't LinuxKit be agnostic of the existence of Moby?
I think the additional confusion here is that LinuxKit (or at least the resulting OS that powers Docker for Mac/Win/etc) used to be internally called Moby, so the build tool just hasn't been renamed.
You can use the LinuxKit components independently, but right now the easy way is to use the Moby tool. Sorry about the confusion though, we are trying to make it clearer.
Are you not concerned that this move (relabeling the open source version different from how the product is known to the broader public) might further diminish the number of contributors that are now already down 50% / back to 2014 levels?
Those stats are wrong. The project has grown massively since 2014 and continues to grow. My guess is that you are looking at the stats for a single repo, when we have broken up the project (and its contribution flow) into many smaller repos.
To answer your question: we think this move will accelerate the growth of the project, because it addresses the two most common concerns from contributors: 1) that the project is too monolithic, and 2) that it's too tied to the Docker product and company.
They're essentially attempting to distinguish their commercial offerings from the upstream open-source project. They are cornering the name "Docker" for themselves since it has a lot of name recognition.
This has pros and cons. Red Hat and Mozilla both learned they had to be aggressive with trademarks, as people were distributing sabotaged builds under their product names.
That's why Firefox in Debian is called IceWeasel, for example; any Firefox build that incorporates external patches (or even one that is built with unofficial options, iirc) must not be called Firefox (barring a waiver from the Mozilla Foundation). This allows Mozilla to pursue people who are distributing contaminated builds of "Firefox".
Changing posture on the Docker trademark will make the arguments of Docker Inc. more persuasive should they find themselves needing to make use of the same sort of remedy. It will also make a separate upstream so that Docker will now be based off the "Moby Framework". If Docker Inc. becomes defunct, "Moby" will be an separate project that lost its primary sponsor, not a defunct project bearing the name of a dead company.
The cons are that it's probably a signal that Docker is going to continue working to monetize aggressively, potentially including spurious legal threats that limit the discoverability of competitors and vendors of alternate tooling. For example, it's possible that Kubernetes will no longer work "with Docker" but "with Moby", and the only orchestration for "Docker" will be docker-swarm.
It's also a potential signal that Docker plans to gear more of its work toward its souped-up closed-source distribution available to paying customers only.
The other pro is that Docker as a product totally sucks, so their alienating people in the community will drive users to other solutions that suck less. I keep hoping the containerization fad will go back in the bottle, and while unlikely, this diminishes faith in it since Docker == containers to a lot of people, so that's good.
I (mostly) think that people are taking it too far. "Containerization" is an old concept that has been done much better more than once (cf. Solaris/Illumos Zones and FreeBSD jails; I'll refrain from classifying classic chroots as "much better" though it's debatable).
Kubernetes is massively overcomplicated and has bitten off an offensively oversized problem space. The arrogance is staggering.
Ultimately, VMs and containers are attempting to solve the same basic problems, and while containers do have some cool features, there is no way that they justify the gross tradeoffs people are making in service of the fad.
I think it's a combination of moving one of the most active repositories on github, opening a very high-profile pull request, and having that pull request linked from the #1 story on Hacker News.
We moved the repository because the first task of the Moby project is to break up the Docker engine monolith into smaller components, and spin them out into separate repos. We started doing that with libnetwork, containerd, distribution etc, but we are not done yet. docker/docker is where that monolith lives, and where its maintainers work.
Once the componentization is complete, Moby will be an integration repository, with the tools necessary to combine upstream components into a complete system. We will use it to assemble Docker, and other system builders can use it to assemble alternatives to Docker.
This redirect will probably prove disruptive, confusing, and frustrating to a lot of people. I assume it's too late now, but you might consider asking github if you can revert to allow github.com/docker/docker to remain as it was for a time, and allow moby/moby to gradually pull in the actual code.
Perhaps there are technical reasons this would be infeasible. The github landing page serves as valuable marketing, branding, and social-proof (stars, etc) even if most folks don't end up looking at the code there.
The original comment being discussed, suggested that Docker is terrible from a security point of view, which frankly I agree with.
Docker only got root privilege isolation (i.e. don't run every container as root on the host) a little over 12 months ago.
Their entire approach seems to be "build whatever buzzword batshit crazy idea cool kids want", with "hey is this shit stable, or even remotely secure?" way down the list of questions even considered during development.
Also: if someone is an OSS maintainer, and their response to bug reports is as bad as Dockers have been, sorry but they deserve to feel bad. The "you" I refer to below is a mythical OSS maintainer, not you specifically, @andrewguenther.
Maintaining software doesn't automatically remove any responsibilities you have to maintain the community. If there are huge issues with the software, you have two choices:
- acknowledge and identify a plan to solve the issues. In this case, assuming the issues are resolved in a timely fashion, I would expect people to remain courteous.
- treat the community as shit, ignore security concerns, and never fix fundamental bugs. In this instance, you should expect people to call your project out for the piece of shit it clearly is.
Let's also be clear here: we're not talking about some tin-pot project with 2 guys hacking away on their lunch breaks to find a cure for cancer, who are being told "this is crap".
Docker is developed by Docker Inc, a company that received nearly $170M in venture capital investment just in 2014-2015, and apparently makes several million a year in revenue.
You're not talking to software. You're talking to human beings about their software. You can criticize Docker's security track record in a more constructive way. There's nothing to be gained by being an asshole.
Someone suggests that Docker's reputation for security is terrible (which it is), and your response is to call someone else in the conversation an asshole, and suggest they should be more constructive?
Here is an updated version with some clarifications:
<<<
Docker is transitioning all of its open source collaborations to the Moby project going forward. During the transition, all open source activity should continue as usual.
IMPORTANT NOTE: if you are a Docker user, this does not change anything. Docker CE continues to exist as a downstream open-source product, with exactly the same interface, packages and release cycle. This change is about the upstream development of the components of Docker, and making that process more open and modular. Moby is not a replacement for Docker: it's a framework to help system engineers build platforms like Docker out of many components. We use Moby to build Docker, but you can use it to build specialized systems other than Docker.
You can learn more at http://mobyproject.org
We are proposing the following list of changes:
- splitting up the engine into more open components - removing the docker UI, SDK etc to keep them in the Docker org - clarifying that the project is not limited to the engine, but to the assembly of all the individual components of the Docker platform - open-source new tools & components which we currently use to assemble the Docker product, but could benefit the community - defining an open, community-centric governance inspired by the Fedora project (a very successful example of balancing the needs of the community with the constraints of the primary corporate sponsor)
>>>
EDIT: I'm happy to answer any follow-up questions here, if it helps clarify.