I'm happy to answer any questions you may have.
Am I on the right track?
I was there when dotCloud announced Docker in 2013 and it was truly pragmatic. I had previously burned my hands on LXC, Juju Charms and Vagrant to containerize the apps, but it was all a house of cards. I remember helping a friend ship his whole Vagrant Box to the interview panel for a job!
It’s incredible to think about how much boilerplate Docker removed right at it's launch. Thank you.
What’s funny, though, is that since it was “just” a lightning talk, it didn’t even make it into the roundup of notes I sent to my team after PyCon. But a mere year later, Docker was how my team (and seemingly everyone else) ran projects locally.
Any interesting memories from that day in 2013? From your perspective, what happened in the year that followed that caused docker to be adopted so widely? Were there other key moments or turning points?
That day is forever engraved in my memory... Off the top of my head:
I signed up for a lightning talk expecting an audience of 10-20 people, for a confidential demo of our unfinished project that we had so far only showed in 1-1 demos. Imagine my surprise when I learned Pycon does lightning talks on the main stage!
The previous speaker didn't show, so I started early. In the recording you can hear me joke that perhaps I should get his time allocation too - the MC was not amused.
I went over time and had to speedtalk through the end.. The MC once again was not amused and practically shoved me off stage :-D
We had a small sponsor booth at the even, where our small marketing team was prepared to pitch our product (DotCloud) but was thoroughly confused when a horde of people showed up after the talk, to ask about "Docker" - which as far as they knew was the founder's side project.
I drove down to Santa Clara on my newly acquired motorcycle - my very first trip on the freeway. I remember telling myself, "today is the day to take all the risks!"
I remember my first time trying to use a container was about 2015 on a beach cafe's wifi in Bocas del Toro, Panama. You can imagine how it went...
After a brief hiatus though I was back to it and since then it's been a core part of many a local dev stack and even production stack (compose) I've set up. I recall the magic moment for me being when I first realised you could mount src to use all your local native dev tools while completely virtualising everything else, including and especially whatever combo of runtimes and dependencies you need.
It was just a lightbulb moment, like it entirely solves 'works on my machine', setting new people up on a project, it just seems like how software engineering ought to work and once you have experienced it it's difficult to go back.
I'm curious, as its creator are you still using it, or as someone who can clearly look ahead to experimental ways of doing things are you on to the next thing whatever that may be? If so, what is it? :-)
Yes, I still use Docker. Although my way of using it has changed over time, like many others I assume.
My new project, Dagger, although it doesn't directly compete with Docker, does implement its own container runtime and orchestrator (for CI pipelines), and we recently added compatibility with Dockerfiles and Docker Compose files... So increasingly I find myself using Dagger instead of Docker, in the context of my CI pipelines.
On the other hand, I use docker compose on almost every personal project - like a minecraft server for my kids, home automation, AI hacks, etc. These days I just let my AI agent set it up for me, though :)
Yeah I've really been enjoying using AI to write docker/compose config and related ad hoc scripts. It was all pretty understandable anyway, but now you can just set up whatever you can think of in literally minutes, even if you just want a particular configuration as a one off. It's a great era for devops!
I take it vito talked you into adopting Garden? </injoke>
You may have already answered this somewhere, so feel free to tell me what to search for or if you have a link to hand, that works too :)
Docker in its earlier incarnations was based on openvz, which predates lxc. The lxc project was initially a kernel development project: a collection of kernel patches to add container support. The userland tool was just a companion. When it became apparent that these patches would "win" the race to be upstreamed even though openvz was much more mature, we switched our tooling to lxc. The patches matured over time, but the userland tool (what we now know as "lxc") was finicky.
The original open-source release of Docker wrapped lxc for low-level container operations. But it was just too unreliable and we swapped it out for an in-house Go implementation called "libcontainer".
As for "commercializing": Docker was famously bad at commercializing anything, we open sourced so much (maybe too much) and still people complained that we were not open enough.
Developers are a tough market to please, especially when your business revolves around open source software.
Unless you have an option to do private cloud hosting (like all these databases and other platforms do), it’s incredibly difficult to monetize.
In the case of docker, I think it’s in the nature of the product that it’s difficult to monetize. Best you can do is enterprise support contracts, but those aren’t good enough to justify all that VC funding.
That's because there was never a proper sysadmin on your team to slap some sense in your decisions.
You've got too opinionated[0], too tied on yourself[1] monster which required too much resources what you can handle alone.
And your market is not the developers but the companies where whose developers have a daytime jobs: https://news.ycombinator.com/item?id=34377674
BTW just for giggles opened hub.docker.com with Network Tools opened:
368 requests, 9.62MB, Finish: 4.50s.
For a thing which should be a static cached copy with everything bundled. Sure, it's a progress from the previous iterations when it could hang for 30 seconds before actually loading... but this is still ridiculous.
[0] https://news.ycombinator.com/item?id=46739153
[1] Eg: the syntax to point the image to a specific registry is something... bad. So bad I just abandoned running my own registry, it wasn't worth it.
PS there is still (AFAIK) no support for a transaction-like image building in Compose. So instead of something like this:
TRANSACTION START
RUNBATCH set -eux
# TODO find a better userspace architecture detection method than querying the kernel
arch="$(uname -m)"
case "$arch" in \
aarch64) gosuArch='arm64' ;
x86_64) gosuArch='amd64' ;
*) echo >&2 "error: unsupported architecture: '$arch'"; exit 1 ;
esac
curl -fL -o /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$gosuArch.asc"
curl -fL -o /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$gosuArch"
export GNUPGHOME="$(mktemp -d)"
gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4
gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu
rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc
chmod +x /usr/local/bin/gosu
gosu --version
gosu nobody true
TRANSACTION END
RUN set -eux; \
# TODO find a better userspace architecture detection method than querying the kernel
arch="$(uname -m)"; \
case "$arch" in \
aarch64) gosuArch='arm64' ;; \
x86_64) gosuArch='amd64' ;; \
*) echo >&2 "error: unsupported architecture: '$arch'"; exit 1 ;; \
esac; \
curl -fL -o /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$gosuArch.asc"; \
curl -fL -o /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$gosuArch"; \
export GNUPGHOME="$(mktemp -d)"; \
gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4; \
gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu; \
rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc; \
chmod +x /usr/local/bin/gosu; \
gosu --version; \
gosu nobody true
I went down the rabbit hole of Incus and once I realized it requires the full network aspects to be configured, then I figured using KVM, QEMU, Libvirt was more practical. Then stumbled into nspawn and it was basically ready to go.
Although, I haven’t done enough research to know if Incus provides certain useful capabilities.
It includes project-level resource separation, advanced networking, clustering, snapshots, and more. Plus, Stéphane Graber is super responsive on the forums and issue tracker. I bounced off of Proxmox, and can't recommend Incus more for hypervisor-type scenarios.
Funny how little has changed except the names, I guess.
What was your imagined use case when you first came up with it?
Did you think Docker as it is used in the industry right now is where you imagined it would be used?
Any use cases for it that the industry missed but you think would have made things even better?
(And as a result that, with ansible, continues to be my preferred way to deploy while the rest of the world passed me by…)
For one, did you get rich beyond all comprehension and did you start a band with your wealth?
Second, do you still program ?
I was lucky enough to get in this industry when Docker was new. Sure beats trying to figure out why X happens on machine A and doesn’t on machine B.
I did not start a band, but I did build a nice house, help my family etc.
I still program everyday, working on my second startup, https://dagger.io (please forgive the plug). We have a very active discord if you want to come say hi.
I guess my follow up question is once you’re financially independent, what continues to motivate you?
Is it fun?
Do you feel like you’re creating tools that the industry needs.
I imagine if I ever get rich I’d work on music and small games.
Video games that you buy once and get to play forever, self host, etc. With a full open source commitment within 5 years or so. I’d probably create a game studio in a lower cost of living country though.
We actually use the same Discord server for community and private team communications. So we are literally there all the time.
None of these things can be transposed today, so I guess I can't answer...
For what it's worth, my current project (dagger) is written in Go.
Another explicit goal was to facilitate contribution. We thought Go syntax was "vanilla" enough to attract a broader set of contributors than a more niche, opinionated language like Erlang, Haskell, Clojure etc. I think that also worked out well - Docker was known for attracting many contributors, many of them making their very first open source contributions. We even ran special meetups to help people become contributors, that was fun.
Thank you.
So, not only did I not anticipate this kind of success, I did not even have a reference frame for what success might even look like. All I knew was, I had an itch that I had to scratch, and I had nothing better to do :)
I didnt get it at that time haha.
When Docker came out it solved so many issues we were having. Thank you for scratching that itch.
I think the breakthrough with Docker was bundling the AUFS-based image layer format with a centralized container image sharing registry. This was revolutionary at the time. But another huge element of success was Solomon and others evangelizing the technology relentlessly at meetups and conferences. I distinctly remember attending a demo in 2012 where the crowd was very excited to see the Docker workflow that we now take for granted, in large part because Solomon was so good at presenting it. At the time, I was working at a cloud PaaS company where we developed a similar technology on top of LXC and BTRFS (but no image registry). Docker really made me think hard about what we were doing, although I didn't end up making any moves.
https://jpetazzo.github.io/2017/02/24/from-dotcloud-to-docke...
> the previous generation of the technology used by dotCloud [...], which was using LXC with AUFS
One small note, the earliest versions (2007-2011) were based on vserver then openvz, which predated lxc. Back then you had to patch the kernel to get container primitives. And since virtualization also required patches (eg. Xen), we had to carefully combine both sets of patches, which was poorly supported because nobody else was crazy enough to stack 2 kinds of virtualization!
https://news.ycombinator.com/item?id=2032821
Then it pivoted to docker:
I’ve been following dagger and it’s been really interesting to see the evolution of the design, and new set of abstractions. There’s also nix which has seen popularity, with their approach of building layered oci containers.
One design principle that was very important to me, was portability and avoiding fragmentation. I wanted every image name to have the same meaning in every engine, so that when writing a Dockerfile, you could have confidence that it would actually behave the same on every docker installation. In this we disagreed with other platforms (like Red Hat) who sought to "embrace and extend" Docker by changing the meaning of short names. We always fought this to avoid fragmentation of the platform. This did not make sense from Red Hat's perspective, since they were the platform - who cared about fragmenting Docker? That led to some negativity which is probably what you're remembering.
At the same time early Docker users wanted the ability to host their own images without being forced to use our central registry. Using Go's system solved both problems at once.
I personally never use the short names and have really come to like your solution. I host my own registry and I can glance at a build or compose file and immediately know if I’m depending on any external resources that I shouldn’t be.
I originally disliked it, but it’s been my favorite namespace solution for a long time now.
> If WASM+WASI existed in 2008, we wouldn't have needed to created Docker. That's how important it is. Webassembly on the server is the future of computing. A standardized system interface was the missing link. Let's hope WASI is up to the task!
How are you feeling about WASI today?
And multi stage builds are just a directed graph short hand for having separate intermediate dockerfiles. There’s nothing special about them either.
If I had to pick one thing that is tricky for someone new to docker to understand, it’s how the filesystem state is cached using the previous state, the env, and the raw command text.
When I was learning Docker I can't tell you how many people would describe it to me by analogy to a standardized shipping container, and start going off about shipping container dimensions and supersized cargo ships and trucks and railcars... and finish by smiling beatifically, like they had just explained the whole thing in a masterstroke. Let's... uh... not do that. :-D
In a problem set, the student might learn a bit about dockerfiles or docker-compose.yml... but only, I think, so they can try building and running a container of their own!
Docker syntax is one of those things that just immediately falls out of my head when I close the editor. I don't care about it at all.
Even better: there are many, many tools that build OCI images without using the Docker syntax. I haven't hand-written a Dockerfile in at least six years.
LLMs have not loved the needle here in my sphere, it was already bouncing off of the zero on the gauge.
edit: this isn't to say the syntax is good or bad (though there are several questionable parts of it) but extremely simple, and concise enough that you can just look up literally any command with great ease. It's surely a large part of its success, for better or for worse.
What made Docker unique were there things:
1. Dockerfile concept. Building images layer by layer. Caching layers. Everything in a single file. It made possible to iterate on image quickly. It made possible to implement CI builds efficiently.
2. Docker network registry concept. Putting universal images onto a server and share them with the world.
3. Momentum. A lot of software that was traditionally complicated to deploy, suddenly appeared in docker servers. I remember how fascinated I was, when I could deploy wordpress website with tiny docker-compose. And how contained it was. I could just delete every trace of it from my host. I always felt uneasy installing various things onto my host and docker just solved that uneasiness.
The main draw for me was that it made containers absolutely trivial. Sure, you could spend ages figuring out how to DIY some isolation, but in practice an awful lot of people stopped at having Apache fork its workers with a non-root uid. You want isolation and a repeatable setup? Spin up another VM and have Ansible set it up! Docker provided a batteries-included lightweight way to isolate and manage multiple applications at once.
Dockerfile? I would've been happy with a Bash script, or handing it a tarball. Caching and fast iteration? Compute is basically free. A single file? Well, there's already a whole bunch of app source code files sitting next to it, so what's a few more in a `setup/` directory? The registry? A shared base could've also been achieved with, say, a `deb` file providing the right dependencies, files, and post-install scripts.
I do agree on the momentum, though. Having every piece of software reduced from a multi-page setup guide to a single command and not risk screwing over your entire machine is absolutely magical.
I worked at Docker from 2014-2019. The most important part is actually written in C.
[I wrote a whole comment here about to explain how the hack works because I was too fixated on how technically incorrect "written in Go" is. Then I went back to read the rest of the article only to see that the paragraph immediately after this kind of explains it in its own way! Let that be a lesson to always finish reading the article before posting about it.]
The first two are just kind of the nature of containers on linux, and applies to any container technology.
Podman does allow you to run it as a non-root user (with some limitations), that was in fact one of the biggest motivations for making podman. This wasn't possible at all in the early days of docker, it wasn't until linux introduced user namespaces, where the root user inside the namespace mapped to a non-root user id outside the namespace that non-root conatiners were possible.
And podman, unfortunately IMO, is also written in go.
Knowing this, you might find rootless Podman's fork-exec method of starting containers to be more sensible. I certainly do. No root, no daemon, no socket.
If you want to learn more about the Linux Kernel namespaces which work together to make a container, this LWN series is really good: https://lwn.net/Articles/531114/
Everything in that list feel natural when you know how things evolves over many years. Maybe it is just me getting older and less curious.
Now it's possible to create containers without elevated privileges (assuming user namespaces are allowed), but that wasn't possible until long after docker had established the client/server model.
rkt (2013) was CoreOS's alternative to Docker and was daemon-less and failed.
Podman came along relatively late (2018) and benefited from a lot of Linux kernel and systemd improvements since Docker was designed.
So while arguably the client/server model is outdated for container management tools, I'm not sure it's weird. This is my workstation for example:
schmichael@fedora:~$ ps -e -o comm= | rg 'd$' | wc -l
67
[1] "Docker implemented in around 100 lines of bash" https://github.com/p8952/bocker
One of the gnarliest issues I recall was where network namespaces could occasionally not be cleaned up correctly. So "randomly" (but more likely under certain loads) the networking of a reused thread would get jacked and Docker could no longer pull images.
(rule broken by almost every language in common use, unfortunately)
Because most web servers use OS threads. Nginx, Apache, litespeed, they all spin up `core count` threads but they can all work single threaded just fine as well.
NGINX uses a state machine per task, which as far as I know is kind of what Rust's async functions end up desugaring to.
Try to make your own VM runtime next, you might be surprised!
A few cool tricks:
`range shell python:3.12` - opens a shell image without pulling the entire image.
`range shell ubuntu --commands` - you are in Ubuntu. Type any command Ubuntu doesn't have and it comes from the image that has it. On the fly.
`range run --network none ubuntu` - network-isolated sandbox
range run ebs://snap-... -- cat /etc/os-release - opens an EBS snapshot with no EC2 instance
Another cool trick:
`range run python:3.12 --mount hf://moonshotai/Kimi-K2-Instruct:/model -- du -sh /model` says 959G instantly :)
Check it out here - https://getrange.sh
Most importantly, Range doesn't need any daemon, any container runtime, any registry or any image conversion. One binary only. On a Mac it boots its own tiny Linux VM per session, and on Linux it uses KVM.
Range always reads only the bytes your program touches. Nothing else. You can prob shell into a 1TB image from a Raspberry Pi :)
Containers From Scratch • Liz Rice • GOTO 2018
This is very important for debugging! Not just that process, but the processes running in that Docker can be found in /proc.
Why this is useful is that you can run gdb outside of that docker and point it to a process in the docker, including setting the file system root to the docker one, so that it gdb reads the correct objects from the correct paths, to get the right symbols and addresses.
It's not a vm: It's all namespaces, baby! Unshare one or more of your mnt, pid, net, ipc, uts, user, cgroup, or time namespaces and you have a "container".
Performance is basically irrelevant because you're not really interacting with the running application. You're running as root, so security-critical. You're talking to the internet for things like fetching images. You need rather complicated business logic to deal with all the possible options, protocols, and file formats.
Sure, you could do it in C, but dealing with dependencies is a major PITA and you're permanently one tiny mistake away from getting pwned, while not needing any of C's strengths. Go is safe, easy to write, and ships as a single batteries-included binary: what's not to like?
Go is good for writing web servers. For system software? Not so much. 90% of Unix is written with C. It's not that bad.
I love Go and use it for everything I can but will absolutely concede this point. It's clearly possible to write the most successful container runtimes on earth in Go, but you absolutely have to fight the language/runtime sometimes. Since containers are based on processes, handing off a container spec to a minimal rust runtime seems like what we should all be doing by now. crun has just proven good enough that AFAIK no one has RIIR.
And implying the system kernel still has to run the container kinda ignores kata containers and gvisor runtimes.
> Did you know that databases are just a file?
oOoOoOoOo
Mostly joking. It's cool to see tech demystified, but I feel like a lot of things we use and take for granted are like this.
Side note almost nothing about Docker is weird. This article chooses rather boring design decisions and picks on them.
"It's an idempotent language for OS level dependencies so you can be sure exactly what is running on your server"
Gimme gimme gimme!
Other weird facts about OSes: a filesystem is a database, the Linux kernel is a monolithic program, "hidden" dot-files were a Unix bug that got carried forward, deleting a file doesn't remove the file from your machine, virtual memory is a lie, your computer can't run multiple programs at once (it's just running one at a time real fast), in 12 years most systems will crash due to one number getting too big, you technically never need to reboot because you can launch a different version of Linux while already running Linux, you can freeze a program to a file and then later resume running that same program in the same state (but nobody uses it), files aren't actually stored in directories, there's a maximum number of files (and size of file) on most filesystems (which you can use to lock up a machine, despite file quotas existing and nobody using them), you can lock up most systems by forking too many times / using too much RAM / accessing disk too much, there's a default number of open files and/or TCP ports which is artificially low and you can use to again lock up a machine, pipes are just two file descriptors connected to one memory buffer, an IP address is just one long number split up into bytes, etc.
This is not a slight on Docker itself. It’s a slight on the conditions that induced Docker to exist.
One of the great selling points of one of my projects is “this is easy to run without needing Docker”.
And folders are graphical representations of directories--sometimes, not always--and not directories themselves. When will the newbies get this?
I know a lot about how http works, and it comes in handy, but I can understand how juniors get away with not knowing it as fully as I do.