Docker-in-Docker

Say you’re hosting a bunch of apps as Docker containers, possibly via Traefik and you want to control those containers from some kind of Admin interface, which too is running in a Docker environment (so that the Traefik routing to the admin interface is easily implemented).

The way to do this is to expose /var/run/docker.sock to the container itself. Of course this is an inherent security issue since it allows the container to control host; use only for containers that you absolutely trust.

Plain Docker

Just run

$ docker run --rm -ti -v /var/run/docker.sock:/var/run/docker.sock ubuntu /bin/bash

Unfortunately, the Ubuntu image won’t have docker-cli installed. There are ways to install the docker-cli only, or via docker java client library. But, when running Ubuntu-on-Ubuntu, the easiest way is to mount the docker binary itself into the container:

$ docker run --rm -ti -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker ubuntu /bin/bash
# docker ps
CONTAINER ID   IMAGE       COMMAND              CREATED          STATUS          PORTS                                   NAMES
2c2e38401d58   ubuntu      "/bin/bash"          45 seconds ago   Up 44 seconds                                           cranky_maxwell
# docker run -d --name my-apache-app -p 8080:80 httpd:2.4
# docker ps
CONTAINER ID   IMAGE       COMMAND              CREATED          STATUS          PORTS                                   NAMES
562cfc3bb562   httpd:2.4   "httpd-foreground"   3 seconds ago    Up 3 seconds    0.0.0.0:8080->80/tcp, :::8080->80/tcp   my-apache-app
2c2e38401d58   ubuntu      "/bin/bash"          45 seconds ago   Up 44 seconds                                           cranky_maxwell

Even though the httpd container was started from an Ubuntu Docker container, httpd container is actually running on the host machine:

  • We’re controlling the docker daemon running on the host machine; we’re just controlling it from a docker container.
  • Similar to when you ssh to a remote machine and run docker.
  • it’s not even possible to start a container within a container since the Docker daemon only runs on the host machine.

You can verify the httpd is running on the host machine, by exiting the Ubuntu container (which is immediately killed and removed because of --rm), and listing running docker containers from the host machine:

# exit
exit
$ docker ps
CONTAINER ID   IMAGE       COMMAND              CREATED              STATUS              PORTS                                   NAMES
562cfc3bb562   httpd:2.4   "httpd-foreground"   About a minute ago   Up About a minute   0.0.0.0:8080->80/tcp, :::8080->80/tcp   my-apache-app

Go to localhost:8080, to see the httpd still running.

Docker-Compose

Unfortunately the trick with mounting the docker-compose binary to your container won’t work anymore since docker-compose is a python script and requires other files to be present as well.

However, that is just a minor inconvenience since docker-compose is just a front for docker, so you should be able to achieve the same thing with just the docker binary.

docker-buildx

It’s even possible to build images via docker build from within docker container, including the buildx extension:

$ docker run --rm -ti -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker -v /usr/libexec/docker/cli-plugins/docker-buildx:/usr/libexec/docker/cli-plugins/docker-buildx ubuntu /bin/bash

You can for example build vaadin-boot-example-gradle:

$ sudo apt update && sudo apt install -y git
$ git clone https://github.com/mvysny/vaadin-boot-example-gradle/
$ cd vaadin-boot-example-gradle
$ docker build --cache-from "type=local,src=/root/build-cache" --cache-to "type=local,dest=/root/build-cache" -t test/vaadin-boot-example-gradle:latest .

The build will be performed outside of the docker container: docker client will zip & send the folder to the docker daemon to perform the build. The --cache-from and --cache-to will however refer path from within the docker container; if you wish to have a persistent cache you can simply mount the path as a volume.

Permission issues

In certain docker images (e.g. Jenkins), the main process is not running as root but rather as a regular user. In such case mounting docker.sock as volume isn’t enough and Jenkins will print: “ERROR: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head “http://%2Fvar%2Frun%2Fdocker.sock/_ping”: dial unix /var/run/docker.sock: connect: permission denied”

You can verify that by running:

$ docker container exec CONTAINER_ID ps -ef n
     UID     PID    PPID  C STIME TTY      STAT   TIME CMD
    1000       1       0  0 12:17 ?        Ss     0:00 /usr/bin/tini -- /usr/local/bin/jenkins.sh
    1000       7       1  3 12:17 ?        Sl     0:24 java -Duser.home=/var/jenkins_home -Djava.awt.headless=true -Xmx512m -Djenkins.model.Jenkins.slaveAgentPort=50000 -Dhudson.lifecycle=hudson.lifecycle.ExitLifecycle -jar /usr/share/jenkins/jenkins.war
    1000     375       0  0 12:31 ?        Rs     0:00 ps -ef n

Notice the UID of 1000, which is not 0, which is the UID of the root user.

One solution is to run Jenkins as root via docker run -u 0, but a better way is to use --group-add 125, where 125 is the group ID of the docker group on the host system. You can obtain the group id by running getent group docker.

Also see docker-compose: group_add on StackOverflow.

Written on November 18, 2024