Setting up Rootless Docker in Rootless Docker

Friday 02 October 2026 · 59 mins read · Viewed 2 times

Table of contents 🔗

Introduction 🔗

Everything start with this question: How can I run multiple CI workers on the same node with a high level of security and isolation?

Naively, you use ephemeral CI workers inside a container: when the worker finishes its job, the process ends and the container is restarted with a clean state. Think about GitHub Actions Runner Controller (ARC): ephemeral CI workers inside a Kubernetes cluster, automatically scaled to the needs of the workload and automatically restarted when the job ends.

When you start using GitHub ARC, you encounter now this issue: How can I use Docker buildx inside a container?

Most people would suggest to either:

  • Use Docker in Docker: The host docker daemon is rootful and launches ephemeral CI workers with a restart policy. Then, a second docker daemon is launched inside a container, on the host docker. The CI workers use the docker daemon socket of the containerized docker to build images.
  • Use Rootless Docker in Docker: The host docker daemon is rootless (not started by root or by the init system) and launches ephemeral CI workers with a restart policy. Then, a second docker daemon is launched inside a container, on the host docker. The CI workers use the docker daemon socket of the containerized docker to build images.
  • Use an alternative to Docker to build images like Buildah: The host docker daemon is rootless and the CI workers use Buildah to build images.

We'll comment on each of those options, then I'll show you how to set up Rootless Docker in Rootless Docker, which is the best option when you want to use buildx optimizations and buildx github actions.

Docker in Docker: Easy to set up, but an inscure setup 🔗

Here's how to run Docker in Docker:

  1. Start a rootful Docker daemon: By default, when installing docker, the docker daemon runs as root. The daemon is started by systemd and can be checked with systemctl status docker.service or systemctl status dockerd.

    You might think that you are running rootless, but to be able to run Docker commands, you had to join the docker group.

    Joining the docker groups allows you to run Docker commands without sudo, but the daemon is still running as root.

  2. Start a Docker DinD container:

    1docker run -d \
2   --privileged \
3   --name docker-dind \
4   docker:dind

    You can already see the vulnerability: it uses --privileged. This flag:

    • Disables Seccomp: Remove restrictions on system calls like mount, pivot_root, ... which are required to setup a user namespaces and container filesystems.
    • Disables AppArmor/SELinux: Also remove additional restrictions on system calls, linux capabilities and file access.
    • Grants all Linux Capabilities: It grants capabilities like SYS_ADMIN, NET_ADMIN, ..., basically bypass permission checks and is in practically equivalent to running as a superuser.
    • Exposes sensitive mounts:
      • It mounts /dev from the host: every devices on the host are exposed, including drives, input devices, ... which are unused by Docker, so no reason to expose them.
      • It mounts /sys/fs/cgroup from the host: allows to create new cgroup which are used to isolate and track CPU, memory, network, ... of the nested container.
      • It remove the masks on /proc and /sys from the host: allows to read and write on /proc.

    So, in other word, when you are running CI jobs (or any multi-tenant workload), the nested containers are able to actually elevate their privileges, and possibly escape the container isolation.

In fact, there is no point to continue showing you how to run CI actions. I will show you how to escape the container isolation:

1# Enter the dind container to emulate a workload. THis can be the CI worker, or
2# anything else.
3docker exec -it docker-dind sh
4
5# Mount the host drive
6mount /dev/nvme0n1p1 /mnt
7
8# Yay, you've successfully escaped the container isolation!

Of course, you may say that "the nested docker daemon will launch secure containers, so it's not a problem". But in fact it is: the isolation you wanted from docker in docker isn't actually achieved, and it is in fact a useless method of isolation.

Here's a possible attack scenario if you were to run a GitHub Actions worker alongside Docker in Docker. The Github Actions worker is not privileged but the DinD container is.

Assuming this setup:

 1+-------------------------------------------------------------------+
 2|                        Rootful Docker Host                        |
 3|                                                                   |
 4|   +-----------------------+               +-------------------+   |
 5|   |  GH Action Runner     |               |    DinD Container |   |
 6|   |  Container            |               |  (Docker-in-Docker)   |
 7|   |                       |  Docker       |                   |   |
 8|   |   +---------------+   |  Socket       |   +-----------+   |   |
 9|   |   | Docker CLI    |---|---------------|---| Docker    |   |   |
10|   |   +---------------+   |  /var/run/    |   | Daemon    |   |   |
11|   |                       |  docker.sock  |   +-----------+   |   |
12|   +-----------------------+               +-------------------+   |
13|                                                                   |
14+-------------------------------------------------------------------+
15
  1. GitHub Actions executes docker run --rm --privileged alpine /bin/sh -c "mount /dev/nvme0n1p1 /mnt && ls /mnt"
  2. Since the docker daemon is running inside a privileged container, and the GitHub Actions worker is executing a privileged command, the attacker will actually succeed to mount the host drive.

Entire PoC:

1docker run -d \
2  --privileged \
3  --name docker-dind \
4  -v shared_socket:/var/run \
5  docker:dind
6
7docker run --rm \
8  -v shared_socket:/var/run \
9  docker run --rm --privileged alpine /bin/sh -c "mount /dev/nvme0n1p1 /mnt && ls /mnt"

If we assume that docker run --rm --privileged alpine /bin/sh -c "mount /dev/nvme0n1p1 /mnt && ls /mnt" was a CI job, the attacker has successfully escaped the container isolation.

Therefore, Docker in Docker is unsafe and unsuitable for isolation

Rootless Docker in Docker 🔗

Understanding the differences between Rootful Docker and Rootless Docker 🔗

Rootful Docker 🔗

To create a container, Docker needs to:

  • Create a network layer by:

    • Setting up a bridge network interface
    • Setting up a virtual ethernet device (veth)
    • Wire the veth into the container namespace (ip link set veth-guest netns mycontainer)
    • Do the classic network interface configuration (IP configuration, routing, ...)
    • Setting up firewall, NAT and port forwarding rules with iptables

  • Create a namespace with flags (like pid, ipc, uts, net ...) using the clone system call.

    • The process will be running in a "namespace": an abstraction that allows to isolate certain aspects of Linux like:

      • Cgroup root directory
      • IPC, POSIX system queues
      • Network devices, stacks, ports and etc.
      • Mount points
      • PID (process IDs)
      • Time (boot and monotonic cloks) (Not used by Docker)
      • User and group IDs (Used by Rootless Docker. It'll be explained later.)
      • UTS (Hostname and NIS domain name)

    • It is possible to test this by using:

      1# Docker rootful equivalent
2unshare --uts --ipc --net --pid --fork --mount-proc --cgroup --mount

  • Setup cgroups to track CPU, memory, network, ...

    • It is possible to test this by setting up:

       1# As root
 2mkdir /sys/fs/cgroup/fake-docker-container
 3
 4# Limit memory to 512 MB in bytes = 536870912
 5echo "536870912" | sudo tee /sys/fs/cgroup/fake-docker-container/memory.max
 6
 7# Limit cpu to 50000us quota per 100000us period
 8echo "50000 100000" | sudo tee /sys/fs/cgroup/fake-docker-container/cpu.max
 9
10# Launch your namespace
11# Don't specify --cgroup here because we are using a custom cgroup
12unshare --uts --ipc --net --pid --fork --mount-proc --mount /bin/sh -c "sleep infinity" &
13FAKE_CONTAINER_PID=$!
14
15# Attach the namespace to the cgroups
16echo "$FAKE_CONTAINER_PID" | sudo tee /sys/fs/cgroup/fake-docker-container/cgroup.procs
17# NB: Docker doesn't use two processes to set up cgroups. We're doing like
18# this because we can't move a namespace to a cgroup before it is created
19# due to `unshare`.

  • Mount the container image file system with a read-only layer and a writable layer. This is done by using OverlayFS.

    • It is possible to test this by using:

       1# As root
 2# lower: read-only layer
 3# upper: writable layer containing changes (additions, modifications, deletions)
 4# work: working directory for overlayfs (not used by the user)
 5# merged: the merged directory
 6mkdir -p /tmp/overlay-demo/{lower,upper,work,merged}
 7
 8# Add some files to the lower (read-only) layer
 9echo "Original content from Base Image" > /tmp/overlay-demo/lower/base_file.txt
10echo "File to be deleted" > /tmp/overlay-demo/lower/delete_me.txt
11
12# Mount the overlay
13mount -t overlay overlay \
14  -o lowerdir=/tmp/overlay-demo/lower,upperdir=/tmp/overlay-demo/upper,workdir=/tmp/overlay-demo/work \
15  /tmp/overlay-demo/merged
16
17# Check the content of the merged directory
18ls /tmp/overlay-demo/merged
19
20# Delete a file and check if it is present on the merged directory and the
21# lower directory
22rm /tmp/overlay-demo/merged/delete_me.txt
23stat /tmp/overlay-demo/merged/delete_me.txt
24stat /tmp/overlay-demo/lower/delete_me.txt
25
26# If you've setup the namespace from earlier, if you exit the namespace,
27# the mounted overlay will be unmounted.

  • Drop privileges and linux capabilities

  • Pivot the root filesystem and unmount the old root filesystem. Docker does this basically:

    1pivor_root /tmp/overlay-demo/merged /tmp/overlay-demo/merged/old_root
2cd /
3# --lazy: detach the filesystem now, clean up things later
4umount --lazy /tmp/overlay-demo/merged/old_root
5rmdir /old_root

    (Why pivot_root over chroot is something you can google yourself. But I'm sure you guessed it: for security reasons.)

In all these steps, here's a comprehensive list of all syscalls used in Docker's container setup, organized by step:

  1. Create Namespaces

    1unshare()           # Create new namespaces (PID, IPC, UTS, NET, Mount, Cgroup)
2fork()              # Create child process (when using --fork flag)

  2. Setup Cgroups

    1mkdir() / mkdirat()         # Create cgroup directories
2open() / openat()           # Open cgroup files for writing
3write()                     # Write limits to cgroup files
4close()                     # Close file descriptors
5statfs()                    # Check cgroup filesystem info

  3. Mount OverlayFS

    1mkdir() / mkdirat()         # Create lower, upper, work, merged directories
2mount()                     # Mount overlay filesystem
3open() / openat()           # Create files in lower layer
4write()                     # Write file contents

  4. Manage Files in Overlay

    1unlink() / unlinkat()       # Delete files
2stat() / lstat() / fstat()  # Check file status/metadata

  5. Drop Privileges & Capabilities

    1setuid() / setuid32()       # Change effective user ID
2setgid() / setgid32()       # Change effective group ID
3setgroups()                 # Set supplementary group IDs
4capset()                    # Drop Linux capabilities
5prctl()                     # Process control (PR_CAPBSET_DROP, etc.)
6seccomp()                   # Setup seccomp filter (optional)

  6. Pivot Root & Cleanup

    1pivot_root()        # Pivot to new root filesystem # Requires CAP_SYS_ADMIN + real root
2chdir() / fchdir()  # Change directory
3umount() / umount2() # Unmount old root (with MNT_LAZY)
4rmdir()             # Remove old_root directory

  7. Execute Container Init

    1execve()            # Execute container's init process

(I might have missed some syscalls. If you find any, please let me know.)

Rootless docker 🔗

Leveraging user namespaces to avoid root 🔗

When using rootless docker, you run docker as a non-root user (duh!), but this means some commands will simply not work like mount or pivot_root which are vital to the container setup process.

As you saw earlier, we can use --user in unshare to create a user namespace. In this user namespace, we can map the current user ID to the root user ID using --map-root-user. It is a "fake" root in some sense, but it cannot really call privileged command like mount:

1unshare --user --map-root-user /bin/sh -c "mount --make-private / && touch /tmp/resolv.conf && mount --bind /etc/resolv.conf /tmp/resolv.conf"
2# mount: /: permission denied.

But in a mount namespace, we can do it:

1unshare --mount --user --map-root-user /bin/sh -c "mount --make-private / && touch /tmp/resolv.conf && mount --bind /etc/resolv.conf /tmp/resolv.conf && cat /tmp/resolv.conf"
2# ok

However, this is not enough to set up a rootless container: we need a UID/GID namespace in which we are allowed to use. For that, we set up not only a user namespace but also UID and GID mapping.

This is something that is done once by the root user to provide a specific user "UID/GID" space. The root user must set up:

1# As root
2# Format is <user>:<start-uid>:<count>
3echo "my-user:100000:65535" >> /etc/subuid
4echo "my-user:100000:65535" >> /etc/subgid

The mapping will looks like:

1# Host     Container
2100000 --> 0
3165535 --> 65535

User namespace with UID/GID mapping allows us to create rootless containers.

Using user namespaces, we'll be able to replace every rootful component with a rootless equivalent.

Using slirp4netns to connect the external network to the namespace 🔗

To set up the network in a rootless environment, we cannot use veth and bridge anymore since it requires the root privileges. Instead, we can leverage TAP devices by using slirp4netns. Of course, you can't set up the network without a user namespace, but basically, the setup looks like this:

 1unshare --user --map-root-user \
 2  --net \
 3  --mount \
 4  /bin/sh -c "sleep infinity" &
 5FAKE_CONTAINER_PID=$!
 6
 7slirp4netns --configure --mtu=65520 --disable-host-loopback $FAKE_CONTAINER_PID tap0 &
 8
 9# Enter back the user namespace
10nsenter --net --user --preserve-credentials -t $FAKE_CONTAINER_PID
11
12# Check the network status
13ip a
14# 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
15#     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
16#     inet 127.0.0.1/8 scope host lo
17#        valid_lft forever preferred_lft forever
18#     inet6 ::1/128 scope host proto kernel_lo
19#        valid_lft forever preferred_lft forever
20# 2: tap0: <BROADCAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UNKNOWN group default qlen 1000
21#     link/ether 42:2b:eb:14:1d:a8 brd ff:ff:ff:ff:ff:ff
22#     inet 10.0.2.100/24 brd 10.0.2.255 scope global tap0
23#        valid_lft forever preferred_lft forever
24#     inet6 fe80::402b:ebff:fe14:1da8/64 scope link proto kernel_ll
25#        valid_lft forever preferred_lft forever
26
27# Test network access
28ping 8.8.8.8 -c 4
29# PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
30# 64 bytes from 8.8.8.8: icmp_seq=1 ttl=255 time=28.0 ms
31# 64 bytes from 8.8.8.8: icmp_seq=2 ttl=255 time=18.7 ms
32# 64 bytes from 8.8.8.8: icmp_seq=3 ttl=255 time=17.0 ms
33# 64 bytes from 8.8.8.8: icmp_seq=4 ttl=255 time=21.9 ms
34#
35# --- 8.8.8.8 ping statistics ---
36# 4 packets transmitted, 4 received, 0% packet loss, time 3001ms
37# rtt min/avg/max/mdev = 17.005/21.398/28.034/4.208 ms
Setting up cgroups 🔗
NOTE

Since I'm running Gentoo Linux with OpenRC, my setup on user cgroups is different from a SystemD setup. It'll be assumptions for this section.

SystemD mount user cgroups at /sys/fs/cgroup/user.slice/user-<UID>.slice where the user can create their own cgroups.

You can test using:

1mkdir -p /sys/fs/cgroup/user.slice/user-$(id -u).slice/my-container
2echo 100M > /sys/fs/cgroup/user.slice/user-$(id -u).slice/my-container/memory.max
3echo $FAKE_CONTAINER_PID | tee /sys/fs/cgroup/user.slice/user-$(id -u).slice/my-container/cgroup.procs
Setting up the container filesystem 🔗

OverlayFS cannot be used without root, so we need to use FUSE (Filesystem in USErspace). To do that, we can use fuse-overlayfs:

 1# lower: read-only layer
 2# upper: writable layer containing changes (additions, modifications, deletions)
 3# work: working directory for overlayfs (not used by the user)
 4# merged: the merged directory
 5mkdir -p /tmp/overlay-demo/fuse/{lower,upper,work,merged}
 6
 7# Add some files to the lower (read-only) layer
 8echo "Original content from Base Image" > /tmp/overlay-demo/fuse/lower/base_file.txt
 9echo "File to be deleted" > /tmp/overlay-demo/fuse/lower/delete_me.txt
10
11# Mount the overlay
12fuse-overlayfs \
13  -o lowerdir=/tmp/overlay-demo/fuse/lower,upperdir=/tmp/overlay-demo/fuse/upper,workdir=/tmp/overlay-demo/fuse/work \
14  /tmp/overlay-demo/fuse/merged
15
16# Check the content of the merged directory
17ls /tmp/overlay-demo/fuse/merged
18
19# Delete a file and check if it is present on the merged directory and the
20# lower directory
21rm /tmp/overlay-demo/fuse/merged/delete_me.txt
22stat /tmp/overlay-demo/fuse/merged/delete_me.txt
23stat /tmp/overlay-demo/fuse/lower/delete_me.txt
24
25# If you need to unmount: fusermount -u /tmp/overlay-demo/fuse/merged

At this point, we've got everything to make a container working. pivot_root works in a user namespace, so we can also switch the root.

Running rootless docker in docker 🔗

Here's how to run Rootless Docker in Docker:

 1docker run --rm -v shared_socket:/work alpine chown 1000:1000 /work
 2
 3docker run -d --name docker-dind \
 4  --security-opt seccomp=unconfined \
 5  --security-opt apparmor=unconfined \
 6  --security-opt systempaths=unconfined \
 7  --device /dev/fuse \
 8  --device /dev/net/tun \
 9  -v shared_socket:/var/run/user/1000 \
10  docker:dind-rootless
11
12# Test it
13docker run --rm \
14  -v shared_socket:/var/run \
15  docker run --rm alpine echo "Hello World"

To explain some of the flags:

1  --security-opt seccomp=unconfined \
2  --security-opt apparmor=unconfined \

is required to allow the namespace to run system calls like mount or pivot_root. It does not provide any additional privileges, but remove common protections for a container (it is not expected for a container to run mount or pivot_root).

1  --security-opt systempaths=unconfined \

similar to seccomp=unconfined and apparmor=unconfined, it allows the namespace to mount /sys, which are required for cgroups to work. This is also a common protection since there are no reasons for a container to interact with /sys..., but this isn't the case for Rootless Docker in Docker.

1  --device /dev/fuse \

As we said earlier, for fuse-overlayfs to work, we need to mount /dev/fuse.

1  --device /dev/net/tun \

And for slirp4netns to work, we need to mount /dev/net/tun.

You can test the PoC from earlier and see that it doesn't work:

 1docker run --rm \
 2  -v shared_socket:/var/run \
 3  docker run --rm --privileged alpine /bin/sh -c "mount /dev/nvme0n1p1 /mnt && ls /mnt"
 4# docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "sysfs" to rootfs at "/sys": mount src=sysfs, dst=/sys, dstFd=/proc/thread-self/fd/15, flags=MS_NOSUID|MS_NODEV|MS_NOEXEC: operation not permitted
 5
 6# And if running without privileges:
 7docker run --rm \
 8  -v shared_socket:/var/run \
 9  docker run --rm alpine /bin/sh -c "mount /dev/nvme0n1p1 /mnt && ls /mnt"
10mount: permission denied (are you root?)

We have now a proper protection layer for nested containers. But, what if the attacker was able to escape the nested container and the DinD container? He would be root!

So, time to run Rootless Docker in Rootless Docker.

Rootless Docker in Docker 🔗

If you try to run Rootless Docker in Rootless Docker, you'll get the following error:

 1docker run --rm -v shared_socket:/work alpine chown 1000:1000 /work
 2
 3docker run -d --name docker-dind \
 4  --security-opt seccomp=unconfined \
 5  --security-opt apparmor=unconfined \
 6  --security-opt systempaths=unconfined \
 7  --device /dev/fuse \
 8  --device /dev/net/tun \
 9  -v shared_socket:/var/run/user/1000 \
10  docker:dind-rootless
11
12docker logs docker-dind
13# [rootlesskit:parent] error: failed to setup UID/GID map: newuidmap 75 [0 1000 1 1 100000 65536] failed: newuidmap: write to uid_map failed: Operation not permitted

Why is that? Remember, we are pre-allocated a certain range of UIDs and GIDs, between 0 and 65535.

The error tells that rootlesskit (the process responsible for setting up rootless in Docker) is unable to setup the UID/GID mapping. Looking at the parameters sent in that error:

1newuidmap 75 0 1000 1 1 100000 65536

Breaking this down:

Parameter Value Meaning
PID 75 Process ID to set up UID mapping for
ID-inside-ns 0 UID 0 inside the user namespace
ID-outside-ns 1000 Maps to UID 1000 outside the namespace
Length 1 Length of this mapping (just 1 user)
ID-inside-ns 1 UID 1 inside the namespace
ID-outside-ns 100000 Maps to UID 100000 outside the namespace
Length 65536 Length of this mapping (65,536 users)

Or, in other word, rootlesskit is trying to map 0 (container-side) to 1000 (host-side), and 1-65536 (container-side) to 100000-165536 (host-side).

However, we just said earlier that we are allocated between 0 and 65535, so 100000-165536 are actually out of range! So, we need to override /etc/subuid and /etc/subgid inside the container (or allow more UIDs/GIDs on the host)!

 1# rootless is the user in docker:dind-rootless
 2echo "rootless:30000:20000" > /tmp/subuid
 3echo "rootless:30000:20000" > /tmp/subgid
 4# Mapping:
 5# 0 -> 30000
 6# 20000 -> 50001
 7# Above 20000, the container isn't allowed to chmod, chown, su, etc... because
 8# it is outside of the user namespace (we're hitting the host UIDs/GIDs)
 9
10docker run -d --name docker-dind \
11  --security-opt seccomp=unconfined \
12  --security-opt apparmor=unconfined \
13  --security-opt systempaths=unconfined \
14  --device /dev/fuse \
15  --device /dev/net/tun \
16  -v shared_socket:/var/run/user/1000 \
17  -v /tmp/subuid:/etc/subuid \
18  -v /tmp/subgid:/etc/subgid \
19  docker:dind-rootless
20
21# Test it
22docker run --rm \
23  -v shared_socket:/var/run \
24  docker run --rm alpine echo "Hello World"

Aaaand it works! Pretty cool right?

Setting up GitHub Actions runners with Rootless Docker 🔗

Assuming an ephemeral GitHub Actions Runner setup, your script should looks like:

 1REPO="org/repo"
 2GH_RUNNER_NAME="rootless-docker-runner"
 3LABELS='["self-hosted","linux","x64"]'
 4ENCODED_JIT_CONFIG=$(curl -fsSL -X "POST" \
 5    -H "Authorization: Bearer ${GH_TOKEN}" \
 6    -H "Accept: application/vnd.github+json" \
 7    "https://api.github.com/repos/${REPO}/actions/runners/generate-jitconfig" -d '{
 8    "name": "'"$GH_RUNNER_NAME"'",
 9    "runner_group_id": 1,
10    "labels": '"$LABELS"',
11    "work_folder": "/home/runner/_work"
12  }' | jq -r '.encoded_jit_config')
13
14docker run --rm -v work:/work alpine chown 1001:1001 /work
15docker run --rm -v shared_socket:/work alpine chown 1001:1001 /work
16
17docker run --rm --name "runner-container" \
18  -v "work:/home/runner/_work" \
19  -v "shared_socket:/run/user/1001" \
20  -e DOCKER_HOST=unix:///run/user/1001/docker.sock \
21  -e TMPDIR=/home/runner/_work/tmp \
22  "ghcr.io/actions/actions-runner:2.337.0" \
23  bash -c "mkdir -p \$TMPDIR && chmod 755 \$TMPDIR && exec ./run.sh --jitconfig $ENCODED_JIT_CONFIG" &
24RUNNER_PID=$!
25
26wait "$RUNNER_PID"

To make volume mounts work, the volume work will be shared between the runner and the rootless Docker-in-Docker container.

You can already tell there will be an issue with UID and GID: the runner is running with UID and GID 1001, but the docker host is running with UID and GID 1000. So, we need to change the UID and GID of the docker daemon to 1001.

To do this, we need to override /etc/group and /etc/passwd inside the container, and set up proper permissions in the working directories.

 1cat <<EOF > /tmp/group
 2root:x:0:root
 3bin:x:1:root,bin,daemon
 4daemon:x:2:root,bin,daemon
 5sys:x:3:root,bin
 6adm:x:4:root,daemon
 7tty:x:5:
 8disk:x:6:root
 9lp:x:7:lp
10kmem:x:9:
11wheel:x:10:root
12floppy:x:11:root
13mail:x:12:mail
14news:x:13:news
15uucp:x:14:uucp
16cron:x:16:cron
17audio:x:18:
18cdrom:x:19:
19dialout:x:20:root
20ftp:x:21:
21sshd:x:22:
22input:x:23:
23tape:x:26:root
24video:x:27:root
25netdev:x:28:
26kvm:x:34:kvm
27games:x:35:
28shadow:x:42:
29www-data:x:82:
30users:x:100:games
31ntp:x:123:
32abuild:x:300:
33utmp:x:406:
34ping:x:999:
35nogroup:x:65533:
36nobody:x:65534:
37docker:x:2375:
38dockremap:x:101:dockremap
39rootless:x:1001:
40EOF
41
42cat <<EOF > /tmp/passwd
43root:x:0:0:root:/root:/bin/sh
44bin:x:1:1:bin:/bin:/sbin/nologin
45daemon:x:2:2:daemon:/sbin:/sbin/nologin
46lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
47sync:x:5:0:sync:/sbin:/bin/sync
48shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
49halt:x:7:0:halt:/sbin:/sbin/halt
50mail:x:8:12:mail:/var/mail:/sbin/nologin
51news:x:9:13:news:/usr/lib/news:/sbin/nologin
52uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin
53cron:x:16:16:cron:/var/spool/cron:/sbin/nologin
54ftp:x:21:21::/var/lib/ftp:/sbin/nologin
55sshd:x:22:22:sshd:/dev/null:/sbin/nologin
56games:x:35:35:games:/usr/games:/sbin/nologin
57ntp:x:123:123:NTP:/var/empty:/sbin/nologin
58guest:x:405:100:guest:/dev/null:/sbin/nologin
59nobody:x:65534:65534:nobody:/:/sbin/nologin
60dockremap:x:100:101::/home/dockremap:/sbin/nologin
61rootless:x:1001:1001:Rootless:/home/rootless:/bin/sh
62EOF
63
64docker run --rm -v docker:/work alpine chown 1001:1001 /work
65
66docker run -d --name docker-dind \
67  --security-opt seccomp=unconfined \
68  --security-opt apparmor=unconfined \
69  --security-opt systempaths=unconfined \
70  --device /dev/fuse \
71  --device /dev/net/tun \
72  --user 1001:1001 \
73  -v shared_socket:/var/run/user/1001 \
74  -v work:/home/runner/_work \
75  --mount type=volume,src=docker,dst=/home/rootless/.local/share/docker,volume-nocopy \
76  -v /tmp/subuid:/etc/subuid \
77  -v /tmp/subgid:/etc/subgid \
78  -v /tmp/group:/etc/group \
79  -v /tmp/passwd:/etc/passwd \
80  docker:dind-rootless
NOTE

You need to use volume-nocopy to prevent the container from copying the content and permissions of the /home/rootless/.local/share/docker directory.

And one last thing: if you need to access to the network namespace of the docker in docker container, you'd want to connect the github runner namespace to the docker-in-docker container namespace.

To do that, you need to add --network "container:docker-dind" to the docker run command.

The final script 🔗

  1#!/bin/sh
  2
  3set -ex
  4# Setting up variables
  5: "${REPO?Need REPO}"
  6GH_TOKEN="$(gh auth token)"
  7
  8# Clean up procedure
  9cleanup() {
 10  docker stop runner-container || true
 11  docker rm -f runner-container docker-dind || true
 12  docker volume rm docker work shared_socket || true
 13  if [ -n "${RUNNER_ID:-}" ]; then
 14    curl -fsSL -X "DELETE" \
 15      -H "Authorization: Bearer ${GH_TOKEN}" \
 16      -H "Accept: application/vnd.github+json" \
 17      "https://api.github.com/repos/${REPO}/actions/runners/${RUNNER_ID}"
 18  fi
 19}
 20trap cleanup EXIT INT TERM
 21
 22# Setting up volumes and permissions
 23docker run --rm -v work:/work alpine chown 1001:1001 /work
 24docker run --rm -v shared_socket:/work alpine chown 1001:1001 /work
 25docker run --rm -v docker:/work alpine chown 1001:1001 /work
 26
 27# Setting up the UID/GID mapping + passwd + group
 28echo "rootless:30000:20000" >/tmp/subuid
 29echo "rootless:30000:20000" >/tmp/subgid
 30
 31cat <<EOF >/tmp/group
 32root:x:0:root
 33bin:x:1:root,bin,daemon
 34daemon:x:2:root,bin,daemon
 35sys:x:3:root,bin
 36adm:x:4:root,daemon
 37tty:x:5:
 38disk:x:6:root
 39lp:x:7:lp
 40kmem:x:9:
 41wheel:x:10:root
 42floppy:x:11:root
 43mail:x:12:mail
 44news:x:13:news
 45uucp:x:14:uucp
 46cron:x:16:cron
 47audio:x:18:
 48cdrom:x:19:
 49dialout:x:20:root
 50ftp:x:21:
 51sshd:x:22:
 52input:x:23:
 53tape:x:26:root
 54video:x:27:root
 55netdev:x:28:
 56kvm:x:34:kvm
 57games:x:35:
 58shadow:x:42:
 59www-data:x:82:
 60users:x:100:games
 61ntp:x:123:
 62abuild:x:300:
 63utmp:x:406:
 64ping:x:999:
 65nogroup:x:65533:
 66nobody:x:65534:
 67docker:x:2375:
 68dockremap:x:101:dockremap
 69rootless:x:1001:
 70EOF
 71
 72cat <<EOF >/tmp/passwd
 73root:x:0:0:root:/root:/bin/sh
 74bin:x:1:1:bin:/bin:/sbin/nologin
 75daemon:x:2:2:daemon:/sbin:/sbin/nologin
 76lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
 77sync:x:5:0:sync:/sbin:/bin/sync
 78shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
 79halt:x:7:0:halt:/sbin:/sbin/halt
 80mail:x:8:12:mail:/var/mail:/sbin/nologin
 81news:x:9:13:news:/usr/lib/news:/sbin/nologin
 82uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin
 83cron:x:16:16:cron:/var/spool/cron:/sbin/nologin
 84ftp:x:21:21::/var/lib/ftp:/sbin/nologin
 85sshd:x:22:22:sshd:/dev/null:/sbin/nologin
 86games:x:35:35:games:/usr/games:/sbin/nologin
 87ntp:x:123:123:NTP:/var/empty:/sbin/nologin
 88guest:x:405:100:guest:/dev/null:/sbin/nologin
 89nobody:x:65534:65534:nobody:/:/sbin/nologin
 90dockremap:x:100:101::/home/dockremap:/sbin/nologin
 91rootless:x:1001:1001:Rootless:/home/rootless:/bin/sh
 92EOF
 93
 94# Register the runner
 95JIT_RESPONSE=$(curl -fsSL -X "POST" \
 96  -H "Authorization: Bearer ${GH_TOKEN}" \
 97  -H "Accept: application/vnd.github+json" \
 98  "https://api.github.com/repos/${REPO}/actions/runners/generate-jitconfig" -d '{
 99    "name": "rootless-docker-runner",
100    "runner_group_id": 1,
101    "labels": ["self-hosted","linux","x64"],
102    "work_folder": "/home/runner/_work"
103  }')
104
105RUNNER_ID=$(echo "$JIT_RESPONSE" | jq -r '.runner.id')
106ENCODED_JIT_CONFIG=$(echo "$JIT_RESPONSE" | jq -r '.encoded_jit_config')
107
108docker run -d --name docker-dind \
109  --security-opt seccomp=unconfined \
110  --security-opt apparmor=unconfined \
111  --security-opt systempaths=unconfined \
112  --device /dev/fuse \
113  --device /dev/net/tun \
114  --user 1001:1001 \
115  -v shared_socket:/var/run/user/1001 \
116  -v work:/home/runner/_work \
117  --mount type=volume,src=docker,dst=/home/rootless/.local/share/docker,volume-nocopy \
118  -v /tmp/subuid:/etc/subuid \
119  -v /tmp/subgid:/etc/subgid \
120  -v /tmp/group:/etc/group \
121  -v /tmp/passwd:/etc/passwd \
122  docker:dind-rootless &
123
124sleep 5
125
126docker run --pull=always --rm --name "runner-container" \
127  --network "container:docker-dind" \
128  -v "work:/home/runner/_work" \
129  -v "shared_socket:/run/user/1001" \
130  -e DOCKER_HOST=unix:///run/user/1001/docker.sock \
131  -e TMPDIR=/home/runner/_work/tmp \
132  "ghcr.io/actions/actions-runner:2.337.0" \
133  bash -c "mkdir -p \$TMPDIR && chmod 755 \$TMPDIR && exec ./run.sh --jitconfig $ENCODED_JIT_CONFIG" &
134RUNNER_PID=$!
135
136wait "$RUNNER_PID"
137

Simulate a GitHub jobs:

 1# Enter the container
 2docker exec -it runner-container bash
 3
 4# Healthcheck docker
 5docker ps -a
 6# CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 7
 8# Run a server and mount a dir
 9docker run --rm -d --name nginx -p 80:80 -v /home/runner/_work/test:/test nginx
10
11# Test
12curl http://localhost
13# <!DOCTYPE html>
14# <html>
15# <head>
16# <title>Welcome to nginx!</title>
17# <style>
18# html { color-scheme: light dark; }
19# body { width: 35em; margin: 0 auto;
20# font-family: Tahoma, Verdana, Arial, sans-serif; }
21# </style>
22# </head>
23# <body>
24# <h1>Welcome to nginx!</h1>
25# <p>If you see this page, nginx is successfully installed and working.
26# Further configuration is required for the web server, reverse proxy,
27# API gateway, load balancer, content cache, or other features.</p>
28#
29# <p>For online documentation and support please refer to
30# <a href="https://nginx.org/">nginx.org</a>.<br/>
31# To engage with the community please visit
32# <a href="https://community.nginx.org/">community.nginx.org</a>.<br/>
33# For enterprise grade support, professional services, additional
34# security features and capabilities please refer to
35# <a href="https://f5.com/nginx">f5.com/nginx</a>.</p>
36#
37# <p><em>Thank you for using nginx.</em></p>
38# </body>
39# </html>
40
41# Test volume sharing
42docker exec -it nginx touch /test/test
43
44stat /home/runner/_work/test/test

Pretty good huh? And everything is rootless and 100% secure!

Conclusion 🔗

Rootless Docker in Rootless Docker is not something impossible. A good understanding of rootless technologies allows you to setup a secure Docker-in-Docker by leveraging Linux user namespaces, UID/GID mappings, FUSE overlayFS and slirp4netns.

In multi-tenant workload or CI jobs, privilege escalation vulnerabilities plague standard Docker in Docker configurations, while maintaining full Docker functionality.

With Rootless Docker in Rootless Docker, we have clear advantages over the classic Docker-in-Docker and Buildah:

  • True isolation: Even an attacker would breach all the isolations, it would never reach the root user.
  • No security compromises: --privileged is never used and no sensitive mounts are used. We're running unprivileged workloads on unprivileged environments.
  • Buildx compatibility: Rootless Docker in Rootless Docker allows you to use Docker buildx, which has many optimizations like layer squashing, cache mounts and more.
  • Production-Ready: With a multi-layered isolation, multi-tenant workloads are possible and can run on any environment, even in production. Common usecases being CI runners, but also AI workloads! You wouldn't want your Agent to escape you containment, would you?

While the setup requires more careful configuration than traditional Docker-in-Docker, especially around UID/GID mappings and volume permissions, the investment in security is well worth it, especially when running untrusted or multi-tenant workloads. For anyone building secure CI/CD infrastructure, Rootless Docker in Rootless Docker is no longer an optional optimization but a key part of the solution.