Docker Overlay2 Storage Driver — Layers, Disk Usage, Troubleshooting

Every Docker homelab eventually hits a storage wall. Containers stop writing. docker compose up fails with “no space left on device”. You check df -h and there’s 40% free. The culprit? Docker’s overlay2 storage driver and its layer-based architecture — a system where disk usage isn’t always visible through traditional filesystem tools.

This article describes the classic overlay2 driver. Fresh Docker Engine 29 installations use the containerd image store by default; upgraded installations can retain classic drivers. Check docker info before using driver-specific paths or options. See Docker’s backend distinction. Copy-on-write affects usage, but a storage error also requires checking the actual mount, inodes, quotas, and application path.

This guide covers the overlay2 architecture from the ground up — how layers are structured, how to measure real disk usage, how to troubleshoot the most common storage failures, and how to tune performance when Docker is backed by ZFS on a Proxmox host.

How Overlay2 Works — Layer Architecture

Docker overlay2 uses the Linux kernel’s overlay filesystem to present a unified view of multiple directory layers. Each container image is built from a stack of read-only layers, with a thin read-write layer on top for the container itself.

The Four Directories

The overlay mount exposes lower, upper, merged, and work paths. These are mount roles, not four literal directories with these names under every layer:

lowerdir — the base image layers (read-only, shared across containers)
upperdir — the container’s writable layer (one per container)
mergeddir — the unified view presented to the container
workdir — internal overlay filesystem metadata

Check the active overlay mounts on your system:

1

cat /proc/mounts | grep overlay | head -5

Example output from a homelab Docker host:

overlay /var/lib/docker/overlay2/a1b2c3d4e5f6/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ABC123:/var/lib/docker/overlay2/l/DEF456,upperdir=/var/lib/docker/overlay2/a1b2c3d4e5f6/diff,workdir=/var/lib/docker/overlay2/a1b2c3d4e5f6/work 0 0

Each container gets its own overlay mount. The diff directory under each layer ID is where container writes actually land.

Copy-on-Write in Practice

When a container modifies a file from the image, Docker copies that file from the lower (read-only) layer into the upper (writable) layer — this is copy-on-write (CoW). The original layer remains intact and shareable across containers running the same image.

1
2


# See which layers an image uses
docker inspect alpine:latest --format '{{.GraphDriver.Data}}'

For a running container:

1

docker inspect "<container-name>" --format '{{.GraphDriver.Data}}'

You’ll see something like:

map[LowerDir:/var/lib/docker/overlay2/.../diff MergedDir:/var/lib/docker/overlay2/.../merged UpperDir:/var/lib/docker/overlay2/.../diff WorkDir:/var/lib/docker/overlay2/.../work]

The lower directories form a chain of read-only layers. Docker composes them using : as a separator in the kernel’s overlay mount.

Inspecting and Managing Disk Usage

Standard du and df don’t tell the full story with overlay2. Build cache, dangling layers, and shared image layers all consume space that df on the host filesystem sees, but du in a container does not.

Docker System DF — Your Primary Tool

1

docker system df

Output:

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          34        16        8.712GB   2.145GB (24%)
Containers      22        11        348.2MB   156.8MB (45%)
Local Volumes   12        8         1.234GB   0B (0%)
Build Cache     17        0         4.567GB   4.567GB (100%)

For a per-layer breakdown:

1

docker system df -v

The Build Cache entry is often the biggest surprise. Docker build cache can consume gigabytes of space that df attributes to “used” but no container actually needs.

Layer-Level Inspection

Check how much each image layer adds:

1

docker history --no-trunc nginx:latest

Find the largest layers across all images:

1

docker history --no-trunc nginx:latest | awk '{print $2, $1}' | sort -h | tail -10

Check actual disk usage by overlay2 directories:

1

sudo du -sh /var/lib/docker/overlay2/*/diff | sort -rh | head -15

This shows which container writable layers are consuming the most space. If a container writes large files, its diff directory grows independently of the image layers.

Cleanup Strategies

1
2
3
4
5
6
7
8


# Inventory usage before choosing any cleanup target
docker system df -v

# Remove only build cache
docker buildx prune --all

# Selective prune — keep images used in the last 24h
docker image prune -a --filter "until=24h"

Builds can accumulate cache; image update tools do not necessarily build images. Review cache needs and rollback images before any manual prune:

1

docker system prune -a -f --filter "until=48h"

Add this to a cron job or systemd timer to keep disk usage under control without disrupting active containers.

Zombie Layers — The Overlay2 Leftover

When Docker prunes an image but a container still references one of its layers, the layer sticks around in /var/lib/docker/overlay2/. These are called “zombie layers” — they show up in du but not in docker system df. If your overlay2 directory seems larger than the Docker-reported total, check for these:

1
2
3


# Compare Docker-reported usage with actual directory size
docker system df --format '{{.Type}}: {{.Size}}'
sudo du -sh /var/lib/docker/

A big discrepancy means orphaned layers. A Docker restart or a full prune usually reclaims them:

1
2


sudo systemctl restart docker
docker system prune -a -f

Troubleshooting Common Overlay2 Issues

Inode Exhaustion — The Silent Killer

The most common overlay2 failure in homelabs isn’t disk space — it’s inode exhaustion. Docker overlay2 creates thousands of metadata entries per container. On ext4, the default inode count can be too low for heavy container usage.

1
2


# Check inode usage
df -i /var/lib/docker

If IUse% is near 100% while Use% is low, you’ve hit the inode wall:

Filesystem     Inodes  IUsed   IFree IUse% Mounted on
/dev/sda1      2M      1.98M   20K   99%   /var/lib/docker

Fix: Move Docker storage to a filesystem with more inodes (XFS) or re-create ext4 with a larger inode ratio:

1

mkfs.ext4 -T news /dev/sdX1   # larger inode table

For Proxmox homelabs running Docker in a VM or LXC, ensure the disk backing overlay2 uses XFS or ext4 with adequate inodes.

ZFS Dataset Full

On Proxmox, Docker storage often sits on a ZFS dataset or zvol. ZFS has its own quota system separate from the filesystem:

1
2
3
4
5


# Check ZFS quotas and reservations
zfs get quota,refquota,reservation,refreservation storage/docker

# Check actual space used
zfs list storage/docker

1
2


# Set a quota if one doesn't exist
zfs set quota=50G storage/docker

If you see a “disk full” error in a container but df -h inside the container shows free space, the underlying ZFS dataset may have hit its quota.

Docker Daemon Logs

When container writes fail, the Docker daemon logs often contain the real error:

1

journalctl -u docker -n 50 --no-pager | grep -i error

Common error patterns:

Error processing tar file: archive/tar: invalid tar header
no space left on device
failed to mount overlay: no such file or directory

The “no such file or directory” on overlay mount usually means a kernel module is missing or the overlay filesystem isn’t supported. Check with:

1
2


lsmod | grep overlay
modprobe overlay

Container-Specific Troubleshooting

When a single container reports “no space left” but everything else works:

1
2
3
4
5
6


# Check the container's writable layer size
sudo du -sh /var/lib/docker/overlay2/$(docker inspect "<container>" --format '{{.GraphDriver.Data.UpperDir}}' | xargs dirname | xargs basename)/diff

# Check the container itself
docker exec "<container>" df -h
docker exec "<container>" df -i

Sometimes a container fills its own layer with logs or temp files. If the container writes to a volume mount, the issue is on the host’s filesystem handling that volume.

Performance Tuning

XFS and d_type

Overlay2 requires the underlying filesystem to support d_type (directory entry type). XFS enables this by default. ext4 also supports it. Verify:

1

xfs_info /var/lib/docker | grep ftype

If ftype=0, overlay2 will fall back to a slower compatibility mode. Re-create the filesystem with ftype=1 to avoid this.

Docker on ZFS

In Proxmox homelabs, Docker frequently runs inside a VM or LXC backed by ZFS. The performance characteristics depend on the ZFS recordsize and how Docker’s overlay2 interacts with ZFS’s own CoW:

Default ZFS recordsize (128K): fine for most containers
Database containers (PostgreSQL, MySQL): set recordsize to 16K on the dataset
Media containers (Jellyfin, Plex): set recordsize to 1M on media volumes

1
2


zfs set recordsize=16K storage/docker/db
zfs set recordsize=1M storage/docker/media

For the Docker storage directory itself, keep the default recordsize unless you benchmark and find a clear improvement.

Docker storage options depend on the backend

Do not bypass kernel compatibility checks or replace an existing daemon configuration to enable a guessed storage option. Inspect docker info, the backing filesystem, and the installed daemon’s supported configuration. Classic overlay2.size requires compatible XFS project quotas and limits the writable layer; it does not cap bind mounts, named volumes, or logs. It is not a portable containerd snapshotter option.

Use the quota and backend guide to plan a separate test. A daemon restart affects running services and belongs in a maintenance window with a rollback plan.

Per-Container Resource Limits

For containers with heavy write workloads, mount a dedicated volume or set per-container storage limits in docker-compose.yml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


services:
  database:
    image: postgres:16-alpine
    volumes:
      - postgres-data:/var/lib/postgresql/data
    deploy:
      resources:
        limits:
          memory: 2G

volumes:
  postgres-data:
    driver_opts:
      type: none
      device: /mnt/ssd-fast/postgres
      o: bind

This bypasses overlay2 CoW for the database directory entirely, avoiding double-CoW (ZFS CoW + overlay2 CoW).

Maintenance and Automation

Review cleanup before scheduling

Do not schedule an indiscriminate data-volume cleanup. An unused volume may hold the only copy of a stopped application’s data. Preserve rollback images and inspect stopped containers before removing them. The following timer records usage instead of deleting resources; its unit names are retained from the earlier example, so inspect existing local units before replacing them.

Systemd timer for weekly storage inventory

1
2
3
4
5
6
7
8


# /etc/systemd/system/docker-cleanup.service
[Unit]
Description=Docker storage inventory
After=docker.service

[Service]
Type=oneshot
ExecStart=/usr/bin/docker system df -v

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# /etc/systemd/system/docker-cleanup.timer
[Unit]
Description=Weekly Docker storage inventory

[Timer]
OnCalendar=weekly
Persistent=true

[Install]
WantedBy=timers.target

1
2


sudo systemctl daemon-reload
sudo systemctl enable --now docker-cleanup.timer

Storage Monitoring with Prometheus

For homelabs running Prometheus and node_exporter, enable the textfile collector to track Docker overlay2 usage:

1
2
3
4
5


#!/bin/bash
# /etc/node_exporter/textfile/docker_storage.prom
DOCKER=$(sudo du -sb /var/lib/docker/ 2>/dev/null | cut -f1)
echo "docker_overlay2_usage_bytes $DOCKER" > /var/lib/node_exporter/textfile_collector/docker_storage.prom
echo "docker_overlay2_usage_percent $(df /var/lib/docker/ --output=pcent 2>/dev/null | tail -1 | tr -d ' %')" >> /var/lib/node_exporter/textfile_collector/docker_storage.prom

Run this every 5 minutes via cron and graph it in Grafana alongside other host metrics.

Conclusion

Docker’s overlay2 storage driver is fast and reliable, but its layer-based architecture requires a different approach to disk management than traditional filesystems. Understanding the four overlay directories, monitoring build cache accumulation, checking inodes before disk space, and tuning for your ZFS or XFS backend all make the difference between a Docker host that runs for months and one that mysteriously fills up every week.

Start with docker system df to see your current state. Check your inode usage with df -i /var/lib/docker. Set up automated pruning. And if you’re on a Proxmox-backed homelab, pay attention to ZFS dataset quotas and recordsize tuning. Your containers will thank you.

How Overlay2 Works — Layer Architecture#

The Four Directories#

Copy-on-Write in Practice#

Inspecting and Managing Disk Usage#

Docker System DF — Your Primary Tool#

Layer-Level Inspection#

Cleanup Strategies#

Zombie Layers — The Overlay2 Leftover#

Troubleshooting Common Overlay2 Issues#

Inode Exhaustion — The Silent Killer#

ZFS Dataset Full#

Docker Daemon Logs#

Container-Specific Troubleshooting#

Performance Tuning#

XFS and d_type#

Docker on ZFS#

Docker storage options depend on the backend#

Per-Container Resource Limits#

Maintenance and Automation#

Review cleanup before scheduling#

Systemd timer for weekly storage inventory#

Storage Monitoring with Prometheus#

Conclusion#