Inspect volumes used by docker

Check system free disk space with: sudo df -h. Sample result

udev            3.9G     0  3.9G   0% /dev
tmpfs           788M  1.1M  787M   1% /run
/dev/nvme0n1p1  249G  248G  519M 100% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           788M     0  788M   0% /run/user/1000
/dev/loop1       18M   18M     0 100% /snap/amazon-ssm-agent/1480
/dev/loop3       92M   92M     0 100% /snap/core/8689
/dev/loop0       18M   18M     0 100% /snap/amazon-ssm-agent/1566
overlay         249G  248G  519M 100% /var/lib/docker/overlay2/b291b6cdebfc50321b714b29efabef3e6930e5afcd83adb4dcf7ed3b2379542a/merged
shm              64M     0   64M   0% /var/lib/docker/containers/12c8899f10d85c09fedb033ff973270b009dc95a56fff440d8a5e047e62908ba/mounts/shm
overlay         249G  248G  519M 100% /var/lib/docker/overlay2/d7a6f373654f08913136f0b75ec18ed4b55e9524b670c10050ee6c2a7c3a727d/merged
shm              64M     0   64M   0% /var/lib/docker/containers/f1f28d1f68df6509ab8f479515c0a36703e44472d5a7a327ae34c4a596687151/mounts/shm
overlay         249G  248G  519M 100% /var/lib/docker/overlay2/6e377af8a1ea663eb2dc98f94043c4dab65030243e147fa995a64f775698f3bb/merged
shm              64M     0   64M   0% /var/lib/docker/containers/c6f6797081d08172e4970ba703e852e60abc1dbb5b73c4a0a0f93fb30d694afc/mounts/shm
overlay         249G  248G  519M 100% /var/lib/docker/overlay2/38092b60026552eec6c17c241b81d5bba499cb02dec81274155d2ca273d540d2/merged
shm              64M     0   64M   0% /var/lib/docker/containers/627e95caaa8f825471a2bf2510c60a1751d6d971c8d04d28b86e258bfd0b5c0b/mounts/shm
/dev/loop2       94M   94M     0 100% /snap/core/8935

Notice the mount point at /. There is only 519MB left.

Inspect volume space used by docker volumes with: sudo du -h --max-depth=1 /var/lib/docker/volumes/. Sample result

8.0K	/var/lib/docker/volumes/d3ad1ec03227f0e5d40ae4ef55e3126680dfd7b14bf44b96a88851c15d34dbeb
207G	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66
500M	/var/lib/docker/volumes/bcabcb5ab9da546f63dd4b1e586f92b5e23c7d35abbf94ac1c23768ba70fa3c9
8.0K	/var/lib/docker/volumes/fc0e4477dbf9d2a6ab96e0c0aba326ed1b2b87b806df606b35195016119809cc
28K	/var/lib/docker/volumes/6219de11f6c0d6330d2204cfc1ec650087f14070aac91284d82302324b2ae1a4
602M	/var/lib/docker/volumes/7ba7c256fa8999c0121de311551388e64c86c34ceda55a3a9f9d5b6d7b43d371
12G	/var/lib/docker/volumes/7f92903f752acbf9e40765e561835badb4a5113c4b8b357b96a308c16fcbe25a
219G	/var/lib/docker/volumes

At line number 2, the volume with id 19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66 consumes 207GB.

Which container uses this volume?

Copy the volume id and find the mounting container id with docker ps -a --filter volume=19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66. Sample result

f1f28d1f68df        jenkinsci/blueocean:1.17.0   "/sbin/tini -- /usr/…"   2 weeks ago         Up 2 weeks          8080/tcp, 0.0.0.0:50000->50000/tcp   amazing_mirzakhani
12c8899f10d8        docker:dind                  "dockerd-entrypoint.…"   2 weeks ago         Up 2 weeks          2375-2376/tcp                        wizardly_lovelace

For now, it depends on the container to decide what to do next.

In my case, I chose to inspect sub-directories of the volume with sudo du -h --max-depth=1 /var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data. Sample result

68K	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/secrets
8.0K	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/userContent
516M	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/jobs
19M	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/org.jenkinsci.plugins.github_branch_source.GitHubSCMProbe.cache
12K	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/.config
8.0K	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/caches
206G	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/workspace
89M	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/war
8.0K	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/.groovy
508K	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/fingerprints
283M	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/plugins
4.0K	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/nodes
4.0K	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/workflow-libs
660K	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/logs
48K	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/.java
2.1M	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/updates
160K	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data/users
207G	/var/lib/docker/volumes/19c18daa0e4845547309173f72fd97558dbecf7a8deb3bf46adb9a3b84818c66/_data

Then now I know that the workspace directory needs to be cleaned.