One of my colleagues uses Docker Desktop on MacOS. One day, it reported that all of the 200GiB of reserved space were full.
The docker daemon did not think so:
$ docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 68 18 36GB 26.9GB (74%) Containers 20 20 111.7MB 0B (0%) Local Volumes 56 12 7.947GB 4.992GB (62%) Build Cache 365 0 0B 0B
This are no 200 GiB.
On Linux we can simply inspect /var/lib/docker, but Docker on MacOS is running inside a virtual machine (VM). We got access to that VM with the help of justincormack/nsenter1:
$ docker run -it --rm --privileged --pid=host justincormack/nsenter1 # cd /var/lib/docker/ # du -h -d 1 44.5G ./overlay2 95.6M ./image 7.7G ./volumes 114.4G ./containers 336.0K ./network 166.7G ./var/lib/docker
So containers is very full although docker said it should contain only 111 MiB :)
# cd /var/lib/docker/containers/ # du -h -d 1 56.0K ./e239e861c2a9097f79d3c0c0f98fce7b20ab899674ea7772356d0355d9f688f4 44.0K ./962a1795b5440e9b9900a4743d86f9c18452675a43c379a88aaac2b16b5bd275 [...] 114G ./2f9867b08a714fcfd83b85d5cf7f883be05394970e7e0058747b399f06bd269d [...] # cd /var/lib/docker/containers/2f9867b08a714fcfd83b85d5cf7f883be05394970e7e0058747b399f06bd269d # ls -lah total 114G [...] -rw-r----- 1 root root 114.3G Oct 11 07:01 2f9867b08a714fcfd83b85d5cf7f883be05394970e7e0058747b399f06bd269d-json.log [...]
And there we have it: A log file of 114 GiB.
But which container is that?
$ docker inspect 2f9867b08a714fcfd83b85d5cf7f883be05394970e7e0058747b399f06bd269d | jq '.[].Config.Image' "gitlab-docker.company.com/customer/customer-wordpress-docker/mysql:latest"
This particular MySQL container seems to be very badly configured.