Docker MacOS: No space left

One of my colleagues uses Docker Desktop on MacOS. One day, it reported that all of the 200GiB of reserved space were full.

The docker daemon did not think so:

$ docker system df
Images          68        18        36GB      26.9GB (74%)
Containers      20        20        111.7MB   0B (0%)
Local Volumes   56        12        7.947GB   4.992GB (62%)
Build Cache     365       0         0B        0B

This are no 200 GiB.

On Linux we can simply inspect /var/lib/docker, but Docker on MacOS is running inside a virtual machine (VM). We got access to that VM with the help of justincormack/nsenter1:

$ docker run -it --rm --privileged --pid=host justincormack/nsenter1
# cd /var/lib/docker/
# du -h -d 1
44.5G ./overlay2
95.6M ./image
7.7G ./volumes
114.4G ./containers
336.0K ./network
166.7G ./var/lib/docker

So containers is very full although docker said it should contain only 111 MiB :)

# cd /var/lib/docker/containers/
# du -h -d 1
56.0K ./e239e861c2a9097f79d3c0c0f98fce7b20ab899674ea7772356d0355d9f688f4
44.0K ./962a1795b5440e9b9900a4743d86f9c18452675a43c379a88aaac2b16b5bd275
114G  ./2f9867b08a714fcfd83b85d5cf7f883be05394970e7e0058747b399f06bd269d
# cd /var/lib/docker/containers/2f9867b08a714fcfd83b85d5cf7f883be05394970e7e0058747b399f06bd269d
# ls -lah
total 114G
-rw-r-----    1 root     root      114.3G Oct 11 07:01 2f9867b08a714fcfd83b85d5cf7f883be05394970e7e0058747b399f06bd269d-json.log

And there we have it: A log file of 114 GiB.

But which container is that?

$ docker inspect 2f9867b08a714fcfd83b85d5cf7f883be05394970e7e0058747b399f06bd269d | jq '.[].Config.Image'

This particular MySQL container seems to be very badly configured.

Written by Christian Weiske.

Comments? Please send an e-mail.