Mysterious iowait every night

Every morning I experienced lagginess on our server when fetching mails or sifting through RSS/Atom feeds with TinyTinyRSS. Munin, our server monitoring tool, showed the following CPU load: CPU load

At first I did not read graph correctly and thought the blue bars are user space processes that took a full CPU, but if you exactly compare the colors with the legend you will see that it's iowait.

Thus I installed a cron that logged the iowait every hour:

$ crontab -e
# ....
3 * * * * iotop -b -n 1 > "/home/cweiske/iotop-`date --rfc-3339=seconds`"

When I looked at the correct file (03:03), I saw the following lines:

Total DISK READ: 970.61 K/s | Total DISK WRITE: 0.00 B/s
15969 be/6 nobody    207.99 K/s    0.00 B/s  0.00 % 59.89 % find /\
 -ignore_readdir_race ( -fstype NFS -o -fstype nfs -o -fstype nfs4\
 -o -fstype afs -o -fstype binfmt_misc -o -fstype proc -o -fstype smbfs\
 -o -fstype autofs -o -fstype iso9660 -o -fstype ncpfs -o -fstype coda\
 -o -fstype devpts -o -fstype ftpfs -o -fstype devfs -o -fstype mfs\
 -o -fstype shfs -o -fstype sysfs -o -fstype cifs -o -fstype lustre_lite\
 -o -fstype tmpfs -o -fstype usbfs -o -fstype udf -o -fstype ocfs2\
 -o -type d -regex \(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/afs$\)\
|\(^/var/lib/schroot/mount$\) ) -prune -o -print0

As it turned out, this is the command that the updatedb uses internally to find files to index. This database in turn is used by locate that allows finding files by name verrry quickly.

The cronjob that runs updatedb is located in /etc/cron.daily/locate.

Written by Christian Weiske.

Comments? Please send an e-mail.