Every morning I experienced lagginess on our server when fetching mails
or sifting through RSS/Atom feeds with
TinyTinyRSS.
Munin, our server monitoring tool,
showed the following CPU load:
At first I did not read graph correctly and thought the blue bars are user space processes that took a full CPU, but if you exactly compare the colors with the legend you will see that it's iowait.
Thus I installed a cron that logged the iowait every hour:
$ crontab -e # .... 3 * * * * iotop -b -n 1 > "/home/cweiske/iotop-`date --rfc-3339=seconds`"
When I looked at the correct file (03:03), I saw the following lines:
Total DISK READ: 970.61 K/s | Total DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 15969 be/6 nobody 207.99 K/s 0.00 B/s 0.00 % 59.89 % find /\ -ignore_readdir_race ( -fstype NFS -o -fstype nfs -o -fstype nfs4\ -o -fstype afs -o -fstype binfmt_misc -o -fstype proc -o -fstype smbfs\ -o -fstype autofs -o -fstype iso9660 -o -fstype ncpfs -o -fstype coda\ -o -fstype devpts -o -fstype ftpfs -o -fstype devfs -o -fstype mfs\ -o -fstype shfs -o -fstype sysfs -o -fstype cifs -o -fstype lustre_lite\ -o -fstype tmpfs -o -fstype usbfs -o -fstype udf -o -fstype ocfs2\ -o -type d -regex \(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/afs$\)\ \|\(^/amd$\)\|\(^/alex$\)\|\(^/var/spool$\)\|\(^/sfs$\)\|\(^/media$\)\\ |\(^/var/lib/schroot/mount$\) ) -prune -o -print0
As it turned out, this is the command that the updatedb uses internally to find files to index. This database in turn is used by locate that allows finding files by name verrry quickly.
The cronjob that runs updatedb is located in /etc/cron.daily/locate.