Dear Internet, I'm sorry. My mail server running at mail.cweiske.de sent spam mails for 5 months until I noticed and stopped it.
Introduction
I'm sharing a VPS with two friends (keeps costs low), and we are administrating the Debian server ourselves.
The server used to host our own and friend's web sites, e-mail (~50 accounts), XMPP, and some custom applications that we wrote over the years.
Wave 1
While debugging a mail delivery problem on 2016-10-20 I noticed that one of the users logged into the SMTP server very often and from different IP addresses. Upon closer inspection, I saw that the sender was randomized (randomstring@domain) and the recipients were too many to be legit mails.
After finding that the mail.log file contained over 260k actions of that user, I changed the password of the mailbox.
While my mail server had sent over 2600 mails that day, it only sent 395 the next one. Problem solved.
Wave 2
When I came back from vacation a week later, my mail box contained many mails with server errors (e.g. mail delivery or cron error due to maxed-out resources).
I checked the mail server statistics in awstats and saw that the previous day, one user had sent 93.000 mails, and this day it already was at 35k.
Again I had to change the mail box password. The next day, mail volume was down to a normal 400.
Post mortem
After I stopped the second wave, I had a closer look at the mail server log visualization that awstats generated. I found that the first account password must already have been broken in juny 2016 - the number of mails delivered on my server jumped from 200-700 per day to over 2000:
That first wave was low in volume and used randomized sender addresses. Both helped it to go unnoticed for 5 months.
The second spam wave was high-volume which broke other server processes because our limited resources were all used up. It also used a single sender that made it easy to spot in the log analytics.
I do not know how the spam senders obtained the passwords of those two accounts. It could have been a brute-force attack on weak passwords, or the people used the same password somewhere else and those passwords were leaked.
Changes
We implemented the following changes to our server:
- fail2ban configuration for postfix and courier to stop brute force attacks quickly.
- pflogsumm integration into Munin via postfix_stats, a critical limit on the number of delivered mails and mail notification for warnings and critical states.
