sent spam mails

Dear Internet, I'm sorry. My mail server running at sent spam mails for 5 months until I noticed and stopped it.


I'm sharing a VPS with two friends (keeps costs low), and we are administrating the Debian server ourselves.

The server used to host our own and friend's web sites, e-mail (~50 accounts), XMPP, and some custom applications that we wrote over the years.

Wave 1

While debugging a mail delivery problem on 2016-10-20 I noticed that one of the users logged into the SMTP server very often and from different IP addresses. Upon closer inspection, I saw that the sender was randomized (randomstring@domain) and the recipients were too many to be legit mails.

After finding that the mail.log file contained over 260k actions of that user, I changed the password of the mailbox.

While my mail server had sent over 2600 mails that day, it only sent 395 the next one. Problem solved.

Wave 2

When I came back from vacation a week later, my mail box contained many mails with server errors (e.g. mail delivery or cron error due to maxed-out resources).

I checked the mail server statistics in awstats and saw that the previous day, one user had sent 93.000 mails, and this day it already was at 35k.

Again I had to change the mail box password. The next day, mail volume was down to a normal 400.

Post mortem

After I stopped the second wave, I had a closer look at the mail server log visualization that awstats generated. I found that the first account password must already have been broken in juny 2016 - the number of mails delivered on my server jumped from 200-700 per day to over 2000:

awstats mail server statistics 2016-06 awstats mail server statistics 2016-10

That first wave was low in volume and used randomized sender addresses. Both helped it to go unnoticed for 5 months.

The second spam wave was high-volume which broke other server processes because our limited resources were all used up. It also used a single sender that made it easy to spot in the log analytics.

I do not know how the spam senders obtained the passwords of those two accounts. It could have been a brute-force attack on weak passwords, or the people used the same password somewhere else and those passwords were leaked.


We implemented the following changes to our server:

Munin showing pflogsumm

Written by Christian Weiske.

Comments? Please send an e-mail.