This site is now 100% read-only, and retired.

XML logo

Postgrey eating significant disk IO
Posted by simonw on Wed 5 Sep 2012 at 14:30
Tags: none.
Postfix with Postgrey has been quietly working for a long time, surviving all sort of things that use to kill sendmail. This morning there was some sort of IO performance issue, in large part caused by poor IO performance on the storage back-end, but worsened by Postgrey.

Seems a small botnet was connecting and trying to send to daft addresses, from daft addresses. Typical pattern a bot connecting 18 times in 4 seconds, each time trying a small selection of addresses. The active phases come in bursts with much lower levels of greylisting in between.

My temporary fix was to null route 100 odd bot IP addresses grepped from the mail log, which got rid of worse offenders.

It looks like Postfix has ample features (smtpd_client_connection_rate_limit, smtpd_client_message_rate_limit) which I can use to minimize such abuse, but most are disabled by default, or set to large values to avoid them getting in the way of big email providers. So I'll enable a few of those to mitigate the problem longer term.

http://www.postfix.org/postconf.5.html

The level of problem was quite minor, but I was wondering how widespread the attack is, as it is abuse against a wide selection of domain names. "dilos.com" features big in the spoofed addresses, which is a legitimate sender, but not a domain we usually see much legitimate email from. Issue started about 09:40 this morning UTC+1.

SpamCop stats have a hint of recent up tick, but it is measuring spam reported, and I'm not aware of any of these emails having passed Greylisting so it may not be that there is much more spam around. As noted previously - spam delivery success on some of our servers was below the 1 in a million at one point which means a lot of spam can be attempted before spam delivered becomes an issue.

 

Comments on this Entry

Re: Postgrey eating significant disk IO
Posted by simonw (78.33.xx.xx) on Mon 10 Sep 2012 at 10:32
[ View Weblogs ]
Just noted this "fix" for postgrey in Debian.

I've reverted this change as well to see if it helps with IO performance when under load.

The abuse is ongoing, and this morning briefly reached levels where IO performance impacted other applications to the point a web application could not work despite various additional limits applied via Postfix.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=334430

[ Parent ]