Re: Premature "No Space left on device" on XFS

[Lst_hoe02]

Zitat von Bernhard Schmidt <be...@birkenwald.de>:

Hey,

a small not-quite but a bit postfix related issue.



bulkin --> amavis --> bulkout







lxmhs45:/var/spool/postfix-bulk/postfix-bulkinhss # touch a
touch: cannot touch `a': No space left on device
lxmhs45:/var/spool/postfix-bulk/postfix-bulkinhss # df .
Filesystem           1K-blocks      Used Available Use% Mounted on

lxmhs45:/var/spool/postfix-bulk/postfix-bulkinhss # df -i .
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sdb             10485760 1742528 8743232   17% /var/spool/postfix-bulk

We (or better said: an over-eager third party) have been running some performance tests against our future outbound bulkmail platform (no, not UCE, university stuff), which consists of multiple SLES11.1 VMs with 1GB of RAM and 4 vCPU each, running Postfix 2.8.5 in a multi-instance setup. It is a classic after-queue content filter with amavisd in between two postfix instances doing DKIM signing. bulkin sends everything to amavis, which does the signing and sends everything to bulkout, which discards it (discard: entry in transport map). The third-party has been bombarding the bulkin instance for several hours with 100 parallel threads. There is no chance in hell amavis could cope with that rate, which eventually lead to almost 2 million files in the incoming queue. We expected the system to get slower and slower and slower and eventually fail to accept new mails due to queue_minfree being hit. But it happened much earlier, and in a very unexpected way Oct 6 20:14:33 lxmhs45 postfix-bulkinhss/bounce[23308]: fatal: open lock file pid/unix.defer: cannot create file exclusively: No space left on device Oct 6 20:15:35 lxmhs45 postfix-bulkinhss/bounce[23691]: fatal: open lock file pid/unix.defer: cannot create file exclusively: No space left on device Oct 6 20:16:36 lxmhs45 postfix-bulkinhss/bounce[24479]: fatal: open lock file pid/unix.defer: cannot create file exclusively: No space left on device Oct 6 20:17:37 lxmhs45 postfix-bulkinhss/bounce[24579]: fatal: open lock file pid/unix.defer: cannot create file exclusively: No space left on device Oct 6 20:18:39 lxmhs45 postfix-bulkinhss/bounce[24684]: fatal: open lock file pid/unix.defer: cannot create file exclusively: No space left on device Oct 6 20:19:40 lxmhs45 postfix-bulkinhss/bounce[24847]: fatal: open lock file pid/unix.defer: cannot create file exclusively: No space left on device Oct 6 20:20:21 lxmhs45 postfix-bulkin/bounce[24949]: fatal: open file defer CD0CF2054904: No space left on device Oct 6 20:20:21 lxmhs45 postfix-bulkin/bounce[24950]: fatal: open file defer DEBCD1AB98: No space left on device Oct 6 20:20:21 lxmhs45 postfix-bulkin/bounce[24951]: fatal: open file defer 1800D3019D35: No space left on device Oct 6 20:20:41 lxmhs45 postfix-bulkinhss/bounce[24977]: fatal: open lock file pid/unix.defer: cannot create file exclusively: No space left on device /dev/sdb 10475520 7471160 3004360 72% /var/spool/postfix-bulk

Looks like this problem to me:
http://linux.derkeiler.com/Mailing-Lists/SuSE/2010-09/msg01038.html



But nothing Postfix really Postfix related.

Regards

Andreas

In short XFS with default options limit the number of inodes by not using more than 5% of the volume space for inodes. Have a look at "xfs_growfs -m".

smime.p7s
Description: S/MIME Cryptographic Signature