devel anaconda put forth on 11/8/2009 11:16 AM: > 06.11.09, 02:32, "Victor Duchovni" <victor.ducho...@morganstanley.com>:
>> You are not going to succeed without more spindles and likely more CPUs. >> Just launching a "Perl" hello-world script takes >10ms on a fast machine, >> do that 100 times a second and you've burned a CPU, and non-trivial >> scripts are slower still. I absolutely agree with Victor WRT to the spindles issue, as I've already stated. I don't necessarily agree with his suggestion that you need more CPUs. You've got 8 cores at 2.8GHz, and they're XEONs, so they've got a lot of L2/L3 cache. That is some really serious processing horsepower. Once you get the disk bottleneck solved, I'm guessing you'll find you have plenty of CPU. Victor has vastly more Postfix and Perl experience than me, so I may be talking out of turn here. However, short of some HPC workloads, I've not seen any application stack to date come close to fully saturating a 16 core system. I'm eager to be proven wrong. ;) > As I said above - scripts is not the point at a time. If it will, I'll > rewrite them, so my scripts will listen unix socket and I'll put messages > directly into sockets. > > I think, that things will go faster, if postfix, instead of saving new > messages to queue, will try to deliver it to deliver agent "in memory", and > only if that fails, save the message to queue. Don't you think, this will be > faster? Is that possible? I can make a patch (not for upstream, only for me). I don't think keeping them in memory is going help anything. It will probably make it worse. Your example previously mentioned over 100GB worth of spool files being generated. You've only got 16GB of RAM. Using your new idea, keeping the files resident, you'll run out of RAM and start hammering the swap file. So, you're in the same situation, just a different software component hammering the disks, now the kernel memory manager instead of postfix queue manager. The big downside to this is that your swap partition is probably only somewhere around 16GB. Due to this, you'll run out of swap space and possibly crash the kernel. At least as things currently are, if Postfix runs out of spool space, it won't crash the kernel, just possibly parts of Postfix. It seems that your problem is purely one of lack of I/O bandwidth, basically generating too many files too quickly for the system to dispose of. Either the emails need to be delivered out the network faster to their final destination, or they need to be delivered faster to local accounts, i.e. local disk. One way or another, you've got to deliver them as fast as they come in or you have to spool them in memory or disk. If the destinations are over the wire to another host, you're going to be waiting on downstream MTAs, and emails will have to sit in memory or in spool files until delivered. If the emails are destined locally, then they must be written to local disk quickly. Either way, you need more disk bandwidth. You have three options as I see it: 1. Cut down on the total volume of email files per time step 2. Substantially increase disk subsystem bandwidth via spindles 3. Increase system RAM to 128GB, make 112GB a RAM disk for the spool Option 3 isn't really a valid option because your server probably doesn't have enough sockets to support 128GB of RAM, and even if it did, you could buy 4 or 5 fourteen drive disk arrays for the same money, and just one such array will solve your problem, for a tiny fraction of the cost of that much RAM. So, if you can't accomplish #1 above, you need to accomplish #2, which you're already testing with two additional drives. You'll probably need more. Keep us posted on your progress. I find this case very interesting, given it touches so heavily on hardware performance. (See the right hand side of my email address). ;) -- Stan
