Basically only two possibilities: you're thrashing the poor box to death by running out of memory, or network tests are taking an age. Or maybe both.
How many children? What load average? How much memory on the machine? How much free memory? Do you have a local caching DNS for the various RBL checks? Loren