On Mon, 24 Nov 2008, Neil Gunton wrote:
I think the issue here is the large size of the directory tree itself - simply traversing this seems to be a problem. I started off a du this morning on that tree, at around 9am, and it's now after 12 midday and the command is still not done yet. Meanwhile my iowait has doubled on the server as a result.
Just a random thought... The O(n) directory search/traversal in filesystems only hits you if you have directories with many many files in. If your directories are like the one you sampled, with few items in, then maybe you are thrashing one of the filesystem caches -- inodes, vnodes or such -- while traversing the tree. I don't recall off-hand how you check this, though looking at the output of iostat and vmstat would give you some idea of where the traffic is in the VM and block IO subsystems.
Best wishes, John