[it's good to reply to the list, so I've Cc'd it]
"Ken Naim" <[EMAIL PROTECTED]> wrote:
> I created a test case similar to my nightly job which removes 130 or so 5gb
> files. The apps* files are identical to files I remove nightly and are 4.3
> gb in size, the ctx files are 20mb, and the single letter files are 0 byte
> files. Based on the strace there is a correlation between file size and
> unlink time. 4.3gb takes 10 seconds, 20mb take less than .1 seconds and 0
> byte files take no time.
>
> I appreciare your help.
>
> [EMAIL PROTECTED] foo2]$ strace -T -e trace=file rm -f * >../x.log
> execve("/bin/rm", ["rm", "-f", "apps_ts_tx_data_2.270.632954231",
> "apps_ts_tx_data_2.271.632954231", "apps_ts_tx_data_2.272.632954231",
> "apps_ts_tx_data_2.273.632954231", "apps_ts_tx_data_2.274.632954231",
> "apps_ts_tx_data_2.275.632954271", "apps_ts_tx_data_2.276.632954291",
> "apps_ts_tx_data_2.277.632954313", "apps_ts_tx_data_2.278.632954313",
> "apps_ts_tx_data_2.278.632954333", "apps_ts_tx_data_2.279.632954313",
> "apps_ts_tx_data_2.279.632954353", "apps_ts_tx_data_2.280.632954373",
> "ctxd.367.632955010", ...], [/* 32 vars */]) = 0 <0.000292>
> access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or
> directory) <0.000062>
> open("/etc/ld.so.cache", O_RDONLY) = 3 <0.000062>
> open("/lib/tls/libc.so.6", O_RDONLY) = 3 <0.000064>
> open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3 <0.000070>
> unlink("apps_ts_tx_data_2.270.632954231") = 0 <10.418224>
> unlink("apps_ts_tx_data_2.271.632954231") = 0 <10.691083>
> unlink("apps_ts_tx_data_2.272.632954231") = 0 <9.708125>
> unlink("apps_ts_tx_data_2.273.632954231") = 0 <11.170446>
> unlink("apps_ts_tx_data_2.274.632954231") = 0 <10.192923>
> unlink("apps_ts_tx_data_2.275.632954271") = 0 <9.677868>
> unlink("apps_ts_tx_data_2.276.632954291") = 0 <10.157322>
> unlink("apps_ts_tx_data_2.277.632954313") = 0 <10.624669>
> unlink("apps_ts_tx_data_2.278.632954313") = 0 <10.640957>
> unlink("apps_ts_tx_data_2.278.632954333") = 0 <10.649074>
> unlink("apps_ts_tx_data_2.279.632954313") = 0 <9.764071>
> unlink("apps_ts_tx_data_2.279.632954353") = 0 <9.486272>
> unlink("apps_ts_tx_data_2.280.632954373") = 0 <9.312557>
As I said, it looks like something related to your file system.
What type of file system are you using? "df -T ." will tell you.
> unlink("ctxd.367.632955010") = 0 <0.051140>
> unlink("ctxd.367.632955011") = 0 <0.078666>
> unlink("ctxd.367.632955012") = 0 <0.057871>
> unlink("ctxd.367.632955013") = 0 <0.051694>
> unlink("ctxd.367.632955014") = 0 <0.084658>
> unlink("ctxd.367.632955015") = 0 <0.047987>
> unlink("ctxd.367.632955016") = 0 <0.049673>
> unlink("ctxd.367.632955017") = 0 <0.098318>
> unlink("ctxd.367.632955018") = 0 <0.059037>
> unlink("ctxd.367.632955019") = 0 <0.049887>
> unlink("ctxd.367.632955020") = 0 <0.092457>
> unlink("ctxd.367.632955021") = 0 <0.060425>
> unlink("ctxd.367.632955022") = 0 <0.045415>
> unlink("ctxd.367.632955023") = 0 <0.067629>
> unlink("ctxd.367.632955024") = 0 <0.068119>
> unlink("ctxd.367.632955025") = 0 <0.044039>
> unlink("ctxd.367.632955026") = 0 <0.048564>
> unlink("ctxd.367.632955027") = 0 <0.034952>
> unlink("ctxd.367.632955028") = 0 <0.056535>
> unlink("ctxd.367.632955029") = 0 <0.073922>
> unlink("ctxd.367.632955030") = 0 <0.050084>
> unlink("ctxd.367.632955031") = 0 <0.076422>
> unlink("ctxd.367.632955032") = 0 <0.083707>
> unlink("ctxd.367.632955033") = 0 <0.062325>
> unlink("ctxd.367.632955034") = 0 <0.056119>
> unlink("ctxd.367.632955035") = 0 <0.056179>
> unlink("ctxd.367.632955036") = 0 <0.061726>
> unlink("ctxd.367.632955037") = 0 <0.170615>
> unlink("ctxd.367.632955038") = 0 <0.173498>
> unlink("ctxd.367.632955039") = 0 <0.088021>
> unlink("ctxd.368.632955010") = 0 <0.045662>
> unlink("ctxd.368.632955011") = 0 <0.050733>
> unlink("ctxd.368.632955012") = 0 <0.051693>
> unlink("ctxd.368.632955013") = 0 <0.054854>
> unlink("ctxd.368.632955014") = 0 <0.061138>
> unlink("ctxd.368.632955015") = 0 <0.048361>
> unlink("ctxd.368.632955016") = 0 <0.084508>
> unlink("ctxd.368.632955017") = 0 <0.062291>
> unlink("ctxd.368.632955018") = 0 <0.110628>
> unlink("ctxd.368.632955019") = 0 <0.064505>
> unlink("ctxd.368.632955020") = 0 <0.080547>
> unlink("ctxd.368.632955021") = 0 <0.058048>
> unlink("ctxd.368.632955022") = 0 <0.063159>
> unlink("ctxd.368.632955023") = 0 <0.068933>
> unlink("ctxd.368.632955024") = 0 <0.045993>
> unlink("ctxd.368.632955025") = 0 <0.042045>
> unlink("ctxd.368.632955026") = 0 <0.071292>
> unlink("ctxd.368.632955027") = 0 <0.054626>
> unlink("ctxd.368.632955028") = 0 <0.049143>
> unlink("ctxd.368.632955029") = 0 <0.082537>
> unlink("ctxd.368.632955030") = 0 <0.061396>
> unlink("ctxd.368.632955031") = 0 <0.062972>
> unlink("ctxd.368.632955032") = 0 <0.049971>
> unlink("ctxd.368.632955033") = 0 <0.086166>
> unlink("ctxd.368.632955034") = 0 <0.046925>
> unlink("ctxd.368.632955035") = 0 <0.048068>
> unlink("ctxd.368.632955036") = 0 <0.084247>
> unlink("ctxd.368.632955037") = 0 <0.046254>
> unlink("ctxd.368.632955038") = 0 <0.042681>
> unlink("ctxd.368.632955039") = 0 <0.052280>
> unlink("d") = 0 <0.000059>
> unlink("e") = 0 <0.000050>
> unlink("f") = 0 <0.000048>
> unlink("g") = 0 <0.000049>
> unlink("h") = 0 <0.000047>
> unlink("i") = 0 <0.000046>
> unlink("j") = 0 <0.000046>
> unlink("k") = 0 <0.000050>
> unlink("l") = 0 <0.000045>
> unlink("m") = 0 <0.000041>
> unlink("n") = 0 <0.000035>
> unlink("o") = 0 <0.000033>
> unlink("p") = 0 <0.000036>
> unlink("q") = 0 <0.000035>
> unlink("r") = 0 <0.000034>
> unlink("s") = 0 <0.000034>
> unlink("t") = 0 <0.000035>
> unlink("u") = 0 <0.000036>
> unlink("v") = 0 <0.000035>
> unlink("w") = 0 <0.000035>
> unlink("x") = 0 <0.000035>
> unlink("y") = 0 <0.000034>
> unlink("z")
>
> -----Original Message-----
> From: Philip Rowlands [mailto:[EMAIL PROTECTED]
> Sent: Wednesday, September 26, 2007 8:43 AM
> To: Ken Naim
> Cc: bug-coreutils@gnu.org
> Subject: Re: Rm performance issue
>
> On Wed, 26 Sep 2007, Ken Naim wrote:
>
>> I am removing 300gb of data spread across 130 files within a single
>> directory and the process take just over 2 hours. In my past experiences
>> removing a small number of large files was very quick, almost
> instantaneous.
>> I am running red hat Linux on ibm p series hardware against a san with
> sata
>> and fiber drives. I see this issue on both the sata and fiber side
> although
>> the rm process is slightly faster on fiber.
>
> It's more likely to be caused by your storage system than rm, but here's
> how to tell:
>
> mkdir foo
> touch foo/{a,b,c,d,e,f}
> strace -T -e trace=file rm -rf foo
>
> Try this in a temporary directory on your local disk, to get some idea
> of how long the unlink(2) system call takes. Then try the strace on your
> slow-running rm command, and see how long rm is spending waiting for the
> system call to complete.
>
>
> Cheers,
> Phil
_______________________________________________
Bug-coreutils mailing list
Bug-coreutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-coreutils
