On Tue, January 5, 2010 17:08, David Magda wrote:
> On Tue, January 5, 2010 05:34, Mikko Lammi wrote:
>
>> As a result of one badly designed application running loose for some
>> time,
>> we now seem to have over 60 million files in one directory. Good thing
>> about ZFS is that it allows it without any issues. Unfortunatelly now
>> that
>> we need to get rid of them (because they eat 80% of disk space) it seems
>> to be quite challenging.
>
> How about creating a new data set, moving the directory into it, and then
> destroying it?
>
> Assuming the directory in question is /opt/MYapp/data:
> 1. zfs create rpool/junk
> 2. mv /opt/MYapp/data /rpool/junk/
> 3. zfs destroy rpool/junk
Tried that as well. It's moving individual files to the new directory with
speed approx. 3,000/minute, so it's not any faster than anything that I
can apply directly to the original directory.
I also tried the perl script that does readdir() earlier (it's as slow as
any other application), and switched the zfs dataset parameter "atime" to
off, but that didn't had much effect either.
However when we deleted some other files from the volume and managed to
raise free disk space from 4 GB to 10 GB, the "rm -rf directory" method
started to perform significantly faster. Now it's deleting around 4,000
files/minute (240,000/h - quite an improvement from 10,000/h). I remember
that I saw some discussion related to ZFS performance when filesystem
becomes very full, so I wonder if that was the case here.
Next I'm going to try if that "find ./ -exec {} +" yelds any better
results than "rm -rf" from parent directory. But I guess at some point the
bottleneck will be just CPU (this is a 1-Ghz T1000 system) and disk I/O,
not the ZFS filesystem. I'm just wondering of what kind of figures to
expect.
regards,
Mikko
--
Mikko Lammi | l...@lmmz.net | http://www.lmmz.net
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
