For some reason my previous emails on this topic didn't go out which I just realized based on my post hitting the rsync lists. If this is a duplicate then ignore it.
I've been doing some testing on my CentOS 5 system running a storage array with XFS. There are huge amounts of fragmentation on very large files when creating large files similar to the results I've previously posted with Windows and cygwin. The large amounts of fragmentation have impacted performance. Matt's patch worked great for cygwin (preallocate.diff). The same approach is not working well under CentOS since it writes out all those 0's for the files using the posix_fallocate function. It seems to me that under Linux the ftruncate function may be better suited for the implementation? I tested using ftruncate in lieu of posix_fallocate and it appears to create the files almost instantaneously on both XFS and Ext3 (the only file system formats I've tested). Filefrag reports just a single extent on both file systems (file created was 77GB). Oddly, filefrag reports a perfect fragmentation of 172 with the file I created on Ext2 but it only had 1 extent. I'm not sure I fully understand the "perfect" fragmentation number. The attached file has replaced all the posix_fallocate functions with ftruncate and removed the check for HAVE_POSIX_FALLOCATE. I am getting ready to test this patch on rsync-3.0.0pre10 to see if it works. Thought I would send it out to see if it makes sense to you, Matt, since you developed it. I don't have the skills to do the modifications to check for cygwin versus another distribution to determine if posix_fallocate should be used versus ftruncate. Again, sorry for any duplication of posts. Rob -----Original Message----- From: Jamie Lokier [mailto:[EMAIL PROTECTED] Sent: Sunday, November 18, 2007 10:06 PM To: Matt McCutchen Cc: Rob Bosch; rsync@lists.samba.org Subject: Re: preallocate CPU usage - pre4 Matt McCutchen wrote: > I notice that the Linux kernel 2.6.23 has gained a system call > "fallocate" that preallocates at the filesystem level like Cygwin's > implementation of posix_fallocate; thus, preallocation may become (at > least slightly) helpful on Linux. Unfortunately, neither ext2 nor > reiserfs supports this call yet, at least in the kernel on my computer; > I hope support will be added soon. It won't work on EXT2 because it requires changes to the filesystem format, to reserve space without writing data. So, you can use it with EXT4 instead... I think you can change EXT2/3 to EXT4 without reformatting, just like EXT2 to EXT3. In Linux 2.6.23, fallocate works on EXT4 and OCFS2 filesystems. Support is expected on CIFS eventually. -- Jamie
preallocate-linux.diff
Description: Binary data
-- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
