bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2011-01-30 Thread Jim Meyering
Jim Meyering wrote: ... > I've amended those two commits, rebased, and pushed to > a new branch: fiemap-copy-3. As I mentioned, everything is now on "master". Odd how that works. Minutes after pushing everything, I wondered if there was a NEWS entry for this feature... No. So I wrote this: >Fro

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2011-01-29 Thread Jim Meyering
Pádraig Brady wrote: ... >> +# Require a fiemap-enabled FS. >> +df -T -t btrfs -t xfs -t ext4 -t ocfs2 . \ >> + || skip_ "this file system lacks FIEMAP support" >> + >> +# Create a large-but-sparse file. >> +timeout 1 dd bs=1 seek=1T of=f < /dev/null || framework_failure_ >> + >> +# Nothing can re

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2011-01-29 Thread Pádraig Brady
On 29/01/11 09:47, Jim Meyering wrote: > Jim Meyering wrote: >> Jeff liu wrote: >>> Now make check passed against the following combination: >>> 1. Refresh installed host in Ubuntu10.0.4, >>> filefrag comes from E2fsprogs 1.41.11 && Kernel: 2.6.32-16 >>> 2. filefrag in e2fsprogs-1.4.12 && kernel-2

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2011-01-29 Thread Jim Meyering
Jim Meyering wrote: > Jeff liu wrote: >> Now make check passed against the following combination: >> 1. Refresh installed host in Ubuntu10.0.4, >> filefrag comes from E2fsprogs 1.41.11 && Kernel: 2.6.32-16 >> 2. filefrag in e2fsprogs-1.4.12 && kernel-2.6.36. > [passes] > > Glad to here it passes f

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2011-01-26 Thread Jeff liu
在 2011-1-26,上午11:58, jeff.liu 写道: > Jim Meyering wrote: >> jeff.liu wrote: >>> Jim Meyering wrote: jeff.liu wrote: > AFAICS, the tests passed on all filesystems except ext4, Really? The vast majority of my testing is with ext4 on Fedora 14, and I have seen no failure -- ot

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2011-01-25 Thread jeff.liu
Jim Meyering wrote: > jeff.liu wrote: >> Jim Meyering wrote: >>> jeff.liu wrote: AFAICS, the tests passed on all filesystems except ext4, >>> Really? >>> The vast majority of my testing is with ext4 on Fedora 14, and I have seen >>> no failure -- otherwise I would have mentioned that as a know

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2011-01-25 Thread jeff.liu
Jim Meyering wrote: > jeff.liu wrote: >> AFAICS, the tests passed on all filesystems except ext4, > > Really? > The vast majority of my testing is with ext4 on Fedora 14, and I have seen > no failure -- otherwise I would have mentioned that as a known problem. I have mentioned this issue at: http

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2011-01-25 Thread Jim Meyering
jeff.liu wrote: > AFAICS, the tests passed on all filesystems except ext4, Really? The vast majority of my testing is with ext4 on Fedora 14, and I have seen no failure -- otherwise I would have mentioned that as a known problem. What type of system/kernel are you using? Was your ext4 partition c

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2011-01-24 Thread jeff.liu
Hi Jim, Thanks for your time to help consolidating the code! Is this patchset acceptable to merge into the next official release? AFAICS, the tests passed on all filesystems except ext4, but the result is ok by comparing the file contents, can we take this risk? Another thing is to add solaris

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2011-01-22 Thread Jim Meyering
jeff.liu wrote: > Hi Jim and All, > > Do you have any comments for the current implementation? There have been several releases since we last talked about this, but now is a good time to revive it. I've rebased the fiemap-copy branch and made a few changes: (somewhat sloppy 2nd log entry with the

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-10-11 Thread jeff.liu
Jim Meyering wrote: > jeff.liu wrote: > >> Jim Meyering wrote: >>> jeff.liu wrote: Sorry for the delay. This is the new patch to isolate the stuff regarding to extents reading to a new module. and teach cp(1) to make use of it. >>> Jeff, >>> >>> I applied your patch to my

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-10-11 Thread Jim Meyering
jeff.liu wrote: > Jim Meyering wrote: >> jeff.liu wrote: >>> Sorry for the delay. >>> >>> This is the new patch to isolate the stuff regarding to extents reading to >>> a new module. and teach >>> cp(1) to make use of it. >> >> Jeff, >> >> I applied your patch to my rebased fiemap-copy branch. >>

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-09-29 Thread jeff.liu
Jim Meyering wrote: > jeff.liu wrote: >> Sorry for the delay. >> >> This is the new patch to isolate the stuff regarding to extents reading to a >> new module. and teach >> cp(1) to make use of it. > > Jeff, > > I applied your patch to my rebased fiemap-copy branch. > My first step was to run th

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-09-29 Thread jeff.liu
Jim Meyering wrote: > jeff.liu wrote: >> jeff.liu wrote: >>> Hi Jim, >>> >>> Thanks for your prompt response, I will fix this issue when all review done. >> Hi Jim, >> >> For my current implementation, I just have another thought to remove the >> "char *fname" from struct >> extent_scan, and add a

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-09-28 Thread Jim Meyering
jeff.liu wrote: > jeff.liu wrote: >> Hi Jim, >> >> Thanks for your prompt response, I will fix this issue when all review done. > Hi Jim, > > For my current implementation, I just have another thought to remove the > "char *fname" from struct > extent_scan, and add a new item "int errno" to save t

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-09-28 Thread Jim Meyering
jeff.liu wrote: > Sorry for the delay. > > This is the new patch to isolate the stuff regarding to extents reading to a > new module. and teach > cp(1) to make use of it. Jeff, I applied your patch to my rebased fiemap-copy branch. My first step was to run the usual ./bootstrap && ./configure

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-09-27 Thread jeff.liu
jeff.liu wrote: > Hi Jim, > > Thanks for your prompt response, I will fix this issue when all review done. Hi Jim, For my current implementation, I just have another thought to remove the "char *fname" from struct extent_scan, and add a new item "int errno" to save the errno set by ioctl(2) or

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-09-26 Thread Jim Meyering
Hi Jeff, This function has problems: - the inner "zeros" declaration shadows the outer one and ends up being useless. - the "sizeof zeros" resolves to 4 or 8. obviously not what you intended. ... > static bool > +write_zeros (int fd, uint64_t n_bytes) > { > - bool last = false; > -

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-09-26 Thread jeff.liu
Hi Jim, Thanks for your prompt response, I will fix this issue when all review done. Regards, -Jeff Jim Meyering wrote: > Hi Jeff, > > This function has problems: > - the inner "zeros" declaration shadows the outer one > and ends up being useless. > - the "sizeof zeros" resolves to 4

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-09-26 Thread jeff.liu
Hi Jim, Sorry for the delay. This is the new patch to isolate the stuff regarding to extents reading to a new module. and teach cp(1) to make use of it. Changes: extent-scan.h/extent-scan.c: 1. Removed all the *static* variables from this module. I'd like to introduce two new data stru

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-09-21 Thread jeff.liu
Hi Jim, Thanks for your prompt response and kindly suggestion! I totally agree with your review comments, I will post next round patches according to that soon. Regards, -Jeff Jim Meyering wrote: > jeff.liu wrote: >> Hi Jim and All, >> >> Do you have any comments for the current implementation

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-09-20 Thread Jim Meyering
jeff.liu wrote: > Hi Jim and All, > > Do you have any comments for the current implementation? Hi Jeff, Thanks for the reminder. I've just gone back and looked at those patches: http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/20534/focus=21008 There are some superficial problems. Fir

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-09-20 Thread jeff.liu
Hi Jim and All, Do you have any comments for the current implementation? Sorry for my so delayed asking. Thanks, -Jeff jeff.liu wrote: > Hello All, > > Below is my patches to isolate the extents scan and fetch functions in a new > module to improve its > extendibility. > > It introduce a ne

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-16 Thread Joel Becker
On Sat, Jul 17, 2010 at 10:11:30AM +0800, jeff.liu wrote: > Joel Becker wrote: > > On Fri, Jul 16, 2010 at 08:53:27AM -0700, Paul Eggert wrote: > >> I haven't had time to look at it carefully, but here's a very brief > >> review. The code you sent, like what's in the fiemap branch, has > >> a sepa

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-16 Thread jeff.liu
Joel Becker wrote: > On Sat, Jul 17, 2010 at 10:11:30AM +0800, jeff.liu wrote: >> Joel Becker wrote: >>> On Fri, Jul 16, 2010 at 08:53:27AM -0700, Paul Eggert wrote: I haven't had time to look at it carefully, but here's a very brief review. The code you sent, like what's in the fiemap b

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-16 Thread jeff.liu
Joel Becker wrote: > On Fri, Jul 16, 2010 at 08:53:27AM -0700, Paul Eggert wrote: >> I haven't had time to look at it carefully, but here's a very brief >> review. The code you sent, like what's in the fiemap branch, has >> a separate version of a chunk of copy.c that does both reading >> and writ

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-16 Thread Joel Becker
On Fri, Jul 16, 2010 at 08:53:27AM -0700, Paul Eggert wrote: > I haven't had time to look at it carefully, but here's a very brief > review. The code you sent, like what's in the fiemap branch, has > a separate version of a chunk of copy.c that does both reading > and writing and optimizes both re

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-16 Thread Paul Eggert
On 07/16/10 07:49, jeff.liu wrote: > For now, I am inclined to separate efficient read through fiemap > and improve the write and allocation stuff via fallocate() or other ways > later. I haven't had time to look at it carefully, but here's a very brief review. The code you sent, like what's in

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-16 Thread jeff.liu
Paul Eggert wrote: This doesn't sound right. A FIEMAP_EXTENT_UNWRITTEN extent is all zeros, and so it should act as if it were a hole. The goal is not to copy the exact fiemap structure of the source (that's impossible): the goal is to use as little time and space as pos

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-15 Thread Paul Eggert
>>> This doesn't sound right. A FIEMAP_EXTENT_UNWRITTEN extent is all zeros, >>> and >>> so it should act as if it were a hole. The goal is not to copy the exact >>> fiemap structure of the source (that's impossible): the goal is to use as >>> little time and space as possible. > A FIEMAP_EXTEN

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-15 Thread Paul Eggert
>>> This doesn't sound right. A FIEMAP_EXTENT_UNWRITTEN extent is all zeros, >>> and >>> so it should act as if it were a hole. The goal is not to copy the exact >>> fiemap structure of the source (that's impossible): the goal is to use as >>> little time and space as possible. > A FIEMAP_EXTEN

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-15 Thread Joel Becker
On Thu, Jul 15, 2010 at 12:51:36AM +0100, Pádraig Brady wrote: > On 14/07/10 18:45, Paul Eggert wrote: First and foremost, I re-concur with the broad strokes of the --sparse={always,never,auto} conversation. I think you all knew that, though ;-) > > It's not just fiemap. It's also the S

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-14 Thread jeff.liu
Hi Pádraig and Paul, Thanks for your quick response. Pádraig Brady wrote: > On 14/07/10 18:45, Paul Eggert wrote: I see fiemap just as a way to efficiently detect/read holes, and should have no bearing on the destination. >> Hmm, but the proposal quoted below would mean that fiemap does

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-14 Thread Pádraig Brady
On 14/07/10 18:45, Paul Eggert wrote: >>> I see fiemap just as a way to efficiently detect/read holes, >>> and should have no bearing on the destination. > > Hmm, but the proposal quoted below would mean that fiemap does have a > bearing on the destination, in the --sparse=auto case. > I guess thi

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-14 Thread Paul Eggert
>> I see fiemap just as a way to efficiently detect/read holes, >> and should have no bearing on the destination. Hmm, but the proposal quoted below would mean that fiemap does have a bearing on the destination, in the --sparse=auto case. I guess this is OK, but it should be documented. >> cp --s

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-14 Thread Pádraig Brady
On 14/07/10 07:58, jeff.liu wrote: > Pádraig Brady wrote: > > Hello All, > > I am very sorry for the late response! I have an urgent task need to deliver > in the past few weeks. > > Thanks for all your suggestions, I would like to improve the fiemap-copy > accordingly, so does the > following

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-07-14 Thread jeff.liu
Pádraig Brady wrote: > On 16/06/10 09:49, Jim Meyering wrote: >> Pádraig Brady wrote: >> >>> On 16/06/10 07:45, Jim Meyering wrote: Paul Eggert wrote: > On 06/15/2010 02:43 PM, Jim Meyering wrote: >> I think that copying physical holes via FIEMAP should be the default, >> when >>>

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-16 Thread Tao Ma
On 06/16/2010 05:03 PM, Joel Becker wrote: On Wed, Jun 16, 2010 at 02:57:01PM +0800, jeff.liu wrote: Paul Eggert wrote: For example, if a fiemap_extent has the FIEMAP_EXTENT_UNWRITTEN flag set, cp should treat that as a hole, because the extent is all zeros. (This will greatly help performanc

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-16 Thread Pádraig Brady
On 16/06/10 09:49, Jim Meyering wrote: > Pádraig Brady wrote: > >> On 16/06/10 07:45, Jim Meyering wrote: >>> Paul Eggert wrote: On 06/15/2010 02:43 PM, Jim Meyering wrote: > I think that copying physical holes via FIEMAP should be the default, when > possible. One problem is that th

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-16 Thread Joel Becker
On Wed, Jun 16, 2010 at 02:57:01PM +0800, jeff.liu wrote: > Paul Eggert wrote: > > For example, if a fiemap_extent has the FIEMAP_EXTENT_UNWRITTEN flag > > set, cp should treat that as a hole, because the extent is all zeros. > > (This will greatly help performance in some cases.) Also, if an inpu

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-16 Thread Joel Becker
On Wed, Jun 16, 2010 at 10:49:08AM +0200, Jim Meyering wrote: > > I would suggest not to add options like this, > > and only do what's possible without new options > > as it would push too many implementation details > > to the docs/users IMHO. > > Currently on the fiemap-copy branch, once we get

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-16 Thread Jim Meyering
Pádraig Brady wrote: > On 16/06/10 07:45, Jim Meyering wrote: >> Paul Eggert wrote: >>> On 06/15/2010 02:43 PM, Jim Meyering wrote: I think that copying physical holes via FIEMAP should be the default, when possible. One problem is that the current code on the fiemap-copy branch do

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-16 Thread Pádraig Brady
On 16/06/10 07:45, Jim Meyering wrote: > Paul Eggert wrote: >> On 06/15/2010 02:43 PM, Jim Meyering wrote: >>> I think that copying physical holes via FIEMAP should be the default, when >>> possible. One problem is that the current code on the fiemap-copy branch >>> does not honor --sparse=WHEN wh

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-16 Thread jeff.liu
Paul Eggert wrote: > I looked at the proposed fiemap patch for cp and see another issue > with it, which needs to be thought through more carefully. > > The proposed fiemap code tries to copy the _physical_ holes in the > source file. But currently cp copies the _logical_ holes: that is, it > ski

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-16 Thread jeff.liu
Tao Ma wrote: > Hi Paul, > > On 06/16/2010 05:11 AM, Paul Eggert wrote: >> On 06/10/2010 05:35 PM, Tao Ma wrote: >> >>> there is a flag FIEMAP_EXTENT_DELALLOC in fiemap ... >>> with dd if=/dev/zero of=/mnt/ext4/a bs=1M count=1 seek=1 >>> the ext4 can't return a valid fiemap extent. >> >> Hmm, this

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-16 Thread Jim Meyering
Paul Eggert wrote: > On 06/15/2010 02:43 PM, Jim Meyering wrote: >> I think that copying physical holes via FIEMAP should be the default, when >> possible. One problem is that the current code on the fiemap-copy branch >> does not honor --sparse=WHEN when in fiemap-copying mode. The solution >> w

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-15 Thread Tao Ma
Hi Paul, On 06/16/2010 05:11 AM, Paul Eggert wrote: On 06/10/2010 05:35 PM, Tao Ma wrote: there is a flag FIEMAP_EXTENT_DELALLOC in fiemap ... with dd if=/dev/zero of=/mnt/ext4/a bs=1M count=1 seek=1 the ext4 can't return a valid fiemap extent. Hmm, this sounds like a fairly serious bug, in

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-15 Thread Pádraig Brady
On 15/06/10 23:19, Paul Eggert wrote: > On 06/15/2010 02:43 PM, Jim Meyering wrote: > >> I think that copying physical holes via FIEMAP should be the default, when >> possible. One problem is that the current code on the fiemap-copy branch >> does not honor --sparse=WHEN when in fiemap-copying mo

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-15 Thread Sunil Mushran
On 06/15/2010 03:19 PM, Paul Eggert wrote: On 06/15/2010 02:43 PM, Jim Meyering wrote: I think that copying physical holes via FIEMAP should be the default, when possible. One problem is that the current code on the fiemap-copy branch does not honor --sparse=WHEN when in fiemap-copying mod

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-15 Thread Joel Becker
On Tue, Jun 15, 2010 at 03:30:15PM -0700, Sunil Mushran wrote: > On 06/15/2010 03:19 PM, Paul Eggert wrote: > >This all is getting a bit complicated, I'm afraid. Perhaps it'd be better > >to try this stuff out with a new option, "cp --sparse=extents" say, and > >keep the default as-is for a while?

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-15 Thread Paul Eggert
On 06/15/2010 02:43 PM, Jim Meyering wrote: > I think that copying physical holes via FIEMAP should be the default, when > possible. One problem is that the current code on the fiemap-copy branch > does not honor --sparse=WHEN when in fiemap-copying mode. The solution > would seem to be to chang

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-15 Thread Joel Becker
On Tue, Jun 15, 2010 at 02:28:43PM -0700, Paul Eggert wrote: > I looked at the proposed fiemap patch for cp and see another issue > with it, which needs to be thought through more carefully. > > The proposed fiemap code tries to copy the _physical_ holes in the > source file. But currently cp cop

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-15 Thread Paul Eggert
I looked at the proposed fiemap patch for cp and see another issue with it, which needs to be thought through more carefully. The proposed fiemap code tries to copy the _physical_ holes in the source file. But currently cp copies the _logical_ holes: that is, it skips over writing (and thus attem

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-15 Thread Jim Meyering
Paul Eggert wrote: > I looked at the proposed fiemap patch for cp and see another issue > with it, which needs to be thought through more carefully. > > The proposed fiemap code tries to copy the _physical_ holes in the > source file. But currently cp copies the _logical_ holes: that is, it > skip

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-15 Thread Paul Eggert
On 06/11/2010 01:31 AM, jeff.liu wrote: > + fiemap->fm_flags = FIEMAP_FLAG_SYNC; If I'm reading the Linux source code correctly, this forces all the dirty blocks in the input file to disk, and forces the kernel to wait until all those blocks actually hit the disk. We don't need that; there s

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-15 Thread Paul Eggert
Sunil Mushran wrote: > SEEK_HOLE/DATA also have the same problem with active files. Yes, that's true if 'cp' is copying a file while someone else is writing to it. But the case we're worried about is when 'cp' starts copying a file immediately after someone else has finished writing to it but dat

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-15 Thread Paul Eggert
On 06/10/2010 05:35 PM, Tao Ma wrote: > there is a flag FIEMAP_EXTENT_DELALLOC in fiemap ... > with dd if=/dev/zero of=/mnt/ext4/a bs=1M count=1 seek=1 > the ext4 can't return a valid fiemap extent. Hmm, this sounds like a fairly serious bug, in that it would prevent this part of cp from working

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-13 Thread Jim Meyering
Jim Meyering wrote: ... > So I will apply the test-fixing patch first, then your change, > Jeff, and then rebase to the latest on master. FYI, I did that. In addition, I needed two more changes in order to avoid "make distcheck" failure: >From be5548445e90a36ab5018cac0fb19f2498d0521c Mon Sep 17

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-12 Thread jeff.liu
Eric Sandeen wrote: > jeff.liu wrote: >> Sunil Mushran wrote: >>> On 06/10/2010 04:47 PM, Paul Eggert wrote: On 06/09/2010 11:56 PM, jeff.liu wrote: > Yeah, I just realized that the behaviour I observed is caused by the > delay allocation mechanism of > the particular FS. >

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-11 Thread Eric Sandeen
jeff.liu wrote: > Sunil Mushran wrote: >> On 06/10/2010 04:47 PM, Paul Eggert wrote: >>> On 06/09/2010 11:56 PM, jeff.liu wrote: >>> Yeah, I just realized that the behaviour I observed is caused by the delay allocation mechanism of the particular FS. >>> If the file sys

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-11 Thread Jim Meyering
Paul Eggert wrote: > On 06/09/2010 11:56 PM, jeff.liu wrote: > >> Yeah, I just realized that the behaviour I observed is caused by the delay >> allocation mechanism of >> the particular FS. > > If the file system is using delayed allocation, then can > the fiemap ioctl tell us that a file contains

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-11 Thread Jim Meyering
jeff.liu wrote: > Sunil Mushran wrote: ... >> I guess we'll have to use FIEMAP_FLAG_SYNC. > Hi Sunil, > > Thanks for the comments. > So we can ensure the source file synced before mapping in this way. > > Hi Jim and Paul, > > How about the tiny patch below? ... > Subject: [PATCH 1/1] copy.c: add FI

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-11 Thread jeff.liu
Sunil Mushran wrote: > On 06/10/2010 04:47 PM, Paul Eggert wrote: >> On 06/09/2010 11:56 PM, jeff.liu wrote: >> >>> Yeah, I just realized that the behaviour I observed is caused by the >>> delay allocation mechanism of >>> the particular FS. >>> >> If the file system is using delayed alloc

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-10 Thread Joel Becker
On Thu, Jun 10, 2010 at 04:47:23PM -0700, Paul Eggert wrote: > On 06/09/2010 11:56 PM, jeff.liu wrote: > > > Yeah, I just realized that the behaviour I observed is caused by the delay > > allocation mechanism of > > the particular FS. > > If the file system is using delayed allocation, then can

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-10 Thread Sunil Mushran
On 06/10/2010 04:47 PM, Paul Eggert wrote: On 06/09/2010 11:56 PM, jeff.liu wrote: Yeah, I just realized that the behaviour I observed is caused by the delay allocation mechanism of the particular FS. If the file system is using delayed allocation, then can the fiemap ioctl tell us t

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-10 Thread Tao Ma
Paul Eggert wrote: On 06/09/2010 11:56 PM, jeff.liu wrote: Yeah, I just realized that the behaviour I observed is caused by the delay allocation mechanism of the particular FS. If the file system is using delayed allocation, then can the fiemap ioctl tell us that a file contains a ho

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-10 Thread Paul Eggert
On 06/09/2010 11:56 PM, jeff.liu wrote: > Yeah, I just realized that the behaviour I observed is caused by the delay > allocation mechanism of > the particular FS. If the file system is using delayed allocation, then can the fiemap ioctl tell us that a file contains a hole (because nothing has b

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-09 Thread jeff.liu
Jim Meyering wrote: > jeff.liu wrote: > >> Jim Meyering wrote: >>> Jim Meyering wrote: Subject: [PATCH 01/10] cp: Add FIEMAP support for efficient sparse file copy >>> FYI, using those patches, I ran a test for the first time in a few days: >>> >>> check -C tests TESTS=cp/sparse-fie

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-09 Thread Jim Meyering
jeff.liu wrote: > Jim Meyering wrote: >> Jim Meyering wrote: >>> Subject: [PATCH 01/10] cp: Add FIEMAP support for efficient sparse file copy >> >> FYI, using those patches, I ran a test for the first time in a few days: >> >> check -C tests TESTS=cp/sparse-fiemap VERBOSE=yes >> >> It failed l

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-09 Thread jeff.liu
Jim Meyering wrote: > Jim Meyering wrote: >> Subject: [PATCH 01/10] cp: Add FIEMAP support for efficient sparse file copy > > FYI, using those patches, I ran a test for the first time in a few days: > > check -C tests TESTS=cp/sparse-fiemap VERBOSE=yes > > It failed like this on an ext4 part

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-09 Thread jeff.liu
Jim Meyering wrote: > Paul Eggert wrote: >> Jeff Liu and Jim Meyering wrote: >>> diff --git a/src/fiemap.h b/src/fiemap.h >>> new file mode 100644 >>> index 000..d33293b >>> --- /dev/null >>> +++ b/src/fiemap.h > > Hi Paul, > > Thanks for the review. > >> Why is this file necessary? fiemap.

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-09 Thread jeff.liu
Jim Meyering wrote: > Paul Eggert wrote: >> Jeff Liu and Jim Meyering wrote: >>> diff --git a/src/fiemap.h b/src/fiemap.h >>> new file mode 100644 >>> index 000..d33293b >>> --- /dev/null >>> +++ b/src/fiemap.h > > Hi Paul, > > Thanks for the review. > >> Why is this file necessary? fiemap.

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-09 Thread Jim Meyering
Paul Eggert wrote: > Jeff Liu and Jim Meyering wrote: >> diff --git a/src/fiemap.h b/src/fiemap.h >> new file mode 100644 >> index 000..d33293b >> --- /dev/null >> +++ b/src/fiemap.h Hi Paul, Thanks for the review. > Why is this file necessary? fiemap.h is included only if it exists, > righ

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-08 Thread Paul Eggert
Jeff Liu and Jim Meyering wrote: > diff --git a/src/fiemap.h b/src/fiemap.h > new file mode 100644 > index 000..d33293b > --- /dev/null > +++ b/src/fiemap.h Why is this file necessary? fiemap.h is included only if it exists, right? Shouldn't we just use the kernel's fiemap.h rather than cop

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-08 Thread Jim Meyering
Eric Blake wrote: >> +++ b/tests/cp/sparse-fiemap >> @@ -0,0 +1,56 @@ >> +#!/bin/sh >> +# Test cp --sparse=always through fiemap copy >> + >> +# Copyright (C) 2006-2010 Free Software Foundation, Inc. > > How much of this content comes from other files from 2006, vs. new > content needing only 2010?

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-08 Thread Jim Meyering
Jim Meyering wrote: > Subject: [PATCH 01/10] cp: Add FIEMAP support for efficient sparse file copy FYI, using those patches, I ran a test for the first time in a few days: check -C tests TESTS=cp/sparse-fiemap VERBOSE=yes It failed like this on an ext4 partition using F13: + timeout 10

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-08 Thread Sunil Mushran
On 06/08/2010 05:10 AM, Eric Blake wrote: new file mode 100644 index 000..d33293b --- /dev/null +++ b/src/fiemap.h @@ -0,0 +1,102 @@ +/* FS_IOC_FIEMAP ioctl infrastructure. + Some portions copyright (C) 2007 Cluster File Systems, Inc + Authors: Mark Fasheh +Kalpak Shah +

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-08 Thread Eric Blake
On 06/08/2010 05:33 AM, Jim Meyering wrote: > new file mode 100644 > index 000..d33293b > --- /dev/null > +++ b/src/fiemap.h > @@ -0,0 +1,102 @@ > +/* FS_IOC_FIEMAP ioctl infrastructure. > + Some portions copyright (C) 2007 Cluster File Systems, Inc > + Authors: Mark Fasheh > +

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-08 Thread Jim Meyering
Jim Meyering wrote: > Jim Meyering wrote: > ... Do you know of a tool other than filefrag that I could use? >>> nope. It looks like a small script could filter filefrag -v output, detect split extents and rewrite to make the output match what's expected. Probably not worth

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-06-02 Thread Jim Meyering
Jim Meyering wrote: ... >>> Do you know of a tool other than filefrag that I could use? >> nope. >>> >>> It looks like a small script could filter filefrag -v output, detect >>> split extents and rewrite to make the output match what's expected. >>> Probably not worth it, though, since this is alre

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-30 Thread jeff.liu
Jim Meyering wrote: > Tao Ma wrote: >> Hi Jim, >> On 05/29/2010 12:44 AM, Jim Meyering wrote: >>> Tao Ma wrote: >>> Hi Jim On 05/27/2010 06:30 PM, Jim Meyering wrote: > jeff.liu wrote: >> This is the revised version, it fixed the fiemap-start offset calculation >> approac

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-30 Thread Jim Meyering
Tao Ma wrote: > Hi Jim, > On 05/29/2010 12:44 AM, Jim Meyering wrote: >> Tao Ma wrote: >> >>> Hi Jim >>> >>> On 05/27/2010 06:30 PM, Jim Meyering wrote: jeff.liu wrote: > This is the revised version, it fixed the fiemap-start offset calculation > approach to remove it out > of the

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-30 Thread Tao Ma
Hi Jim, On 05/29/2010 12:44 AM, Jim Meyering wrote: Tao Ma wrote: Hi Jim On 05/27/2010 06:30 PM, Jim Meyering wrote: jeff.liu wrote: This is the revised version, it fixed the fiemap-start offset calculation approach to remove it out of the 'for (i = 0; i< fiemap->fm_mapped_extents; i++)'

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-28 Thread Jim Meyering
Tao Ma wrote: > Hi Jim > > On 05/27/2010 06:30 PM, Jim Meyering wrote: >> jeff.liu wrote: >>> This is the revised version, it fixed the fiemap-start offset calculation >>> approach to remove it out >>> of the 'for (i = 0; i< fiemap->fm_mapped_extents; i++)' loop. >> >> Hi Jeff, >> >> I've include

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-28 Thread Tao Ma
Hi Jim On 05/27/2010 06:30 PM, Jim Meyering wrote: jeff.liu wrote: This is the revised version, it fixed the fiemap-start offset calculation approach to remove it out of the 'for (i = 0; i< fiemap->fm_mapped_extents; i++)' loop. Hi Jeff, I've included below the state of my local changes. Un

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-27 Thread Jim Meyering
Jim Meyering wrote: > jeff.liu wrote: >> Jim Meyering wrote: >>> jeff.liu wrote: This is the revised version, it fixed the fiemap-start offset calculation approach to remove it out of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)' loop. >>> >>> Hi Jeff, >>> >>> I've included b

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-27 Thread Sunil Mushran
On 05/27/2010 11:43 AM, Jim Meyering wrote: Sunil Mushran wrote: Jim Meyering wrote: Hi Jeff, I've included below the state of my local changes. Unfortunately, with that 5-patch series, there is always a test failure on F13/ext4. Maybe someone who knows more about extents can provid

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-27 Thread Jim Meyering
jeff.liu wrote: > Jim Meyering wrote: >> jeff.liu wrote: >>> This is the revised version, it fixed the fiemap-start offset calculation >>> approach to remove it out >>> of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)' loop. >> >> Hi Jeff, >> >> I've included below the state of my local chan

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-27 Thread Jim Meyering
Sunil Mushran wrote: > Jim Meyering wrote: >>> Hi Jeff, >>> >>> I've included below the state of my local changes. >>> Unfortunately, with that 5-patch series, there is always a test failure >>> on F13/ext4. Maybe someone who knows more about extents can provide an >>> explanation? >>> >>> Here's

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-27 Thread Sunil Mushran
Jim Meyering wrote: Hi Jeff, I've included below the state of my local changes. Unfortunately, with that 5-patch series, there is always a test failure on F13/ext4. Maybe someone who knows more about extents can provide an explanation? Here's a small example to demonstrate: Create a file with

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-27 Thread jeff.liu
Jim Meyering wrote: > jeff.liu wrote: >> This is the revised version, it fixed the fiemap-start offset calculation >> approach to remove it out >> of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)' loop. > > Hi Jeff, > > I've included below the state of my local changes. > Unfortunately, wi

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-27 Thread Jim Meyering
jeff.liu wrote: > This is the revised version, it fixed the fiemap-start offset calculation > approach to remove it out > of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)' loop. Hi Jeff, I've included below the state of my local changes. Unfortunately, with that 5-patch series, there is al

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-25 Thread jeff.liu
Jim Meyering wrote: > jeff.liu wrote: > ... >> Please ignore above patch, I just found another issue in 'tests/cp/sparse', >> the new created test >> file also named to 'sparse', so when it running, the `cp/sparse' will be >> truncated to `expr 128 \* >> 1024 + 1`. >> >> Below patch fix it to cre

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-24 Thread Jim Meyering
jeff.liu wrote: ... > Please ignore above patch, I just found another issue in 'tests/cp/sparse', > the new created test > file also named to 'sparse', so when it running, the `cp/sparse' will be > truncated to `expr 128 \* > 1024 + 1`. > > Below patch fix it to create a sparse file 'sparse1' ins

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-24 Thread Jim Meyering
jeff.liu wrote: > Jim Meyering wrote: >> jeff.liu wrote: >> ... > Subject: [PATCH 1/1] tests: add a new test for FIEMAP-copy > > * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a > loopbacked ext4 partition. > * tests/Makefile.am (sparse-fiemap): Reference the n

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-24 Thread jeff.liu
jeff.liu wrote: > Jim Meyering wrote: >> jeff.liu wrote: >>> Jim Meyering wrote: jeff.liu wrote: ... >>> Subject: [PATCH 1/1] tests: add a new test for FIEMAP-copy >>> >>> * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a >>> loopbacked ext4 partition. >>>

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-24 Thread jeff.liu
Jim Meyering wrote: > jeff.liu wrote: >> Jim Meyering wrote: >>> jeff.liu wrote: >>> ... >> Subject: [PATCH 1/1] tests: add a new test for FIEMAP-copy >> >> * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a >> loopbacked ext4 partition. >> * tests/Makefile.am (s

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-21 Thread jeff.liu
Hi Jim, This is the revised version, it fixed the fiemap-start offset calculation approach to remove it out of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)' loop. I have not got a 64bits machine for the testing at the moment, at the following, the first case only run againt x86 with val

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-21 Thread jeff.liu
Jim Meyering wrote: > jeff.liu wrote: > ... >>> [*] I tried to count syscalls with strace but got a segfault. >>> Using valgrind I get errors, so debugged enough to get a clean >>> run, but possibly at the expense of correctness. We'll need more >>> tests to ensure that the non-sparse blocks in th

bug#6131: [PATCH]: fiemap support for efficient sparse file copy

2010-05-21 Thread Jim Meyering
jeff.liu wrote: ... > Sorry for the lack of detailed info for this point, except for removing the > fiemap->fm_start from > the loop, I need to remove "fiemap->fm_start = (fm_ext[i-1].fe_logical + > fm_ext[i-1].fe_length);" > out of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)" as well. >

  1   2   >