Re: Bug#605009: serious performance regression with ext4

2010-11-30 Thread Mike Hommey
On Tue, Nov 30, 2010 at 10:35:11AM +0100, Samuel Thibault wrote: > Mike Hommey, le Tue 30 Nov 2010 10:07:55 +0100, a écrit : > > On Mon, Nov 29, 2010 at 07:18:17AM +0100, Guillem Jover wrote: > > > > What's going on here? sync_file_range() is a Linux specific system > > > > call that has been arou

Re: Bug#605009: serious performance regression with ext4

2010-11-30 Thread Bastian Blank
On Tue, Nov 30, 2010 at 10:35:11AM +0100, Samuel Thibault wrote: > Mike Hommey, le Tue 30 Nov 2010 10:07:55 +0100, a écrit : > > On Mon, Nov 29, 2010 at 07:18:17AM +0100, Guillem Jover wrote: > > > Hmm, ok so what about posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED) > > > instead, skimming over the k

Re: Bug#605009: serious performance regression with ext4

2010-11-30 Thread Samuel Thibault
Mike Hommey, le Tue 30 Nov 2010 10:07:55 +0100, a écrit : > On Mon, Nov 29, 2010 at 07:18:17AM +0100, Guillem Jover wrote: > > > What's going on here? sync_file_range() is a Linux specific system > > > call that has been around for a while. It allows program to control > > > when writeback happen

Re: Bug#605009: serious performance regression with ext4

2010-11-30 Thread Mike Hommey
On Mon, Nov 29, 2010 at 07:18:17AM +0100, Guillem Jover wrote: > > What's going on here? sync_file_range() is a Linux specific system > > call that has been around for a while. It allows program to control > > when writeback happens in a very low-level fashion. The first set of > > sync_file_ran

Re: Bug#605009: serious performance regression with ext4

2010-11-29 Thread Olaf van der Spek
On Mon, Nov 29, 2010 at 8:23 PM, Raphael Hertzog wrote: > (Dropping bug report) > > On Mon, 29 Nov 2010, Olaf van der Spek wrote: >> > Hence the fact that all file system developers, whether they were >> > btrfs developers or XFS developers or ext4 developers, made the joke >> > at the file system

Re: Bug#605009: serious performance regression with ext4

2010-11-29 Thread Raphael Hertzog
(Dropping bug report) On Mon, 29 Nov 2010, Olaf van der Spek wrote: > > Hence the fact that all file system developers, whether they were > > btrfs developers or XFS developers or ext4 developers, made the joke > > at the file system developers summit two years ago, that what the > > application p

Re: Bug#605009: serious performance regression with ext4

2010-11-29 Thread Olaf van der Spek
On Mon, Nov 29, 2010 at 4:18 PM, Ted Ts'o wrote: > Lots of users have complained about the desktop performance problem, > but the reality is we can't really solve that without also taking away > the magic that made (c) happen.  Whether you solve it by using > data=writeback and stick with ext3, or

Re: Bug#605009: serious performance regression with ext4

2010-11-29 Thread Ted Ts'o
On Mon, Nov 29, 2010 at 02:58:16PM +, Ian Jackson wrote: > > This is the standard way that ordinary files for which reliability was > important have been updated on Unix for decades. fsync is for files > which need synchronisation with things external to the computer (or at > least, external

Re: Bug#605009: serious performance regression with ext4

2010-11-29 Thread Olaf van der Spek
On Mon, Nov 29, 2010 at 3:58 PM, Ian Jackson wrote: > Olaf van der Spek writes ("Re: Bug#605009: serious performance regression > with ext4"): >> Probably not an issue for dpkg, but in general: >> Don't you reset meta-data that way? > > Yes.  If you want to

Re: Bug#605009: serious performance regression with ext4

2010-11-29 Thread Ian Jackson
Olaf van der Spek writes ("Re: Bug#605009: serious performance regression with ext4"): > Probably not an issue for dpkg, but in general: > Don't you reset meta-data that way? Yes. If you want to keep the metadata you must copy it. > Require a second file (name), permi

Re: Bug#605009: serious performance regression with ext4

2010-11-29 Thread Olaf van der Spek
On Mon, Nov 29, 2010 at 2:56 PM, Bastian Blank wrote: > Again: Please quote the standard instead of crying. Your view of things > disallows many of the recent improvements in filesystems, so you have to > show evidence. All the databases and other reliable data handing tools > uses fsync since a l

Re: Bug#605009: serious performance regression with ext4

2010-11-29 Thread Olaf van der Spek
On Mon, Nov 29, 2010 at 2:22 PM, Ian Jackson wrote: > Olaf van der Spek writes ("Re: Bug#605009: serious performance regression > with ext4"): >> Are there any plans to provide an API for atomic (non-durable) file >> updates, not involving fsync? > > Yes.  Such

Re: Bug#605009: serious performance regression with ext4

2010-11-29 Thread Michael Biebl
On 29.11.2010 07:18, Guillem Jover wrote: > > Could someone with ext4/btrfs/xfs/etc test w/ and w/o the attached patch > against dpkg? I'm using ext4 (as already mentioned), my small benchmark is (re)installing vim-runtime using dpkg -i 1.15.8.5: real0m9.259s user0m4.212s sys 0m0.75

Re: Bug#605009: serious performance regression with ext4

2010-11-29 Thread Bastian Blank
On Mon, Nov 29, 2010 at 01:22:25PM +, Ian Jackson wrote: > Olaf van der Spek writes ("Re: Bug#605009: serious performance regression > with ext4"): > > Are there any plans to provide an API for atomic (non-durable) file > > updates, not involving fsync? > Yes

Re: Bug#605009: serious performance regression with ext4

2010-11-29 Thread Ian Jackson
Olaf van der Spek writes ("Re: Bug#605009: serious performance regression with ext4"): > On Fri, Nov 26, 2010 at 10:52 PM, Ted Ts'o wrote: > > I am guessing you are doing (a) today --- am I right? =C2=A0(c) or (d) > > would be best. > > Are there any pla

Re: Bug#605009: serious performance regression with ext4

2010-11-28 Thread Guillem Jover
Hi Ted! On Sun, 2010-11-28 at 23:11:52 -0500, Ted Ts'o wrote: > I did some experimenting, and I figured out what was going on. You're > right, (c) doesn't quite work, because delayed allocation meant that > the writeout didn't take place until the fsync() for each file > happened. I didn't see t

Re: Bug#605009: serious performance regression with ext4

2010-11-28 Thread Ted Ts'o
I did some experimenting, and I figured out what was going on. You're right, (c) doesn't quite work, because delayed allocation meant that the writeout didn't take place until the fsync() for each file happened. I didn't see this at first; my apologies. However, this *does* work: extract(a)

Re: Bug#605009: serious performance regression with ext4

2010-11-27 Thread Olaf van der Spek
On Fri, Nov 26, 2010 at 10:52 PM, Ted Ts'o wrote: > I am guessing you are doing (a) today --- am I right?  (c) or (d) > would be best. Are there any plans to provide an API for atomic (non-durable) file updates, not involving fsync? Would be simpler (for apps), faster and more general (because it

Re: Bug#605009: serious performance regression with ext4

2010-11-27 Thread Guillem Jover
Hi! On Fri, 2010-11-26 at 16:52:54 -0500, Ted Ts'o wrote: > On Fri, Nov 26, 2010 at 03:53:27PM +0100, Raphael Hertzog wrote: > > Just to sum up what dpkg --unpack does in 1.15.8.6: > > 1/ set the package status as half-installed/reinst-required > > 2/ extract all the new files as *.dpkg-new > > 3/

Re: Bug#605009: serious performance regression with ext4

2010-11-27 Thread Raphael Hertzog
On Sat, 27 Nov 2010, Jonathan Nieder wrote: > > c) extract(a.dpkg-new); > > extract(b.dpkg-new); > > extract(c.dpkg-new); > > fsync(a.dpkg-new); > > fsync(b.dpkg-new); > > fsync(c.dpkg-new); > > rename(a.dpkg-new, a); > > rename(b.dpkg-new, b); > > rename(c.dpkg-new

Re: Bug#605009: serious performance regression with ext4

2010-11-26 Thread Jonathan Nieder
Hi Ted, Ted Ts'o wrote: > 1) Suppose package contains files "a", "b", and "c". Which are you > doing? > > a) extract a.dpkg-new ; fsync(a.dpkg-new); rename(a.dpkg-new, a); > extract b.dpkg-new ; fsync(b.dpkg-new); rename(b.dpkg-new, b); > extract c.dpkg-new ; fsync(c.dpkg-new); rename(

Re: Bug#605009: serious performance regression with ext4

2010-11-26 Thread Ted Ts'o
On Fri, Nov 26, 2010 at 03:53:27PM +0100, Raphael Hertzog wrote: > Just to sum up what dpkg --unpack does in 1.15.8.6: > 1/ set the package status as half-installed/reinst-required > 2/ extract all the new files as *.dpkg-new > 3/ for all the unpacked files: fsync(foo.dpkg-new) followed by >ren

Re: Bug#605009: serious performance regression with ext4

2010-11-26 Thread Raphael Hertzog
Hi, adding debian-devel, debian-boot, debian-kernel and Theodore Y. Ts'o in CC because I'm fed up with this problem. Sorry for the massive crosspost, you might want to follow up only on -devel and on the bug report. Some clones/reassign should probably result from this discussion anyway. On Fri,