Re: Pre-approval request for dpkg sync() changes for squeeze

On Sun, Nov 21, 2010 at 09:18:04AM +0100, Raphael Hertzog wrote: > On Sun, 21 Nov 2010, Ben Hutchings wrote: > > I'm coming to this late. It sounds like dpkg has changed its behaviour > > several times recently. Please can you summarise dpkg's current and > > proposed use of fsync() vs sync(), an

Re: Pre-approval request for dpkg sync() changes for squeeze

On Sun, 21 Nov 2010, Ben Hutchings wrote: > I'm coming to this late. It sounds like dpkg has changed its behaviour > several times recently. Please can you summarise dpkg's current and > proposed use of fsync() vs sync(), and the reasons for this. Jonathan made a good summary of the history. I s

Re: Pre-approval request for dpkg sync() changes for squeeze

Hi Ben, Ben Hutchings wrote: > On Mon, 2010-11-15 at 19:31 +0100, Philipp Kern wrote: >> and I don't suppose we could make that the default? Is there anything >> else the dpkg developers can try to be portable and still not be >> sacrificing performance? > > I'm coming to this late. It sounds l

Re: Pre-approval request for dpkg sync() changes for squeeze

On Mon, 2010-11-15 at 19:31 +0100, Philipp Kern wrote: > Dear kernel team, > > On Mon, Nov 15, 2010 at 09:58:47AM +0100, Sven Joachim wrote: > > > I'm sorry, I won't have the time to do new benchmarks on this. > > > > > > The only benchmarks we have have been made by Sven Joachim: > > > http://bu

Re: Pre-approval request for dpkg sync() changes for squeeze

Hi! On Mon, 2010-11-15 at 19:31:00 +0100, Philipp Kern wrote: > On Mon, Nov 15, 2010 at 09:58:47AM +0100, Sven Joachim wrote: > > All this is with a standard squeeze kernel on an otherwise idle system. > > It should be noted that with lots of other disk activity such as writing > > to USB disks, t

Re: Pre-approval request for dpkg sync() changes for squeeze

Hi! On Mon, 2010-11-15 at 08:21:05 +0100, Raphael Hertzog wrote: > On Wed, 10 Nov 2010, Philipp Kern wrote: > > On Tue, Nov 09, 2010 at 07:50:23PM +0100, Philipp Kern wrote: > > > On Fri, Oct 22, 2010 at 11:35:54AM +0200, Guillem Jover wrote: > > > > 1) Switch back from sync() to fsync() before

Re: Pre-approval request for dpkg sync() changes for squeeze

Dear kernel team, On Mon, Nov 15, 2010 at 09:58:47AM +0100, Sven Joachim wrote: > > I'm sorry, I won't have the time to do new benchmarks on this. > > > > The only benchmarks we have have been made by Sven Joachim: > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=578635#20 > > (asyncsync is t

Re: Pre-approval request for dpkg sync() changes for squeeze

On 2010-11-15 08:21 +0100, Raphael Hertzog wrote: > On Wed, 10 Nov 2010, Philipp Kern wrote: >> On Tue, Nov 09, 2010 at 07:50:23PM +0100, Philipp Kern wrote: >> > On Fri, Oct 22, 2010 at 11:35:54AM +0200, Guillem Jover wrote: >> > > 1) Switch back from sync() to fsync() before rename() (while ke

Re: Pre-approval request for dpkg sync() changes for squeeze

Hi, On Wed, 10 Nov 2010, Philipp Kern wrote: > On Tue, Nov 09, 2010 at 07:50:23PM +0100, Philipp Kern wrote: > > On Fri, Oct 22, 2010 at 11:35:54AM +0200, Guillem Jover wrote: > > > 1) Switch back from sync() to fsync() before rename() (while keeping > > > the sync() code around for the ben

Re: Pre-approval request for dpkg sync() changes for squeeze

On Tue, Nov 09, 2010 at 07:50:23PM +0100, Philipp Kern wrote: > On Fri, Oct 22, 2010 at 11:35:54AM +0200, Guillem Jover wrote: > > 1) Switch back from sync() to fsync() before rename() (while keeping > > the sync() code around for the benefit of other distributions > > that might not wa

Re: Pre-approval request for dpkg sync() changes for squeeze

Hi, On Fri, Oct 22, 2010 at 11:35:54AM +0200, Guillem Jover wrote: > 1) Switch back from sync() to fsync() before rename() (while keeping > the sync() code around for the benefit of other distributions > that might not want to switch just yet). So to avoid unrelated > I/O when the

Re: Pre-approval request for dpkg sync() changes for squeeze

On 2010-11-08 16:40 +0100, Phillip Susi wrote: > On 11/6/2010 5:41 AM, Sven Joachim wrote: >> For ext4, mounting with the nodelalloc option helps a lot, although this >> option allegedly slows down ext4 in the general case. > > How does that help? Doesn't it just disable the delayed allocator, >

Re: Pre-approval request for dpkg sync() changes for squeeze

On 11/6/2010 5:41 AM, Sven Joachim wrote: > For ext4, mounting with the nodelalloc option helps a lot, although this > option allegedly slows down ext4 in the general case. How does that help? Doesn't it just disable the delayed allocator, forcing the blocks to be allocated when they hit the cach

Re: Pre-approval request for dpkg sync() changes for squeeze

Guillem Jover wrote: > Ah, sorry for the noise. I can see why one would be annoyed with Ted's proposed solutions (e.g., don't crash), yes. Jonathan -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of "unsubscr

Re: Pre-approval request for dpkg sync() changes for squeeze

On Sat, 2010-11-06 at 16:33:14 -0500, Jonathan Nieder wrote: > Guillem Jover wrote: > > The worst that could happen on other file systems w/o the sync()/fsync() > > before rename()s for extracted files was that the dpkg database might > > get slightly out of sync relative to what was installed on d

Re: Pre-approval request for dpkg sync() changes for squeeze

Guillem Jover wrote: > The worst that could happen on other file systems w/o the sync()/fsync() > before rename()s for extracted files was that the dpkg database might > get slightly out of sync relative to what was installed on disk, but > that's at most confusing, nothing compared to getting zer

Re: Pre-approval request for dpkg sync() changes for squeeze

Hi! On Sat, 2010-11-06 at 10:20:29 +0100, Raphael Hertzog wrote: > On Sat, 06 Nov 2010, Guillem Jover wrote: > > Finally, while accepting patch 2) alone might make sense, accepting 1) > > alone would not. > > BTW, I think we should go with patch 2 alone currently (i.e. just add > --force-unsafe-i

Re: Pre-approval request for dpkg sync() changes for squeeze

On Sat, 2010-11-06 at 10:41:46 +0100, Sven Joachim wrote: > On 2010-11-06 08:46 +0100, Guillem Jover wrote: > > On Fri, 2010-11-05 at 23:18:47 +0100, Philipp Kern wrote: > > > On Fri, Oct 22, 2010 at 11:35:54AM +0200, Guillem Jover wrote: > > > What are the implications for ext4 and btrfs? > > > >

Re: Pre-approval request for dpkg sync() changes for squeeze

On 2010-11-06 08:46 +0100, Guillem Jover wrote: > On Fri, 2010-11-05 at 23:18:47 +0100, Philipp Kern wrote: >> On Fri, Oct 22, 2010 at 11:35:54AM +0200, Guillem Jover wrote: > >> What are the implications for ext4 and btrfs? > > They are going to be slower with this change. Are you sure this is t

Re: Pre-approval request for dpkg sync() changes for squeeze

Hi, On Sat, 06 Nov 2010, Guillem Jover wrote: > Finally, while accepting patch 2) alone might make sense, accepting 1) > alone would not. BTW, I think we should go with patch 2 alone currently (i.e. just add --force-unsafe-io). Continuing to use sync() instead of fsync() is the best compromise w

Re: Pre-approval request for dpkg sync() changes for squeeze

On Fri, 2010-11-05 at 23:18:47 +0100, Philipp Kern wrote: > On Fri, Oct 22, 2010 at 11:35:54AM +0200, Guillem Jover wrote: > > Those are related to the fsync()/sync() changes in dpkg from some time > > ago, the patches would: > > > > 1) Switch back from sync() to fsync() before rename() (while k

Re: Pre-approval request for dpkg sync() changes for squeeze

Hi, On Fri, Oct 22, 2010 at 11:35:54AM +0200, Guillem Jover wrote: > I'd like to apply two patches for the next dpkg 1.15.8.6 upload. And > would like to run them through you to avoid having to possibly revert > them from dpkg's sid git branch. > > Those are related to the fsync()/sync() changes

Re: Pre-approval request for dpkg sync() changes for squeeze

On Tue, 26 Oct 2010, Andreas Barth wrote: > * Phillip Susi (ps...@cfl.rr.com) [101025 17:00]: > > On 10/24/2010 1:20 PM, Andreas Barth wrote: > > > I quite often run dpkg installs / upgrades in pbuilder ramdisks. If > > > the sync could be reduced to only that ramdisk, everything would be > > > fin

Re: Pre-approval request for dpkg sync() changes for squeeze

* Phillip Susi (ps...@cfl.rr.com) [101025 17:00]: > On 10/24/2010 1:20 PM, Andreas Barth wrote: > > I quite often run dpkg installs / upgrades in pbuilder ramdisks. If > > the sync could be reduced to only that ramdisk, everything would be > > fine. > > That's exactly the environment where you wou

Re: Pre-approval request for dpkg sync() changes for squeeze

On 10/25/2010 4:28 AM, Goswin von Brederlow wrote: > Except that the fsync() patches from RedHat were never added to the AIO > code in the kernel. The portable way for this is to mmap() the files and > msync() them with MS_ASYNC. But that makes catching write errors a > nightmare. Blast. Then I g

Re: Pre-approval request for dpkg sync() changes for squeeze

On 10/24/2010 1:20 PM, Andreas Barth wrote: > I quite often run dpkg installs / upgrades in pbuilder ramdisks. If > the sync could be reduced to only that ramdisk, everything would be > fine. That's exactly the environment where you would disable syncing entirely since you don't care if the pbuild

Re: Pre-approval request for dpkg sync() changes for squeeze

Phillip Susi writes: > On 10/24/2010 07:16 AM, Goswin von Brederlow wrote: >> Or 5 minutes because sync() also needs to write out the 16GB cache data >> to my usb 1.0 drive that is not involved with dpkg at all. > > True, but that seems a bit of a contrived corner case. Most of the > time when p

Re: Pre-approval request for dpkg sync() changes for squeeze

* Phillip Susi (ps...@cfl.rr.com) [101024 19:15]: > True, but that seems a bit of a contrived corner case. Most of the time > when people are upgrading, I'd wager that they don't have many gb of > dirty cache buffers already sitting in the cache. Though that does make > me wonder why there

Re: Pre-approval request for dpkg sync() changes for squeeze

On 10/24/2010 07:16 AM, Goswin von Brederlow wrote: Or 5 minutes because sync() also needs to write out the 16GB cache data to my usb 1.0 drive that is not involved with dpkg at all. True, but that seems a bit of a contrived corner case. Most of the time when people are upgrading, I'd wager t

Re: Pre-approval request for dpkg sync() changes for squeeze

Phillip Susi writes: > On 10/22/2010 5:35 AM, Guillem Jover wrote: >> 1) Switch back from sync() to fsync() before rename() (while keeping > > Don't you WANT to use sync? If you fsync every file that is going to be > rather slow since it forces a disk write for every file, rather than > allowi

Re: Pre-approval request for dpkg sync() changes for squeeze

On 10/22/2010 5:35 AM, Guillem Jover wrote: > 1) Switch back from sync() to fsync() before rename() (while keeping Don't you WANT to use sync? If you fsync every file that is going to be rather slow since it forces a disk write for every file, rather than allowing writes between each file to be

Re: Pre-approval request for dpkg sync() changes for squeeze

Guillem Jover wrote: > 1) Switch back from sync() to fsync() before rename() (while keeping > the sync() code around for the benefit of other distributions > that might not want to switch just yet). So to avoid unrelated > I/O when there's background work being done for example. T