Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-04-01 Thread Holger Hoffstätte
On 04/01/16 03:01, Dave Chinner wrote: > Can you go back to your original kernel, and lower nr_requests to 8? Sure, did that and as expected it didn't help much. Under prolonged stress it was actually even a bit worse than writeback throttling. IMHO that's not really surprising either, since small

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-04-01 Thread Jens Axboe
On 04/01/2016 12:27 AM, Dave Chinner wrote: On Thu, Mar 31, 2016 at 09:25:33PM -0600, Jens Axboe wrote: On 03/31/2016 06:46 PM, Dave Chinner wrote: virtio in guest, XFS direct IO -> no-op -> scsi in host. That has write back caching enabled on the guest, correct? No. It uses virtio,cache=no

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-04-01 Thread Jens Axboe
On 04/01/2016 12:16 AM, Dave Chinner wrote: On Thu, Mar 31, 2016 at 09:39:25PM -0600, Jens Axboe wrote: On 03/31/2016 09:29 PM, Jens Axboe wrote: I can't seem to reproduce this at all. On an nvme device, I get a fairly steady 60K/sec file creation rate, and we're nowhere near being IO bound. So

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 09:25:33PM -0600, Jens Axboe wrote: > On 03/31/2016 06:46 PM, Dave Chinner wrote: > >>>virtio in guest, XFS direct IO -> no-op -> scsi in host. > >> > >>That has write back caching enabled on the guest, correct? > > > >No. It uses virtio,cache=none (that's the "XFS Direct IO

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 09:39:25PM -0600, Jens Axboe wrote: > On 03/31/2016 09:29 PM, Jens Axboe wrote: > >>>I can't seem to reproduce this at all. On an nvme device, I get a > >>>fairly steady 60K/sec file creation rate, and we're nowhere near > >>>being IO bound. So the throttling has no effect a

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 09:29:30PM -0600, Jens Axboe wrote: > On 03/31/2016 06:56 PM, Dave Chinner wrote: > >I'm not changing the host kernels - it's a production machine and so > >it runs long uptime testing of stable kernels. (e.g. catch slow > >memory leaks, etc). So if you've disabled throttli

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 09:29 PM, Jens Axboe wrote: I can't seem to reproduce this at all. On an nvme device, I get a fairly steady 60K/sec file creation rate, and we're nowhere near being IO bound. So the throttling has no effect at all. That's too slow to show the stalls - your likely concurrency bound

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 09:29 PM, Jens Axboe wrote: I'm not changing the host kernels - it's a production machine and so it runs long uptime testing of stable kernels. (e.g. catch slow memory leaks, etc). So if you've disabled throttling in the guest, I can't test the throttling changes. Right, that'd d

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 06:56 PM, Dave Chinner wrote: On Thu, Mar 31, 2016 at 10:21:04AM -0600, Jens Axboe wrote: On 03/31/2016 08:29 AM, Jens Axboe wrote: What I see in these performance dips is the XFS transaction subsystem stalling *completely* - instead of running at a steady state of around 350,000

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 06:46 PM, Dave Chinner wrote: On Thu, Mar 31, 2016 at 08:29:35AM -0600, Jens Axboe wrote: On 03/31/2016 02:24 AM, Dave Chinner wrote: On Wed, Mar 30, 2016 at 09:07:48AM -0600, Jens Axboe wrote: Hi, This patchset isn't as much a final solution, as it's demonstration of what I bel

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 10:09:56PM +, Holger Hoffstätte wrote: > > Hi, > > Jens mentioned on Twitter I should post my experience here as well, > so here we go. > > I've backported this series (incl. updates) to stable-4.4.x - not too > difficult, minus the NVM part which I don't need anyway

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 10:21:04AM -0600, Jens Axboe wrote: > On 03/31/2016 08:29 AM, Jens Axboe wrote: > >>What I see in these performance dips is the XFS transaction > >>subsystem stalling *completely* - instead of running at a steady > >>state of around 350,000 transactions/s, there are *zero* >

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 08:29:35AM -0600, Jens Axboe wrote: > On 03/31/2016 02:24 AM, Dave Chinner wrote: > >On Wed, Mar 30, 2016 at 09:07:48AM -0600, Jens Axboe wrote: > >>Hi, > >> > >>This patchset isn't as much a final solution, as it's demonstration > >>of what I believe is a huge issue. Since

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Holger Hoffstätte
Hi, Jens mentioned on Twitter I should post my experience here as well, so here we go. I've backported this series (incl. updates) to stable-4.4.x - not too difficult, minus the NVM part which I don't need anyway - and have been running it for the past few days without any problem whatsoever, wi

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 08:29 AM, Jens Axboe wrote: What I see in these performance dips is the XFS transaction subsystem stalling *completely* - instead of running at a steady state of around 350,000 transactions/s, there are *zero* transactions running for periods of up to ten seconds. This co-incides w

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 02:24 AM, Dave Chinner wrote: On Wed, Mar 30, 2016 at 09:07:48AM -0600, Jens Axboe wrote: Hi, This patchset isn't as much a final solution, as it's demonstration of what I believe is a huge issue. Since the dawn of time, our background buffered writeback has sucked. When we do bac

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Wed, Mar 30, 2016 at 09:07:48AM -0600, Jens Axboe wrote: > Hi, > > This patchset isn't as much a final solution, as it's demonstration > of what I believe is a huge issue. Since the dawn of time, our > background buffered writeback has sucked. When we do background > buffered writeback, it shou