summary of current pending changes in aio-next.git

2019-08-08 Thread Benjamin LaHaise
next.git master for you to fetch changes up to 6878ea72a5d1aa6caae86449975a50b7fe9abed5: aio: be defensive to ensure request batching is non-zero instead of BUG_ON() (2013-07-31 10:34:18 -0400) -------- Benjamin LaHaise (4):

Re: [PATCH] aio: Support read/write with non-iter file-ops

2019-07-18 Thread Benjamin LaHaise
On Thu, Jul 18, 2019 at 04:43:52PM -0700, Bjorn Andersson wrote: > On Thu 18 Jul 16:17 PDT 2019, Al Viro wrote: > > > On Thu, Jul 18, 2019 at 04:10:54PM -0700, Bjorn Andersson wrote: > > > Implement a wrapper for aio_read()/write() to allow async IO on files > > > not implementing the iter version

Re: aio poll, io_pgetevents and a new in-kernel poll API V4

2018-01-25 Thread Benjamin LaHaise
> added, which atomically saves and restores the signal mask over the > io_pgetevents system call. It it the logical equivalent to pselect and > ppoll for io_pgetevents. That looks useful. I'll have to look at this in detail. -ben commit a299c474b19107122eae846b53f7

Re: [PATCH 2/6] move _body_io_syscall to the generic syscall.h

2018-01-05 Thread Benjamin LaHaise
On Fri, Jan 05, 2018 at 11:25:17AM -0500, Jeff Moyer wrote: > Christoph Hellwig writes: > > > This way it can be used for the fallback 6-argument version on > > all architectures. > > > > Signed-off-by: Christoph Hellwig > > This is a strange way to do things. However, I was never really sold

Re: [PATCH 1/1] aio: make sure the input "timeout" value is valid

2017-12-13 Thread Benjamin LaHaise
ut what an awful interface we have here! > The user must not only fetch it, they must validate it separately? > And if they forget, then userspace is provoking undefined behaviour? Ugh. > Why not this: That looks far better! Acked-by: Benjamin LaHaise -ben > di

Re: [PATCH 0/5] blkcg: Limit maximum number of aio requests available for cgroup

2017-12-06 Thread Benjamin LaHaise
On Wed, Dec 06, 2017 at 09:19:09PM +0300, Kirill Tkhai wrote: > On 06.12.2017 20:44, Benjamin LaHaise wrote: > > On Wed, Dec 06, 2017 at 06:32:56PM +0100, Oleg Nesterov wrote: > >>>> This memory lives in page-cache/lru, it is visible for shrinker which > >>>&g

Re: [PATCH 0/5] blkcg: Limit maximum number of aio requests available for cgroup

2017-12-06 Thread Benjamin LaHaise
On Wed, Dec 06, 2017 at 06:32:56PM +0100, Oleg Nesterov wrote: > > > This memory lives in page-cache/lru, it is visible for shrinker which > > > will unmap these pages for no reason on memory shortage. IOW, aio fools > > > the kernel, this memory looks reclaimable but it is not. And we only do > >

Re: [PATCH 0/5] blkcg: Limit maximum number of aio requests available for cgroup

2017-12-05 Thread Benjamin LaHaise
On Tue, Dec 05, 2017 at 04:19:56PM +0100, Oleg Nesterov wrote: > On 12/05, Kirill Tkhai wrote: > > > > Currently, aio_nr and aio_max_nr are global. > > Yeah, I too tried to complain 2 years ago... > > > In case of containers this > > means that a single container may occupy all aio requests, whic

Re: [PATCH 0/5] blkcg: Limit maximum number of aio requests available for cgroup

2017-12-04 Thread Benjamin LaHaise
Hi Kirill, On Mon, Dec 04, 2017 at 07:12:51PM +0300, Kirill Tkhai wrote: > Hi, > > this patch set introduces accounting aio_nr and aio_max_nr per blkio cgroup. > It may be used to limit number of aio requests, which are available for > a cgroup, and could be useful for containers. > > The accoun

[GIT PULL] aio: improve aio-nr counting

2017-09-07 Thread Benjamin LaHaise
Hello Linus, Please consider pulling the following change for 4.14 to improve aio-nr counting on large SMP systems which has been in linux-next for quite some time. Cheers, -ben The following changes since commit 569dbb88e80deb68974ef6fdd6a13edb9d686261: Linux 4.13 (2017-09-0

Re: [PATCH] aio: Add command to wait completion of all requests

2017-06-14 Thread Benjamin LaHaise
On Wed, Jun 14, 2017 at 12:11:38PM +0300, Kirill Tkhai wrote: > On 14.06.2017 02:26, Benjamin LaHaise wrote: ... > > Nope. An aio may not complete in a timely fashion, in which case your > > wait for all aios to complete will simply wait forever. I take it this is > > not

Re: [PATCH] aio: Add command to wait completion of all requests

2017-06-13 Thread Benjamin LaHaise
On Tue, Jun 13, 2017 at 07:17:43PM +0300, Kirill Tkhai wrote: > On 13.06.2017 18:26, Benjamin LaHaise wrote: > > On Tue, Jun 13, 2017 at 06:11:03PM +0300, Kirill Tkhai wrote: > > ... > >> The functionality, I did, grew from real need and experience. We try to > >>

Re: [PATCH] aio: Add command to wait completion of all requests

2017-06-13 Thread Benjamin LaHaise
On Tue, Jun 13, 2017 at 06:11:03PM +0300, Kirill Tkhai wrote: ... > The functionality, I did, grew from real need and experience. We try to > avoid kernel modification, where it's possible, but the in-flight aio > requests is not a case suitable for that. What you've done only works for *your* use

Re: [PATCH] aio: Add command to wait completion of all requests

2017-06-13 Thread Benjamin LaHaise
On Fri, Jun 09, 2017 at 12:49:34PM +0300, Kirill Tkhai wrote: > During implementation aio support for Checkpoint-Restore > in Userspace project (CRIU), we found, that current > kernel functionality does not allow to wait for > completion of all submitted requests. Checkpoint software > can't determ

Re: aio: questions with ioctx_alloc() and large num_possible_cpus()

2016-10-05 Thread Benjamin LaHaise
On Wed, Oct 05, 2016 at 02:58:12PM -0300, Mauricio Faria de Oliveira wrote: > Hi Benjamin, > > On 10/05/2016 02:41 PM, Benjamin LaHaise wrote: > >I'd suggest increasing the default limit by changing how it is calculated. > >The current number came about 13 years ago w

Re: aio: questions with ioctx_alloc() and large num_possible_cpus()

2016-10-05 Thread Benjamin LaHaise
On Tue, Oct 04, 2016 at 07:55:12PM -0300, Mauricio Faria de Oliveira wrote: > Hi Benjamin, Kent, and others, > > Would you please comment / answer about this possible problem? > Any feedback is appreciated. I'd suggest increasing the default limit by changing how it is calculated. The current n

Re: linux-next: build failure after merge of the aio tree

2016-03-19 Thread Benjamin LaHaise
On Wed, Mar 16, 2016 at 02:59:38PM +0100, Arnd Bergmann wrote: > On Wednesday 16 March 2016 13:12:36 Andy Shevchenko wrote: > > > > > I've also sent a patch that fixes the link error on ARM and that should > > > work on all other architectures too. > > > > In case of avr32 signalfd_read() fails.

Re: linux-next: build failure after merge of the aio tree

2016-03-15 Thread Benjamin LaHaise
On Tue, Mar 15, 2016 at 04:19:02PM +, Sudip Mukherjee wrote: > On Tue, Mar 15, 2016 at 05:46:34PM +1100, Stephen Rothwell wrote: > > Hi Benjamin, > > > > After merging the aio tree, today's linux-next build (powerpc > > ppc44x_defconfig) failed like this: > > > > fs/built-in.o: In function `a

Re: linux-next: manual merge of the aio tree with the vfs tree

2016-03-15 Thread Benjamin LaHaise
On Tue, Mar 15, 2016 at 05:35:33AM +, Al Viro wrote: > On Mon, Mar 14, 2016 at 11:24:38AM -0400, Benjamin LaHaise wrote: > > On Mon, Mar 14, 2016 at 08:35:23AM +0100, Christoph Hellwig wrote: > > > The aio changes have either been reviewed negatively or not at all. That &g

Re: linux-next: manual merge of the aio tree with the vfs tree

2016-03-15 Thread Benjamin LaHaise
On Tue, Mar 15, 2016 at 05:19:39AM +, Al Viro wrote: > On Tue, Mar 15, 2016 at 05:07:12AM +, Al Viro wrote: > > > There *is* a reason for code review. Or, at least, asking somebody familiar > > with the code you are working with whether some assumption you are making > > is true or false.

aio openat Re: [PATCH 07/13] aio: enabled thread based async fsync

2016-03-14 Thread Benjamin LaHaise
ence of where you are now." fs/aio.c | 122 +- fs/internal.h |1 fs/namei.c| 16 -- fs/open.c |2 include/linux/namei.h |1 5 files changed, 117 insertions(+), 25 deletions(-) commit 5d3d80fc

Re: linux-next: build failure after merge of the aio tree

2016-03-14 Thread Benjamin LaHaise
On Mon, Mar 14, 2016 at 03:49:15PM +1100, Stephen Rothwell wrote: > Hi Ben, ... > OK, so at this point (just to get rid of the build failure I have done this: ... > Well, you need to negotiate that with the affected architectures. I put in a patch to use get_user() for now since the 32 bit archit

Re: [PATCH] x86_32: add support for 64 bit __get_user() v3

2016-03-14 Thread Benjamin LaHaise
Any more comments/feedback on this? Can this be merged for 4.6? -ben On Wed, Mar 09, 2016 at 03:05:56PM -0500, Benjamin LaHaise wrote: > The existing __get_user() implementation does not support fetching > 64 bit values on 32 bit x86. Implement this in a way that do

Re: linux-next: manual merge of the aio tree with the vfs tree

2016-03-14 Thread Benjamin LaHaise
On Mon, Mar 14, 2016 at 08:35:23AM +0100, Christoph Hellwig wrote: > The aio changes have either been reviewed negatively or not at all. That > tree should be dropped. That isn't solely your decision. If you have comments, please provide constructive feedback, as there are users and use-cases

[PATCH] x86_32: add support for 64 bit __get_user() v3

2016-03-09 Thread Benjamin LaHaise
suggested by H. Peter Anvin, which cleans the code up nicely, and fix things to work on x86_64 as well. v3: fix undefined behaviour on 32 bit x86 caused by multiple expansion of ptr in the 64 bit case. Signed-off-by: Benjamin LaHaise diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include

[PATCH] x86_32: add support for 64 bit __get_user() v2

2016-03-09 Thread Benjamin LaHaise
suggested by H. Peter Anvin, which cleans the code up nicely, and fix things to work on x86_64 as well. Tested on both 32 bit and 64 bit x86. Signed-off-by: Benjamin LaHaise diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index a4a30e4..1284da2 100644 --- a/arch/x86

Re: [PATCH] x86_32: add support for 64 bit __get_user()

2016-03-09 Thread Benjamin LaHaise
On Wed, Mar 09, 2016 at 10:22:50AM -0800, H. Peter Anvin wrote: > On 03/09/2016 09:50 AM, Benjamin LaHaise wrote: > > On Wed, Mar 09, 2016 at 09:36:30AM -0800, H. Peter Anvin wrote: > >> On March 9, 2016 9:22:25 AM PST, Benjamin LaHaise wrote: > >>> The existing __

Re: [PATCH] x86_32: add support for 64 bit __get_user()

2016-03-09 Thread Benjamin LaHaise
On Wed, Mar 09, 2016 at 09:36:30AM -0800, H. Peter Anvin wrote: > On March 9, 2016 9:22:25 AM PST, Benjamin LaHaise wrote: > >The existing __get_user() implementation does not support fetching > >64 bit values on 32 bit x86. Implement this in a way that does not > >g

[PATCH] x86_32: add support for 64 bit __get_user()

2016-03-09 Thread Benjamin LaHaise
The existing __get_user() implementation does not support fetching 64 bit values on 32 bit x86. Implement this in a way that does not generate any incorrect warnings as cautioned by Russell King. Test code available at http://www.kvack.org/~bcrl/x86_32-get_user.tar . Signed-off-by: Benjamin

Re: linux-next: Tree for Feb 24

2016-02-24 Thread Benjamin LaHaise
On Wed, Feb 24, 2016 at 07:32:17AM +0100, Sedat Dilek wrote: > On Wed, Feb 24, 2016 at 6:34 AM, Stephen Rothwell > wrote: > > Hi all, > > > > Changes since 20160223: > > > ... > > The aio tree still had a build failure so I used the version from > > next-20160111. > > > > Might be good to poke t

Re: linux-next: build failure after merge of the aio tree

2016-02-04 Thread Benjamin LaHaise
On Thu, Feb 04, 2016 at 04:17:42PM +, Russell King - ARM Linux wrote: > __gu_val will be 32-bit, even when you're wanting a 64-bit quantity. > That's where the fun and games start... Okay, I figured out how to do it: instead of using a 64 bit unsigned long long for __gu_val, use an array of 2

Re: linux-next: build failure after merge of the aio tree

2016-02-04 Thread Benjamin LaHaise
On Thu, Feb 04, 2016 at 04:17:42PM +, Russell King - ARM Linux wrote: > That's the easy bit! > > The problem you're going to run into is here: > > #define __get_user_nocheck(x, ptr, size)\ > ({

Re: linux-next: build failure after merge of the aio tree

2016-02-04 Thread Benjamin LaHaise
On Thu, Feb 04, 2016 at 04:17:42PM +, Russell King - ARM Linux wrote: > On Thu, Feb 04, 2016 at 11:01:01AM -0500, Benjamin LaHaise wrote: > > On Thu, Feb 04, 2016 at 02:39:07PM +, Russell King - ARM Linux wrote: > > > However, this one should warn: > > > &g

Re: linux-next: build failure after merge of the aio tree

2016-02-04 Thread Benjamin LaHaise
On Thu, Feb 04, 2016 at 02:39:07PM +, Russell King - ARM Linux wrote: > However, this one should warn: > > int test_wrong(char **v, const char **p) > { return __get_user(*v, p); } > > Good luck (I think you'll need lots of it to get a working solution)! :) This works with your test cases on

Re: linux-next: build failure after merge of the aio tree

2016-02-04 Thread Benjamin LaHaise
On Thu, Feb 04, 2016 at 02:12:53PM +, Russell King - ARM Linux wrote: > Hence, __get_user() on x86-32 with a 64-bit quantity results in > __get_user_bad() being called, which is an undefined function. > Only if you build with x86-64 support enabled (iow, CONFIG_X86_32 not > defined) then you ge

Re: linux-next: build failure after merge of the aio tree

2016-02-04 Thread Benjamin LaHaise
On Thu, Feb 04, 2016 at 01:50:56PM +, Russell King - ARM Linux wrote: > > I am still convinced that this is an architecture issue. Given that 64 bit > > values work in the *get_user implementations on other architectures, I see > > no reason there should need to be a workaround for this in c

Re: linux-next: build failure after merge of the aio tree

2016-02-04 Thread Benjamin LaHaise
On Thu, Feb 04, 2016 at 01:19:59PM +1100, Stephen Rothwell wrote: > Hi Benjamin, > > On Fri, 29 Jan 2016 13:03:39 +0100 Geert Uytterhoeven > wrote: > > > > On Fri, Jan 29, 2016 at 12:30 PM, Russell King - ARM Linux > > wrote: > > >> Background: new aio code is adding __get_user() calls referenc

Re: [mm -next] mapping->tree_lock inconsistent lock state

2016-02-03 Thread Benjamin LaHaise
On Wed, Feb 03, 2016 at 01:45:09PM -0500, Johannes Weiner wrote: > On Thu, Feb 04, 2016 at 12:36:33AM +0900, Sergey Senozhatsky wrote: > > Hello, > > > > next-20160203 > > > > [ 3587.997451] = > > [ 3587.997453] [ INFO: inconsistent lock state ] > > [ 3587.997456]

Re: [PATCH 07/13] aio: enabled thread based async fsync

2016-01-22 Thread Benjamin LaHaise
On Sat, Jan 23, 2016 at 03:24:49PM +1100, Dave Chinner wrote: > On Wed, Jan 20, 2016 at 04:56:30PM -0500, Benjamin LaHaise wrote: > > On Thu, Jan 21, 2016 at 08:45:46AM +1100, Dave Chinner wrote: > > > Filesystems *must take locks* in the IO path. We have to serialise > >

Re: [PATCH -next] aio: Fix compile error due to unexpected use of cmpxchg()

2016-01-22 Thread Benjamin LaHaise
; fs/aio.c:1499:1: error: cast specifies function type > Fixes: 6a81013efc40 ("aio: add support for IOCB_CMD_POLL via aio thread > helper") Ah, I was wondering if there was an easy way to fix this bug. Applied -- thanks! -ben > Cc: Benjamin LaHaise > Signed-o

Re: [PATCH 07/13] aio: enabled thread based async fsync

2016-01-20 Thread Benjamin LaHaise
On Thu, Jan 21, 2016 at 08:45:46AM +1100, Dave Chinner wrote: > Filesystems *must take locks* in the IO path. We have to serialise > against truncate and other operations at some point in the IO path > (e.g. block mapping vs concurrent allocation and/or removal), and > that can only be done sanely

Re: [PATCH 07/13] aio: enabled thread based async fsync

2016-01-20 Thread Benjamin LaHaise
On Wed, Jan 20, 2016 at 12:29:32PM -0800, Linus Torvalds wrote: > On Wed, Jan 20, 2016 at 11:59 AM, Dave Chinner wrote: > >> > >> Are there other users outside of Solace? It would be good to get comments.. > > > > I know of quite a few storage/db products that use AIO. The most > > recent high pro

Re: int overflow in io_getevents

2016-01-07 Thread Benjamin LaHaise
and that negative or other invalid values fail with EINVAL (download from http://www.kvack.org/~bcrl/aio-io_getevents-timespec.c ). -ben -- "Thought is the essence of where you are now." commit 49b78150bc5762c58cfb8b19a859c354cf1a71ac Author: Benjamin LaHaise Date: Th

Re: int overflow in io_getevents

2016-01-07 Thread Benjamin LaHaise
ess my concerns. Updated commit below. -ben -- "Thought is the essence of where you are now." commit 3a55c535cf3836257434518bd6bc11464c493492 Author: Benjamin LaHaise Date: Thu Jan 7 10:25:44 2016 -0500 aio: handle integer overflow in io_getevents() timespec usage

Re: int overflow in io_getevents

2016-01-06 Thread Benjamin LaHaise
patch below if there are no objections. -ben -- "Thought is the essence of where you are now." commit 4304367826d0df42086ef24428c6c277acd822a9 Author: Benjamin LaHaise Date: Wed Jan 6 12:46:12 2016 -0500 aio: handle integer overflow in io_getevents() timespec usage Dmitry Vyu

Re: use-after-free in sock_wake_async

2015-11-24 Thread Benjamin LaHaise
On Tue, Nov 24, 2015 at 04:30:01PM -0500, Jason Baron wrote: > So looking at this trace I think its the other->sk_socket that gets > freed and then we call sk_wake_async() on it. > > We could I think grab the socket reference there with unix_state_lock(), > since that is held by unix_release_sock(

Re: [PATCH] aio: Add few code style fixes

2015-11-11 Thread Benjamin LaHaise
On Wed, Nov 11, 2015 at 11:28:08PM +0800, 刘长冬 wrote: > >From 1609d68dee344925d182631922cd98790109588b Mon Sep 17 00:00:00 2001 > From: Liu Changdong > Date: Tue, 10 Nov 2015 00:04:18 +0800 > Subject: [PATCH] aio: Add few code style fixes > > Add a blank line after declarations > > Signed-off-by:

Re: [PATCH v2] mm-move-mremap-from-file_operations-to-vm_operations_struct-fix

2015-07-21 Thread Benjamin LaHaise
On Tue, Jul 21, 2015 at 05:29:03PM +0200, Oleg Nesterov wrote: > Andrew, > > this replaces > > mm-move-mremap-from-file_operations-to-vm_operations_struct-fix.patch > > in -mm tree nacked by Benjamin. > > fs/aio.c can't be compiled if CONFIG_MMU=n, filemap_page_mkwrite() is > not defined

Re: [PATCH] mm-move-mremap-from-file_operations-to-vm_operations_struct-fix

2015-07-20 Thread Benjamin LaHaise
On Mon, Jul 20, 2015 at 09:24:40PM +0200, Oleg Nesterov wrote: > On 07/20, Oleg Nesterov wrote: > > > > But again, again, please ignore. This all is off-topic and my understanding > > is very limited. > > Yes, yes, but sorry for noise and let me repeat... > > This memory lives in page-cache/lru,

Re: [PATCH] mm-move-mremap-from-file_operations-to-vm_operations_struct-fix

2015-07-20 Thread Benjamin LaHaise
On Mon, Jul 20, 2015 at 07:33:11PM +0200, Oleg Nesterov wrote: > Hi Jeff, > > On 07/20, Jeff Moyer wrote: > > > > Hi, Oleg, > > > > Oleg Nesterov writes: > > > > > Shouldn't we account aio events/pages somehow, say per-user, or in > > > mm->pinned_vm ? > > > > Ages ago I wrote a patch to account

Re: [PATCH] mm-move-mremap-from-file_operations-to-vm_operations_struct-fix

2015-07-17 Thread Benjamin LaHaise
On Fri, Jul 17, 2015 at 07:27:26PM +0200, Oleg Nesterov wrote: > Benjamin, > > it seems that we do not understand each other, ... > > > > Either try to fix it correctly, > > And I think this fix is correct. In a sense that we only add > filemap_page_mkwrite() to make the linker happy, it can neve

Re: [PATCH] mm-move-mremap-from-file_operations-to-vm_operations_struct-fix

2015-07-17 Thread Benjamin LaHaise
On Fri, Jul 17, 2015 at 01:52:28AM +0200, Oleg Nesterov wrote: > On 07/16, Andrew Morton wrote: > > > > On Fri, 17 Jul 2015 01:14:05 +0200 Oleg Nesterov wrote: > > > > > fs/aio.c can't be compiled if CONFIG_MMU=n, filemap_page_mkwrite() > > > is not defined in this case. Add yet another "must not

Re: [PATCH v5 0/3] aio: ctx->dead cleanups

2015-06-22 Thread Benjamin LaHaise
Hi Oleg, On Fri, Jun 19, 2015 at 08:18:40PM +0200, Oleg Nesterov wrote: > > > As Jeff suggested, 1/3 was changed to simply remove the ->dead check. > > > I also updated the changelog. > > > > > > Jeff, I preserved your acks in 2-3. > > > > also remove ctx->dead setting in ioctx_alloc(). > > Remov

Re: [GIT PULL] aio: fix sleeping while TASK_INTERRUPTIBLE

2015-02-03 Thread Benjamin LaHaise
On Wed, Feb 04, 2015 at 09:23:12AM +1100, Dave Chinner wrote: > On Mon, Feb 02, 2015 at 10:45:42AM -0800, Linus Torvalds wrote: > > On Sun, Feb 1, 2015 at 9:54 PM, Dave Chinner wrote: > > > > > > Simple enough - the patch below removes the warning from generic/036 > > > for me. > > > > So because

Re: [GIT PULL] aio: fix sleeping while TASK_INTERRUPTIBLE

2015-02-01 Thread Benjamin LaHaise
On Sun, Feb 01, 2015 at 03:33:25PM -0800, Linus Torvalds wrote: > On Sun, Feb 1, 2015 at 2:14 PM, Benjamin LaHaise wrote: > > > > It's ugly, but it actually is revealing a bug. Spurious wake ups caused > > by the task already being added to ctx->wait when callin

Re: [GIT PULL] aio: fix sleeping while TASK_INTERRUPTIBLE

2015-02-01 Thread Benjamin LaHaise
On Sun, Feb 01, 2015 at 01:01:06PM -0800, Linus Torvalds wrote: > On Sun, Feb 1, 2015 at 6:40 AM, Benjamin LaHaise wrote: > > > > Chris Mason (1): > > fs/aio: fix sleeping while TASK_INTERRUPTIBLE > > Ugh. > > This patch is too ugly to live. As far as I ca

[GIT PULL] aio: fix sleeping while TASK_INTERRUPTIBLE

2015-02-01 Thread Benjamin LaHaise
Hello Linus, The following changes since commit 5f785de588735306ec4d7c875caf9d28481c8b21: aio: Skip timer for io_getevents if timeout=0 (2014-12-13 17:50:20 -0500) are available in the git repository at: git://git.kvack.org/~bcrl/aio-fixes.git master for you to fetch changes up to f84249f4

Re: [v3.19-rc1] read_events => aio_read_events => WARNING: "do not call blocking ops when !TASK_RUNNING"

2014-12-24 Thread Benjamin LaHaise
On Wed, Dec 24, 2014 at 02:50:02PM +0800, Fam Zheng wrote: > Could you trace down to the source code context of this io_getevents call in > mysqld? The change merely made io_getevents return a lot more quickly than > before, so it seems like that a polling loop is spinning too fast as a result. T

Re: [GIT PULL] aio: changes for 3.19

2014-12-14 Thread Benjamin LaHaise
On Mon, Dec 15, 2014 at 01:02:08AM +0200, Kirill A. Shutemov wrote: > But it seems the problem is bigger than what the patch fixes. To me we are > too permisive on what vma can be remapped. > > How can we know that it's okay to move vma around for random driver which > provide .mmap? Or I miss som

Re: [GIT PULL] aio: changes for 3.19

2014-12-14 Thread Benjamin LaHaise
d in commit e4a0d3e720e7e508749c1439b5ba3aff56c92976. Signed-off-by: Benjamin LaHaise Documentation/filesystems/vfs.txt |4 fs/aio.c |8 2 files changed, 12 insertions(+) diff --git a/Documentation/filesystems/vfs.txt b/Documentation/filesystems/vfs.t

Re: [GIT PULL] aio: changes for 3.19

2014-12-14 Thread Benjamin LaHaise
On Sun, Dec 14, 2014 at 01:47:32PM -0800, Linus Torvalds wrote: > On Sun, Dec 14, 2014 at 12:22 PM, Benjamin LaHaise wrote: > > > > Pavel Emelyanov (1): > > aio: Make it possible to remap aio ring > > So quite frankly, I think this should have had more acks

[GIT PULL] aio: changes for 3.19

2014-12-14 Thread Benjamin LaHaise
Hello Linus & everyone, The following changes since commit b2776bf7149bddd1f4161f14f79520f17fc1d71d: Linux 3.18 (2014-12-07 14:21:05 -0800) are available in the git repository at: git://git.kvack.org/~bcrl/aio-next.git master for you to fetch changes up to 5f785de588735306ec4d7c875caf9d284

[GIT PULL] aio: dirty page accounting fix

2014-11-24 Thread Benjamin LaHaise
Good morning Linus and all, The following changes since commit 6098b45b32e6baeacc04790773ced9340601d511: aio: block exit_aio() until all context requests are completed (2014-09-04 16:54:47 -0400) are available in the git repository at: git://git.kvack.org/~bcrl/aio-fixes.git master for yo

Re: [PATCH] aio: Skip timer for io_getevents if timeout=0

2014-11-06 Thread Benjamin LaHaise
Hi Fam, On Thu, Nov 06, 2014 at 08:44:36PM +0800, Fam Zheng wrote: > In this case, it is basically a polling. Let's not involve timer at all > because that would hurt performance for application event loops. > > In an arbitrary test I've done, io_getevents syscall elapsed time > reduces from 5000

Re: [PATCH V2] aio: fix uncorrent dirty pages accouting when truncating AIO ring buffer

2014-11-06 Thread Benjamin LaHaise
Hi Gu, On Thu, Nov 06, 2014 at 05:46:21PM +0800, Gu Zheng wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=86831 > > Markus reported that when shutting down mysqld (with AIO support, > on a ext3 formatted Harddrive) leads to a negative number of dirty pages > (underrun to the counter). The ne

Re: [PATCH] aio: fix uncorrent dirty pages accouting when truncating AIO ring buffer

2014-11-05 Thread Benjamin LaHaise
On Wed, Nov 05, 2014 at 05:53:11PM +0800, Gu Zheng wrote: > ping... I need someone a bit more familiar with this area of code to chime in on reviewing this. Andrew, can you provide any feedback on this fix? -ben > On 10/31/2014 06:07 PM, Gu Zheng wrote: > > > https://bugzilla.

Re: [PATCH] aio: Fix return code of io_submit() (RFC)

2014-10-03 Thread Benjamin LaHaise
On Fri, Oct 03, 2014 at 12:13:39PM -0600, Jens Axboe wrote: > On 2014-10-03 12:08, Kent Overstreet wrote: > >io_submit() could return -EAGAIN on memory allocation failure when it > >should > >really have been returning -ENOMEM. This could confuse applications (i.e. > >fio) > >since -EAGAIN means

Re: [PATCH] aio: Fix return code of io_submit() (RFC)

2014-10-03 Thread Benjamin LaHaise
omehow > come to rely on the current behaviour... > > Signed-off-by: Kent Overstreet > Cc: Benjamin LaHaise > Cc: Zach Brown > Cc: Jeff Moyer > Cc: Jens Axboe > Cc: Slava Pestov > --- > fs/aio.c | 26 +- > 1 file changed, 13 inser

Re: [RFC 1/2] aio: async readahead

2014-09-19 Thread Benjamin LaHaise
On Fri, Sep 19, 2014 at 04:26:12AM -0700, Christoph Hellwig wrote: > Requiring the block mappings to be entirely async is why we never went > for full buffered aio. What would seem more useful is to offload all > readahead to workqueues to make sure they never block the caller for > sys_readahead

[RFC 2/2] ext4: async readpage for indirect style inodes

2014-09-17 Thread Benjamin LaHaise
Hi all, And here is the version of readpage for ext3/ext4 that performs async metadata reads for old-style indirect block based ext3/ext4 filesystems. This version only includes the changes against ext4 -- the changes to ext3 are pretty much identical. This is only an RFC and has at least on

[RFC 1/2] aio: async readahead

2014-09-17 Thread Benjamin LaHaise
Hi Milosz et al, This code is probably relevant to the non-blocking read thread. A non-blocking read is pretty useless without some way to trigger and become aware of data being read into the page cache, and the attached patch is one way to do so. The changes below introduce an async readahea

Re: [PATCH 7/7] check for O_NONBLOCK in all read_iter instances

2014-09-17 Thread Benjamin LaHaise
On Wed, Sep 17, 2014 at 09:47:02AM -0400, Theodore Ts'o wrote: ... > % git version > git version 2.1.0 > > Perhaps you and other people are using your own scripts, and not using > git send-email? That would be because none of my systems have git 2.1.0 on them. Fedora (and EPEL) appear to still

Re: [PATCH 7/7] check for O_NONBLOCK in all read_iter instances

2014-09-17 Thread Benjamin LaHaise
On Wed, Sep 17, 2014 at 07:42:54AM +1000, Dave Chinner wrote: > [Please don't top post!] > > On Tue, Sep 16, 2014 at 03:45:52PM -0400, Milosz Tanski wrote: > > On Tue, Sep 16, 2014 at 3:27 PM, Jeff Moyer wrote: > > > Christoph Hellwig writes: > > > > > > Hrm, you're not Christoph... > > > > I am

Re: [PATCH] aio: block exit_aio() until all context requests are completed

2014-09-04 Thread Benjamin LaHaise
On Thu, Sep 04, 2014 at 08:47:19AM +0800, Gu Zheng wrote: > That's good, but I did not find it, have you sent it to the open list? Sorry, the aio list wasn't cc'd on that one -- it was on one of the other development lists. > IMO, there's no need to check the return value of kill_ioctx(), becaus

[GIT PULL] aio: 2 small fixes for 3.17

2014-09-04 Thread Benjamin LaHaise
The following changes since commit 7be141d0554921751db103b54e9f794956aa4f65: Merge branch 'x86-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip (2014-08-24 16:17:41 -0700) are available in the git repository at: git://git.kvack.org/~bcrl/aio-fixes.git master for

Re: [PATCH] aio: block exit_aio() until all context requests are completed

2014-09-03 Thread Benjamin LaHaise
Hi Gu, On Wed, Sep 03, 2014 at 05:45:44PM +0800, Gu Zheng wrote: > It seems that exit_aio() also needs to wait for all iocbs to complete (like > io_destroy), but we missed the wait step in current implemention, so fix > it in the same way as we did in io_destroy. I actually already have another v

Re: [PATCH v1 5/9] block: loop: convert to blk-mq

2014-08-27 Thread Benjamin LaHaise
On Wed, Aug 27, 2014 at 08:08:59PM +0400, Maxim Patlasov wrote: ... > 1) /dev/loop0 of 3.17.0-rc1 with Ming's patches applied -- 11K iops > 2) the same as above, but call loop_queue_work() directly from > loop_queue_rq() -- 270K iops > 3) /dev/nullb0 of 3.17.0-rc1 -- 380K iops > > Taking into acc

Re: Revert "aio: fix aio request leak when events are reaped by user space"

2014-08-25 Thread Benjamin LaHaise
On Mon, Aug 25, 2014 at 03:06:12PM +, Elliott, Robert (Server Storage) wrote: > Using this version of the patch, I ran into this crash after 36 > hours of scsi-mq testing over the weekend. ... > > local_irq_save(flags); > kcpu = this_cpu_ptr(ctx->cpu); > kcpu->reqs_ava

Re: Revert "aio: fix aio request leak when events are reaped by user space"

2014-08-24 Thread Benjamin LaHaise
On Fri, Aug 22, 2014 at 02:43:56PM -0700, Linus Torvalds wrote: > Ugh. > > Ben, at this point my gut feel is that we should just revert the > original "fix", and you should take a much deeper look at this all. > The original "fix" was more broken then the leak it purported to fix, > and now the pa

Re: Revert "aio: fix aio request leak when events are reaped by user space"

2014-08-24 Thread Benjamin LaHaise
hat address your concerns? Cheers, -ben -- "Thought is the essence of where you are now." SNIP: from git://git.kvack.org/~bcrl/aio-fixes.git >From 46faee0c63f6712874215b7ca878c52a0960c5da Mon Sep 17 00:00:00 2001 From: Benjamin LaHaise Date: Sun, 24 Aug 2014 1

Re: Revert "aio: fix aio request leak when events are reaped by user space"

2014-08-22 Thread Benjamin LaHaise
On Fri, Aug 22, 2014 at 07:15:02PM +0300, Dan Aloni wrote: > Sorry, I was waiting for a new patch from your direction, I should > have replied earlier. What bothered me about the patch you sent is that > completed_events is added as a new field but nothing assigns to it, so I > wonder how it can b

Re: Revert "aio: fix aio request leak when events are reaped by user space"

2014-08-22 Thread Benjamin LaHaise
On Tue, Aug 19, 2014 at 08:46:51PM -0400, Benjamin LaHaise wrote: > You can trigger the behaviour with fio by using userspace event reaping. > Adding a test case for that behaviour to libaio would be a good idea. > I thought about how to fix this, and it isn't actually that hard

Re: Revert "aio: fix aio request leak when events are reaped by user space"

2014-08-19 Thread Benjamin LaHaise
On Tue, Aug 19, 2014 at 08:14:26PM +0300, Dan Aloni wrote: > On Tue, Aug 19, 2014 at 12:54:04PM -0400, Benjamin LaHaise wrote: > > On Tue, Aug 19, 2014 at 07:37:33PM +0300, Dan Aloni wrote: > > > Some testing I've done today indicates that the original commit broke > &g

Re: Revert "aio: fix aio request leak when events are reaped by user space"

2014-08-19 Thread Benjamin LaHaise
On Tue, Aug 19, 2014 at 07:37:33PM +0300, Dan Aloni wrote: > Some testing I've done today indicates that the original commit broke > AIO with regard to users that overflow the maximum number of request > per IO context (where -EAGAIN is returned). > > In fact, it did worse - the attached C program

[GIT PULL] aio changes for 3.17

2014-08-15 Thread Benjamin LaHaise
to 00fefb9cf2b5493a86912de55ba912bdfae4a207: aio: use iovec array rather than the single one (2014-07-24 10:59:40 -0400) Benjamin LaHaise (2): Merge ../aio-fixes aio: remove no longer needed preempt_disable() Gu Zheng

Re: [PATCH 7/7] aio: use iovec array rather than the single one

2014-08-15 Thread Benjamin LaHaise
On Fri, Aug 15, 2014 at 06:35:25PM -0600, Linus Torvalds wrote: > Ben, > > *please* stop doing this kind of crap. > > You're creating fake emails with invalid senders. That's simply not > acceptable. ... I'll admit this was my mistake and I should have caught it while reviewing the emails. At

[PATCH 3/7] aio: remove no longer needed preempt_disable()

2014-08-15 Thread Benjamin LaHaise
Based on feedback from Jens Axboe on 263782c1c95bbddbb022dc092fd89a36bb8d5577, clean up get/put_reqs_available() to remove the no longer needed preempt_disable() and preempt_enable() pair. Signed-off-by: Benjamin LaHaise Cc: Jens Axboe --- fs/aio.c | 10 ++ 1 file changed, 2

[PATCH 0/7] aio: changes for 3.17

2014-08-15 Thread Benjamin LaHaise
Hello Linus, Please pull the following changes from the aio-next git tree at git://git.kvack.org/~bcrl/aio-next.git . Most of the changes this time around are code cleanups. -ben Benjamin LaHaise (1): aio: remove no longer needed preempt_disable() Gu Zheng (4): aio

Re: [RESEND PATCH 2/4] aio: use the macro rather than the inline magic number

2014-07-24 Thread Benjamin LaHaise
On Wed, Jul 23, 2014 at 06:03:52PM +0800, Gu Zheng wrote: > Replace the inline magic number with the ready-made macro(AIO_RING_MAGIC), > just clean up. Thanks Gu -- applied to my aio-next tree at git://git.kvack.org/~bcrl/aio-next.git . -ben > Reviewed-by: Jeff Moyer > Signed

Re: [RESEND PATCH 3/4] aio: fix some comments

2014-07-24 Thread Benjamin LaHaise
On Wed, Jul 23, 2014 at 06:03:53PM +0800, Gu Zheng wrote: > The function comments of aio_run_iocb and aio_read_events are out of date, so > fix them here. Thanks Gu -- applied to my aio-next tree at git://git.kvack.org/~bcrl/aio-next.git . -ben > Reviewed-by: Jeff Moyer > Sig

Re: [RESEND PATCH 1/4] aio: remove the needless registration of ring file's private_data

2014-07-24 Thread Benjamin LaHaise
On Wed, Jul 23, 2014 at 06:03:51PM +0800, Gu Zheng wrote: > Remove the registration of ring file's private_data, we do not use > it. Thanks Gu -- applied to my aio-next tree at git://git.kvack.org/~bcrl/aio-next.git . -ben > Reviewed-by: Jeff Moyer > Signed-off-by: Gu Zheng >

Re: [RESEND PATCH 4/4] aio: use iovec array rather than the single one

2014-07-24 Thread Benjamin LaHaise
On Wed, Jul 23, 2014 at 06:03:54PM +0800, Gu Zheng wrote: > Previously, we only offer a single iovec to handle all the read/write cases, > so > the PREADV/PWRITEV request always need to alloc more iovec buffer when copying > user vectors. > If we use a tmp iovec array rather than the single one, s

Re: [PATCH 3/4] aio: fix some comments

2014-07-22 Thread Benjamin LaHaise
On Tue, Jul 22, 2014 at 10:40:03AM +0800, Gu Zheng wrote: > Signed-off-by: Gu Zheng Again, you're missing a commit message here. Please resubmit with a commit message. -ben > --- > fs/aio.c |7 +++ > 1 files changed, 3 insertions(+), 4 deletions(-) > > diff --git a/f

Re: [PATCH 2/4] aio: use the macro rather than the inline magic number

2014-07-22 Thread Benjamin LaHaise
On Tue, Jul 22, 2014 at 10:40:02AM +0800, Gu Zheng wrote: > Signed-off-by: Gu Zheng You're missing a commit message here. The patch is otherwise fine. -ben > --- > fs/aio.c |2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/fs/aio.c b/fs/aio.c > in

Re: [PATCH] aio: protect reqs_available updates from changes in interrupt handlers

2014-07-14 Thread Benjamin LaHaise
On Mon, Jul 14, 2014 at 07:30:43PM +0200, Jens Axboe wrote: > Why not just makes this: > > local_irq_save(flags); > kcpu = this_put_ptr(...); > ... > > and get rid of the preemption bits, it's a bit redundant now you need to > kill local interrupts anyway. *nod*. I'll add the

Re: scsi-mq V2

2014-07-14 Thread Benjamin LaHaise
Hi Robert, On Sun, Jul 13, 2014 at 05:15:15PM +, Elliott, Robert (Server Storage) wrote: > > > I will see if that solves the problem with the scsi-mq-3 tree, or > > > at least some of the bisect trees leading up to it. > > > > scsi-mq-3 is still going after 45 minutes. I'll leave it running

[PATCH] aio: protect reqs_available updates from changes in interrupt handlers

2014-07-14 Thread Benjamin LaHaise
the per cpu batch updates of reqs_available. Many thanks to Robert and folks for testing and tracking this down. Reported-by: Robert Elliot Tested-by: Robert Elliot Signed-off-by: Benjamin LaHaise Cc: Jens Axboe , Christoph Hellwig Cc: sta...@vger.kenel.org --- fs/aio.c | 7 +++ 1 file

Re: scsi-mq V2

2014-07-11 Thread Benjamin LaHaise
On Fri, Jul 11, 2014 at 02:33:12PM +, Elliott, Robert (Server Storage) wrote: > That ran 9 total hours with no problem. > > Rather than revert in the bisect trees, I added just this single additional > patch to the no-rebase tree, and the problem appeared: Can you try the below totally untes

Re: scsi-mq V2

2014-07-10 Thread Benjamin LaHaise
On Thu, Jul 10, 2014 at 02:36:40PM +, Elliott, Robert (Server Storage) wrote: > > > > -Original Message- > > From: Jens Axboe [mailto:ax...@kernel.dk] > > Sent: Thursday, 10 July, 2014 8:53 AM > > To: Christoph Hellwig; Benjamin LaHaise > >

Re: scsi-mq V2

2014-07-10 Thread Benjamin LaHaise
On Thu, Jul 10, 2014 at 03:48:10PM +0200, Jens Axboe wrote: > On 2014-07-10 15:44, Benjamin LaHaise wrote: > >On Thu, Jul 10, 2014 at 03:39:57PM +0200, Jens Axboe wrote: > >>That's how fio always runs, it sets up the context with the exact queue > >>depth that it

  1   2   3   4   >