Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-10 Thread Phil Sutter
On Thu, Sep 10, 2015 at 04:03:44PM +0800, Herbert Xu wrote: > On Tue, Sep 01, 2015 at 04:51:24PM +0200, Thomas Graf wrote: > > > > 1. The current in-kernel self-test > > 2. bind_netlink.c: https://github.com/tgraf/rhashtable > > I can't reproduce it: I can't speak for netlink, but if you apply p

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-10 Thread Herbert Xu
On Tue, Sep 01, 2015 at 04:51:24PM +0200, Thomas Graf wrote: > > 1. The current in-kernel self-test > 2. bind_netlink.c: https://github.com/tgraf/rhashtable I can't reproduce it: $ while :; do ./bind_netlink 1 12345; done Ports successfully created, terminating Create 1 ports Created 100

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-02 Thread Thomas Graf
On 09/02/15 at 10:00am, Herbert Xu wrote: > On Tue, Sep 01, 2015 at 04:51:24PM +0200, Thomas Graf wrote: > > > > 1. The current in-kernel self-test > > 2. bind_netlink.c: https://github.com/tgraf/rhashtable > > Thanks, I will try to reproduce this. The path in question is: int rhashtable_insert_

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-01 Thread Herbert Xu
On Tue, Sep 01, 2015 at 04:51:24PM +0200, Thomas Graf wrote: > > 1. The current in-kernel self-test > 2. bind_netlink.c: https://github.com/tgraf/rhashtable Thanks, I will try to reproduce this. -- Email: Herbert Xu Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-01 Thread Thomas Graf
On 09/01/15 at 10:16pm, Herbert Xu wrote: > On Tue, Sep 01, 2015 at 04:13:05PM +0200, Thomas Graf wrote: > > > > You can easily trigger this outside of the testsuite as well. Open > > 10K Netlink sockets in a loop and the creation of the sockets will > > fail way before memory pressure kicks in. >

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-01 Thread Herbert Xu
On Tue, Sep 01, 2015 at 04:13:05PM +0200, Thomas Graf wrote: > > You can easily trigger this outside of the testsuite as well. Open > 10K Netlink sockets in a loop and the creation of the sockets will > fail way before memory pressure kicks in. That means our implementation is buggy. Do you have

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-01 Thread Thomas Graf
On 09/01/15 at 10:03pm, Herbert Xu wrote: > On Tue, Sep 01, 2015 at 03:56:18PM +0200, Phil Sutter wrote: > > > > Looking at rhashtable_test.c, I see the initial table size is 8 entries. > > 70% of that is 5.6 entries, so background expansion is started after the > > 6th entry has been added, right

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-01 Thread Herbert Xu
On Tue, Sep 01, 2015 at 03:56:18PM +0200, Phil Sutter wrote: > > Looking at rhashtable_test.c, I see the initial table size is 8 entries. > 70% of that is 5.6 entries, so background expansion is started after the > 6th entry has been added, right? Given there are 10 threads running > which try to

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-01 Thread Phil Sutter
On Tue, Sep 01, 2015 at 09:50:19PM +0800, Herbert Xu wrote: > On Tue, Sep 01, 2015 at 03:43:11PM +0200, Phil Sutter wrote: > > > > Hmm. Since memory allocation is first tried with GFP_ATOMIC set and upon > > failure retried in background, this seems like a situation which might > > happen during no

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-01 Thread Herbert Xu
On Tue, Sep 01, 2015 at 03:43:11PM +0200, Phil Sutter wrote: > > Hmm. Since memory allocation is first tried with GFP_ATOMIC set and upon > failure retried in background, this seems like a situation which might > happen during normal use. If that already indicates a severe problem, > why retry in b

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-01 Thread Phil Sutter
On Tue, Sep 01, 2015 at 09:00:57PM +0800, Herbert Xu wrote: > On Tue, Sep 01, 2015 at 02:46:48PM +0200, Phil Sutter wrote: > > > > This is not an inherent behaviour of the implementation but general > > agreement. The insertion may fail non-permanently (returning -EBUSY), > > users are expected to

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-01 Thread Eric Dumazet
On Tue, 2015-09-01 at 21:00 +0800, Herbert Xu wrote: > On Tue, Sep 01, 2015 at 02:46:48PM +0200, Phil Sutter wrote: > > > > This is not an inherent behaviour of the implementation but general > > agreement. The insertion may fail non-permanently (returning -EBUSY), > > users are expected to handle

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-01 Thread Herbert Xu
On Tue, Sep 01, 2015 at 02:46:48PM +0200, Phil Sutter wrote: > > This is not an inherent behaviour of the implementation but general > agreement. The insertion may fail non-permanently (returning -EBUSY), > users are expected to handle this by retrying the operation. Absolutely not. The only reas

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-01 Thread Phil Sutter
On Tue, Sep 01, 2015 at 07:43:00PM +0800, Herbert Xu wrote: > On Mon, Aug 31, 2015 at 01:00:12PM +0200, Phil Sutter wrote: > > > > The variable would be used to track if the worker has failed to allocate > > memory in background. > > > > Since the failing insertion will be retried, subsequent ins

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-09-01 Thread Herbert Xu
On Mon, Aug 31, 2015 at 01:00:12PM +0200, Phil Sutter wrote: > > The variable would be used to track if the worker has failed to allocate > memory in background. > > Since the failing insertion will be retried, subsequent inserts are not > necessary unrelated. If an insertion fails it is never r

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-08-31 Thread Phil Sutter
On Sun, Aug 30, 2015 at 03:47:17PM +0800, Herbert Xu wrote: > Phil Sutter wrote: > > > > Should we introduce a new field to struct rhashtable to track the > > internal state? This might allow to clean up some rather obscure tests, > > e.g. whether a table resize is in progress or not. > > Why wou

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-08-30 Thread Herbert Xu
Phil Sutter wrote: > > Should we introduce a new field to struct rhashtable to track the > internal state? This might allow to clean up some rather obscure tests, > e.g. whether a table resize is in progress or not. Why would we want to do that? The deferred expansion is done on a best effort bas

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-08-29 Thread Phil Sutter
On Sat, Aug 29, 2015 at 12:43:03AM +0200, Thomas Graf wrote: > On 08/28/15 at 03:34pm, Phil Sutter wrote: > > Quite ugly, IMHO: rhashtable_insert_fast() may return -ENOMEM as > > non-permanent error, if allocation in GFP_ATOMIC failed. In this case, > > allocation in GFP_KERNEL is retried by rht_de

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-08-28 Thread Thomas Graf
On 08/28/15 at 03:34pm, Phil Sutter wrote: > Quite ugly, IMHO: rhashtable_insert_fast() may return -ENOMEM as > non-permanent error, if allocation in GFP_ATOMIC failed. In this case, > allocation in GFP_KERNEL is retried by rht_deferred_worker(). Sadly, > there is no way to determine if that has al

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-08-28 Thread Phil Sutter
On Fri, Aug 28, 2015 at 01:13:20PM +0200, Phil Sutter wrote: > On Fri, Aug 28, 2015 at 01:09:29PM +0200, Thomas Graf wrote: > > On 08/28/15 at 12:28pm, Phil Sutter wrote: > > > After adding cond_resched() calls to threadfunc(), a surprisingly high > > > rate of insert failures occurred probably due

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-08-28 Thread Phil Sutter
On Fri, Aug 28, 2015 at 01:09:29PM +0200, Thomas Graf wrote: > On 08/28/15 at 12:28pm, Phil Sutter wrote: > > After adding cond_resched() calls to threadfunc(), a surprisingly high > > rate of insert failures occurred probably due to table resizes getting a > > better chance to run in background. T

Re: [PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-08-28 Thread Thomas Graf
On 08/28/15 at 12:28pm, Phil Sutter wrote: > After adding cond_resched() calls to threadfunc(), a surprisingly high > rate of insert failures occurred probably due to table resizes getting a > better chance to run in background. To not soften up the remaining > tests, retry inserts until they eithe

[PATCH 2/3] rhashtable-test: retry insert operations in threads

2015-08-28 Thread Phil Sutter
After adding cond_resched() calls to threadfunc(), a surprisingly high rate of insert failures occurred probably due to table resizes getting a better chance to run in background. To not soften up the remaining tests, retry inserts until they either succeed or fail permanently. Signed-off-by: Phil