Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-08-17 Thread Eric W. Biederman
wi...@casper.infradead.org writes: > On Mon, Jun 22, 2020 at 10:20:40AM -0500, Eric W. Biederman wrote: >> Junxiao Bi writes: >> > On 6/20/20 9:27 AM, Matthew Wilcox wrote: >> >> On Fri, Jun 19, 2020 at 05:42:45PM -0500, Eric W. Biederman wrote: >> >>> Junxiao Bi writes: >> Still high lock

Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-25 Thread Junxiao Bi
On 6/22/20 5:47 PM, Matthew Wilcox wrote: On Sun, Jun 21, 2020 at 10:15:39PM -0700, Junxiao Bi wrote: On 6/20/20 9:27 AM, Matthew Wilcox wrote: On Fri, Jun 19, 2020 at 05:42:45PM -0500, Eric W. Biederman wrote: Junxiao Bi writes: Still high lock contention. Collect the following hot path.

Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-22 Thread Matthew Wilcox
On Sun, Jun 21, 2020 at 10:15:39PM -0700, Junxiao Bi wrote: > On 6/20/20 9:27 AM, Matthew Wilcox wrote: > > On Fri, Jun 19, 2020 at 05:42:45PM -0500, Eric W. Biederman wrote: > > > Junxiao Bi writes: > > > > Still high lock contention. Collect the following hot path. > > > A different location thi

Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-22 Thread Junxiao Bi
On 6/22/20 8:20 AM, ebied...@xmission.com wrote: If I understand correctly, the Java VM is not exiting. Just some of it's threads. That is a very different problem to deal with. That are many optimizations that are possible when_all_ of the threads are exiting that are not possible when_many

Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-22 Thread willy
On Mon, Jun 22, 2020 at 10:20:40AM -0500, Eric W. Biederman wrote: > Junxiao Bi writes: > > On 6/20/20 9:27 AM, Matthew Wilcox wrote: > >> On Fri, Jun 19, 2020 at 05:42:45PM -0500, Eric W. Biederman wrote: > >>> Junxiao Bi writes: > Still high lock contention. Collect the following hot path.

Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-22 Thread Eric W. Biederman
Junxiao Bi writes: > On 6/20/20 9:27 AM, Matthew Wilcox wrote: > >> On Fri, Jun 19, 2020 at 05:42:45PM -0500, Eric W. Biederman wrote: >>> Junxiao Bi writes: Still high lock contention. Collect the following hot path. >>> A different location this time. >>> >>> I know of at least exit_signa

Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-22 Thread Eric W. Biederman
Masahiro Yamada writes: > On Fri, Jun 19, 2020 at 11:14 PM Eric W. Biederman > wrote: >> >> >> Junxiao Bi reported: >> > When debugging some performance issue, i found that thousands of threads >> > exit >> > around same time could cause a severe spin lock contention on proc dentry >> > "/proc

Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-21 Thread Masahiro Yamada
On Fri, Jun 19, 2020 at 11:14 PM Eric W. Biederman wrote: > > > Junxiao Bi reported: > > When debugging some performance issue, i found that thousands of threads > > exit > > around same time could cause a severe spin lock contention on proc dentry > > "/proc/$parent_process_pid/task/", that's b

Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-21 Thread Junxiao Bi
On 6/20/20 9:27 AM, Matthew Wilcox wrote: On Fri, Jun 19, 2020 at 05:42:45PM -0500, Eric W. Biederman wrote: Junxiao Bi writes: Still high lock contention. Collect the following hot path. A different location this time. I know of at least exit_signal and exit_notify that take thread wide lo

Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-20 Thread Matthew Wilcox
On Fri, Jun 19, 2020 at 05:42:45PM -0500, Eric W. Biederman wrote: > Junxiao Bi writes: > > Still high lock contention. Collect the following hot path. > > A different location this time. > > I know of at least exit_signal and exit_notify that take thread wide > locks, and it looks like exit_mm

Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-19 Thread Eric W. Biederman
Junxiao Bi writes: > On 6/19/20 10:24 AM, ebied...@xmission.com wrote: > >> Junxiao Bi writes: >> >>> Hi Eric, >>> >>> The patch didn't improve lock contention. >> Which raises the question where is the lock contention coming from. >> >> Especially with my first variant. Only the last thread to

Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-19 Thread Junxiao Bi
On 6/19/20 10:24 AM, ebied...@xmission.com wrote: Junxiao Bi writes: Hi Eric, The patch didn't improve lock contention. Which raises the question where is the lock contention coming from. Especially with my first variant. Only the last thread to be reaped would free up anything in the cac

Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-19 Thread Eric W. Biederman
Junxiao Bi writes: > Hi Eric, > > The patch didn't improve lock contention. Which raises the question where is the lock contention coming from. Especially with my first variant. Only the last thread to be reaped would free up anything in the cache. Can you comment out the call to proc_flush_p

Re: [PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-19 Thread Junxiao Bi
Hi Eric, The patch didn't improve lock contention.    PerfTop:   48925 irqs/sec  kernel:95.6%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 104 CPUs)

[PATCH] proc: Avoid a thundering herd of threads freeing proc dentries

2020-06-19 Thread Eric W. Biederman
Junxiao Bi reported: > When debugging some performance issue, i found that thousands of threads exit > around same time could cause a severe spin lock contention on proc dentry > "/proc/$parent_process_pid/task/", that's because threads needs to clean up > their pid file from that dir when exit.