On Fri, 2005-04-01 at 06:30 +0200, Ingo Molnar wrote:
> * Lee Revell <[EMAIL PROTECTED]> wrote:
>
> > > > ah - cool! This was a 100 MB writeout so having 3.7 msecs to process
> > > > 20K+ pages is not unreasonable. To break the latency, can i just do a
> > > > simple lock-break, via the patch be
fr den 01.04.2005 Klokka 09:16 (-0700) skreiv Orion Poplawski:
> Just a question - would these changes be expected to improve NFS client
> *read* access at all, or just write?
Just write.
Cheers,
Trond
--
Trond Myklebust <[EMAIL PROTECTED]>
-
To unsubscribe from this list: send the line "un
Ingo Molnar wrote:
* Lee Revell <[EMAIL PROTECTED]> wrote:
This fixes all the NFS related latency problems I was seeing. Now the
longest latency from an NFS kernel compile with "make -j64" is 391
usecs in get_swap_page.
great! The latest patches (-42-08 and later) have the reworked
nfs_scan_li
* Lee Revell <[EMAIL PROTECTED]> wrote:
> > > ah - cool! This was a 100 MB writeout so having 3.7 msecs to process
> > > 20K+ pages is not unreasonable. To break the latency, can i just do a
> > > simple lock-break, via the patch below?
> >
> > with this patch the worst-case latency during NFS
On Thu, 2005-03-31 at 16:50 +0200, Ingo Molnar wrote:
> * Ingo Molnar <[EMAIL PROTECTED]> wrote:
>
> > > So the overhead you are currently seeing should just be that of
> > > iterating through the list, locking said requests and adding them to
> > > our private list.
> >
> > ah - cool! This was
to den 31.03.2005 Klokka 17:10 (+0200) skreiv Ingo Molnar:
> * Trond Myklebust <[EMAIL PROTECTED]> wrote:
>
> > > would it be safe to collect locked entries into a separate, local list,
> > > so that the restart would only see newly added entries? Then once the
> > > moving of all entries has be
* Ingo Molnar <[EMAIL PROTECTED]> wrote:
> > The good news, though, is that because requests on the "commit" list
> > do not remain locked for long without being removed from the list, you
> > should rarely have to skip them. IOW restarting from the head of the
> > list is pretty much the same
* Trond Myklebust <[EMAIL PROTECTED]> wrote:
> > would it be safe to collect locked entries into a separate, local list,
> > so that the restart would only see newly added entries? Then once the
> > moving of all entries has been done, all the locked entries could be
> > added back to the comm
to den 31.03.2005 Klokka 16:58 (+0200) skreiv Ingo Molnar:
> would it be safe to collect locked entries into a separate, local list,
> so that the restart would only see newly added entries? Then once the
> moving of all entries has been done, all the locked entries could be
> added back to th
to den 31.03.2005 Klokka 16:54 (+0200) skreiv Ingo Molnar:
> i think it's incorrect in its current form, because 'pos' is not
> necessarily safe and might be removed from the commit_list?
Right.
> the whole loop could be a "while (!list_empty(head))" thing, if it wasnt
> for the possibility fo
* Trond Myklebust <[EMAIL PROTECTED]> wrote:
> Errm... That looks a bit unsafe. What is there then to stop another
> process from removing "pos" while you are scheduling? It seems to me
> that you should really start afresh in that case.
yeah.
> The good news, though, is that because requests
to den 31.03.2005 Klokka 16:39 (+0200) skreiv Ingo Molnar:
> * Trond Myklebust <[EMAIL PROTECTED]> wrote:
>
> > > your patch works fine here - but there are still latencies in
> > > nfs_scan_commit()/nfs_scan_list(): see the attached 3.7 msec latency
> > > trace. It happened during a simple big-
* Ingo Molnar <[EMAIL PROTECTED]> wrote:
> [...] To break the latency, can i just do a simple lock-break, via the
> patch below?
i think it's incorrect in its current form, because 'pos' is not
necessarily safe and might be removed from the commit_list?
the whole loop could be a "while (!list
* Ingo Molnar <[EMAIL PROTECTED]> wrote:
> > So the overhead you are currently seeing should just be that of
> > iterating through the list, locking said requests and adding them to
> > our private list.
>
> ah - cool! This was a 100 MB writeout so having 3.7 msecs to process
> 20K+ pages is
* Trond Myklebust <[EMAIL PROTECTED]> wrote:
> > your patch works fine here - but there are still latencies in
> > nfs_scan_commit()/nfs_scan_list(): see the attached 3.7 msec latency
> > trace. It happened during a simple big-file writeout and is easily
> > reproducible. Could the nfsi->commi
to den 31.03.2005 Klokka 15:58 (+0200) skreiv Ingo Molnar:
> * Trond Myklebust <[EMAIL PROTECTED]> wrote:
>
> > > + int res;
> > ^^^ Should be "int res = 0;"
>
> your patch works fine here - but there are still latencies in
> nfs_scan_commit()/nfs_scan_list(): see the attached 3.7 msec
* Trond Myklebust <[EMAIL PROTECTED]> wrote:
> > + int res;
> ^^^ Should be "int res = 0;"
your patch works fine here - but there are still latencies in
nfs_scan_commit()/nfs_scan_list(): see the attached 3.7 msec latency
trace. It happened during a simple big-file writeout and i
to den 31.03.2005 Klokka 06:58 (-0500) skreiv Trond Myklebust:
>
> @@ -563,11 +566,14 @@ static int
> nfs_scan_commit(struct inode *inode, struct list_head *dst, unsigned long
> idx_start, unsigned int npages)
> {
> struct nfs_inode *nfsi = NFS_I(inode);
> - int res;
> - res
to den 31.03.2005 Klokka 09:30 (+0200) skreiv Ingo Molnar:
> > And presumably that list-based code rarely hits the worst-case, else
> > it would have been changed by now.
>
> that was my other point in a previous mail: most write benchmarks do
> continuous append writes, and CPU overhead easily
* Ingo Molnar <[EMAIL PROTECTED]> wrote:
> ok, find a new latency trace attached (1.6 msecs). I generated write
> loads, and the nfs_scan_list_dirty() latency is gone and indeed
> nfs_scan_list() generates the worst latency - while processing 8535
> pages in one critical section.
here's a mor
* Ingo Molnar <[EMAIL PROTECTED]> wrote:
> could you check the -41-23 -RT kernel at the usual place:
>
>http://redhat.com/~mingo/realtime-preempt/
>
> i've added Trond's radix lookup code, plus the lockbreaker patch.
>
> (one thing that is not covered yet is nfs_scan_list() - that still sc
* Lee Revell <[EMAIL PROTECTED]> wrote:
> > Yes. Together with the radix tree-based sorting of dirty requests,
> > that's pretty much what I've spent most of today doing. Lee, could you
> > see how the attached combined patch changes your latency numbers?
>
> Different code path, and the latency
* Andrew Morton <[EMAIL PROTECTED]> wrote:
> Well. The radix-tree approach's best-case is probably quite a lot
> worse than the list-based approach's best-case. It hits a lot more
> cachelines and involves a lot more code.
The list-based approach's best-case are large continuous append write
Ingo Molnar <[EMAIL PROTECTED]> wrote:
>
>
> * Trond Myklebust <[EMAIL PROTECTED]> wrote:
>
> > > The 7 ms are spent in this loop:
> >
> > Which is basically confirming what the guys from Bull already told me,
> > namely that the radix tree tag stuff is much less efficient that what
> > w
* Ingo Molnar <[EMAIL PROTECTED]> wrote:
> i think all it needs now is a lock-breaker in the main radix-lookup
> loop in nfs_scan_lock_dirty(), or a latency-oriented reduction in the
> npages argument, to make the loop bounded. [...]
can nfsi->req_lock be dropped within nfs_scan_dirty()? Or do
* Lee Revell <[EMAIL PROTECTED]> wrote:
> > Is a bunch of gobbledygook. Hows about you interpret it for us?
>
> Sorry. When I summarized them before, Ingo just asked for the full
> verbose trace.
please send non-verbose traces if possible. The verbose traces are
useful when it's not clear w
* Trond Myklebust <[EMAIL PROTECTED]> wrote:
> > The 7 ms are spent in this loop:
>
> Which is basically confirming what the guys from Bull already told me,
> namely that the radix tree tag stuff is much less efficient that what
> we've got now. I never saw their patches, though, so I was curio
on den 30.03.2005 Klokka 21:47 (-0500) skreiv Lee Revell:
> On Wed, 2005-03-30 at 18:39 -0800, Andrew Morton wrote:
> > Lee Revell <[EMAIL PROTECTED]> wrote:
> > >
> > > > Yes. Together with the radix tree-based sorting of dirty requests,
> > > > that's pretty much what I've spent most of today do
On Wed, 2005-03-30 at 18:39 -0800, Andrew Morton wrote:
> Lee Revell <[EMAIL PROTECTED]> wrote:
> >
> > > Yes. Together with the radix tree-based sorting of dirty requests,
> > > that's pretty much what I've spent most of today doing. Lee, could you
> > > see how the attached combined patch chang
Lee Revell <[EMAIL PROTECTED]> wrote:
>
> > Yes. Together with the radix tree-based sorting of dirty requests,
> > that's pretty much what I've spent most of today doing. Lee, could you
> > see how the attached combined patch changes your latency numbers?
> >
>
> Different code path, and the
On Wed, 2005-03-30 at 16:14 -0500, Trond Myklebust wrote:
> on den 30.03.2005 Klokka 11:56 (-0800) skreiv Andrew Morton:
> > > That's normal and cannot be avoided: when writing, we have to look for
> > > the existence of old nfs_page requests. The reason is that if one does
> > > exist, we must eit
on den 30.03.2005 Klokka 11:56 (-0800) skreiv Andrew Morton:
> > That's normal and cannot be avoided: when writing, we have to look for
> > the existence of old nfs_page requests. The reason is that if one does
> > exist, we must either coalesce our new dirty area into it or if we
> > can't, we mus
Trond Myklebust <[EMAIL PROTECTED]> wrote:
>
> on den 30.03.2005 Klokka 09:26 (-0500) skreiv Lee Revell:
> > On Tue, 2005-03-29 at 18:18 -0500, Trond Myklebust wrote:
> > > ty den 29.03.2005 Klokka 18:04 (-0500) skreiv Lee Revell:
> > > > I am seeing long latencies in the NFS client code. Attached
Ingo Molnar <[EMAIL PROTECTED]> wrote:
>
> > No. Coalescing on the client makes tons of sense. The overhead of
> > sending 8 RPC requests for 4k writes instead of sending 1 RPC request
> > for a single 32k write is huge: among other things, you end up tying up
> > 8 RPC slots on the client + 8 n
On Wed, 2005-03-30 at 09:50 -0500, Trond Myklebust wrote:
> on den 30.03.2005 Klokka 09:26 (-0500) skreiv Lee Revell:
> > On Tue, 2005-03-29 at 18:18 -0500, Trond Myklebust wrote:
> > > ty den 29.03.2005 Klokka 18:04 (-0500) skreiv Lee Revell:
> > > > I am seeing long latencies in the NFS client co
on den 30.03.2005 Klokka 09:26 (-0500) skreiv Lee Revell:
> On Tue, 2005-03-29 at 18:18 -0500, Trond Myklebust wrote:
> > ty den 29.03.2005 Klokka 18:04 (-0500) skreiv Lee Revell:
> > > I am seeing long latencies in the NFS client code. Attached is a ~1.9
> > > ms latency trace.
> >
> > What kind
On Tue, 2005-03-29 at 18:18 -0500, Trond Myklebust wrote:
> ty den 29.03.2005 Klokka 18:04 (-0500) skreiv Lee Revell:
> > I am seeing long latencies in the NFS client code. Attached is a ~1.9
> > ms latency trace.
>
> What kind of workload are you using to produce these numbers?
>
Here is the o
* Trond Myklebust <[EMAIL PROTECTED]> wrote:
> > the comment suggests that this is optimized for append writes (which is
> > quite common, but by far not the only write workload) - but the
> > worst-case behavior of this code is very bad. How about disabling this
> > sorting altogether and ben
on den 30.03.2005 Klokka 10:02 (+0200) skreiv Ingo Molnar:
> the comment suggests that this is optimized for append writes (which is
> quite common, but by far not the only write workload) - but the
> worst-case behavior of this code is very bad. How about disabling this
> sorting altogether an
* Trond Myklebust <[EMAIL PROTECTED]> wrote:
> ty den 29.03.2005 Klokka 18:32 (-0500) skreiv Lee Revell:
> > On Tue, 2005-03-29 at 18:18 -0500, Trond Myklebust wrote:
> > > ty den 29.03.2005 Klokka 18:04 (-0500) skreiv Lee Revell:
> > > > I am seeing long latencies in the NFS client code. Attach
On Tue, 2005-03-29 at 18:34 -0500, Trond Myklebust wrote:
> ty den 29.03.2005 Klokka 18:32 (-0500) skreiv Lee Revell:
> > On Tue, 2005-03-29 at 18:18 -0500, Trond Myklebust wrote:
> > > ty den 29.03.2005 Klokka 18:04 (-0500) skreiv Lee Revell:
> > > > I am seeing long latencies in the NFS client co
ty den 29.03.2005 Klokka 18:32 (-0500) skreiv Lee Revell:
> On Tue, 2005-03-29 at 18:18 -0500, Trond Myklebust wrote:
> > ty den 29.03.2005 Klokka 18:04 (-0500) skreiv Lee Revell:
> > > I am seeing long latencies in the NFS client code. Attached is a ~1.9
> > > ms latency trace.
> >
> > What kind
On Tue, 2005-03-29 at 18:18 -0500, Trond Myklebust wrote:
> ty den 29.03.2005 Klokka 18:04 (-0500) skreiv Lee Revell:
> > I am seeing long latencies in the NFS client code. Attached is a ~1.9
> > ms latency trace.
>
> What kind of workload are you using to produce these numbers?
>
Just a kernel
ty den 29.03.2005 Klokka 18:04 (-0500) skreiv Lee Revell:
> I am seeing long latencies in the NFS client code. Attached is a ~1.9
> ms latency trace.
What kind of workload are you using to produce these numbers?
Cheers,
Trond
--
Trond Myklebust <[EMAIL PROTECTED]>
-
To unsubscribe from this
44 matches
Mail list logo
