Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-02-20 Thread Peter Zijlstra
On Tue, Feb 18, 2014 at 07:54:34PM +, Waskiewicz Jr, Peter P wrote: > On Tue, 2014-02-18 at 20:35 +0100, Peter Zijlstra wrote: > > On Tue, Feb 18, 2014 at 05:29:42PM +, Waskiewicz Jr, Peter P wrote: > > > > Its not a problem that changing the task:RMID map is expensive, what is > > > > a pr

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-02-18 Thread Waskiewicz Jr, Peter P
On Tue, 2014-02-18 at 20:35 +0100, Peter Zijlstra wrote: > On Tue, Feb 18, 2014 at 05:29:42PM +, Waskiewicz Jr, Peter P wrote: > > > Its not a problem that changing the task:RMID map is expensive, what is > > > a problem is that there's no deterministic fashion of doing it. > > > > We are goin

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-02-18 Thread Peter Zijlstra
On Tue, Feb 18, 2014 at 05:29:42PM +, Waskiewicz Jr, Peter P wrote: > > Its not a problem that changing the task:RMID map is expensive, what is > > a problem is that there's no deterministic fashion of doing it. > > We are going to add to the SDM that changing RMID's often/frequently is > not

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-02-18 Thread Waskiewicz Jr, Peter P
On Mon, 2014-01-27 at 18:34 +0100, Peter Zijlstra wrote: Hi Peter, First of all, sorry for the delay in responding. I've been talking with the CPU architects to make sure we're going down the right path here before coming back to this. Responses below. > On Tue, Jan 14, 2014 at 09:58:26AM -0800

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-27 Thread Peter Zijlstra
On Tue, Jan 14, 2014 at 09:58:26AM -0800, H. Peter Anvin wrote: > On 01/12/2014 11:55 PM, Peter Zijlstra wrote: > > > > The problem is, since there's a limited number of RMIDs we have to > > rotate at some point, but since changing RMIDs is nondeterministic we > > can't. > > > > This is fundamen

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-14 Thread Waskiewicz Jr, Peter P
On Mon, 2014-01-13 at 08:55 +0100, Peter Zijlstra wrote: > On Fri, Jan 10, 2014 at 06:55:11PM +, Waskiewicz Jr, Peter P wrote: > > I've spoken with the CPU architect, and he's set me straight. I was > > getting some simulation data and reality mixed up, so apologies. > > > > The cacheline is

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-14 Thread H. Peter Anvin
On 01/12/2014 11:55 PM, Peter Zijlstra wrote: > > The problem is, since there's a limited number of RMIDs we have to > rotate at some point, but since changing RMIDs is nondeterministic we > can't. > This is fundamentally the crux here. RMIDs are quite expensive for the hardware to implement, s

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-12 Thread Peter Zijlstra
On Fri, Jan 10, 2014 at 06:55:11PM +, Waskiewicz Jr, Peter P wrote: > I've spoken with the CPU architect, and he's set me straight. I was > getting some simulation data and reality mixed up, so apologies. > > The cacheline is tagged with the RMID being tracked when it's brought > into the cac

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-10 Thread Waskiewicz Jr, Peter P
On Tue, 2014-01-07 at 22:12 +0100, Peter Zijlstra wrote: > Maybe its me (its late) but I can't follow. > > So if every cacheline is tagged with both CR3 and RMID (on all levels -- > I get that it needs to propagate etc..) then you can, upon observing a > new CR3,RMID pair, iterate the entire cache

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-07 Thread Peter Zijlstra
On Tue, Jan 07, 2014 at 03:15:52PM +, Waskiewicz Jr, Peter P wrote: > > Still confused here. So what you're saying is that cachelines get tagged > > with {CR3,RMID} and when they observe the same CR3 with a different RMID > > the hardware will iterate the entire cache and update all tuples? > >

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-07 Thread Waskiewicz Jr, Peter P
On Tue, 2014-01-07 at 09:34 +0100, Peter Zijlstra wrote: > On Mon, Jan 06, 2014 at 10:45:24PM +, Waskiewicz Jr, Peter P wrote: > > > Since its a very limited resource that seems like a weird assumption to > > > me; there's plenty scenarios in which you'd want to re-use RMIDs that > > > belong t

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-07 Thread Peter Zijlstra
On Mon, Jan 06, 2014 at 10:45:24PM +, Waskiewicz Jr, Peter P wrote: > > Since its a very limited resource that seems like a weird assumption to > > me; there's plenty scenarios in which you'd want to re-use RMIDs that > > belong to a still running context. > > I think I see what you're really

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Waskiewicz Jr, Peter P
On Mon, 2014-01-06 at 23:12 +0100, Peter Zijlstra wrote: > On Mon, Jan 06, 2014 at 09:48:29PM +, Waskiewicz Jr, Peter P wrote: > > The cacheline is tagged internally with the RMID as part of the waymask > > for the thread in the core. > > > > > Without a wipe you keep having stale entries of t

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Peter Zijlstra
On Mon, Jan 06, 2014 at 09:48:29PM +, Waskiewicz Jr, Peter P wrote: > On Mon, 2014-01-06 at 22:26 +0100, Peter Zijlstra wrote: > > On Mon, Jan 06, 2014 at 08:10:45PM +, Waskiewicz Jr, Peter P wrote: > > > There is one per logical CPU. However, in the current generation, they > > > report o

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Waskiewicz Jr, Peter P
On Mon, 2014-01-06 at 22:26 +0100, Peter Zijlstra wrote: > On Mon, Jan 06, 2014 at 08:10:45PM +, Waskiewicz Jr, Peter P wrote: > > There is one per logical CPU. However, in the current generation, they > > report on the usage of the same L3 cache. But the CPU takes care of the > > resolution

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Peter Zijlstra
On Mon, Jan 06, 2014 at 08:10:45PM +, Waskiewicz Jr, Peter P wrote: > There is one per logical CPU. However, in the current generation, they > report on the usage of the same L3 cache. But the CPU takes care of the > resolution of which MSR write and read comes from the logical CPU, so > soft

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Waskiewicz Jr, Peter P
On Mon, 2014-01-06 at 19:06 +0100, Peter Zijlstra wrote: > On Mon, Jan 06, 2014 at 04:47:57PM +, Waskiewicz Jr, Peter P wrote: > > > As is I don't really see a good use for RMIDs and I would simply not use > > > them. > > > > If you want to use CQM in the hardware, then the RMID is how you get

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Peter Zijlstra
On Mon, Jan 06, 2014 at 04:47:57PM +, Waskiewicz Jr, Peter P wrote: > > As is I don't really see a good use for RMIDs and I would simply not use > > them. > > If you want to use CQM in the hardware, then the RMID is how you get the > cache usage data from the CPU. If you don't want to use CQM

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Waskiewicz Jr, Peter P
On Mon, 2014-01-06 at 18:53 +0100, Peter Zijlstra wrote: > On Mon, Jan 06, 2014 at 04:47:57PM +, Waskiewicz Jr, Peter P wrote: > > > Yeah that's not accurate, nor desired I think, because you get into > > > horrible problems with hierarchies, do child groups belong to your RMID > > > or not? >

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Peter Zijlstra
On Mon, Jan 06, 2014 at 04:47:57PM +, Waskiewicz Jr, Peter P wrote: > > Yeah that's not accurate, nor desired I think, because you get into > > horrible problems with hierarchies, do child groups belong to your RMID > > or not? > > I'd rather not support a child group of a child group. Only g

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Waskiewicz Jr, Peter P
On Mon, 2014-01-06 at 17:41 +0100, Peter Zijlstra wrote: > On Mon, Jan 06, 2014 at 04:34:04PM +, Waskiewicz Jr, Peter P wrote: > > On Mon, 2014-01-06 at 12:16 +0100, Peter Zijlstra wrote: > > > On Sun, Jan 05, 2014 at 05:23:07AM +, Waskiewicz Jr, Peter P wrote: > > > > The CPU side is easy

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Waskiewicz Jr, Peter P
On Mon, 2014-01-06 at 12:08 +0100, Peter Zijlstra wrote: > On Fri, Jan 03, 2014 at 12:34:41PM -0800, Peter P Waskiewicz Jr wrote: > > The CPU features themselves are relatively straight-forward, but > > the presentation of the data is less straight-forward. Since this > > tracks cache usage and oc

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Peter Zijlstra
On Mon, Jan 06, 2014 at 04:34:04PM +, Waskiewicz Jr, Peter P wrote: > On Mon, 2014-01-06 at 12:16 +0100, Peter Zijlstra wrote: > > On Sun, Jan 05, 2014 at 05:23:07AM +, Waskiewicz Jr, Peter P wrote: > > > The CPU side is easy and clean. When something in the software wants to > > > monitor

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Waskiewicz Jr, Peter P
On Mon, 2014-01-06 at 12:16 +0100, Peter Zijlstra wrote: > On Sun, Jan 05, 2014 at 05:23:07AM +, Waskiewicz Jr, Peter P wrote: > > The CPU side is easy and clean. When something in the software wants to > > monitor when a particular task is scheduled and started, write whatever > > RMID that t

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Peter Zijlstra
On Sun, Jan 05, 2014 at 05:23:07AM +, Waskiewicz Jr, Peter P wrote: > The processor doesn't need to understand the grouping at all, but it > also isn't tracking things per-process that are rolled up later. > They're tracked via the RMID resource in the hardware, which could > correspond to a si

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-06 Thread Peter Zijlstra
On Fri, Jan 03, 2014 at 12:34:41PM -0800, Peter P Waskiewicz Jr wrote: > The CPU features themselves are relatively straight-forward, but > the presentation of the data is less straight-forward. Since this > tracks cache usage and occupancy per process (by swapping Resource > Monitor IDs, or RMIDs

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-04 Thread Waskiewicz Jr, Peter P
On Sat, 2014-01-04 at 17:50 -0500, Tejun Heo wrote: > Hello, Hi Tejun, > On Sat, Jan 04, 2014 at 10:43:00PM +, Waskiewicz Jr, Peter P wrote: > > Simply put, when we want to allocate an RMID for monitoring httpd > > traffic, we can create a new child in the subsystem hierarchy, and > > assign

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-04 Thread Tejun Heo
Hello, On Sat, Jan 04, 2014 at 10:43:00PM +, Waskiewicz Jr, Peter P wrote: > Simply put, when we want to allocate an RMID for monitoring httpd > traffic, we can create a new child in the subsystem hierarchy, and > assign the httpd processes to it. Then the RMID can be assigned to the > subsys

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-04 Thread Waskiewicz Jr, Peter P
On Sat, 2014-01-04 at 11:10 -0500, Tejun Heo wrote: > Hello, Hi Tejun, > On Fri, Jan 03, 2014 at 12:34:41PM -0800, Peter P Waskiewicz Jr wrote: > > The CPU features themselves are relatively straight-forward, but > > the presentation of the data is less straight-forward. Since this > > tracks ca

Re: [PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-04 Thread Tejun Heo
Hello, On Fri, Jan 03, 2014 at 12:34:41PM -0800, Peter P Waskiewicz Jr wrote: > The CPU features themselves are relatively straight-forward, but > the presentation of the data is less straight-forward. Since this > tracks cache usage and occupancy per process (by swapping Resource > Monitor IDs,

[PATCH 0/4] x86: Add Cache QoS Monitoring (CQM) support

2014-01-03 Thread Peter P Waskiewicz Jr
This patchset adds support for the new Cache QoS Monitoring (CQM) feature found in future Intel Xeon processors. CQM allows a process, or set of processes, to be tracked by the CPU to determine the cache usage of that task group. Using this data from the CPU, software can be written to extract th