Re: [perf-discuss] NUMA ptools and ISM segments

On Wed, Oct 05, 2005 at 08:53:17AM -0700, David McDaniel (damcdani) wrote: | Are the sparc TLBs transparent as to instrucitons or data? At one | point I though I read that I and D tlbs were separate entities. I'm only talking about the SPARC DTLB here, since that is what you were concerned with

RE: [perf-discuss] NUMA ptools and ISM segments

M > To: Marc Rocas > Cc: perf-discuss@opensolaris.org; [EMAIL PROTECTED]; Kit Chow > Subject: Re: [perf-discuss] NUMA ptools and ISM segments > > On Tue, Oct 04, 2005 at 09:54:17PM -0400, Marc Rocas wrote: > | > | That's correct. We've measured 3-5% improvement with 4

Re: [perf-discuss] NUMA ptools and ISM segments

On Tue, Oct 04, 2005 at 09:54:17PM -0400, Marc Rocas wrote: | | That's correct. We've measured 3-5% improvement with 4MB pages on SPARC and | simply wanted to carry over to amd64. We have not carried out the | measurements on amd64 to compare 4KB vs. 2MB pages since the ratio is the | same as in S

Re: [perf-discuss] NUMA ptools and ISM segments

On 10/4/05, Jonathan Chew <[EMAIL PROTECTED]> wrote: Marc Rocas wrote On 09/27/05 21:27,:> I spoke to one of our I/O guys to see whether there is a prescribed way> to allocate physical memory in the least significant 4 gig for DMA from> userland.  Solaris doesn't provide an existing way

Re: [perf-discuss] NUMA ptools and ISM segments

Marc Rocas wrote On 09/27/05 21:27,: > > On 9/26/05, *jonathan chew* <[EMAIL PROTECTED] > > wrote: > > > There may be a slightly better way to allocate memory from the lgroup > containing the least significant physical memory using > lgrp_affinity_set(3LGRP)

Re: [perf-discuss] NUMA ptools and ISM segments

On 9/26/05, jonathan chew <[EMAIL PROTECTED]> wrote: There may be a slightly better way to allocate memory from the lgroupcontaining the least significant physical memory usinglgrp_affinity_set(3LGRP), meminfo(2), and madvise(MADV_ACCESS_LWP).Unfortunately, none of these ways are guaranteed to alwa

Re: [perf-discuss] NUMA ptools and ISM segments

Marc Rocas wrote: All, I ended up creating a DISM segment, migrating the creating thread to the lgrp containing cpu0, madvising, touching the memory, and MC_LOCK'ing per Jonathan's suggestions and it all worked out. The memory was exclusively allocated from lgrp 1 (cpu0) using 2M pages per

Re: [perf-discuss] NUMA ptools and ISM segments

All, I ended up creating a DISM segment, migrating the creating thread to the lgrp containing cpu0, madvising, touching the memory, and MC_LOCK'ing per Jonathan's suggestions and it all worked out. The memory was exclusively allocated from lgrp 1 (cpu0) using 2M pages per the output of 'pmap -Ls $

Re: [perf-discuss] NUMA ptools and ISM segments

> "Marc" == Marc Rocas writes: Marc> Lastly, I tried relocating the memory by following the instructions in Marc> Alexander Kolbasov's blog (Memory Placement) by writing a simple app that Marc> attached to the existing segment with a few sleep() calls to allow me to Marc> type the fo

Re: [perf-discuss] NUMA ptools and ISM segments

Joe Bonasera wrote: The x86 HAT (D)ISM code is tricky. Here's why: pagesize is 4K, a large page is 2Meg (usually). So a pagetable covers either an entire 512 4K pages aligned at 2M or 512 2M pages aligned at 1Gig. To share a page table, the (D)ISM segment has to be either a multiple of (512

Re: [perf-discuss] NUMA ptools and ISM segments

The x86 HAT (D)ISM code is tricky. Here's why: pagesize is 4K, a large page is 2Meg (usually). So a pagetable covers either an entire 512 4K pages aligned at 2M or 512 2M pages aligned at 1Gig. To share a page table, the (D)ISM segment has to be either a multiple of (512) 4K pages aligned to

Re: [perf-discuss] NUMA ptools and ISM segments

jonathan chew wrote: It sounds like Eric Lowe has a theory as to why madvise(MADV_ACCESS_*) and pmadvise(1) didn't work for migrating your ISM segment. Joe and Nils are experts on the x86/AMD64 HAT and may be able to comment on Eric's theory that the lack of dynamic ISM unmap is preventing

Re: [perf-discuss] NUMA ptools and ISM segments

Joe Bonasera wrote: jonathan chew wrote: It sounds like Eric Lowe has a theory as to why madvise(MADV_ACCESS_*) and pmadvise(1) didn't work for migrating your ISM segment. Joe and Nils are experts on the x86/AMD64 HAT and may be able to comment on Eric's theory that the lack of dynamic I

Re: [perf-discuss] NUMA ptools and ISM segments

Eric, Bart, and Jonathan, Thanks for your quick replies. See my comments below:On 9/16/05, jonathan chew <[EMAIL PROTECTED]> wrote: Marc Rocas wrote:> I've been playing around with the tools on a Stinger box and I think > their pretty cool!I'm glad that you like them.  We like them too and think t

Re: [perf-discuss] NUMA ptools and ISM segments

PPS I forgot to ask what you did to test whether pmadvise(1) would migrate your ISM segment and how you know that it didn't. jonathan chew wrote: Marc Rocas wrote: I've been playing around with the tools on a Stinger box and I think their pretty cool! I'm glad that you like them. We l

Re: [perf-discuss] NUMA ptools and ISM segments

Marc Rocas wrote: I've been playing around with the tools on a Stinger box and I think their pretty cool! I'm glad that you like them. We like them too and think that they are fun to play with besides being useful for observability and experimenting with performance. I hope that our MPO o

Re: [perf-discuss] NUMA ptools and ISM segments

Marc Rocas wrote: I have an ISM segment that gets consumed by HW that is limited to 32-bit addressing and thus have a need to control the physical range that backs the segment. At this point, it would seem that I need to allocate the memory (about 300MB) in the kernel and map it back to user-

Re: [perf-discuss] NUMA ptools and ISM segments

On Thu, Sep 15, 2005 at 11:55:21PM -0400, Marc Rocas wrote: | | The one question I have is whether ISM (Intimate Shared Memory) segments are | immune to being coerced to relocate via pmadvise(3c)? I've tried it without | success. A quick look at the seg_spt.c code seemed to indicate that when an