Marc Rocas wrote:
I've been playing around with the tools on a Stinger box and I think
their pretty cool!
I'm glad that you like them. We like them too and think that they are
fun to play with besides being useful for observability and
experimenting with performance. I hope that our MPO overview, man
pages, and our examples on how to use the tools were helpful.
The one question I have is whether ISM (Intimate Shared Memory)
segments are immune to being coerced to relocate via pmadvise(3c)?
I've tried it without success. A quick look at the seg_spt.c code
seemed to indicate that when an spt segment is created its lgroup
policy is set to LGRP_MEM_POLICY_DEFAULT that will result in
randomized allocation for segments greater than 8MB. I've verified as
much using the NUMA enhanced pmap(3c) command.
It sounds like Eric Lowe has a theory as to why madvise(MADV_ACCESS_*)
and pmadvise(1) didn't work for migrating your ISM segment. Joe and
Nils are experts on the x86/AMD64 HAT and may be able to comment on
Eric's theory that the lack of dynamic ISM unmap is preventing page
migration from working.
I'll see whether I can reproduce the problem.
Which version of Solaris are you using?
I have an ISM segment that gets consumed by HW that is limited to
32-bit addressing and thus have a need to control the physical range
that backs the segment. At this point, it would seem that I need to
allocate the memory (about 300MB) in the kernel and map it back to
user-land but I would lose the use of 2MB pages since I have not quite
figured out how to allocate memory using a particular page size. Have
I misinterpreted the code? Do I have other options?
Bart's suggestion is the simplest (brute force) way. Eric's suggestion
sounded a little painful. I have a couple of other options below, but
can't think of a nice way to specify that your application needs low
physical memory. So, I want to understand what you are doing better to
see if there is any better way.
Can you please tell me more about your situation and requirements? What
is the hardware that needs the 32-bit addressing (framebuffer, device
needing DMA, etc.)? Besides needing to be in low physical memory, does
it need to be wired down and shared?
Jonathan
PS
Assuming that you can change your code, here are a couple of other
options that are less painful than Eric's suggestion but still aren't
very elegant because of the low physical memory requirement:
- Use DISM instead of ISM which is Dynamic Intimate Shared Memory and is
pageable (see SHM_PAGEABLE flag to shmat(2))
OR
- Use mmap(2) and the MAP_ANON (and MAP_SHARED if you need shared
memory) flag to allocate (shared) anonymous memory
- Call memcntl(2) with MC_HAT_ADVISE to specify that you want large pages
AND
- Call madvise(MADV_ACCESS_LWP) on your mmap-ed or DISM segment to say
that the next thread to access it will use it a lot
- Access it from CPU 0 on your Stinger. I don't like this part because
this is hardware implementation specific. It turns out that the low
physical memory usually lives near CPU 0 on an Opteron box. You can use
liblgrp(3LIB) to discover which leaf lgroup contains CPU 0 and
lgrp_affinity_set(3LGRP) to set a strong affinity for that lgroup (which
will set the home lgroup for the thread to that lgroup). Alternatively,
you can use processor_bind(2) to bind/unbind to CPU 0.
- Use the MC_LOCK flag to memcntl(2) to lock down the memory for your
segment if you want the physical memory to stay there (until you unlock it)
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org
