PPS
I forgot to ask what you did to test whether pmadvise(1) would migrate
your ISM segment and how you know that it didn't.
jonathan chew wrote:
Marc Rocas wrote:
I've been playing around with the tools on a Stinger box and I think
their pretty cool!
I'm glad that you like them. We like them too and think that they are
fun to play with besides being useful for observability and
experimenting with performance. I hope that our MPO overview, man
pages, and our examples on how to use the tools were helpful.
The one question I have is whether ISM (Intimate Shared Memory)
segments are immune to being coerced to relocate via pmadvise(3c)?
I've tried it without success. A quick look at the seg_spt.c code
seemed to indicate that when an spt segment is created its lgroup
policy is set to LGRP_MEM_POLICY_DEFAULT that will result in
randomized allocation for segments greater than 8MB. I've verified as
much using the NUMA enhanced pmap(3c) command.
It sounds like Eric Lowe has a theory as to why madvise(MADV_ACCESS_*)
and pmadvise(1) didn't work for migrating your ISM segment. Joe and
Nils are experts on the x86/AMD64 HAT and may be able to comment on
Eric's theory that the lack of dynamic ISM unmap is preventing page
migration from working.
I'll see whether I can reproduce the problem.
Which version of Solaris are you using?
I have an ISM segment that gets consumed by HW that is limited to
32-bit addressing and thus have a need to control the physical range
that backs the segment. At this point, it would seem that I need to
allocate the memory (about 300MB) in the kernel and map it back to
user-land but I would lose the use of 2MB pages since I have not
quite figured out how to allocate memory using a particular page
size. Have I misinterpreted the code? Do I have other options?
Bart's suggestion is the simplest (brute force) way. Eric's
suggestion sounded a little painful. I have a couple of other options
below, but can't think of a nice way to specify that your application
needs low physical memory. So, I want to understand what you are
doing better to see if there is any better way.
Can you please tell me more about your situation and requirements?
What is the hardware that needs the 32-bit addressing (framebuffer,
device needing DMA, etc.)? Besides needing to be in low physical
memory, does it need to be wired down and shared?
Jonathan
PS
Assuming that you can change your code, here are a couple of other
options that are less painful than Eric's suggestion but still aren't
very elegant because of the low physical memory requirement:
- Use DISM instead of ISM which is Dynamic Intimate Shared Memory and
is pageable (see SHM_PAGEABLE flag to shmat(2))
OR
- Use mmap(2) and the MAP_ANON (and MAP_SHARED if you need shared
memory) flag to allocate (shared) anonymous memory
- Call memcntl(2) with MC_HAT_ADVISE to specify that you want large pages
AND
- Call madvise(MADV_ACCESS_LWP) on your mmap-ed or DISM segment to say
that the next thread to access it will use it a lot
- Access it from CPU 0 on your Stinger. I don't like this part
because this is hardware implementation specific. It turns out that
the low physical memory usually lives near CPU 0 on an Opteron box.
You can use liblgrp(3LIB) to discover which leaf lgroup contains CPU 0
and lgrp_affinity_set(3LGRP) to set a strong affinity for that lgroup
(which will set the home lgroup for the thread to that lgroup).
Alternatively, you can use processor_bind(2) to bind/unbind to CPU 0.
- Use the MC_LOCK flag to memcntl(2) to lock down the memory for your
segment if you want the physical memory to stay there (until you
unlock it)
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org
