Hello again,
> It's possible that glibc is creating too many memory arenas. Are you
> setting/exporting MALLOC_ARENA_MAX to something sane before calling
> the JVM? You can check that in /proc/<pid>/environ.
I checked and we have the default value of 4.
> I would also turn on -XX:NativeMemoryTracking=summary and use jcmd to
> check out native memory usage from the JVM's perspective.
>
> -Mark
I have turned on `-XX:NativeMemoryTracking=summary` and have more
information to share.
Right now one of my cassandra nodes is using `22467424` of RSS memory
(around 22GB) according to `ps`.
`nodetool info` tells me it should be using around 17GB :
```
Heap Memory (MB) : 3129.20 / 12288.00
Off Heap Memory (MB) : 5165.15
```
Finally this is the output of the jcmd :
```
sudo -u cassandra jcmd 30952 VM.native_memory summary.diff
30952:
Native Memory Tracking:
Total: reserved=15534476KB +402527KB, committed=14271664KB +418143KB
- Java Heap (reserved=12582912KB, committed=12582912KB)
(mmap: reserved=12582912KB, committed=12582912KB)
- Class (reserved=1120624KB +30810KB,
committed=75960KB +29914KB)
(classes #7034 +231)
(malloc=31088KB +26714KB #166677 +123037)
(mmap: reserved=1089536KB +4096KB,
committed=44872KB +3200KB)
- Thread (reserved=184281KB +37799KB,
committed=184281KB +37799KB)
(thread #627 +144)
(stack: reserved=182728KB +37440KB,
committed=182728KB +37440KB)
(malloc=818KB +190KB #3142 +720)
(arena=735KB +169 #1253 +288)
- Code (reserved=260864KB +3698KB,
committed=67296KB +20210KB)
(malloc=11264KB +3698KB #14716 +4048)
(mmap: reserved=249600KB,
committed=56032KB +16512KB)
- GC (reserved=676609KB +121116KB,
committed=676609KB +121116KB)
(malloc=176897KB +121116KB #163106 +113361)
(mmap: reserved=499712KB, committed=499712KB)
- Compiler (reserved=2017KB +728KB, committed=2017KB +728KB)
(malloc=1887KB +728KB #3621 +1189)
(arena=131KB #3)
- Internal (reserved=654807KB +203186KB,
committed=654803KB +203186KB)
(malloc=654803KB +203186KB #284904 +219680)
(mmap: reserved=4KB, committed=0KB)
- Symbol (reserved=16381KB +140KB, committed=16381KB +140KB)
(malloc=14582KB +140KB #65637 +989)
(arena=1798KB #1)
- Native Memory Tracking (reserved=11060KB +7254KB,
committed=11060KB +7254KB)
(malloc=63KB +14KB #685 +148)
(tracking overhead=10997KB +7240KB)
- Arena Chunk (reserved=312KB -2205KB, committed=312KB -2205KB)
(malloc=312KB -2205KB)
- Unknown (reserved=24608KB, committed=32KB)
(mmap: reserved=24608KB, committed=32KB)
```
I made a baseline shortly after restarting the cassandra node.
Unless I am mistaken somewhere this result is very surprising, the jvm
is using memory that the native memory tracking doesn't see ?
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
