Re: [ceph-users] osd_memory_target exceeding on Luminous OSD BlueStore

2019-04-10 Thread Mark Nelson
In fact the autotuner does it itself every time it tunes the cache size: https://github.com/ceph/ceph/blob/master/src/os/bluestore/BlueStore.cc#L3630 Mark On 4/10/19 2:53 AM, Frédéric Nass wrote: Hi everyone, So if the kernel is able to reclaim those pages, is there still a point in runni

Re: [ceph-users] osd_memory_target exceeding on Luminous OSD BlueStore

2019-04-10 Thread Frédéric Nass
Hi everyone, So if the kernel is able to reclaim those pages, is there still a point in running the heap release on a regular basis? Regards, Frédéric. Le 09/04/2019 à 19:33, Olivier Bonvalet a écrit : Good point, thanks ! By making memory pressure (by playing with vm.min_free_kbytes), memo

Re: [ceph-users] osd_memory_target exceeding on Luminous OSD BlueStore

2019-04-09 Thread Olivier Bonvalet
Good point, thanks ! By making memory pressure (by playing with vm.min_free_kbytes), memory is freed by the kernel. So I think I essentially need to update monitoring rules, to avoid false positive. Thanks, I continue to read your resources. Le mardi 09 avril 2019 à 09:30 -0500, Mark Nelson a

Re: [ceph-users] osd_memory_target exceeding on Luminous OSD BlueStore

2019-04-09 Thread Mark Nelson
My understanding is that basically the kernel is either unable or uninterested (maybe due to lack of memory pressure?) in reclaiming the memory .  It's possible you might have better behavior if you set /sys/kernel/mm/khugepaged/max_ptes_none to a low value (maybe 0) or maybe disable transparen

Re: [ceph-users] osd_memory_target exceeding on Luminous OSD BlueStore

2019-04-09 Thread Olivier Bonvalet
Well, Dan seems to be right : _tune_cache_size target: 4294967296 heap: 6514409472 unmapped: 2267537408 mapped: 4246872064 old cache_size: 2845396873 new cache size: 2845397085 So we have 6GB in heap, but "only" 4GB mapped. But "ceph tell osd.* heap release" shou

Re: [ceph-users] osd_memory_target exceeding on Luminous OSD BlueStore

2019-04-09 Thread Olivier Bonvalet
Thanks for the advice, we are using Debian 9 (stretch), with a custom Linux kernel 4.14. But "heap release" didn't help. Le lundi 08 avril 2019 à 12:18 +0200, Dan van der Ster a écrit : > Which OS are you using? > With CentOS we find that the heap is not always automatically > released. (You can

Re: [ceph-users] osd_memory_target exceeding on Luminous OSD BlueStore

2019-04-08 Thread Mark Nelson
One of the difficulties with the osd_memory_target work is that we can't tune based on the RSS memory usage of the process. Ultimately it's up to the kernel to decide to reclaim memory and especially with transparent huge pages it's tough to judge what the kernel is going to do even if memory h

Re: [ceph-users] osd_memory_target exceeding on Luminous OSD BlueStore

2019-04-08 Thread Dan van der Ster
Which OS are you using? With CentOS we find that the heap is not always automatically released. (You can check the heap freelist with `ceph tell osd.0 heap stats`). As a workaround we run this hourly: ceph tell mon.* heap release ceph tell osd.* heap release ceph tell mds.* heap release -- Dan O

[ceph-users] osd_memory_target exceeding on Luminous OSD BlueStore

2019-04-06 Thread Olivier Bonvalet
Hi, on a Luminous 12.2.11 deploiement, my bluestore OSD exceed the osd_memory_target : daevel-ob@ssdr712h:~$ ps auxw | grep ceph-osd ceph3646 17.1 12.0 6828916 5893136 ? Ssl mars29 1903:42 /usr/bin/ceph-osd -f --cluster ceph --id 143 --setuser ceph --setgroup ceph ceph3991 1