Sergio, if you do some online searching about ‘bufferbloat’ in networking, you’ll find the background to help explain what motivates networking changes. Actual investigation of network performance can get a bit gnarly. The TL;DR summary is that big buffers function like big queues, and thus attempts to speed up throughput can cause things stuck in a queue to have higher latency. With very fast networks, there isn’t as much need to have big buffers. Imagine having a coordinator node waiting to respond to a query but can’t because a bunch of merkel trees are sitting in the tcp buffer waiting to be sent out. Sometimes total latency doesn’t fairly measure actual effort to do the work, some of that can be time spent sitting waiting in the buffer to be shipped out back to the client.
From: Sergio <lapostadiser...@gmail.com> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Monday, October 21, 2019 at 4:54 PM To: "user@cassandra.apache.org" <user@cassandra.apache.org> Subject: Re: Cassandra Recommended System Settings Message from External Sender Thanks Elliott! How do you know if there is too much RAM used for those settings? Which metrics do you keep track of? What would you recommend instead? Best, Sergio On Mon, Oct 21, 2019, 1:41 PM Elliott Sims <elli...@backblaze.com<mailto:elli...@backblaze.com>> wrote: Based on my experiences, if you have a new enough kernel I'd strongly suggest switching the TCP scheduler algorithm to BBR. I've found the rest tend to be extremely sensitive to even small amounts of packet loss among cluster members where BBR holds up well. High ulimits for basically everything are probably a good idea, although "unlimited" may not be purely optimal for all cases. The TCP keepalive settings are probably only necessary for traffic buggy/misconfigured firewalls, but shouldn't really do any harm on a modern fast network. The TCP memory settings are pretty aggressive and probably result in unnecessary RAM usage. The net.core.rmem_default/net.core.wmem_default settings are overridden by the TCP-specific settings as far as I know, so they're not really relevant/helpful for Cassandra The net.ipv4.tcp_rmem/net.ipv4.tcp_wmem max settings are pretty aggressive. That works out to something like 1Gbps with 130ms latency per TCP connection, but on a local LAN with latencies <1ms it's enough buffer for over 100Gbps per TCP session. A much smaller value will probably make more sense for most setups. On Mon, Oct 21, 2019 at 10:21 AM Sergio <lapostadiser...@gmail.com<mailto:lapostadiser...@gmail.com>> wrote: Hello! This is the kernel that I am using Linux 4.16.13-1.el7.elrepo.x86_64 #1 SMP Wed May 30 14:31:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux Best, Sergio Il giorno lun 21 ott 2019 alle ore 07:30 Reid Pinchback <rpinchb...@tripadvisor.com<mailto:rpinchb...@tripadvisor.com>> ha scritto: I don't know which distro and version you are using, but watch out for surprises in what vm.swappiness=0 means. In older kernels it means "only use swap when desperate". I believe that newer kernels changed to have 1 mean that, and 0 means to always use the oomkiller. Neither situation is strictly good or bad, what matters is what you intend the system behavior to be in comparison with whatever monitoring/alerting you have put in place. R On 10/18/19, 9:04 PM, "Sergio Bilello" <lapostadiser...@gmail.com<mailto:lapostadiser...@gmail.com>> wrote: Message from External Sender Hello everyone! Do you have any setting that you would change or tweak from the below list? sudo cat /proc/4379/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size unlimited unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 32768 32768 processes Max open files 1048576 1048576 files Max locked memory unlimited unlimited bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals unlimited unlimited signals Max msgqueue size unlimited unlimited bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us These are the sysctl settings default['cassandra']['sysctl'] = { 'net.ipv4.tcp_keepalive_time' => 60, 'net.ipv4.tcp_keepalive_probes' => 3, 'net.ipv4.tcp_keepalive_intvl' => 10, 'net.core.rmem_max' => 16777216, 'net.core.wmem_max' => 16777216, 'net.core.rmem_default' => 16777216, 'net.core.wmem_default' => 16777216, 'net.core.optmem_max' => 40960, 'net.ipv4.tcp_rmem' => '4096 87380 16777216', 'net.ipv4.tcp_wmem' => '4096 65536 16777216', 'net.ipv4.ip_local_port_range' => '10000 65535', 'net.ipv4.tcp_window_scaling' => 1, 'net.core.netdev_max_backlog' => 2500, 'net.core.somaxconn' => 65000, 'vm.max_map_count' => 1048575, 'vm.swappiness' => 0 } Am I missing something else? Do you have any experience to configure CENTOS 7 for JAVA HUGE PAGES https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_dse_5.1_dse-2Dadmin_datastax-5Fenterprise_config_configRecommendedSettings.html-23CheckJavaHugepagessettings&d=DwIBaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=zke-WpkD1c6Qt1cz8mJG0ZQ37h8kezqknMSnerQhXuU&s=b6lGdbtv1SN9opBsIOFRT6IX6BroMW-8Tudk9qEh3bI&e= OPTIMIZE SSD https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_dse_5.1_dse-2Dadmin_datastax-5Fenterprise_config_configRecommendedSettings.html-23OptimizeSSDs&d=DwIBaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=zke-WpkD1c6Qt1cz8mJG0ZQ37h8kezqknMSnerQhXuU&s=c0S3S3V_0YHVMx2I-pyOh24MiQs1D-L73JytaSw648M&e= https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_dse_5.1_dse-2Dadmin_datastax-5Fenterprise_config_configRecommendedSettings.html&d=DwIBaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=zke-WpkD1c6Qt1cz8mJG0ZQ37h8kezqknMSnerQhXuU&s=PZFG6SXF6dL5LRJ-aUoidHnnLGpKPbpxdKstM8M9JMk&e= We are using AWS i3.xlarge instances Thanks, Sergio --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org<mailto:user-unsubscr...@cassandra.apache.org> For additional commands, e-mail: user-h...@cassandra.apache.org<mailto:user-h...@cassandra.apache.org>
