I don’t see jemalloc anywhere in your setup scripts. Preferably use the latest upstream jemalloc version available.
Ondřej -- Ondřej Surý — ISC (He/Him) My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours. > On 2. 8. 2022, at 17:14, Dmitri Pavlov <dpav...@perforce.com> wrote: > > Hi, > > Thank you very much for your feedback, Ondrej. > > Sharing the steps. Very simple: configure -> make -> make install. Very > simple configuration. Just the zone file is big. Please, see the attached. > > 1. I followed the instructions from here > https://bind9.readthedocs.io/en/v9_18_4/chapter10.html > No git. > > 2. Rocky 9 > uname -a > Linux server.test.zone 5.14.0-70.17.1.el9_0.x86_64 #1 SMP PREEMPT Wed Jul 13 > 18:23:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux > Package gcc-11.2.1-9.4.el9.x86_64 > > 9.18.5 > ============= OUTPUT ============= > pmap -x 27821 | tail -n 1 > total kB 34853832 29425312 29416180 > > smemstat -p named > PID Swap USS PSS RSS User Command > 27821 0.0 B 28.1 G 28.1 G 28.1 G root > /usr/local/sbin/named > Total: 0.0 B 28.1 G 28.1 G 28.1 > ============= ============= ============= > > 9.16.21 > ============= OUTPUT ============= > pmap -x 18026 | tail -n 1 > total kB 28161152 27414164 27406208 > > smemstat -p named > PID Swap USS PSS RSS User Command > 18026 0.0 B 26.1 G 26.1 G 26.1 G root > /usr/local/sbin/named > Total: 0.0 B 26.1 G 26.1 G 26.1 G > ============= ============= ============= > > 2 GB diff is still there. > > Steps have been attached. > > Will be very grateful, could you please, have a look at the steps provided? > They are quite straightforward and self-explanatory. What is that "magic > souse" I should apply, when there is Rocky 9 or RHEL 9 machine in front of > me. I believe if I replicate your setup/steps, most likely I'll get the same > output. However, if looking at the articles available to the wide audience > (like this https://bind9.readthedocs.io/en/v9_18_4/chapter10.html or any > other(?)), what should my steps be to get the expected smaller memory > footprint. Or maybe there are obvious flaws in the experiment? > > Thanking you very much for patience and cooperation, > Dmitri. > > > > > -----Original Message----- > From: Ondřej Surý <ond...@isc.org> > Sent: Monday, August 1, 2022 8:18 PM > To: Dmitri Pavlov <dpav...@perforce.com> > Cc: bind-users@lists.isc.org > Subject: Re: High memory consumption in bind 9.18.2 > > $ wc -l gen.db > 100000000 gen.db > > generated with: > > #!/bin/env python3 > > for x in range(0, 10000): > for y in range(0, 2500): > print(f"az{x}-{y} IN A 10.53.0.1") > print(f"bz{x}-{y} IN A 10.53.0.2") > print(f"ca{x}-{y} IN A 10.53.0.3") > print(f"xh{x}-{y} IN CNAME az{x}-{y}”) > > ## BIND 9.16(git) > > 01-Aug-2022 19:06:27.151 starting BIND 9.16.32-dev (Extended Support Version) > <id:c863061a13> > 01-Aug-2022 19:06:27.151 running on Linux x86_64 5.10.0-16-amd64 #1 SMP > Debian 5.10.127-1 (2022-06-30) > 01-Aug-2022 19:06:27.151 built with 'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast -ggdb > -Wno-deprecated-declarations -fno-omit-frame-pointer > -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 > -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' > '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' > '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' > '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:' > 01-Aug-2022 19:06:27.151 running as: named -c named.conf -g -p 12345 > 01-Aug-2022 19:06:27.151 compiled by GCC 12.0.1 20220319 (experimental) > [master r12-7719-g8ca61ad148f] > 01-Aug-2022 19:06:27.151 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul > 2022 > 01-Aug-2022 19:06:27.151 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022 > 01-Aug-2022 19:06:27.151 compiled with libxml2 version: 2.9.10 > 01-Aug-2022 19:06:27.151 linked to libxml2 version: 20910 > 01-Aug-2022 19:06:27.151 compiled with json-c version: 0.15 > 01-Aug-2022 19:06:27.151 linked to json-c version: 0.15 > 01-Aug-2022 19:06:27.151 compiled with zlib version: 1.2.11 > 01-Aug-2022 19:06:27.151 linked to zlib version: 1.2.11 > > $ smem -P name[d] -a > SWAP:193304 USS:30451812 PSS:30452272 RSS:30454872 > > ## BIND 9.18(git) > > 01-Aug-2022 18:12:57.227 starting BIND 9.18.6-dev (Stable Release) > <id:097a57e> > 01-Aug-2022 18:12:57.227 running on Linux x86_64 5.10.0-16-amd64 #1 SMP > Debian 5.10.127-1 (2022-06-30) > 01-Aug-2022 18:12:57.227 built with 'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast -ggdb > -Wno-deprecated-declarations -fno-omit-frame-pointer > -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 > -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' > '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' > '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' > '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:' > 01-Aug-2022 18:12:57.227 running as: named -c named.conf -g -p 12345 > 01-Aug-2022 18:12:57.227 compiled by GCC 12.0.1 20220319 (experimental) > [master r12-7719-g8ca61ad148f] > 01-Aug-2022 18:12:57.227 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul > 2022 > 01-Aug-2022 18:12:57.227 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022 > 01-Aug-2022 18:12:57.227 compiled with libxml2 version: 2.9.10 > 01-Aug-2022 18:12:57.227 linked to libxml2 version: 20910 > 01-Aug-2022 18:12:57.227 compiled with json-c version: 0.15 > 01-Aug-2022 18:12:57.227 linked to json-c version: 0.15 > 01-Aug-2022 18:12:57.227 compiled with zlib version: 1.2.11 > 01-Aug-2022 18:12:57.227 linked to zlib version: 1.2.11 > > $ smem -P name[d] -a > SWAP:645960 USS:29446788 PSS:29447529 RSS:29451056 > > ## BIND 9.19(git) > > 01-Aug-2022 18:18:20.127 starting BIND 9.19.4-dev (Development Release) > <id:fa1f35b> > 01-Aug-2022 18:18:20.127 running on Linux x86_64 5.10.0-16-amd64 #1 SMP > Debian 5.10.127-1 (2022-06-30) > 01-Aug-2022 18:18:20.127 built with 'CC=gcc-12' 'LD=' 'CFLAGS=-Ofast -ggdb > -Wno-deprecated-declarations -fno-omit-frame-pointer > -fno-optimize-sibling-calls -DISC_MEM_USE_INTERNAL_MALLOC=0 > -DISC_MEM_TRACKLINES=1' 'LDFLAGS=' '--enable-developer' '--enable-warn-error' > '--with-openssl' '--with-zlib' '--with-libxml2' '--with-json-c' > '--with-readline' '--with-libidn2' '--enable-dnstap' '--with-libtool' > '--without-make-clean' 'PKG_CONFIG_PATH=/home/ondrej/.local/lib/pkgconfig:' > 01-Aug-2022 18:18:20.127 running as: named -c named.conf -g -p 12345 > 01-Aug-2022 18:18:20.127 compiled by GCC 12.0.1 20220319 (experimental) > [master r12-7719-g8ca61ad148f] > 01-Aug-2022 18:18:20.127 compiled with OpenSSL version: OpenSSL 3.0.5 5 Jul > 2022 > 01-Aug-2022 18:18:20.127 linked to OpenSSL version: OpenSSL 3.0.5 5 Jul 2022 > 01-Aug-2022 18:18:20.127 compiled with libxml2 version: 2.9.10 > 01-Aug-2022 18:18:20.127 linked to libxml2 version: 20910 > 01-Aug-2022 18:18:20.127 compiled with json-c version: 0.15 > 01-Aug-2022 18:18:20.127 linked to json-c version: 0.15 > 01-Aug-2022 18:18:20.127 compiled with zlib version: 1.2.11 > 01-Aug-2022 18:18:20.127 linked to zlib version: 1.2.11 > > $ smem -P name[d] -a > SWAP:1032108 USS:29062164 PSS:29062968 RSS:29066580 > > So, it’s (USS+Swap): > 9.16(git): 30645116 > 9.18(git): 30092748 > 9.19(git): 30094272 > > So, no, I am unable to reproduce the results and yet again, I see a lower > memory usage with 9.18+. > > But I see a common pattern here. I think both you and the OP were using > CentOS/RHEL 7 which is using GCC 4.8. > > GCC was improved significantly since then. I would suggest to repeat the > experiment on RHEL 9 if you can reproduce the same results. > > Ondrej > -- > Ondřej Surý (He/Him) > ond...@isc.org > > My working hours and your working hours may be different. Please do not feel > obligated to reply outside your normal working hours. > >>> On 1. 8. 2022, at 17:46, Ondřej Surý <ond...@isc.org> wrote: >>> >>>> On 1. 8. 2022, at 17:19, Dmitri Pavlov <dpav...@perforce.com> wrote: >>> >>> I’m pretty much sure you will get the same results in your lab. >> >> I don’t want to delve into vague description of your experiment. >> You’ll have to share the exact scripts. >> >> Even this: >> >>> just the time-consuming task is to generate 100 MIL A + CNAME dummy records >>> into the zone file. >> >> could be violently different. Are those random names? Or >> deterministically generated? What are the CNAME targets? >> >> What is the exact `named.conf`? >> >> What are the exact library versions? What is the exact ./configure >> invocation? >> >> How do you execute `named`? >> >> And yet again - the memory statistics from the statschannel could show >> the breakdown of the internal memory contexts and also show if there’s >> a difference between the internally reported memory usage and >> externally reported memory usage, e.g. is this something allocated in named, >> or is this allocated in the libraries? >> >>> Should the memory reduction apply to our experiment? >> >> The question doesn’t really make sense. We have not measured any >> increase in our test scenarios, which doesn’t mean you can’t find different >> scenarios with a memory increase. >> >> Ondrej >> -- >> Ondřej Surý (He/Him) >> ond...@isc.org >> >> My working hours and your working hours may be different. Please do not feel >> obligated to reply outside your normal working hours. >> > > > > CAUTION: This email originated from outside of the organization. Do not click > on links or open attachments unless you recognize the sender and know the > content is safe. > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. >
<<attachment: STEPS.zip>>
-- Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
