Using Redis (Was: Significant memory usage)

2025-07-02 Thread Ondřej Surý
Redis (nor any other nosql database) doesn’t really have the properties we need for finding closest enclosers, doing partial matches nor really the performance.Ondrej --Ondřej Surý — ISC (He/Him)My working hours and your working hours may be different. Please do not feel obligated to reply outside

Re: Significant memory usage

2025-07-02 Thread Matthias Fechner
Am 02.07.2025 um 09:46 schrieb Doug Freed: You are correct; the syntax of response-policy is very unique. response-policy {   zone "foo" ; } ; It is the semicolon which ends a statement, not the closing curly bracket of a block, which is why all blocks have to end in a semicolon too. thank

Re: Significant memory usage

2025-07-02 Thread Carlos Horowicz via bind-users
Ondřej By the way, have you ever considered using Redis as an in-memory cache database? I’ve been thinking about offloading some of the TTL expiry and cache management to Redis. In some customer environments, the query volume is extremely high — we’re using Mellanox CX-6 25G

Re: Significant memory usage

2025-07-01 Thread Doug Freed
On 7/1/25 23:55, Lee wrote: On Tue, Jul 1, 2025 at 11:14 PM Matthias Fechner wrote: Am 01.07.2025 um 22:23 schrieb Lee: response-policy { zone "rpz.foo"; zone "rpz.bar"; zone "rpz.pgl"; } break-dnssec yes recursive-only no qname-wait-recurse no; should these 3 line

Re: Significant memory usage

2025-07-01 Thread Lee
On Tue, Jul 1, 2025 at 11:14 PM Matthias Fechner wrote: > > Am 01.07.2025 um 22:23 schrieb Lee: > >response-policy { zone "rpz.foo"; zone "rpz.bar"; zone "rpz.pgl"; } > > break-dnssec yes > > recursive-only no > > qname-wait-recurse no; > > should these 3 lines (break-dnssec

Re: Significant memory usage

2025-07-01 Thread Ondřej Surý
> On 2. 7. 2025, at 0:14, OwN-3m-All wrote: > > I wonder if other memory issues users are complaining about are related. I don’t know. You were the first one to actually provided a reproducer and a usable test case. Despite your exaggeration about “countless” reports there were not that many o

Re: Significant memory usage

2025-07-01 Thread Matthias Fechner
Am 01.07.2025 um 22:23 schrieb Lee: response-policy { zone "rpz.foo"; zone "rpz.bar"; zone "rpz.pgl"; } break-dnssec yes recursive-only no qname-wait-recurse no; should these 3 lines (break-dnssec , ...) not inside the response-policy block? Otherwise it is applied to the

Re: Significant memory usage

2025-07-01 Thread OwN-3m-All
Thank you Ondrej! I changed my scripts to apply the hosts in this manner moving forward. Everything appears to be working as before, with significantly less memory usage, which is awesome! Still, I think the memory usage for the way I had it setup before shouldn't drastically increase in new ver

Re: Significant memory usage

2025-07-01 Thread Ondřej Surý
Good point. As this is local setup, it makes much sense to use qname-wait-recurse no; this saves both time and bandwidth as this is of no concern (from documentation): > No DNS records are needed for a QNAME or Client-IP trigger; the name or IP > address itself is sufficient, so in principle the

Re: Significant memory usage

2025-07-01 Thread Carlos Horowicz via bind-users
Ondřej, I usually include *qname-wait-recurse no* after the *response-policy { ... } *block, hoping to avoid issues where SERVFAILs, lame delegations, or firewalled authoritative servers might interfere with RPZ responses. I’m not entirely sure if I’m just being a bit /superstitious/ about tha

Re: Significant memory usage

2025-07-01 Thread Lee
On Tue, Jul 1, 2025 at 2:33 PM OwN-3m-All wrote: > > No, I'm not asking you to prioritize anything. I'm just saying that > previously valid and memory performant setups are not performing well on the > newest versions of bind (using too much memory). c'est la vie > I created this setup based o

Re: Significant memory usage

2025-07-01 Thread OwN-3m-All
Also, 127.0.0.1 (localhost) needs to be returned for these hosts, not a NXDOMAIN response. Would that impact it? -- Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at ht

Re: Significant memory usage

2025-07-01 Thread Ondřej Surý
You'll have to experiment a bit (and I mean read the documentation[1]) as I am writing this from top of my head, 1. You need to create RPZ zone like this: $TTL 604800 $ORIGIN adaway.rpz. @ IN SOA localhost. root.localhost. (1 604800 86400 2419200 604800 ) @ IN NS localhost. ad-assets.futurecdn.n

Re: Significant memory usage

2025-07-01 Thread OwN-3m-All
No, I'm not asking you to prioritize anything. I'm just saying that previously valid and memory performant setups are not performing well on the newest versions of bind (using too much memory). I created this setup based on guides I found online. So, if this is not the proper way to do it, what

Re: Significant memory usage

2025-07-01 Thread Carlos Horowicz via bind-users
Apparently you have 295108 zones, maybe you can try one single rpz zone with all 295108 fqdn's like . 12724[.]xyz IN CNAME . 21736[.[xyz IN CNAME . . instead of one zone per fqdn, and see if the memory footprint changes (both VMEM and RES) Good luck! Carlos Horowicz Planisys On 0

Re: Significant memory usage

2025-07-01 Thread Ondřej Surý
Hi, thanks for providing a reproducer. Just to give some rough numbers for the various branches we have (9.18, 9.20 and development): BIND 9.18 (bind-9.18 branch HEAD) $ smem -P name[d] PID User Command Swap USS PSS RSS 450020 ondrej named0 3233560 3234515

Re: Significant memory usage

2025-07-01 Thread OwN-3m-All
>> Apologies if I misunderstood your setup. I’ve also encountered memory issues in recent BIND versions — BIND 9.18.33 on Debian 12 is a tremendous beast, capable of handling millions of QPS — but after reducing logging (including DNSTAP) and disabling serve-stale, I saw a significant improvement

Re: Significant memory usage

2025-07-01 Thread Carlos Horowicz via bind-users
Hello there, I’m not a BIND developer either, but I was intrigued when you mentioned /millions of zone entries/. Are you referring to millions of individual zones, rather than consolidating entries into a single RPZ zone? Apologies if I misunderstood your setup. I’ve also encountered memory

Re: Significant memory usage

2025-07-01 Thread OwN-3m-All
Can we quit pretending that the newest versions of bind aren't memory hogs? We shouldn't have to provide the technical details as to why the newest versions of bind use so much ram. We don't know. We're just end users. However, with millions of zone entries (used as an ad blocking DNS server) l

Re: Significant memory usage

2025-06-09 Thread Ondřej Surý
Ok. And what are your observations? Or do you expect us to debug your issue and interpret the outputs you send here for you? As a side note, the rndc outputs you are pasting into your emails are mostly useless to debug memory issues. Ondrej -- Ondřej Surý — ISC (He/Him) My working hours and y

Re: Significant memory usage

2025-06-09 Thread Philip Prindeville via bind-users
I’ve been running for 12+ hours with a max-cache-size of 256M (since I’m on a machine with 2GB that does a lot of data reduction as it’s a honeypot firewall). This is what I’ve collected. +++ Statistics Dump +++ (1749514002) ++ Incoming Requests ++ 203077 QUERY 12014

Re: Significant memory usage

2025-06-09 Thread Philip Prindeville via bind-users
I take it back, it does at the beginning of the manual. It would be helpful if all references to had links back to the explanation. Odd that when you specify a percentage the effective amount is logged by named, but when you specify an absolute amount it’s not. > On Jun 8, 2025, at 10:46 P

Re: Significant memory usage

2025-06-09 Thread Sten Carlsen
Actually it does https://bind9.readthedocs.io/en/v9.20.9/reference.html#term-sizeval -- Best regards Sten Carlsen No improvements come from shouting: "MALE BOVINE MANURE!!!" > On 9 Jun 2025, at 06.47, Philip Prindeville via bind-users > wrote: > > I read: > > https://bind9.readthedocs.io/en

Re: Significant memory usage

2025-06-08 Thread Philip Prindeville via bind-users
I read: https://bind9.readthedocs.io/en/v9.20.9/reference.html#namedconf-statement-max-cache-size and it doesn’t explain the notation for . > On Jun 8, 2025, at 10:39 PM, Ondřej Surý wrote: > > What If you actually read the manual that I sent you - syntax of sizeval is > explained there. >

Re: Significant memory usage

2025-06-08 Thread Ondřej Surý
What If you actually read the manual that I sent you - syntax of sizeval is explained there. -- Ondřej Surý — ISC (He/Him) My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours. > On 9. 6. 2025, at 6:34, Philip Prindevi

Re: Significant memory usage

2025-06-08 Thread Philip Prindeville via bind-users
Maybe GB is the only unit it groks. Jun 8 22:31:52 OpenWrt named[19145]: /etc/bind/named.conf:42: expected integer and optional unit or percent near ‘1536MB’ Nope: Jun 8 22:32:48 OpenWrt named[19609]: /etc/bind/named.conf:43: expected integer and optional unit or percent near ‘2GB' > On

Re: Significant memory usage

2025-06-08 Thread Ondřej Surý
It does have the effect. Also there’s BIND 9 ARM at https://bind9.readthedocs.io/en/v9.20.9/ -- Ondřej Surý — ISC (He/Him) My working hours and your working hours may be differentw . Please do not feel obligated to reply outside your normal working hours. > On 9. 6. 2025, at 6:20, Philip Prinde

Re: Significant memory usage

2025-06-08 Thread Philip Prindeville via bind-users
Jun 8 22:22:10 OpenWrt named[15142]: /etc/bind/named.conf:42: expected integer and optional unit or percent near '1638MB' > On Jun 8, 2025, at 10:17 PM, Ondřej Surý wrote: > > Yes, there's no math involved, it just honors the limit. > > FTR you can also say: > > max-cache-size 2GB; > > You

Re: Significant memory usage

2025-06-08 Thread Philip Prindeville via bind-users
I’ll try to get a smoking gun. How do you configure an explicit number of bytes with max-cache-size? The manpage says: max-cache-size ( default | unlimited | | ); but doesn’t explain the syntax of “sizeval”. I tried “1638M” but that doesn’t seem to have an effect. > On Jun 8, 2025, at 10

Re: Significant memory usage

2025-06-08 Thread Ondřej Surý
Yes, there's no math involved, it just honors the limit. FTR you can also say: max-cache-size 2GB; You don't have to specify it to the last byte. Ondrej -- Ondřej Surý (He/Him) ond...@isc.org My working hours and your working hours may be different. Please do not feel obligated to reply outsi

Re: Significant memory usage

2025-06-08 Thread Ondřej Surý
I don't see anything wrong with the memory in the attached file - 13MB doesn't seem to be causing any havoc. And it roughly matches what I am seeing here with fresh named instance on 64-bit machine: $ smem -P name[d] PID User Command Swap USS PSS RSS

Re: Significant memory usage

2025-06-08 Thread Philip Prindeville via bind-users
Odd. I tried: max-cache-size 1717986918; and restarted and I don’t see anything in the logs about it. But I did when I used a percentage. > On Jun 8, 2025, at 10:02 PM, Ondřej Surý wrote: > > The 1.7GB is what the system is reporting. That’s why I asked as I’ve seen > OpenWRT repo

Re: Significant memory usage

2025-06-08 Thread Ondřej Surý
The 1.7GB is what the system is reporting. That’s why I asked as I’ve seen OpenWRT reporting weird or no values before. 171MB cache is little on a low side and negative effects from overmem LRU cleaning will going to hurt the performance. I would suggest to set a fixed size for the cache - 1.6G

Re: Significant memory usage

2025-06-08 Thread Philip Prindeville via bind-users
This is on an embedded system, i.e. a 4-core AMD64 low-power machine with 16GB of memory, that uses 2GB of that as a tmpfs. 90% would cripple the system. I’m going to try 10% (after all, it’s only doing name service for 200 machines, maybe 450 RRs, and more than have of the machines are IoTs t

Re: Significant memory usage

2025-06-08 Thread Philip Prindeville via bind-users
Working on it: https://github.com/openwrt/packages/pull/26721 Here’s my statistics-channel output: named-stats.xml Description: XML document > On May 18, 2025, at 10:30 PM, Ondřej Surý wrote: > > Well, you’ve provided basically nothing as leads, so it is hard to tell > what’s going on w

Re: Significant memory usage

2025-06-08 Thread Ondřej Surý
Does the named report proper max-cache-size into the log when starting? Something like: 'max-cache- size 90%' - setting to 86522MB (out of 96136MB) Ondrej -- Ondřej Surý — ISC (He/Him) My working hours and your working hours may be different. Please do not feel obligated to reply outside your

Re: Significant memory usage

2025-06-08 Thread Philip Prindeville via bind-users
bind-users" >>> Sent: Sunday, May 18, 2025 5:20:59 PM >>> Subject: Significant memory usage >> >>> What I’ve noticed is that at startup I’m using about 33K pages as the VSZ >>> (per >>> top on x86_64 hardware). >> >> VSZ (virtual

Re: Significant memory usage

2025-06-08 Thread Philip Prindeville via bind-users
> On May 21, 2025, at 3:38 PM, Ben Scott wrote: > > - Original Message - >> From: "Philip Prindeville via bind-users" >> To: "bind-users" >> Sent: Sunday, May 18, 2025 5:20:59 PM >> Subject: Significant memory usage > >>

Re: Significant memory usage

2025-06-08 Thread Philip Prindeville via bind-users
> On May 21, 2025, at 3:38 PM, Ben Scott wrote: > > - Original Message - >> From: "Philip Prindeville via bind-users" >> To: "bind-users" >> Sent: Sunday, May 18, 2025 5:20:59 PM >> Subject: Significant memory usage > >>

Re: Significant memory usage

2025-05-22 Thread Petr Špaček
On 5/21/25 23:38, Ben Scott wrote: What I’ve noticed is that at startup I’m using about 33K pages as the VSZ (per top on x86_64 hardware). VSZ (virtual size) just counts the number of virtual memory pages associated with the process in some way. That includes RAM, but also memory mapped fi

Re: Significant memory usage

2025-05-21 Thread Ben Scott
- Original Message - > From: "Philip Prindeville via bind-users" > To: "bind-users" > Sent: Sunday, May 18, 2025 5:20:59 PM > Subject: Significant memory usage > What I’ve noticed is that at startup I’m using about 33K pages as the VSZ (per > to

Re: Significant memory usage

2025-05-18 Thread Ondřej Surý
Well, you’ve provided basically nothing as leads, so it is hard to tell what’s going on with just output of named -V. I would suggest to recompile names with jemalloc enabled and then use jemalloc profiling to see where the memory goes. See https://www.isc.org/blogs/2023-BIND-memory-management-

Significant memory usage

2025-05-18 Thread Philip Prindeville via bind-users
Hi, I’ve been running 9.20.7 on OpenWRT for over a year without issues. Yesterday, however, it died because of OOM (out-of-memory) kill from the kernel. I use it with isc-dhcp service for IPv4, and do split horizon for internal domains. What I’ve noticed is that at startup I’m using about 33K