Sharing zones between views to conserve memory
Hi
I've come across an interesting scalability issue with regards to how
our organization uses BIND. I'm putting up the question here, but I
have a sneaky suspicion that I'll have to solve this problem in the
source code. The way we use BIND seems to be slightly non-obvious, and
I'm really after any creative or bright ideas that would help me in
reducing memory use, but still keeping the functionality there.
The use case:
We have one BIND server (running 9.9) that serves a number of
subdivisions within our company. Each subdivision has a response
policy that refers to a few "category" zones (of which there are about
90). The issue is, that these zones are LARGE and almost impossible to
clean up. There's a huge commonality between subdivisions and zones,
however (many subdivisions use the largest zone file with 1.7m entries
or so).
The current setup:
I've set up BIND to do something like this:
view "subdivision-01" {
match-clients { 10.0.1.0/24; };
zone "category-01" {type master; file "/etc/bind/cat01.rpz";
allow-query { none; }; };
...
zone "category-90" {type master; file "/etc/bind/cat90.rpz"; allow-query
zone "whitelist" {type master; file "/etc/bind/whitelist.rpz"; allow-query
{ none; }; };
response-policy {
zone "whitelist" policy no-op;
zone "category-01" policy CNAME policy.example.com;
...
zone "category-90" policy CNAME policy.example.com;
};
include "/etc/bind/named.conf.default-zones";
include "/etc/bind/zones.rfc1918";
};
I normally make a copy of that setup for each subdivision, omitting
the categories that aren't used in the subdivision.
The problem:
Each subdivision added in this way, increases BIND's memory use by
almost a gigabyte! Most subdivisions use the category with 1.7m
entries, and that's a big culprit. It looks a lot like BIND makes a
new copy of the zone per view. Making the memory requirements an order
of magnitude less for the zone, or making it independent of the number
of views would both solve the problem.
So, here's my question: is there a way to share zones between views to
conserve memory? The hypothetical way would be to put the policy zones
in a "common" view, and just set response-policy uniquely for each
view. Like I said, I have a sneaky suspicion that it's not possible
with 9.9...
Any other bright ideas, suggestions or general flames would be
welcome! I'm not married to this specific setup in the least: any tool
that does the job is valid.
As an off-topic aside, it seems that the policy no-op statement only
works in bind 9.9, and not 9.8. I'm running Ubuntu 12.04 (pity me!)
and 9.8 seems to have slight issues with a response-policy
Thanks for reading this far, I feel I've succeeded in my goal if I've
caused at least one sysadmin or developer to overheat for a couple of
minutes.
Jan Gutter
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
from this list
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: Sharing zones between views to conserve memory
Thanks for the suggestions!
I'm currently investigating two options: the local view and forwarded
zones, and I'm going to check out if I can write a fast DLZ lookup to
share the RPZ zones between the views. Caching is not a big problem
here, the "shared zones" should only change about once per month.
I'm having a little trouble with the first option though. I changed my
config to look like this:
view "rpz" { //(master rpz view, zone files loaded into memory only once)
match-clients { 127.0.0.1/32; };
zone "cat01.rpz" {type master; file
"/etc/bind/category-zones/cat01.rpz"; };
...
zone "cat99.rpz" {type master; file
"/etc/bind/category-zones/cat99.rpz"; };
};
view "subdivision-01" { //subdivisions forward to master rpz view
match-clients { 192.168.1.0/24; };
zone "cat01.rpz" {type forward; forward only; forwarders {
127.0.0.1; }; };
...
zone "cat99.rpz" {type forward; forward only; forwarders {
127.0.0.1; }; };
zone "custom-whitelist.rpz" {type master; file
"/etc/bind/override-zones/subdivision-01-whitelist.rpz"; allow-query {
none; }; };
zone "custom-blacklist.rpz" {type master; file
"/etc/bind/override-zones/subdivision-01-blacklist.rpz"; allow-query {
none; }; };
response-policy {
zone "custom-whitelist.rpz" policy no-op;
zone "custom-blacklist.rpz" policy CNAME policy.example.com;
zone "cat01.rpz" policy CNAME policy.example.com;
...
zone "cat99.rpz" policy CNAME policy.example.com;
};
include "/etc/bind/named.conf.default-zones";
include "/etc/bind/zones.rfc1918";
};
So, named-checkconf validates the config file OK, but when I try to
start bind, I get an error:
unknown zone 'cat01.rpz'
I've disabled DNSSEC and other stuff for testing, and I verified if I
take out the response-policy section, that I can query the RPZ zone
and it forwards correctly (host example-entry.com.cat01.rpz works
fine, on the DNS server and on the client).
However, it seems RPZ doesn't like "forward" type zones in the
response-policy stanza. I have a nasty feeling I'm missing something
obvious, though.
My next bright idea is to turn all the zone entries back into type
master, and try to figure out if I can write a simple, fast, DLZ
entry. I'm not planning to use something like postgresql as a backend:
a memory cache should work fine. If loads all the millions of zone
entries ONCE, I don't mind it taking up a lot of RAM.
Again, thanks for your suggestions, every lateral idea helps! In the
end the solution might be something stupid, or take a lot of work.
Either way, I'm grateful!
Jan Gutter
On Thu, Jan 10, 2013 at 1:37 AM, Sten Carlsen wrote:
> IIRC provided you do NOT update the common zones dynamically, you can share
> the files. This is a dirty solution, the risk is that on e view may change a
> file and the other views using it will be out of sync.
>
>
>
> On 10/01/13 0:34, Kevin Darcy wrote:
>
> On 1/9/2013 10:57 AM, Carl Byington wrote:
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On Wed, 2013-01-09 at 14:37 +0200, Jan Gutter wrote:
>
> So, here's my question: is there a way to share zones between views to
> conserve memory?
>
> One way is to put the master copy of those large zones in one view, then
> define those zones in the other views as type 'forward', forwarding to
> localhost.
>
> Yeah, I've done that before, but it has some undesirable caching behavior,
> i.e. entries get cached in the view which forwards, so clients can get
> responses from data that is not as "fresh" as they might get if the data
> were authoritative (assuming NOTIFY is in place, or some sort of fast
> replication mechanism). Also, some things care whether the AA flag is set in
> responses.
>
> I've also been told that at least one BIND-based DNS management system, to
> which I'm planning to migrate, will divide up the available memory for named
> equally, among all of the views defined in the config. So, if that's true --
> and I haven't been able to independently confirm it -- then there would be a
> limit to how much memory a given view can have, regardless of what tricks
> and/or stratagems one uses. (I'm hoping it isn't true, since I have some
> environments where my named.conf has 5 views defined in it, 2 of which are
> non-recursive, and another 1 of which sees an extremely small volume and
> diversity of queries, therefore it would be quite painful for the 2
> remaining "real" recursive views to be limited to only 1/5th
Re: Sharing zones between views to conserve memory
On Thu, Jan 10, 2013 at 11:17 AM, Jan Gutter wrote:
> Thanks for the suggestions!
>
> I'm currently investigating two options: the local view and forwarded
> zones, and I'm going to check out if I can write a fast DLZ lookup to
> share the RPZ zones between the views. Caching is not a big problem
> here, the "shared zones" should only change about once per month.
> However, it seems RPZ doesn't like "forward" type zones in the
> response-policy stanza. I have a nasty feeling I'm missing something
> obvious, though.
Hah, after a bit of source-code examination and googling, I found the
following paragraph:
3.2. Designated RPZs must be primary or secondary zones, since RPZs
cannot be queried on the wire, only searched in the recursive server's
own storage. A "zone" statement must therefore be given for the RPZ,
with all necessary "masters" clauses, each having all necessary "key"
subclauses. It is often a good idea to include "allow-query {none;};"
in the zone statement to refuse ordinary, non-rewriting queries of the
policy data.
quoted from ftp://ftp.isc.org/isc/dnsrpz/isc-tn-2010-1.txt
I guess I'm going to have to investigate the DLZ option then.
(Un)Fortunately, some other priority work has come up, so I'm just
adding more RAM for a stop-gap and will look at it again in a month or
so.
http://xkcd.com/979/
Thanks again for all your feedback!
Jan Gutter
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
from this list
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
