On Fri, 16 Apr 2021 at 02:21, Kevin 'ldir' Darbyshire-Bryant <l...@darbyshire-bryant.me.uk> wrote: > > > On 14 Apr 2021, at 00:34, Simon Kelley <si...@thekelleys.org.uk> wrote: > > > > Tagging onto the end of the thread just to report the results of my > > research. > > > > This started because of problems with the OOM killer in a > > resource-constrained system that was prompting OOM death when it spawned > > sub-processes to handle TCP connections. I proposed a trick of putting > > the large in-memory dataset in sub-process, so that the main dnsmasq > > process which forks to handle TCP requests stays small. Reading around, > > that probably won't work: the OOM killer weights size of all children > > when looking for a victim. I think that trying to outsmart the OOM > > killer is probably a hiding to nothing. It's possible for the OS to > > protect critical daemons using oom_adj and friends, and supporting that > > in OpenWRT looks like a better way to go. > > > > We then moved onto the fact that adblocking involves thousands of lines of > > > > local=/example.com/ > > > > and the code supporting that didn't really envisage such large numbers. > > I think improvements can be made there, and I'll look at doing that in > > more detail. > > Hi Simon, > > Thanks for picking this up again. There are multiple interlinked problems: > > (ab)use of local=/example.com/ or similar for adblock lists which leads to > memory usage & (apparently) slower response time due to the linear search of > ’servers’ handling the local domain. I’m sure the ‘local’ list could be put > into some sort of tree/hash structure to speed that up but the memory > consumption will inherently be there to some (hopefully optimised) extent. > The list has to exist :-) > > TCP requests causing a fork of dnsmasq with all that ‘local/server’ list > memory usage, up to 21 times. I’m looking at my APU 2 running openwrt at the > moment with a 46000 line (small) adblock ‘address=/foo.bar/‘ list - dnsmasq > consuming circa 12MB. 21*12MB 252MB isn’t going to cause my APU to sweat > memory wise but a lesser device could very well ask ‘Where am I going to find > this extra 228MB of memory from then?’. > > I don’t agree that your ‘large dataset sub-process with small tcp handling > children’ won’t work. Yes linux oom killer takes into account all the > children, but the tcp children are going to be much(much!) smaller, > presumably the size of a basic dnsmasq instance of which most of it will be > program text. A base dnsmasq on said APU2 takes 2.5MB (the above example > would then total 12MB + 20*2.5=62MB) That’s quite a difference on a > constrained system between asking for lumps of 12MB vs 2.5MB. Whether this > approach makes sense from a latency perspective I don’t know… I’m assuming > that large process is acting as an ‘address validator’ and hence the > requesting children will need to wait for its ‘yes/no’ answer subject to > process scheduling etc. > > Tony mentioned another possible solution which avoids the sub-process malarky > of marking the ’server list’ memory in some way that the OS knows it’s shared > and therefore don’t need to have ‘real memory’ for all those tcp > sub-processes. I’m sure he’ll be along in a minute to explain further. >
Hi Simon, Kevin, I wrote a test program that emulates dnsmasq execution when loading large blocklists and then forking children to handle TCP DNS requests. It allocates memory and initializes it, then sets it read-only, and finally forks a number of child processes. At each step, it prints committed memory from /proc/meminfo, tracking the risk of OOM. The option "--private" uses private anonymous memory and reproduces normal dnsmasq behaviour; it can easily trigger OOM on small memory systems (e.g. 128MB). The other option "--shared" demonstrates how using shared anonymous memory reduces the overall commit and can work on even small systems. I also captured some log files from running on a "big" 8GB system and a "small" 128MB OpenWrt system. Using shared memory on the "small" system, I can allocate 64 MB without issue, while doing the same using private memory is a reliable way to trigger OOM. For more details see the head commit and related files here: https://github.com/guidosarducci/dnsmasq/commits/oom_alloc_test/ Best regards, Tony _______________________________________________ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
