Just comments of no help I suspect. Why can't the limit be soft coded? I run machines with 100s of gig of memory and don't need a limit. IBM AIX which handles memory very well.
Some day I would like to discuss with someone who looks over this project (which I have in the past contributed to) about memory management. In the old days it was "very bad" to use memory and not return it. Memory was at a premium. It was usually called "memory leaks." In the current world machines have more memory than they need (or can get it for few dollars). With multi-cores and multi-processors people don't realize (this is over simplified) that in order to take memory or return memory from the O/S requires a "context switch" which is more or less a term to mean the machine must switch to single user mode to make the memory operations. All of my code recently deliberately uses memory leaks. If I have a program running that will process 1,000s of XML files with 1,000 of nodes ... there is zero need to get a tiny slip of memory ofr some purpose and return it. Make the pointer static and if it exists, reuse it. To do this well, keep a static for the size as well and if too small use realloc(). Eventually it will be big enough. This can provide very dramatic performance increases on a machine that has thousands of processes parsing XML. It could easily be implemented as an option (by having free() set the pointer to NULL if it really does a free, and if it does not make the free() a noop. Not releasing the memory is in fact a memory leak ... but if you are repeating the creation/return of memory a lot it is not a big deal for memory usage, and it is a big deal for a large multi-user system (my largest is 16,000 users on one box -- a big IBM). Imaging 16,000 users stuck single threaded because of memory management. Just a thought. Many machines do not handle the creation/destruction of memory all that well and this could allow you to set your own parameters -- perhaps on maximum memory used (although the O/S usually has soft and hard limits for that) or perhaps on number of nodes or ... Eric VICS, LLC Eric S Eberhard 2933 W Middle Verde Rd Camp Verde, AZ 86322 928-567-3727 (land line) 928-301-7537 (cell phone) http://www.vicsmba.com https://www.facebook.com/groups/286143052248115 -----Original Message----- From: xml <xml-boun...@gnome.org> On Behalf Of Nick Sent: Wednesday, December 1, 2021 9:53 PM To: xml@gnome.org Subject: [xml] Unexpected Nodeset Limit for XPath I have a large XML file, but a fairly basic and flat layout. Depending on the number of elements, I'm hitting a nodeset limit. I'm surprised to hit the nodeset limit because it seems like the xpath can ignore (ie. not store) all of the "Term" elements--which are causing the error. If I reduce the number of "Term" elements then it works fine. libxml2 version 2.9.12 I can repro the issue with xmllint Here's a basic XML to demonstrate the behavior (this example is stripped down for demonstration purposes, the XPath makes more sense in the real-world use case): <?xml version="1.0" encoding="UTF-8"?> <Elements id="5M_Elements"> <Group> <Term>1</Term> <Term>2</Term> <Term>3</Term> <!-- 5 Million more Term elements... --> </Group> </Elements> Here's what I get for the XPath: xmllint --xpath '//*[@id="5M_Elements"]' ./LibXml_NodeSetLimit.xml >/dev/null XPath error : Memory allocation failed : growing nodeset hit limit growing nodeset hit limit ^ XPath evaluation failure Can the XPath code not count all of these elements toward the node limit? Thanks, Nick _______________________________________________ xml mailing list, project page http://xmlsoft.org/ xml@gnome.org https://mail.gnome.org/mailman/listinfo/xml _______________________________________________ xml mailing list, project page http://xmlsoft.org/ xml@gnome.org https://mail.gnome.org/mailman/listinfo/xml
