rzo1 commented on PR #2263: URL: https://github.com/apache/stormcrawler/pull/2263#issuecomment-6387381296
Thanks for tracking this down. I'd rather not bring Xerces back, though: 2.12.2 is from 2022, Xerces hasn't had a release since, and putting it on the classpath swaps out the SAX parser for everything in the worker, not just sitemaps and feeds. The JDK parser can handle this with two system properties. I tried it on JDK 25.0.3 with a sitemap of 34,000 URLs and three `&` each. With the defaults it fails as you describe. Raising only one of the limits isn't enough, because the second one kicks in at 100,000 too. With both it parses fine: ``` -Djdk.xml.maxGeneralEntitySizeLimit=0 -Djdk.xml.totalEntitySizeLimit=0 ``` So my suggestion would be to document these for the worker JVMs (`topology.worker.childopts`) in the configuration docs and the archetypes' `crawler-conf.yaml`, with a note in the 4.0 migration notes for anyone running on Java 24+. Longer term, crawler-commons could set the two limits on its own parser in `SiteMapParser.processXml()` when DOCTYPEs are disallowed (the default). Without a DTD there are no custom entities, so it's safe, and then nobody needs the worker options for sitemaps. ROME would need the same for feeds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
