dpol1 opened a new issue, #2057: URL: https://github.com/apache/stormcrawler/issues/2057
FetcherBolt is over 1100 lines now, FetcherThread alone is ~430 and mixes robots, sitemaps, crawl-delay, protocol calls and status emission. #867 and #504 would add more on top. Plan: characterization tests first, then extract the queue classes as-is (same package, package-private), then thin down FetcherThread step by step. No behaviour change — anything fishy found along the way gets its own issue (first one: #XXXX). No new public API, extensibility stays on the streams. Maybe later a common fetch kernel with SimpleFetcherBolt, but that's a separate discussion. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
