jnioche opened a new pull request, #2280: URL: https://github.com/apache/stormcrawler/pull/2280
Raises the ceiling that reading and parsing the input put on the throughput reported by `StatusUpdaterBenchmark`, i.e. its `baseline`. - **Streaming JSON parsing**: the URLItem lines are read token by token with a `JsonParser` instead of building a full `JsonNode` tree to extract three fields. The parser reads the UTF-8 bytes directly, which also skips the UTF-16 decoding done by the `InputStreamReader`. The semantics are unchanged (discovered takes precedence over known, `refetchableFromDate` over `refetchable_from_date`, int64 values as strings, non-textual URL is invalid). - **Buffered input**: the file stream is wrapped in a 64 KB `BufferedInputStream` and `GZIPInputStream` uses a 64 KB buffer instead of the default 512 bytes. - **Parse pool**: the reading thread now only decompresses and hands blocks of whole lines to a pool of parser threads, which parse and route the URLs to the instances by hash as before. The size of the pool defaults to half the number of cores and can be set with `status.updater.benchmark.parsers`; it is printed in the summary. The URLs are no longer handed to the instances in the exact order of the input, a given URL still always goes to the same instance. Adds `StatusUpdaterBenchmarkTest` for the parsing. Rough figures for `baseline` with 4 instances on 3M generated mixed lines (20 cores, ~1 s runs so noisy): | | OPS | |---|---| | before | ~1.56M | | 1 parser | ~2.26M | | 2 parsers | ~3.39M | | 4 parsers | ~3.11M | Beyond 2 parsers the single decompressing thread is likely the limit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
