The GitHub Actions job "Java CI with Maven" on stormcrawler.git/main has failed. Run started by GitHub user rzo1 (triggered by rzo1).
Head commit for run: 1c6f1ae750f812ac085063a6cac5624ef8269f39 / Abhinav <[email protected]> Use batch endpoint in URLFrontier module for DISCOVERED urls (#2117) * Use batched PutDiscovered endpoint for DISCOVERED URLs in URLFrontier StatusUpdaterBolt Discovered URLs, which are the bulk of what a crawl writes, are now grouped into batches and sent on the PutDiscovered endpoint introduced in URLFrontier 2.6, amortising the per-message cost that limits the ingestion rate. Known URLs keep using the streaming PutURLs endpoint. - new config key urlfrontier.batch.size (default 100, 0 disables batching) - partially filled batches are flushed after 1s by a scheduled flusher, and whenever a known URL arrives (natural end-of-page boundary) or permits run short - batches are acked as a whole through BatchAck: one status per URL in the order they were sent, each resolved against the waitAck cache - frontiers without PutDiscovered (pre-2.6) are detected via UNIMPLEMENTED and the bolt falls back to sending discovered URLs individually - flow control follows the PutURLs client shipped with URLFrontier: throttled sends wait on a monitor woken by acks and by the transports' on-ready notifications instead of polling - requires urlfrontier-API 2.6 (wire compatible with 2.5 servers) Fixes #2062 * Address review: no tuple stranding, honest throttling metric, docs and formatting - fail the tuples of a batch that was never sent (interrupted flush) or whose stream died right away instead of leaving them to the waitAck cache expiry, which would outlive Storm's tuple timeouts - measure the time actually spent waiting in the throttle loop: an ack can wake the wait well before the timeout, and the timeSpentThrottling metric must keep meaning something - document that the permit tryAcquire is deliberately outside the flow monitor (a racing notifyAll costs one backstop interval, never a deadlock) and the batchLock then sendLock acquisition order - document urlfrontier.batch.size in configuration.adoc - reformat with the google-java-format version the git-code-format-maven-plugin embeds Report URL: https://github.com/apache/stormcrawler/actions/runs/34452848970 With regards, GitHub Actions via GitBox
