dev  

Messages by Date

• 2026/07/27 [PR] #2026 - http.agent.name is dropped from the robots.txt match set when http.robots.agents is set [stormcrawler] via GitHub
• 2026/07/27 [I] http.agent.name is dropped from the robots.txt match set when http.robots.agents is set [stormcrawler] via GitHub
• 2026/07/26 [GH] (stormcrawler/dependabot/maven/software.amazon.awssdk-bom-2.49.3): Workflow run "Java CI with Maven" is working again! GitBox
• 2026/07/26 [GH] (stormcrawler/dependabot/maven/storm-client.version-3.0.0): Workflow run "Java CI with Maven" failed! GitBox
• 2026/07/26 [PR] build(deps): bump org.apache.maven.plugins:maven-jar-plugin from 3.5.0 to 3.5.1 [stormcrawler] via GitHub
• 2026/07/26 [PR] build(deps): bump storm-client.version from 2.8.8 to 3.0.0 [stormcrawler] via GitHub
• 2026/07/26 [PR] build(deps): bump tika.version from 3.3.1 to 3.3.2 [stormcrawler] via GitHub
• 2026/07/26 [PR] build(deps): bump software.amazon.awssdk:bom from 2.48.3 to 2.49.3 [stormcrawler] via GitHub
• 2026/07/26 Re: [I] [Bug report] URLFilterBolt seems to emit unanchored tuples [stormcrawler] via GitHub
• 2026/07/26 Re: [I] [Bug report] URLFilterBolt seems to emit unanchored tuples [stormcrawler] via GitHub
• 2026/07/25 Re: [I] [Bug report] URLFilterBolt seems to emit unanchored tuples [stormcrawler] via GitHub
• 2026/07/24 [I] [Bug report] URLFilterBolt seems to emit unanchored [stormcrawler] via GitHub
• 2026/07/24 Re: Release 3.7.0? Richard Zowalla
• 2026/07/24 Re: [PR] Enforce robots.txt Crawl-delay in URLFrontier queues [stormcrawler] via GitHub
• 2026/07/24 Release 3.7.0? Julien Nioche
• 2026/07/23 Re: [DISCUSS] Storm 3 meets StormCrawler Richard Zowalla
• 2026/07/23 Re: [I] Deprecate HTTPClient protocol implementation [stormcrawler] via GitHub
• 2026/07/23 Re: [PR] Mark HTTPClient as deprecated [stormcrawler] via GitHub
• 2026/07/23 Re: [PR] Allow indexer bolts to use a metadata value as the document ID [stormcrawler] via GitHub
• 2026/07/23 [GH] (stormcrawler/2013): Workflow run "Java CI with Maven" is working again! GitBox
• 2026/07/23 [GH] (stormcrawler/2013): Workflow run "Java CI with Maven" failed! GitBox
• 2026/07/23 [PR] Mark HTTPClient as deprecated [stormcrawler] via GitHub
• 2026/07/23 Re: [PR] Mark CloudSearch as deprecated [stormcrawler] via GitHub
• 2026/07/23 Re: [I] Deprecate AWS CloudSearch [stormcrawler] via GitHub
• 2026/07/23 Re: [I] Deprecate Selenium module [stormcrawler] via GitHub
• 2026/07/23 Re: [PR] Mark Selenium module as deprecated [stormcrawler] via GitHub
• 2026/07/23 [PR] Mark Selenium module as deprecated [stormcrawler] via GitHub
• 2026/07/23 [PR] Mark CloudSearch as deprecated [stormcrawler] via GitHub
• 2026/07/23 Re: [DISCUSS] Storm 3 meets StormCrawler Julien Nioche
• 2026/07/23 [PR] Upgrade to Apache Storm 3.0.0 and Java 25 [stormcrawler] via GitHub
• 2026/07/23 Re: [PR] Allow indexer bolts to use a metadata value as the document ID [stormcrawler] via GitHub
• 2026/07/23 [DISCUSS] Storm 3 meets StormCrawler Richard Zowalla
• 2026/07/23 [I] Deprecate AWS CloudSearch [stormcrawler] via GitHub
• 2026/07/23 [I] Upgrade to Apache Storm 3.0.0 [stormcrawler] via GitHub
• 2026/07/23 [I] Deprecate Selenium module [stormcrawler] via GitHub
• 2026/07/23 [I] Deprecate HTTPClient protocol implementation [stormcrawler] via GitHub
• 2026/07/22 Re: [PR] #108 Normalize misspelled HTTP headers [stormcrawler] via GitHub
• 2026/07/22 Re: [PR] Allow indexer bolts to use a metadata value as the document ID [stormcrawler] via GitHub
• 2026/07/22 [PR] Allow indexer bolts to use a metadata value as the document ID [stormcrawler] via GitHub
• 2026/07/21 Re: [I] Remove needsDOM() in ParseFilter [stormcrawler] via GitHub
• 2026/07/21 Re: [PR] Resolve relative outlinks against <base href> in JSoupParserBolt [stormcrawler] via GitHub
• 2026/07/21 [PR] Resolve relative outlinks against <base href> in JSoupParserBolt [stormcrawler] via GitHub
• 2026/07/21 Re: [I] Remove needsDOM() in ParseFilter [stormcrawler] via GitHub
• 2026/07/21 Re: [I] SubDocumentsFilterTest should not use a sitemap file [stormcrawler] via GitHub
• 2026/07/21 Re: [PR] Fix #304: Use HTML test document instead of sitemap XML in SubDocumentsFilterTest [stormcrawler] via GitHub
• 2026/07/21 Re: [PR] Fix #304: Use HTML test document instead of sitemap XML in SubDocumentsFilterTest [stormcrawler] via GitHub
• 2026/07/21 Re: [PR] Enforce robots.txt Crawl-delay in URLFrontier queues [stormcrawler] via GitHub
• 2026/07/21 Re: [PR] build(deps): bump actions/checkout from 7.0.0 to 7.0.1 [stormcrawler] via GitHub
• 2026/07/21 [PR] build(deps): bump actions/checkout from 7.0.0 to 7.0.1 [stormcrawler] via GitHub
• 2026/07/21 Re: [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/07/21 [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/07/21 Re: [PR] build(deps): bump langchain4j.version from 1.17.2 to 1.18.0 [stormcrawler] via GitHub
• 2026/07/21 Re: [PR] build(deps): bump software.amazon.awssdk:bom from 2.47.5 to 2.48.3 [stormcrawler] via GitHub
• 2026/07/21 Re: [PR] build(deps): bump actions/setup-java from 5.5.0 to 5.6.0 [stormcrawler] via GitHub
• 2026/07/20 Re: Feedback on Website Refresh Richard Zowalla
• 2026/07/20 Re: Feedback on Website Refresh Dávid Szigecsán
• 2026/07/20 Re: [PR] Fix #304: Use HTML test document instead of sitemap XML in SubDocumentsFilterTest [stormcrawler] via GitHub
• 2026/07/20 Re: [PR] Enforce robots.txt Crawl-delay in URLFrontier queues [stormcrawler] via GitHub
• 2026/07/19 [PR] build(deps): bump software.amazon.awssdk:bom from 2.47.5 to 2.48.3 [stormcrawler] via GitHub
• 2026/07/19 [PR] build(deps): bump langchain4j.version from 1.17.2 to 1.18.0 [stormcrawler] via GitHub
• 2026/07/19 Re: [PR] Fix #304: Use HTML test document instead of sitemap XML in SubDocumentsFilterTest [stormcrawler] via GitHub
• 2026/07/19 Re: [PR] #108 Normalize misspelled HTTP headers [stormcrawler] via GitHub
• 2026/07/19 Re: [PR] bugfix - max content not applied for robots.txt when redirecting [stormcrawler] via GitHub
• 2026/07/18 [PR] #108 Normalize misspelled HTTP headers [stormcrawler] via GitHub
• 2026/07/18 [PR] Fix #304: Use HTML test document instead of sitemap XML in SubDocumentsFilterTest [stormcrawler] via GitHub
• 2026/07/18 [PR] bugfix - max content not applied for robots.txt when redirecting [stormcrawler] via GitHub
• 2026/07/17 Re: [PR] #2001 Document adaptive scheduler metadata [stormcrawler] via GitHub
• 2026/07/17 Re: [PR] #2001 Document adaptive scheduler metadata [stormcrawler] via GitHub
• 2026/07/17 Re: [I] [Improvement] Document metadata to be persisted for adaptive scheduling [stormcrawler] via GitHub
• 2026/07/17 Re: Feedback on Website Refresh Richard Zowalla
• 2026/07/17 [PR] build(deps): bump actions/setup-java from 5.5.0 to 5.6.0 [stormcrawler] via GitHub
• 2026/07/17 Re: Feedback on Website Refresh Richard Zowalla
• 2026/07/17 Re: [PR] #2001 Document adaptive scheduler metadata [stormcrawler] via GitHub
• 2026/07/16 Re: [PR] #2001 Document adaptive scheduler metadata [stormcrawler] via GitHub
• 2026/07/16 Feedback on Website Refresh Richard Zowalla
• 2026/07/16 Re: [PR] #2001 Document adaptive scheduler metadata [stormcrawler] via GitHub
• 2026/07/16 [PR] #2001 Document adaptive scheduler metadata [stormcrawler] via GitHub
• 2026/07/16 [I] [Improvement] Document metadata to be persisted for adaptive scheduling [stormcrawler] via GitHub
• 2026/07/16 Re: [PR] [WARC] Format WARC-IP-Address header values following the WARC community recommendations [stormcrawler] via GitHub
• 2026/07/16 Re: [PR] [WARC] Format WARC-IP-Address header values following the WARC community recommendations [stormcrawler] via GitHub
• 2026/07/15 Re: [PR] Enforce robots.txt Crawl-delay in URLFrontier queues [stormcrawler] via GitHub
• 2026/07/15 [PR] Enforce robots.txt Crawl-delay in URLFrontier queues [stormcrawler] via GitHub
• 2026/07/14 Re: [D] Deprecate HTTPClient protocol implementation [stormcrawler] via GitHub
• 2026/07/14 Re: [D] Deprecate Selenium module? [stormcrawler] via GitHub
• 2026/07/14 Re: [PR] [WARC] Format WARC-IP-Address header values following the WARC community recommendations [stormcrawler] via GitHub
• 2026/07/14 Re: [PR] [WARC] Format WARC-IP-Address header values following the WARC community recommendations [stormcrawler] via GitHub
• 2026/07/14 Re: [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/07/14 Re: [D] Deprecate Selenium module? [stormcrawler] via GitHub
• 2026/07/14 Re: [D] Deprecate HTTPClient protocol implementation [stormcrawler] via GitHub
• 2026/07/14 [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/07/14 Re: [PR] build(deps): bump software.amazon.awssdk:bom from 2.46.21 to 2.47.5 [stormcrawler] via GitHub
• 2026/07/14 Re: [PR] build(deps): bump selenium.version from 4.45.0 to 4.46.0 [stormcrawler] via GitHub
• 2026/07/14 Re: [PR] build(deps): bump junit.version from 6.1.1 to 6.1.2 [stormcrawler] via GitHub
• 2026/07/14 Re: [PR] build(deps): bump langchain4j.version from 1.17.1 to 1.17.2 [stormcrawler] via GitHub
• 2026/07/14 [I] [Improvement] WARC writer: WARC-Protocol header to follow WARC field proposals [stormcrawler] via GitHub
• 2026/07/14 [PR] [WARC] Format WARC-IP-Address header values following the WARC community recommendations [stormcrawler] via GitHub
• 2026/07/13 Re: [PR] Verbatim headers for Playwright [stormcrawler] via GitHub
• 2026/07/13 [DISCUSS] Deprecating the Selenium module and the HTTPClient protocol implementation Richard Zowalla
• 2026/07/13 [GH] (stormcrawler/verbatimHeadersPW): Workflow run "Java CI with Maven" is working again! GitBox
• 2026/07/13 [GH] (stormcrawler/verbatimHeadersPW): Workflow run "Java CI with Maven" failed! GitBox
• 2026/07/13 [PR] Verbatim headers for Playwright [stormcrawler] via GitHub
• 2026/07/13 Re: [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/07/13 [D] Deprecate HTTPClient protocol implementation [stormcrawler] via GitHub
• 2026/07/13 [D] Deprecate Selenium module? [stormcrawler] via GitHub
• 2026/07/13 [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/07/13 Re: [PR] Workaround to activate LogViewer in docker compose [stormcrawler] via GitHub
• 2026/07/13 Re: [PR] Adaptive back-off for rate-limited hosts via the queue stream [stormcrawler] via GitHub
• 2026/07/13 Re: [I] Fetcher: optionally slow down fetching from hosts with repeated exceptions [stormcrawler] via GitHub
• 2026/07/13 Re: [PR] Adaptive back-off for rate-limited hosts via the queue stream [stormcrawler] via GitHub
• 2026/07/13 [PR] Workaround to activate LogViewer in docker compose [stormcrawler] via GitHub
• 2026/07/13 Re: [PR] Adaptive back-off for rate-limited hosts via the queue stream [stormcrawler] via GitHub
• 2026/07/12 Re: [PR] Adaptive back-off for rate-limited hosts via the queue stream [stormcrawler] via GitHub
• 2026/07/12 [PR] build(deps): bump software.amazon.awssdk:bom from 2.46.21 to 2.47.5 [stormcrawler] via GitHub
• 2026/07/12 [PR] build(deps): bump selenium.version from 4.45.0 to 4.46.0 [stormcrawler] via GitHub
• 2026/07/12 [PR] build(deps): bump junit.version from 6.1.1 to 6.1.2 [stormcrawler] via GitHub
• 2026/07/12 [PR] build(deps): bump langchain4j.version from 1.17.1 to 1.17.2 [stormcrawler] via GitHub
• 2026/07/10 Re: [PR] Adaptive back-off for rate-limited hosts via the queue stream [stormcrawler] via GitHub
• 2026/07/10 Re: [PR] Adaptive back-off for rate-limited hosts via the queue stream [stormcrawler] via GitHub
• 2026/07/09 Re: [PR] Adaptive back-off for rate-limited hosts via the queue stream [stormcrawler] via GitHub
• 2026/07/09 Re: [PR] Adaptive back-off for rate-limited hosts via the queue stream [stormcrawler] via GitHub
• 2026/07/09 Re: [PR] Pluggable robot rule parser [stormcrawler] via GitHub
• 2026/07/09 Re: [PR] Pluggable robot rule parser [stormcrawler] via GitHub
• 2026/07/09 Re: [PR] Pluggable robot rule parser [stormcrawler] via GitHub
• 2026/07/09 [GH] (stormcrawler/pluggableRobotRuleParser): Workflow run "Java CI with Maven" is working again! GitBox
• 2026/07/09 Re: [PR] Pluggable robot rule parser [stormcrawler] via GitHub
• 2026/07/09 Re: [PR] Pluggable robot rule parser [stormcrawler] via GitHub
• 2026/07/09 Re: [PR] Pluggable robot rule parser [stormcrawler] via GitHub
• 2026/07/09 [GH] (stormcrawler/pluggableRobotRuleParser): Workflow run "Java CI with Maven" failed! GitBox
• 2026/07/09 [PR] Pluggable robot rule parser [stormcrawler] via GitHub
• 2026/07/09 Re: [PR] Don't trigger mvn workflow for changes to the docs [stormcrawler] via GitHub
• 2026/07/09 Re: [PR] Adding Green Web Foundation to Powered By [stormcrawler] via GitHub
• 2026/07/09 [PR] Don't trigger mvn workflow for changes to the docs [stormcrawler] via GitHub
• 2026/07/08 [PR] Adding Green Web Foundation to Powered By [stormcrawler] via GitHub
• 2026/07/08 [PR] Adaptive back-off for rate-limited hosts via the queue stream [stormcrawler] via GitHub
• 2026/07/08 Re: [PR] build(deps): bump actions/setup-java from 5.4.0 to 5.5.0 [stormcrawler] via GitHub
• 2026/07/08 [GH] (stormcrawler/feat/1106-adaptive-host-backoff): Workflow run "Java CI with Maven" failed! GitBox
• 2026/07/08 [PR] build(deps): bump actions/setup-java from 5.4.0 to 5.5.0 [stormcrawler] via GitHub
• 2026/07/08 [PR] Feature/store request url [stormcrawler] via GitHub
• 2026/07/08 Re: [PR] Port QueueBolt (queue stream) to the opensearch-java module [stormcrawler] via GitHub
• 2026/07/08 Re: [PR] Port QueueBolt (queue stream) to the opensearch-java module [stormcrawler] via GitHub
• 2026/07/07 [PR] Port QueueBolt (queue stream) to the opensearch-java module [stormcrawler] via GitHub
• 2026/07/07 Re: [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/07/07 [GH] (stormcrawler/main): Workflow run "CI Main - Regenerate License" is working again! GitBox
• 2026/07/07 [GH] (stormcrawler/main): Workflow run "CI Main - Regenerate License" failed! GitBox
• 2026/07/07 [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/07/07 Re: [PR] build(deps): bump langchain4j.version from 1.17.0 to 1.17.1 [stormcrawler] via GitHub
• 2026/07/07 Re: [PR] build(deps): bump opensearch.version from 2.19.5 to 2.19.6 [stormcrawler] via GitHub
• 2026/07/07 Re: [PR] build(deps): bump com.microsoft.playwright:playwright from 1.60.0 to 1.61.0 [stormcrawler] via GitHub
• 2026/07/07 Re: [PR] build(deps): bump software.amazon.awssdk:bom from 2.46.17 to 2.46.21 [stormcrawler] via GitHub
• 2026/07/07 [I] Propagate robots.txt crawl-delay to URLFrontier via setDelay [stormcrawler] via GitHub
• 2026/07/06 Re: [PR] Block the URLFrontier queue on Retry-After via the queue stream [stormcrawler] via GitHub
• 2026/07/06 Re: [I] support retry-after in FetcherBolt [stormcrawler] via GitHub
• 2026/07/06 Re: [PR] Block the URLFrontier queue on Retry-After via the queue stream [stormcrawler] via GitHub
• 2026/07/06 Re: [PR] Block the URLFrontier queue on Retry-After via the queue stream [stormcrawler] via GitHub
• 2026/07/06 Re: [PR] Block the URLFrontier queue on Retry-After via the queue stream [stormcrawler] via GitHub
• 2026/07/05 [GH] (stormcrawler/dependabot/maven/langchain4j.version-1.17.1): Workflow run "Java CI with Maven" is working again! GitBox
• 2026/07/05 [GH] (stormcrawler/dependabot/maven/com.microsoft.playwright-playwright-1.61.0): Workflow run "Java CI with Maven" failed! GitBox
• 2026/07/05 [PR] build(deps): bump opensearch.version from 2.19.5 to 2.19.6 [stormcrawler] via GitHub
• 2026/07/05 [PR] build(deps): bump software.amazon.awssdk:bom from 2.46.17 to 2.46.21 [stormcrawler] via GitHub
• 2026/07/05 [PR] build(deps): bump com.microsoft.playwright:playwright from 1.60.0 to 1.61.0 [stormcrawler] via GitHub
• 2026/07/05 [PR] build(deps): bump langchain4j.version from 1.17.0 to 1.17.1 [stormcrawler] via GitHub
• 2026/07/05 Re: [PR] Block the URLFrontier queue on Retry-After via the queue stream [stormcrawler] via GitHub
• 2026/07/04 Re: [PR] Block the URLFrontier queue on Retry-After via the queue stream [stormcrawler] via GitHub
• 2026/07/04 Re: [I] [Improvement] Add configuration to stop parsing PDFs after X pages [stormcrawler] via GitHub
• 2026/07/03 Re: [PR] Block the URLFrontier queue on Retry-After via the queue stream [stormcrawler] via GitHub
• 2026/07/03 Re: [PR] Block the URLFrontier queue on Retry-After via the queue stream [stormcrawler] via GitHub
• 2026/07/03 Re: [PR] Block the URLFrontier queue on Retry-After via the queue stream [stormcrawler] via GitHub
• 2026/07/03 Re: [PR] Block the URLFrontier queue on Retry-After via the queue stream [stormcrawler] via GitHub
• 2026/07/03 Re: [I] Send host info on a specific stream [stormcrawler] via GitHub
• 2026/07/03 Re: [PR] Queue stream [stormcrawler] via GitHub
• 2026/07/03 Re: [PR] Queue stream [stormcrawler] via GitHub
• 2026/07/02 [GH] (stormcrawler/990): Workflow run "Java CI with Maven" is working again! GitBox
• 2026/07/02 [GH] (stormcrawler/990): Workflow run "Java CI with Maven" failed! GitBox
• 2026/07/02 Re: [I] Send host info on a specific stream [stormcrawler] via GitHub
• 2026/07/02 [PR] Emit Retry-After host back-off and block the URLFrontier queue [stormcrawler] via GitHub
• 2026/06/30 Re: [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/06/30 [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/06/30 Re: [PR] build(deps): bump langchain4j.version from 1.16.3 to 1.17.0 [stormcrawler] via GitHub
• 2026/06/30 Re: [PR] build(deps): bump org.apache:apache from 38 to 39 [stormcrawler] via GitHub
• 2026/06/29 Re: [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/06/29 [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/06/29 Re: [PR] build(deps): bump actions/cache from 6.0.0 to 6.1.0 [stormcrawler] via GitHub
• 2026/06/29 Re: [PR] build(deps): bump junit.version from 6.1.0 to 6.1.1 [stormcrawler] via GitHub
• 2026/06/29 Re: [PR] build(deps): bump software.amazon.awssdk:bom from 2.46.15 to 2.46.17 [stormcrawler] via GitHub
• 2026/06/29 [PR] build(deps): bump actions/cache from 6.0.0 to 6.1.0 [stormcrawler] via GitHub
• 2026/06/28 [PR] build(deps): bump org.apache:apache from 38 to 39 [stormcrawler] via GitHub
• 2026/06/28 [PR] build(deps): bump langchain4j.version from 1.16.3 to 1.17.0 [stormcrawler] via GitHub
• 2026/06/28 [PR] build(deps): bump software.amazon.awssdk:bom from 2.46.15 to 2.46.17 [stormcrawler] via GitHub
• 2026/06/28 [PR] build(deps): bump junit.version from 6.1.0 to 6.1.1 [stormcrawler] via GitHub
• 2026/06/28 Re: [PR] build(deps): bump actions/setup-java from 5.3.0 to 5.4.0 [stormcrawler] via GitHub
• 2026/06/28 Re: [PR] build(deps): bump actions/cache from 5.0.5 to 6.0.0 [stormcrawler] via GitHub
• 2026/06/26 [PR] build(deps): bump actions/setup-java from 5.3.0 to 5.4.0 [stormcrawler] via GitHub
• 2026/06/24 [PR] build(deps): bump actions/cache from 5.0.5 to 6.0.0 [stormcrawler] via GitHub
• 2026/06/24 Re: [PR] use placeholder instead of older version of archetypes in README [stormcrawler] via GitHub
• 2026/06/24 [PR] use placeholder instead of older version of archetypes in README [stormcrawler] via GitHub
• 2026/06/22 Re: [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/06/22 [PR] Regenerated License file after dependency upgrades [stormcrawler] via GitHub
• 2026/06/22 Re: [PR] #466 - Handle text/plain content in JSoupParserBolt [stormcrawler] via GitHub
• 2026/06/22 Re: [PR] #466 - Handle text/plain content in JSoupParserBolt [stormcrawler] via GitHub
• 2026/06/22 Re: [I] JSoup parser to handle text/plain [stormcrawler] via GitHub

• Earlier messages
• Later messages