dev
Thread
Date
Earlier messages
Messages by Thread
[PR] opensearch-java: send bulk requests with the OpenSearchAsyncClient [stormcrawler]
via GitHub
Re: [PR] opensearch-java: send bulk requests with the OpenSearchAsyncClient [stormcrawler]
via GitHub
Re: [PR] opensearch-java: send bulk requests with the OpenSearchAsyncClient [stormcrawler]
via GitHub
Re: [PR] opensearch-java: send bulk requests with the OpenSearchAsyncClient [stormcrawler]
via GitHub
[PR] URLFrontier StatusUpdaterBolt: cheaper bookkeeping of the acks [stormcrawler]
via GitHub
Re: [PR] URLFrontier StatusUpdaterBolt: cheaper bookkeeping of the acks [stormcrawler]
via GitHub
[PR] StatusUpdaterBenchmark: run several instances of the status updater in parallel [stormcrawler]
via GitHub
Re: [PR] StatusUpdaterBenchmark: run several instances of the status updater in parallel [stormcrawler]
via GitHub
[PR] build(deps): bump org.apache.commons:commons-lang3 from 3.20.0 to 3.21.0 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump org.apache.commons:commons-lang3 from 3.20.0 to 3.21.0 [stormcrawler]
via GitHub
[PR] build(deps): bump com.puppycrawl.tools:checkstyle from 14.1.0 to 14.3.0 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump com.puppycrawl.tools:checkstyle from 14.1.0 to 14.3.0 [stormcrawler]
via GitHub
[PR] build(deps): bump com.google.guava:guava from 33.7.1-jre to 33.7.2-jre [stormcrawler]
via GitHub
Re: [PR] build(deps): bump com.google.guava:guava from 33.7.1-jre to 33.7.2-jre [stormcrawler]
via GitHub
[PR] build(deps): bump git-code-format-maven-plugin.version from 6.1 to 6.2 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump git-code-format-maven-plugin.version from 6.1 to 6.2 [stormcrawler]
via GitHub
[PR] build(deps): bump software.amazon.awssdk:bom from 2.55.5 to 2.55.10 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump software.amazon.awssdk:bom from 2.55.5 to 2.55.10 [stormcrawler]
via GitHub
[PR] build(deps): bump langchain4j.version from 1.20.0 to 1.20.2 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump langchain4j.version from 1.20.0 to 1.20.2 [stormcrawler]
via GitHub
[PR] #2214 Manage the Jackson dataformat/datatype versions in the root pom [stormcrawler]
via GitHub
Re: [PR] #2214 Manage the Jackson dataformat/datatype versions in the root pom [stormcrawler]
via GitHub
[GH] (stormcrawler/statusupdaterbenchmark): Workflow run "Java CI with Maven" failed!
GitBox
[PR] Add StatusUpdaterBenchmark to measure the performance of status updater backends [stormcrawler]
via GitHub
Re: [PR] Add StatusUpdaterBenchmark to measure the performance of status updater backends [stormcrawler]
via GitHub
Re: [PR] Add StatusUpdaterBenchmark to measure the performance of status updater backends [stormcrawler]
via GitHub
[I] Keep opensearch-java and remove opensearch module? [stormcrawler]
via GitHub
Re: [I] Keep 'opensearch-java' and remove the older 'opensearch' module? [stormcrawler]
via GitHub
Re: [I] Keep 'opensearch-java' and remove the older 'opensearch' module? [stormcrawler]
via GitHub
[I] Migrate core from HttpClient 4.x to HttpClient 5.x [stormcrawler]
via GitHub
Re: [I] Migrate core from HttpClient 4.x to HttpClient 5.x [stormcrawler]
via GitHub
[PR] Upgrade to Tika 4.1.0 [stormcrawler]
via GitHub
Re: [PR] Upgrade to Tika 4.1.0 [stormcrawler]
via GitHub
[PR] #2057 FetcherBolt: move the fetcher internals to o.a.s.fetcher [stormcrawler]
via GitHub
Re: [PR] #2057 FetcherBolt: move the fetcher internals to o.a.s.fetcher [stormcrawler]
via GitHub
Re: [PR] #2057 FetcherBolt: move the fetcher internals to o.a.s.fetcher [stormcrawler]
via GitHub
Re: [PR] #2057 FetcherBolt: move the fetcher internals to o.a.s.fetcher [stormcrawler]
via GitHub
[I] Upgrade to Tika 4.1.0 [stormcrawler]
via GitHub
Re: [I] Upgrade to Tika 4.1.0 [stormcrawler]
via GitHub
Re: [I] Upgrade to Tika 4.1.0 [stormcrawler]
via GitHub
[I] [Bug report] BasicURLNormalizer folds the query into the path when it rebuilds a URL [stormcrawler]
via GitHub
[PR] #2057 FetcherBolt: extract the sitemap autodiscovery [stormcrawler]
via GitHub
Re: [PR] #2057 FetcherBolt: extract the sitemap autodiscovery [stormcrawler]
via GitHub
[I] URLNormalizer The query is split on `;` as well as `&`, which rewrites URLs [stormcrawler]
via GitHub
[PR] #2197 BasicURLNormalizer keeps the query when rebuilding a URL [stormcrawler]
via GitHub
Re: [PR] #2197 BasicURLNormalizer keeps the query when rebuilding a URL [stormcrawler]
via GitHub
Re: [PR] #2197 BasicURLNormalizer keeps the query when rebuilding a URL [stormcrawler]
via GitHub
[PR] #2057 FetcherBolt: extract the crawl-delay policy [stormcrawler]
via GitHub
Re: [PR] #2057 FetcherBolt: extract the crawl-delay policy [stormcrawler]
via GitHub
[PR] #2057 FetcherBolt: extract the response and failure handling [stormcrawler]
via GitHub
Re: [PR] #2057 FetcherBolt: extract the response and failure handling [stormcrawler]
via GitHub
Re: [PR] #2057 FetcherBolt: extract the response and failure handling [stormcrawler]
via GitHub
Re: [PR] #2057 FetcherBolt: extract the response and failure handling [stormcrawler]
via GitHub
Re: [PR] #2057 FetcherBolt: extract the response and failure handling [stormcrawler]
via GitHub
Re: [PR] #2057 FetcherBolt: extract the response and failure handling [stormcrawler]
via GitHub
[PR] #2057 FetcherBolt: extract the robots.txt lookup [stormcrawler]
via GitHub
Re: [PR] #2057 FetcherBolt: extract the robots.txt lookup [stormcrawler]
via GitHub
[PR] #2057 FetcherBolt: move the fetch queue classes to their own files [stormcrawler]
via GitHub
Re: [PR] #2057 FetcherBolt: move the fetch queue classes to their own files [stormcrawler]
via GitHub
[PR] #2057 FetcherBolt: add characterization tests [stormcrawler]
via GitHub
Re: [PR] #2057 FetcherBolt: add characterization tests [stormcrawler]
via GitHub
[I] [Bug report] BasicURLNormalizer unmangleQueryString inserts a second `?` when a query value contains a `/` [stormcrawler]
via GitHub
[PR] build(deps): bump com.github.ben-manes.caffeine:caffeine from 3.2.4 to 3.3.0 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump com.github.ben-manes.caffeine:caffeine from 3.2.4 to 3.3.0 [stormcrawler]
via GitHub
[PR] build(deps): bump org.apache:apache from 39 to 40 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump org.apache:apache from 39 to 40 [stormcrawler]
via GitHub
[PR] build(deps): bump software.amazon.awssdk:bom from 2.55.0 to 2.55.5 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump software.amazon.awssdk:bom from 2.55.0 to 2.55.5 [stormcrawler]
via GitHub
[PR] build(deps): bump git-code-format-maven-plugin.version from 6.0 to 6.1 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump git-code-format-maven-plugin.version from 6.0 to 6.1 [stormcrawler]
via GitHub
[PR] build(deps): bump org.mockito:mockito-core from 5.23.0 to 5.24.0 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump org.mockito:mockito-core from 5.23.0 to 5.24.0 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump org.mockito:mockito-core from 5.23.0 to 5.24.0 [stormcrawler]
via GitHub
[PR] build(deps): bump org.slf4j:slf4j-simple from 2.0.19 to 2.0.20 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump org.slf4j:slf4j-simple from 2.0.19 to 2.0.20 [stormcrawler]
via GitHub
[PR] build(deps-dev): bump de.thetaphi:forbiddenapis from 3.10 to 3.11 [stormcrawler]
via GitHub
Re: [PR] build(deps-dev): bump de.thetaphi:forbiddenapis from 3.10 to 3.11 [stormcrawler]
via GitHub
[PR] Bring back the logo in the README [stormcrawler]
via GitHub
Re: [PR] Bring back the logo in the README [stormcrawler]
via GitHub
[PR] #2185 URLFrontier Spout: depth metrics to observe the breadth of the crawl [stormcrawler]
via GitHub
Re: [PR] #2185 URLFrontier Spout: depth metrics to observe the breadth of the crawl [stormcrawler]
via GitHub
Re: [PR] #2185 URLFrontier Spout: depth metrics to observe the breadth of the crawl [stormcrawler]
via GitHub
Re: [PR] #2185 URLFrontier Spout: depth metrics to observe the breadth of the crawl [stormcrawler]
via GitHub
[I] URLFrontier Spout: depth metrics to observe and drive the breadth of the crawl [stormcrawler]
via GitHub
Re: [I] URLFrontier Spout: depth metrics to observe and drive the breadth of the crawl [stormcrawler]
via GitHub
Re: [I] URLFrontier Spout: depth metrics to observe and drive the breadth of the crawl [stormcrawler]
via GitHub
Re: [I] URLFrontier Spout: depth metrics to observe and drive the breadth of the crawl [stormcrawler]
via GitHub
Re: [I] URLFrontier Spout: depth metrics to observe and drive the breadth of the crawl [stormcrawler]
via GitHub
Re: [I] URLFrontier Spout: depth metrics to observe and drive the breadth of the crawl [stormcrawler]
via GitHub
Re: [I] URLFrontier Spout: depth metrics to observe and drive the breadth of the crawl [stormcrawler]
via GitHub
Re: [I] URLFrontier Spout: depth metrics to observe and drive the breadth of the crawl [stormcrawler]
via GitHub
Re: [I] URLFrontier Spout: depth metrics to observe and drive the breadth of the crawl [stormcrawler]
via GitHub
[GH] (stormcrawler/fix/2058-queue-timeout-metric): Workflow run "Java CI with Maven" is working again!
GitBox
[GH] (stormcrawler/fix/2058-queue-timeout-metric): Workflow run "Java CI with Maven" failed!
GitBox
[PR] #2058 FetcherBolt: count the URLs dropped by fetcher.timeout.queue [stormcrawler]
via GitHub
Re: [PR] #2058 FetcherBolt: count the URLs dropped by fetcher.timeout.queue [stormcrawler]
via GitHub
[GH] (stormcrawler/tika-pipes-parser-timeout): Workflow run "Java CI with Maven" is working again!
GitBox
[GH] (stormcrawler/tika-pipes-parser-timeout): Workflow run "Java CI with Maven" is working again!
GitBox
[GH] (stormcrawler/tika-pipes-parser-timeout): Workflow run "Java CI with Maven" is working again!
GitBox
[GH] (stormcrawler/tika-pipes-parser-timeout): Workflow run "Java CI with Maven" failed!
GitBox
[GH] (stormcrawler/tika-pipes-parser-timeout): Workflow run "Java CI with Maven" failed!
GitBox
[GH] (stormcrawler/tika-pipes-parser-timeout): Workflow run "Java CI with Maven" failed!
GitBox
[PR] Tika pipes parser timeout [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
Re: [PR] 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs [stormcrawler]
via GitHub
[GH] (stormcrawler/fix-2097-tika-parse-timeout): Workflow run "Java CI with Maven" failed!
GitBox
[PR] #2093 LLM text extractor: keep page markers out of the prompt and clean the reply [stormcrawler]
via GitHub
Re: [PR] #2093 LLM text extractor: keep page markers out of the prompt and clean the reply [stormcrawler]
via GitHub
Re: [PR] #2093 LLM text extractor: keep page markers out of the prompt and clean the reply [stormcrawler]
via GitHub
Re: [PR] #2093 LLM text extractor: keep page markers out of the prompt and clean the reply [stormcrawler]
via GitHub
Re: [PR] #2093 LLM text extractor: keep page markers out of the prompt and clean the reply [stormcrawler]
via GitHub
[PR] #2097 Tika ParserBolt: optional timeout for the parse with parser.tika.timeout [stormcrawler]
via GitHub
Re: [PR] #2097 Tika ParserBolt: optional timeout for the parse with parser.tika.timeout [stormcrawler]
via GitHub
Re: [PR] #2097 Tika ParserBolt: optional timeout for the parse with parser.tika.timeout [stormcrawler]
via GitHub
Re: [PR] #2097 Tika ParserBolt: optional timeout for the parse with parser.tika.timeout [stormcrawler]
via GitHub
Re: [PR] #2097 Tika ParserBolt: optional timeout for the parse with parser.tika.timeout [stormcrawler]
via GitHub
Re: [PR] #2097 Tika ParserBolt: optional timeout for the parse with parser.tika.timeout [stormcrawler]
via GitHub
Re: [PR] #2097 Tika ParserBolt: optional timeout for the parse with parser.tika.timeout [stormcrawler]
via GitHub
Re: [PR] #2097 Tika ParserBolt: optional timeout for the parse with parser.tika.timeout [stormcrawler]
via GitHub
Re: [PR] #2097 Tika ParserBolt: optional timeout for the parse with parser.tika.timeout [stormcrawler]
via GitHub
Re: [PR] #2097 Tika ParserBolt: optional timeout for the parse with parser.tika.timeout [stormcrawler]
via GitHub
Re: [PR] #2097 Tika ParserBolt: optional timeout for the parse with parser.tika.timeout [stormcrawler]
via GitHub
Re: [PR] #2097 Tika ParserBolt: optional timeout for the parse with parser.tika.timeout [stormcrawler]
via GitHub
[GH] (stormcrawler/fix-2098-urlfrontier-tls): Workflow run "Java CI with Maven" is working again!
GitBox
[GH] (stormcrawler/fix-2103-opensearch-credentials-scope): Workflow run "Java CI with Maven" failed!
GitBox
sending fetched pages to a remote tika over grpc
Davide Polato
[PR] build(deps): bump software.amazon.awssdk:bom from 2.54.16 to 2.55.0 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump software.amazon.awssdk:bom from 2.54.16 to 2.55.0 [stormcrawler]
via GitHub
[PR] build(deps): bump com.microsoft.playwright:playwright from 1.62.0 to 1.63.0 [stormcrawler]
via GitHub
Re: [PR] build(deps): bump com.microsoft.playwright:playwright from 1.62.0 to 1.63.0 [stormcrawler]
via GitHub
[PR] #2097 Tika ParserBolt: limit the extracted text with parser.tika.text.maxlength [stormcrawler]
via GitHub
Re: [PR] #2097 Tika ParserBolt: limit the extracted text with parser.tika.text.maxlength [stormcrawler]
via GitHub
Re: [PR] #2097 Tika ParserBolt: limit the extracted text with parser.tika.text.maxlength [stormcrawler]
via GitHub
[PR] #2098 URLFrontier: optional TLS for the channels to the frontier [stormcrawler]
via GitHub
Re: [PR] #2098 URLFrontier: optional TLS for the channels to the frontier [stormcrawler]
via GitHub
Re: [PR] #2098 URLFrontier: optional TLS for the channels to the frontier [stormcrawler]
via GitHub
Re: [PR] #2098 URLFrontier: optional TLS for the channels to the frontier [stormcrawler]
via GitHub
Re: [PR] #2098 URLFrontier: optional TLS for the channels to the frontier [stormcrawler]
via GitHub
[PR] #2103 OpenSearchConnection: send the credentials only to the configured addresses [stormcrawler]
via GitHub
Re: [PR] #2103 OpenSearchConnection: send the credentials only to the configured addresses [stormcrawler]
via GitHub
Re: [PR] #2103 OpenSearchConnection: send the credentials only to the configured addresses [stormcrawler]
via GitHub
[PR] #2166 SiteMapParserBolt: cap the number of URLs emitted per sitemap [stormcrawler]
via GitHub
Re: [PR] #2166 SiteMapParserBolt: cap the number of URLs emitted per sitemap [stormcrawler]
via GitHub
[GH] (stormcrawler/2095-playwright-ip-filter): Workflow run "Java CI with Maven" is working again!
GitBox
[GH] (stormcrawler/fix-2099-metrics-header-parse): Workflow run "Java CI with Maven" is working again!
GitBox
[GH] (stormcrawler/fix-2091-reserved-metadata-keys): Workflow run "Java CI with Maven" failed!
GitBox
[GH] (stormcrawler/fix-2091-reserved-metadata-keys): Workflow run "Java CI with Maven" failed!
GitBox
[PR] #2101 HostURLFilter: drop the source host memoization shared by all fetcher threads [stormcrawler]
via GitHub
Re: [PR] #2101 HostURLFilter: drop the source host memoization shared by all fetcher threads [stormcrawler]
via GitHub
[PR] #2091 Protocols: reject metadata keys arriving in a response header [stormcrawler]
via GitHub
Re: [PR] #2091 Protocols: reject metadata keys arriving in a response header [stormcrawler]
via GitHub
Re: [PR] #2091 Protocols: reject metadata keys arriving in a response header [stormcrawler]
via GitHub
Re: [PR] #2091 Protocols: reject metadata keys arriving in a response header [stormcrawler]
via GitHub
[PR] #2099 Fetcher bolts: skip a metrics.* value which is not a number [stormcrawler]
via GitHub
Re: [PR] #2099 Fetcher bolts: skip a metrics.* value which is not a number [stormcrawler]
via GitHub
Re: [PR] #2099 Fetcher bolts: skip a metrics.* value which is not a number [stormcrawler]
via GitHub
Re: [PR] #2099 Fetcher bolts: skip a metrics.* value which is not a number [stormcrawler]
via GitHub
[PR] #2100 SiteMapParserBolt: read the content type under the protocol prefix [stormcrawler]
via GitHub
Re: [PR] #2100 SiteMapParserBolt: read the content type under the protocol prefix [stormcrawler]
via GitHub
Re: [PR] #2100 SiteMapParserBolt: read the content type under the protocol prefix [stormcrawler]
via GitHub
Re: [PR] #2100 SiteMapParserBolt: read the content type under the protocol prefix [stormcrawler]
via GitHub
[PR] #2090 Fetcher bolts: drop a set-header value coming from a response [stormcrawler]
via GitHub
Re: [PR] #2090 Fetcher bolts: drop a set-header value coming from a response [stormcrawler]
via GitHub
[I] [Improvement] Add a cap on the number of outlinks emitted per sitemap [stormcrawler]
via GitHub
Re: [I] [Improvement] Add a cap on the number of outlinks emitted per sitemap [stormcrawler]
via GitHub
[GH] (stormcrawler/perf/fetch-timeout-call-cancel): Workflow run "Java CI with Maven" failed!
GitBox
[GH] (stormcrawler/meta-charset-overflow): Workflow run "Java CI with Maven" is working again!
GitBox
[GH] (stormcrawler/meta-charset-overflow): Workflow run "Java CI with Maven" is working again!
GitBox
[PR] #2095 Apply the IP address filter to the requests made by the Playwright browser [stormcrawler]
via GitHub
Re: [PR] #2095 Apply the IP address filter to the requests made by the Playwright browser [stormcrawler]
via GitHub
Re: [PR] #2095 Apply the IP address filter to the requests made by the Playwright browser [stormcrawler]
via GitHub
Re: [PR] #2095 Apply the IP address filter to the requests made by the Playwright browser [stormcrawler]
via GitHub
[PR] #2162 Send credential headers from http.custom.headers only to the hosts in http.custom.headers.hosts [stormcrawler]
via GitHub
Re: [PR] #2162 Send credential headers from http.custom.headers only to the hosts in http.custom.headers.hosts [stormcrawler]
via GitHub
[I] Scope credential headers in http.custom.headers to hosts [stormcrawler]
via GitHub
Re: [I] Scope credential headers in http.custom.headers to hosts [stormcrawler]
via GitHub
[PR] #2089 Send http.basicauth credentials only to the hosts in http.basicauth.hosts [stormcrawler]
via GitHub
Re: [PR] #2089 Send http.basicauth credentials only to the hosts in http.basicauth.hosts [stormcrawler]
via GitHub
Re: [PR] #2089 Send http.basicauth credentials only to the hosts in http.basicauth.hosts [stormcrawler]
via GitHub
Earlier messages