Re: [PR] SOLR-16903: Migrate off java.io.File to java.nio.file.Path from core files [solr]
dsmiley commented on code in PR #2924:
URL: https://github.com/apache/solr/pull/2924#discussion_r1936591016
##
solr/core/src/java/org/apache/solr/schema/ManagedIndexSchemaFactory.java:
##
@@ -410,7 +410,9 @@ private void upgradeToManagedSchema() {
"nor under SolrConfig.getConfigDir() or the current directory. ",
"PLEASE REMOVE THIS FILE.");
} else {
- Path upgradedSchemaFile = Path.of(nonManagedSchemaFile +
UPGRADED_SCHEMA_EXTENSION);
+ Path upgradedSchemaFile =
+ nonManagedSchemaFile.resolveSibling(
Review Comment:
I love the use of resolveSibling where you've introduced it!
##
solr/core/src/java/org/apache/solr/core/DirectoryFactory.java:
##
@@ -338,59 +335,66 @@ public String getDataHome(CoreDescriptor cd) throws
IOException {
}
public void cleanupOldIndexDirectories(
- final String dataDirPath, final String currentIndexDirPath, boolean
afterCoreReload) {
+ final String dataDirPath, final String currentIndexDirPath, boolean
afterCoreReload)
+ throws IOException {
-// TODO SOLR-8282 move to PATH
-File dataDir = new File(dataDirPath);
-if (!dataDir.isDirectory()) {
+Path dataDirFile = Path.of(dataDirPath);
+if (!Files.isDirectory(dataDirFile)) {
log.debug(
"{} does not point to a valid data directory; skipping clean-up of
old index directories.",
dataDirPath);
return;
}
-final File currentIndexDir = new File(currentIndexDirPath);
-File[] oldIndexDirs =
-dataDir.listFiles(
-new FileFilter() {
- @Override
- public boolean accept(File file) {
-String fileName = file.getName();
-return file.isDirectory()
-&& !file.equals(currentIndexDir)
-&& (fileName.equals("index") ||
fileName.matches(INDEX_W_TIMESTAMP_REGEX));
- }
-});
+final Path currentIndexDir = Path.of(currentIndexDirPath);
+List dirsList;
+try (Stream oldIndexDirs = Files.list(dataDirFile)) {
+ dirsList =
+ oldIndexDirs
+ .filter(
+ (file) -> {
+String fileName = file.getFileName().toString();
+return Files.isDirectory(file)
+&& !file.equals(currentIndexDir)
+&& (fileName.equals("index") ||
fileName.matches(INDEX_W_TIMESTAMP_REGEX));
+ })
+ .sorted()
+ .toList();
+}
+;
-if (oldIndexDirs == null || oldIndexDirs.length == 0)
+if (dirsList.isEmpty()) {
return; // nothing to do (no log message needed)
-
-List dirsList = Arrays.asList(oldIndexDirs);
-dirsList.sort(Collections.reverseOrder());
Review Comment:
I just noticed this reverseOrder; you didn't retain this detail.
##
solr/core/src/java/org/apache/solr/core/SolrCore.java:
##
@@ -1354,16 +1353,18 @@ public void initializeMetrics(SolrMetricsContext
parentContext, String scope) {
Category.CORE.toString());
}
-// TODO SOLR-8282 move to PATH
// initialize disk total / free metrics
-Path dataDirPath = Paths.get(dataDir);
-File dataDirFile = dataDirPath.toFile();
-parentContext.gauge(
-() -> dataDirFile.getTotalSpace(), true, "totalSpace",
Category.CORE.toString(), "fs");
-parentContext.gauge(
-() -> dataDirFile.getUsableSpace(), true, "usableSpace",
Category.CORE.toString(), "fs");
+Path dataDirFile = Paths.get(dataDir);
+
+// Do not pre-compute the data directory total/usable space on
initialization. Calculated when
+// metric is fetched
+// TODO Move metrics initialization calls to after the data directory is
created to fetch true
Review Comment:
If you want to move it, I think it's okay to do it in this PR as it seems
like a minor detail. But I'm confused at the current state; where is the
calculation occurring? I see hard-coded 0 now.
##
solr/core/src/java/org/apache/solr/core/SolrPaths.java:
##
@@ -94,7 +94,7 @@ public static void assertNoPathTraversal(Path pathToAssert) {
/** Asserts that a path is not a Windows UNC path */
public static void assertNotUnc(Path pathToAssert) {
-if (OS.isFamilyWindows() && pathToAssert.toString().startsWith("")) {
+if (OS.isFamilyWindows() || pathToAssert.toString().startsWith("")) {
Review Comment:
but OR means that this would consistently fail on Windows *even if the path
isn't UNC*
##
solr/core/src/java/org/apache/solr/core/SolrResourceLoader.java:
##
@@ -372,12 +372,16 @@ public InputStream openResource(String resource) throws
IOException {
// Delegate to the class loader (looking into $INSTANCE_DIR/lib jars).
// We need a ClassLoader-compatible (forward-slashes) path here!
-InputStream is = classLoader.
Re: [PR] SOLR-17447 : Support to early terminate a search based on maxHits per collector. [solr]
gus-asf commented on PR #2960: URL: https://github.com/apache/solr/pull/2960#issuecomment-2625369713 > (not thinking of backwards compatibility here, just thinking holistically) > > I appreciate that there are very different causes of a terminated request, and we want to differentiate them in case the client cares and for diagnostics. But semantically to the caller, the cause is irrelevant (right?), so I think a simple flag/enum should be returned _separate_ for the cause. I'm not arguing the cause should be hidden / indiscernible. I understand, once upon a time, there was a single cause of "terminate early" but as Solr has matured to do so under growing conditions, we shouldn't be held back by segment terminated early as the one and only terminate-early. Let's step back and change names / semantics if needed. Solr 10 is coming. If by "caller" you mean the person/programmer/code that is leveraging solr to make a fabulous profit or save the world then there are two questions to answer: 1) can we trust these results as 100% definitive 2) what can be done to obtain definitive results if that's important. For number 1 we don't need to know why we just need to know if it's trustworthy, that's entirely binary as you suggest. However if we need to adjust things to supply definitive results (for that one case the customer really cares about, etc) then we need to know which nob to turn, and thus it matters which option is the limiting factor (one might set multiple limits or be using limits with this feature). If by caller you mean the next (few) method(s) up the stack, then there is at least one important distinction between this and limits. In the limits case, we typically want to stop all work ASAP once the limit is hit. In this case the intent is that all shards should return up to the limiting number of hits. We wouldn't want to stop or terminate any other subrequests for this feature. Notably this feature would not protect the users from a shard living on a critically limping and slow server, whereas timeAllowed should ensure some sort of response even if one shard has nearly (but not quite) ground to a halt. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17623: Simple ordered map should implement map [solr]
dsmiley commented on PR #3048: URL: https://github.com/apache/solr/pull/3048#issuecomment-2625396992 I removed the asShallowMap returning SimpleOrderedMap so such a decision can be in Solr 10. I intend on merging this when the tests pass and back porting to 9x. Today/tomorrow. Thanks again for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17637) HttpShardHandler can wait indefinitely when retrying after certain failures
[ https://issues.apache.org/jira/browse/SOLR-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922513#comment-17922513 ] ASF subversion and git services commented on SOLR-17637: Commit db0cbd38d512da22e99fbbb0e4f65fc720de5441 in solr's branch refs/heads/main from Houston Putman [ https://gitbox.apache.org/repos/asf?p=solr.git;h=db0cbd38d51 ] SOLR-17637: Fix LBHttpSolrClient & HttpShardHandler bug (#3147) This bug causes async requests to be uncompleted in some error scenarios. The HttpShardHandler can hang indefinitely when this happens. > HttpShardHandler can wait indefinitely when retrying after certain failures > --- > > Key: SOLR-17637 > URL: https://issues.apache.org/jira/browse/SOLR-17637 > Project: Solr > Issue Type: Bug >Affects Versions: 9.8 >Reporter: Houston Putman >Priority: Critical > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > If a The LBHttp2SolrClient is used to do an asynchronous request, then any > error that has a null message can result in that request not having its > future completed. The HttpShardHandler loops indefinitely until all responses > have been processed. So if a future is not completed, then its response will > not be added to the list. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17637: Fix LBHttpSolrClient & HttpShardHandler bug [solr]
HoustonPutman merged PR #3147: URL: https://github.com/apache/solr/pull/3147 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17637) HttpShardHandler can wait indefinitely when retrying after certain failures
[ https://issues.apache.org/jira/browse/SOLR-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922516#comment-17922516 ] ASF subversion and git services commented on SOLR-17637: Commit 973ef46fa0f6e48b2d2204ffcaeee9dbb6053d41 in solr's branch refs/heads/branch_9x from Houston Putman [ https://gitbox.apache.org/repos/asf?p=solr.git;h=973ef46fa0f ] SOLR-17637: Fix LBHttpSolrClient & HttpShardHandler bug (#3147) This bug causes async requests to be uncompleted in some error scenarios. The HttpShardHandler can hang indefinitely when this happens. (cherry picked from commit db0cbd38d512da22e99fbbb0e4f65fc720de5441) > HttpShardHandler can wait indefinitely when retrying after certain failures > --- > > Key: SOLR-17637 > URL: https://issues.apache.org/jira/browse/SOLR-17637 > Project: Solr > Issue Type: Bug >Affects Versions: 9.8 >Reporter: Houston Putman >Priority: Critical > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > If a The LBHttp2SolrClient is used to do an asynchronous request, then any > error that has a null message can result in that request not having its > future completed. The HttpShardHandler loops indefinitely until all responses > have been processed. So if a future is not completed, then its response will > not be added to the list. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17623: Simple ordered map should implement map [solr]
renatoh commented on PR #3048: URL: https://github.com/apache/solr/pull/3048#issuecomment-2625544960 > I removed the asShallowMap returning SimpleOrderedMap so such a decision can be in Solr 10. I intend on merging this when the tests pass and back porting to 9x. Today/tomorrow. Thanks again for your contribution! You are welcome, thanks for your support! Now it is failing with: Detected license header issues (skip with -Pvalidation.rat.failOnError=false): Unknown license: /home/runner/work/solr/solr/solr/solrj/src/test/org/apache/solr/common/util/SimpleOrderedMapTest.java No idea what this means -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17619) Generate CHANGELOG.md (formerly CHANGES.txt) via logchange
[ https://issues.apache.org/jira/browse/SOLR-17619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922529#comment-17922529 ] Jan Høydahl commented on SOLR-17619: Or we could start encouraging committers to also use the PR checklist that contributors use, and add a bullet point «I have added logchange file and considered ref guide major changes» > Generate CHANGELOG.md (formerly CHANGES.txt) via logchange > -- > > Key: SOLR-17619 > URL: https://issues.apache.org/jira/browse/SOLR-17619 > Project: Solr > Issue Type: Task > Components: Build >Reporter: David Smiley >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > The [logchange|https://github.com/logchange/logchange] tool helps projects > maintain a log of changes. Each change (e.g. from a PR) will no longer edit a > CHANGES.txt file; instead it will include a _new_ YAML file in an appropriate > directory with others for the next Solr version. The release process will > execute the tool, which will build a CHANGELOG.md file and will probably also > do something with the YAML files (remove?). > Decide the most convenient way for us to run the tool for change authors. > Could a gradle task do it? See [this > issue|https://github.com/logchange/logchange/issues/397] filed on the > logchange project. > Outcome of this issue: > * a logchange tool configuration file -- logchange-config.yml > * Solr 10's CHANGES.txt entries converted to YAML. (start this issue by > doing only a few before over-investing) > * a dev-docs page > ** for contributors/everyone: basic info explaining how each new change > should be recorded. Explain how to run the tool to generate the YAML file. > What field(s) matter the most; what should be ignored. Link to further > details. > ** for release manager: how to produce CHANGELOG.md. Link to further > details. Ultimately this will probably move to the release wizard in some > fashion. > TBD: CHANGES.txt < 10 and CHANGELOG.md > 10 ? > TBD: changes.html generation in the release process will be removed or will > change. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Resolved] (SOLR-17637) HttpShardHandler can wait indefinitely when retrying after certain failures
[ https://issues.apache.org/jira/browse/SOLR-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Houston Putman resolved SOLR-17637. --- Fix Version/s: 9.8.1 Assignee: Houston Putman Resolution: Fixed > HttpShardHandler can wait indefinitely when retrying after certain failures > --- > > Key: SOLR-17637 > URL: https://issues.apache.org/jira/browse/SOLR-17637 > Project: Solr > Issue Type: Bug >Affects Versions: 9.8 >Reporter: Houston Putman >Assignee: Houston Putman >Priority: Critical > Labels: pull-request-available > Fix For: 9.8.1 > > Time Spent: 1h > Remaining Estimate: 0h > > If a The LBHttp2SolrClient is used to do an asynchronous request, then any > error that has a null message can result in that request not having its > future completed. The HttpShardHandler loops indefinitely until all responses > have been processed. So if a future is not completed, then its response will > not be added to the list. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17642) Knn queries use multi threaded search unintentionally
[
https://issues.apache.org/jira/browse/SOLR-17642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-17642:
-
Affects Version/s: 9.8
9.7
> Knn queries use multi threaded search unintentionally
> ---
>
> Key: SOLR-17642
> URL: https://issues.apache.org/jira/browse/SOLR-17642
> Project: Solr
> Issue Type: Bug
>Affects Versions: 9.7, 9.8
>Reporter: Varun Thacker
>Priority: Major
>
> First reported on
> [https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh|https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh,],
> Dr. Andreas Moll reported a failing query.
> I was able to reproduce is consistently locally on 9.7 and 9.8 using Dr.
> Andreas Moll's repro steps.
> There are two bugs in play here
> *The Solr side bug:*
> * Solr 9.7 added SOLR-13350 but disabled multi-threaded search by default. I
> can confirm the failing query follows this code path
>
> {code:java}
> if (!MultiThreadedSearcher.allowMT(pf.postFilter, cmd)) {
> log.trace("SINGLE THREADED search, skipping collector manager in
> getDocListNC");{code}
>
> * However when following that code path the query goes to
> AbstractKnnVectorQuery and this is where executor thread pool gets used
> {code:java}
> TimeLimitingKnnCollectorManager knnCollectorManager =
> new TimeLimitingKnnCollectorManager(
> getKnnCollectorManager(k, indexSearcher), indexSearcher.getTimeout());
> TaskExecutor taskExecutor = indexSearcher.getTaskExecutor();
> {code}
> * This executor was added in SOLR-13350 in CoreContainer(
> indexSearcherExecutor) . If you follow the SolrIndexSearcher code that calls
> {code:java}
> In SolrIndexSearcher.java
> super(wrapReader(core, r),
> core.getCoreContainer().getIndexSearcherExecutor());{code}
> {code:java}
> In IndexSearcher.java
> this.executor = executor;
> this.taskExecutor =
> executor == null ? new TaskExecutor(Runnable::run) : new
> TaskExecutor(executor);{code}
> I think the idea in SOLR-13350 was create an indexSearcherExector, but only
> use it when multiThreaded=true as a query param (default=false). The fact we
> initialised indexSearcherExector means all KNN queries use that thread pool
> and there is no way to disable that.
>
> *Now the Lucene Bug:*
> [~benwtrent] noted on
> [https://lists.apache.org/thread/ftm6f32f2nsoys6noxx8p2ygpwnlfjc9] that
> Multi-threaded vector search over multiple segments can lead to inconsistent
> results
>
> So the fact Solr by default always uses multi threaded search 9.7+ triggers
> the lucene bug. But I think in Solr we should also have a way to not use
> multi-threaded KNN like we do for other queries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17623: Simple ordered map should implement map [solr]
dsmiley commented on PR #3048: URL: https://github.com/apache/solr/pull/3048#issuecomment-2625909437 As you are a very new contributor, I can tell by your changes and your last question that you have not been running `./gradlew precommit`, which will detect issues that you can correct before pushing. BTW all source files need a header. IntelliJ does it automatically for me, as I've configured it to. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16932: expose `AsyncTracker`'s max outstanding async req limit config in `Http2SolrClient` [solr]
github-actions[bot] commented on PR #2313: URL: https://github.com/apache/solr/pull/2313#issuecomment-2625974050 This PR is now closed due to 60 days of inactivity after being marked as stale. Re-opening this PR is still possible, in which case it will be marked as active again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16953: Grouped fields are handled in edismax clauses [solr]
github-actions[bot] commented on PR #1893: URL: https://github.com/apache/solr/pull/1893#issuecomment-2625974639 This PR is now closed due to 60 days of inactivity after being marked as stale. Re-opening this PR is still possible, in which case it will be marked as active again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16427: Enable error-prone ClassInitializationDeadlock rule [solr]
github-actions[bot] commented on PR #2881: URL: https://github.com/apache/solr/pull/2881#issuecomment-2625973468 This PR has had no activity for 60 days and is now labeled as stale. Any new activity will remove the stale label. To attract more reviewers, please tag people who might be familiar with the code area and/or notify the d...@solr.apache.org mailing list. To exempt this PR from being marked as stale, make it a draft PR or add the label "exempt-stale". If left unattended, this PR will be closed after another 60 days of inactivity. Thank you for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17015: Test and fix bug in MoveReplicaCmd for PRS collections [solr]
github-actions[bot] closed pull request #1986: SOLR-17015: Test and fix bug in MoveReplicaCmd for PRS collections URL: https://github.com/apache/solr/pull/1986 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16953: Grouped fields are handled in edismax clauses [solr]
github-actions[bot] closed pull request #1893: SOLR-16953: Grouped fields are handled in edismax clauses URL: https://github.com/apache/solr/pull/1893 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-16903) Use Path instead of File in Java Code
[ https://issues.apache.org/jira/browse/SOLR-16903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922399#comment-17922399 ] David Smiley commented on SOLR-16903: - BTW I don't think module-by-module is generally the way to go. I think it should be mostly by converting SolrCore methods, then CoreContainer methods, etc.. SolrCore data dir & DirectoryFactory deserves isolation. Any way, it's perhaps too late to organize the approach as the floodgates have opened to address them in every which way that occurs to contributors. > Use Path instead of File in Java Code > - > > Key: SOLR-16903 > URL: https://issues.apache.org/jira/browse/SOLR-16903 > Project: Solr > Issue Type: Improvement >Affects Versions: 9.3 >Reporter: Eric Pugh >Priority: Minor > Labels: newdev, pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > As a community, we have decided to migrate to using java.nio.file.Path in > place of java.io.File in our codebase. > This ticket is to go through the codebase and make that change. We'd also > like to add the java.io.File pattern to our forbidden-apis setup. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17447 : Support to early terminate a search based on maxHits per collector. [solr]
gus-asf commented on PR #2960: URL: https://github.com/apache/solr/pull/2960#issuecomment-2624803443 > > Overall this makes sense to me. Thanks for the nice contribution! I'd prefer if @gus-asf could take a look at some aspects since he worked on cpuAllowed recently. BTW it's clear you need to run `./gradlew tidy` > > Addressed. > > I suggest renaming the param "maxHitsPerShard" to simply "maxHits" or "maxHitsTerminateEarly" and document that it's per-shard and best-effort; that more hits may ultimately be detected in aggregate. But maybe others disagree. > > Renamed to maxHits > maybe shardMaxHits? Brevity is nice, but I think it's helpful to have a name that is at least slightly self documenting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-XXXXX: Support different filesystem implementations with EmbeddedSolrServer [solr]
dsmiley commented on PR #3146: URL: https://github.com/apache/solr/pull/3146#issuecomment-2624996827 Probably need to wait for @mlbiscoc in https://github.com/apache/solr/pull/2924 which is touching lots of stuff. Then pick a particular String based API endpoint and convert it in an isolated PR relating to that one method or class (if there are multiple). I think it's easier to review like this rather than touching everything. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17640) Support different filesystem implementations with EmbeddedSolrServer
Andrey Bozhko created SOLR-17640: Summary: Support different filesystem implementations with EmbeddedSolrServer Key: SOLR-17640 URL: https://issues.apache.org/jira/browse/SOLR-17640 Project: Solr Issue Type: Improvement Components: Embedded Affects Versions: 9.8 Reporter: Andrey Bozhko EmbeddedSolrServer has a public constructor `EmbeddedSolrServer(Path solrHome, String defaultCoreName)` - which suggests that any Path implementation is acceptable (e.g., ZipPath, S3Path, and not just UnixPath/WindowsPath). It would be good to try this out and see if EmbeddedSolrServer actually works with solrHome paths in filesystems other than the default one. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17447 : Support to early terminate a search based on maxHits per collector. [solr]
gus-asf commented on code in PR #2960:
URL: https://github.com/apache/solr/pull/2960#discussion_r1935817655
##
solr/core/src/java/org/apache/solr/search/QueryResult.java:
##
@@ -22,6 +22,7 @@ public class QueryResult {
// Object for back compatibility so that we render true not "true" in json
private Object partialResults;
private Boolean segmentTerminatedEarly;
Review Comment:
I agree it's good to know why so that the corrective action is clear, but
terminateEarly is unlike maxHits (or shardMaxHits) so the relationship takes
special knowledge to appreciate. If something is going to report a reason I'd
want it to include the same phrase... i.e. "maxHitsReached" and definately
would not want to have to "just know" that terminateEarly has nothing to do
with segmentTerminateEarly despite strong similarity in naming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17492: Introduce recommendations of WAYS of running Solr from small to massive [solr]
epugh commented on PR #2783: URL: https://github.com/apache/solr/pull/2783#issuecomment-2624787972 This remains on my "must do" list for Solr 10, and I will pick it up as we get closer ;-). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-XXXXX: Support different filesystem implementations with EmbeddedSolrServer [solr]
AndreyBozhko commented on PR #3146: URL: https://github.com/apache/solr/pull/3146#issuecomment-2624829745 Thanks - I just opened SOLR-17640 before reading your comment. Do you suggest this should go under SOLR-16903 instead? Or would you prefer to see a more comprehensive refactoring for Paths#get usage in the codebase? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17641) Support Java 24
Houston Putman created SOLR-17641: - Summary: Support Java 24 Key: SOLR-17641 URL: https://issues.apache.org/jira/browse/SOLR-17641 Project: Solr Issue Type: Bug Components: SolrCLI, Tests Reporter: Houston Putman The main thing here is to disable the Security Manager for Java 24+, both for the CLI and testing. Reference: https://github.com/apache/lucene/issues/14184 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-7504) Atomic Update causes solr.CountFieldValuesUpdateProcessorFactory to be wrong
[
https://issues.apache.org/jira/browse/SOLR-7504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922371#comment-17922371
]
Paul Heijman commented on SOLR-7504:
This is still an issue in 9.6.1.
It has nothing to do with setting fields to null:
{code:json}
{
"content_key": "ICS_100.RPT",
"researchpagekeycontentcategorynames": {
"add": {
"test"
}
}
}
{code}
will also set key_content_count to one...
> Atomic Update causes solr.CountFieldValuesUpdateProcessorFactory to be wrong
>
>
> Key: SOLR-7504
> URL: https://issues.apache.org/jira/browse/SOLR-7504
> Project: Solr
> Issue Type: Bug
> Components: SearchComponents - other
>Affects Versions: 4.10.4, 5.0, 5.1
>Reporter: Jay
>Priority: Major
> Fix For: 4.10.4, 5.1
>
>
> Below is a snippet of my solrconfig.xml
> {code}
>
>
>
> .*_count
>
>
>
> id
>
>
>
> researchpagekeycontentcategorynames
> key_content_count
>
>
> key_content_count
>
>
> key_content_count
> 0
>
> {code}
> Performing an Atomic Update like this causes that key_content_count to be one
> (1) when the expected value should be zero (0).
> To my knowledge setting a value to null should zero out the count, correct?
> {code}
> curl -s 'http://localhost:9080/solr/ndr/update' -H
> 'Content-type:application/json' -d
> '[{"content_key":"ICS_100.RPT","researchpagekeycontentcategorynames":{"set":null}}]'
> {code}
> Thanks for looking into this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-XXXXX: Support different filesystem implementations with EmbeddedSolrServer [solr]
epugh commented on code in PR #3146: URL: https://github.com/apache/solr/pull/3146#discussion_r1935674798 ## solr/CHANGES.txt: ## @@ -34,6 +34,8 @@ Improvements * SOLR-17516: `LBHttp2SolrClient` is now generic, adding support for `HttpJdkSolrClient`. (James Dyer) +* SOLR-X: Support different filesystem implementations with EmbeddedSolrServer (Andrey Bozhko) Review Comment: DOn't we have a `GH` or `GITHUB` type name for a PR with no JIRA? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-XXXXX: Support different filesystem implementations with EmbeddedSolrServer [solr]
dsmiley commented on code in PR #3146:
URL: https://github.com/apache/solr/pull/3146#discussion_r1935720303
##
solr/core/src/java/org/apache/solr/schema/ManagedIndexSchemaFactory.java:
##
@@ -148,12 +147,11 @@ public String lookupZKManagedSchemaPath() {
*/
public Path lookupLocalManagedSchemaPath() {
final Path legacyManagedSchemaPath =
-Paths.get(
Review Comment:
Absolutely; same with Path.of. _Ultimately_ we should ForbiddenApi check
this so that it's rare to call this except for a few authorized places.
##
solr/core/src/test/org/apache/solr/client/solrj/embedded/TestEmbeddedSolrServerConstructors.java:
##
@@ -58,4 +63,63 @@ public void testNodeConfigConstructor() throws Exception {
assertEquals(1, server.query("newcore", new
SolrQuery("*:*")).getResults().getNumFound());
}
}
+
+ @SuppressWarnings("removal")
+ @Test
+ public void testPathConstructorZipFS() throws Exception {
+assumeTrue(
+"""
+Test only works without Security Manager due to
SecurityConfHandlerLocal
+missing permission to read /1/2/3/4/security.json file""",
+System.getSecurityManager() == null);
Review Comment:
CoreContainer.solrHome is a Path so hopefully it won't be too huge to switch
it's getter to use Path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[I] Scaling Down Causes "Down" Replicas [solr-operator]
Kamalsaiperla opened a new issue, #754: URL: https://github.com/apache/solr-operator/issues/754 **Environment** Solr Operator Version: 0.8.1 → 0.9.0 (same issue) Solr Image Version: 9.6.1 Platform: GKE Custom Plugins: Yes HPA Configuration: Configured for CPU-based scaling **Issue Description** When scaling up (averageUtilization=10%), Solr pods successfully scale to the maxReplicas (10) without issues. However, when scaling down (averageUtilization=80%), Solr does not reduce the number of pods, and several shards show "Down" replicas. **Steps to Reproduce** Deploy Solr Operator (0.8.1, later tested with 0.9.0) with Solr 9.6.1. Configure an HPA with CPU-based scaling. Create collections and insert documents. Test 1: Decrease averageUtilization to 10% → Pods scale up to 10 (expected behavior). Test 2: Increase averageUtilization to 80% → Pods do not scale down, and some shards show "Down" replicas. **Expected Behavior** When increasing averageUtilization, pods should scale down as per HPA settings. Shards should not end up in "Down" state. **Observed Behavior** Pods remain at max (10). Some shards have "Down" replicas. **Additional Information** Upgrading the Solr Operator from 0.8.1 to 0.9.0 did not resolve the issue. Manually deleting some pods forces the cluster to recover, but this should not be required. Screenshots https://github.com/user-attachments/assets/bfdd6427-c575-428f-bb58-11dcdf6ac751"; /> https://github.com/user-attachments/assets/7b6f8935-fa20-4fd8-928b-d4d102f2eb1c"; /> https://github.com/user-attachments/assets/d0697535-9258-49ae-a250-34dbdf739f03"; /> https://github.com/user-attachments/assets/c1ea2c9a-d2d5-488f-abfb-22617d410f19"; /> https://github.com/user-attachments/assets/9bca0ee1-5477-4d93-bb57-02b570a33113"; /> **Logs:** `2025-01-30 16:58:24.643 ERROR (qtp1155769010-5575-search-solrcloud-4.csr-58880) [c:l5RecommendationCollection s:shard2 r:core_node502 x:l5RecommendationCollection_shard2_replica_n501 t:search-solrcloud-4.csr-58880] o.a.s.u.UpdateLog Exception reading versions from log => java.io.EOFException at org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:79) java.io.EOFException: null at org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:79) ~[?:?] at org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:239) ~[?:?] at org.apache.solr.update.TransactionLog$FSReverseReader.(TransactionLog.java:889) ~[?:?] at org.apache.solr.update.TransactionLog.getReverseReader(TransactionLog.java:705) ~[?:?] at org.apache.solr.update.UpdateLog$RecentUpdates.update(UpdateLog.java:1613) ~[?:?] at org.apache.solr.update.UpdateLog$RecentUpdates.(UpdateLog.java:1528) ~[?:?] at org.apache.solr.update.UpdateLog.getRecentUpdates(UpdateLog.java:1727) ~[?:?] at org.apache.solr.handler.component.RealTimeGetComponent.processGetVersions(RealTimeGetComponent.java:1262) ~[?:?] at org.apache.solr.handler.component.RealTimeGetComponent.process(RealTimeGetComponent.java:161) ~[?:?] at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:465) ~[?:?] at org.apache.solr.handler.RealTimeGetHandler.handleRequestBody(RealTimeGetHandler.java:43) ~[?:?] at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:226) ~[?:?] at org.apache.solr.core.SolrCore.execute(SolrCore.java:2886) ~[?:?] at org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:910) ~[?:?] at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:596) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:262) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:219) ~[?:?] at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:249) ~[?:?] at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:215) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) ~[?:?] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.20.jar:10.0.20] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.20.jar:10.0.20] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.20.jar:10.0.20] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.20.jar:10.0.20] at org.eclipse.jett
Re: [PR] SOLR-17447 : Support to early terminate a search based on maxHits per collector. [solr]
gus-asf commented on code in PR #2960:
URL: https://github.com/apache/solr/pull/2960#discussion_r1935871739
##
solr/core/src/java/org/apache/solr/search/EarlyTerminatingCollector.java:
##
@@ -29,11 +30,15 @@
*/
public class EarlyTerminatingCollector extends FilterCollector {
+ private final int chunkSize = 100; // Check across threads only at a chunk
size
Review Comment:
How was this chosen? Should it be tuneable via env/sysprop?
##
solr/solr-ref-guide/modules/query-guide/pages/common-query-parameters.adoc:
##
@@ -399,6 +399,23 @@ For example, setting `cpuAllowed=500` gives a limit of at
most 500 ms of CPU tim
All other considerations regarding partial results listed for the
`timeAllowed` parameter apply here, too.
+
+== maxHits Parameter
+
+[%autowidth,frame=none]
+|===
+|Optional |Default: `false`
+|===
+
+This parameter specifies the max number of hits a searcher will iterate
through before early terminating the search.
+The count is per shard and across all threads involved in case of
multi-threaded search. This parameter works
+in conjunction with other parameters that could early terminate a search, ex:
_timeAllowed_ etc. In case the search
+was early terminated due to it exceeding maxHits a _terminatedEarly_ header in
the response will be set along with
+_partialResults_ to indicate the same. Note that the _partialResults_ flag
could be set in the absence of the _maxHits_
+parameter due to other limits like _timeAllowed_ or _cpuAllowed_.
+Note : the hits counted will need not be exactly equal to the maxHits
provided, however it will be within range of this value.
+
+
Review Comment:
I have a similar concern as Houston regarding helping the user understand
when to use this, here's my attempt to rewrite your text based on what I've
gathered... of course if I misunderstood something or I'm too wordy we should
fix that ;) As a side note I think we're supposed to write these as "ventilated
prose" which is one sentence per line, no line breaks in a sentence.
```
== maxHits Parameter
[%autowidth,frame=none]
|===
|Optional |Default: `false`
|===
This parameter specifies the max number of hits a searcher will iterate
through.
Searchers will arbitrarily ignore any number of additional hits beyond this
value.
Practically speaking, the count is per shard and across all threads involved
in case of multi-threaded search.
The intention of this feature is to favor speed over perfect relevancy &
recall.
The trade-off is that if one shard contains many relevant hits and another
contains a few less relevant hits the less relevant hits from the second shard
may get returned instead of the more relevant hits that were clustered in the
first shard.
This parameter works in conjunction with other parameters that could early
terminate a search, ex: _timeAllowed_ etc.
In case the search was early terminated due to it exceeding maxHits a
_terminatedEarly_ header in the response will be set along with
_partialResults_ to indicate the same.
The _partialResults_ flag could be set in the absence of the _maxHits_
parameter due to other limits like _timeAllowed_ or _cpuAllowed_.
```
I have to admit I don't understand this bit:
```
NOTE: the hits counted will need not be exactly equal to the maxHits
provided, however it will be within range of this value.
```
##
solr/core/src/java/org/apache/solr/search/QueryCommand.java:
##
@@ -194,7 +195,8 @@ public QueryCommand setNeedDocSet(boolean needDocSet) {
}
public boolean getTerminateEarly() {
-return (flags & SolrIndexSearcher.TERMINATE_EARLY) != 0;
+return (flags & SolrIndexSearcher.TERMINATE_EARLY) != 0
Review Comment:
As I noted in another comment I think this and the Lucene feature are
significantly different. I don't like conflating them.
##
solr/core/src/java/org/apache/solr/search/MultiThreadedSearcher.java:
##
@@ -190,10 +194,16 @@ public ScoreMode scoreMode() {
static class SearchResult {
final ScoreMode scoreMode;
private final Object[] result;
+final boolean isTerminatedEarly;
public SearchResult(ScoreMode scoreMode, Object[] result) {
+ this(scoreMode, result, false);
Review Comment:
My IDE claims this unused. Since this is a package private inner class we
probably don't need to maintain this constructor for the API.
##
solr/core/src/java/org/apache/solr/search/EarlyTerminatingCollector.java:
##
@@ -61,11 +69,25 @@ public LeafCollector getLeafCollector(LeafReaderContext
context) throws IOExcept
public void collect(int doc) throws IOException {
super.collect(doc);
numCollected++;
-if (maxDocsToCollect <= numCollected) {
+terminatedEarly = maxDocsToCollect <= numCollected;
+if (numCollected % chunkSize == 0) {
+ pendingDocsToCollect.add(chunkSize);
+
Re: [PR] SOLR-17447 : Support to early terminate a search based on maxHits per collector. [solr]
dsmiley commented on PR #2960: URL: https://github.com/apache/solr/pull/2960#issuecomment-2625311557 (not thinking of backwards compatibility here, just thinking holistically) I appreciate that there are very different causes of a terminated request, and we want to differentiate them in case the client cares and for diagnostics. But semantically to the caller, the cause is irrelevant (right?), so I think a simple flag/enum should be returned _separate_ for the cause. I'm not arguing the cause should be hidden / indiscernible. I understand, once upon a time, there was a single cause of "terminate early" but as Solr has matured to do so under growing conditions, we shouldn't be held back by segment terminated early as the one and only terminate-early. Let's step back and change names / semantics if needed. Solr 10 is coming. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17637: Fix LBHttpSolrClient & HttpShardHandler bug [solr]
gus-asf commented on PR #3147: URL: https://github.com/apache/solr/pull/3147#issuecomment-2625307927 Nice catch! thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-XXXXX: Support different filesystem implementations with EmbeddedSolrServer [solr]
dsmiley commented on code in PR #3146: URL: https://github.com/apache/solr/pull/3146#discussion_r1936130337 ## solr/CHANGES.txt: ## @@ -34,6 +34,8 @@ Improvements * SOLR-17516: `LBHttp2SolrClient` is now generic, adding support for `HttpJdkSolrClient`. (James Dyer) +* SOLR-X: Support different filesystem implementations with EmbeddedSolrServer (Andrey Bozhko) Review Comment: I believe he put this here as a TODO. We have quite a number of existing JIRAs to pick from for this theme :-). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17637) HttpShardHandler can wait indefinitely when retrying after certain failures
[ https://issues.apache.org/jira/browse/SOLR-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922527#comment-17922527 ] ASF subversion and git services commented on SOLR-17637: Commit fc1d3e1656afcb08b4a731a4f3db5bd0950a0ce1 in solr's branch refs/heads/branch_9_8 from Houston Putman [ https://gitbox.apache.org/repos/asf?p=solr.git;h=fc1d3e1656a ] SOLR-17637: Fix LBHttpSolrClient & HttpShardHandler bug (#3147) This bug causes async requests to be uncompleted in some error scenarios. The HttpShardHandler can hang indefinitely when this happens. (cherry picked from commit db0cbd38d512da22e99fbbb0e4f65fc720de5441) > HttpShardHandler can wait indefinitely when retrying after certain failures > --- > > Key: SOLR-17637 > URL: https://issues.apache.org/jira/browse/SOLR-17637 > Project: Solr > Issue Type: Bug >Affects Versions: 9.8 >Reporter: Houston Putman >Priority: Critical > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > If a The LBHttp2SolrClient is used to do an asynchronous request, then any > error that has a null message can result in that request not having its > future completed. The HttpShardHandler loops indefinitely until all responses > have been processed. So if a future is not completed, then its response will > not be added to the list. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16932: expose `AsyncTracker`'s max outstanding async req limit config in `Http2SolrClient` [solr]
github-actions[bot] closed pull request #2313: SOLR-16932: expose `AsyncTracker`'s max outstanding async req limit config in `Http2SolrClient` URL: https://github.com/apache/solr/pull/2313 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17015: Test and fix bug in MoveReplicaCmd for PRS collections [solr]
github-actions[bot] commented on PR #1986: URL: https://github.com/apache/solr/pull/1986#issuecomment-2625974285 This PR is now closed due to 60 days of inactivity after being marked as stale. Re-opening this PR is still possible, in which case it will be marked as active again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17582) CLUSTERSTATUS API could stream the response to save memory
[ https://issues.apache.org/jira/browse/SOLR-17582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922517#comment-17922517 ] David Smiley commented on SOLR-17582: - Yes you are seeing my vision here :-). Also see SOLR-17635. We might simply not bother about detecting the client version in Solr 10, telling folks that in some cases (CloudSolrClient and maybe more as time goes) to use Solr 9.9 SolrJ to compatibly converse with Solr 10, if the client can't upgrade to SolrJ 10 yet for some reason. > CLUSTERSTATUS API could stream the response to save memory > -- > > Key: SOLR-17582 > URL: https://issues.apache.org/jira/browse/SOLR-17582 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Reporter: David Smiley >Priority: Minor > Labels: pull-request-available > Fix For: 9.9 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > The CLUSTERSTATUS API currently creates the entire response before submitting > it. If there are many thousands of collections, this will use up memory. It > could easily be modified to stream the response to compute each collection > information just-in-time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17642) Knn queries use multi threaded search unintentionally
Varun Thacker created SOLR-17642:
Summary: Knn queries use multi threaded search unintentionally
Key: SOLR-17642
URL: https://issues.apache.org/jira/browse/SOLR-17642
Project: Solr
Issue Type: Bug
Reporter: Varun Thacker
First reported on
[https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh|https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh,],
Dr. Andreas Moll reported a failing query.
I was able to reproduce is consistently locally on 9.7 and 9.8 using Dr.
Andreas Moll's repro steps.
There are two bugs in play here
*The Solr side bug:*
* Solr 9.7 added https://issues.apache.org/jira/browse/SOLR-13350 but disabled
multi-threaded search by default. I can confirm the failing query follows this
code path
{code:java}
if (!MultiThreadedSearcher.allowMT(pf.postFilter, cmd)) {
log.trace("SINGLE THREADED search, skipping collector manager in
getDocListNC");{code}
* However when following that code path the query goes to
AbstractKnnVectorQuery and this is where executor thread pool gets used
{code:java}
TimeLimitingKnnCollectorManager knnCollectorManager =
new TimeLimitingKnnCollectorManager(
getKnnCollectorManager(k, indexSearcher), indexSearcher.getTimeout());
TaskExecutor taskExecutor = indexSearcher.getTaskExecutor();
{code}
* This executor was added in SOLR-13350 in CoreContainer(
indexSearcherExecutor) . If you follow the SolrIndexSearcher code that calls
{code:java}
In SolrIndexSearcher.java
super(wrapReader(core, r),
core.getCoreContainer().getIndexSearcherExecutor());{code}
{code:java}
In IndexSearcher.java
this.executor = executor;
this.taskExecutor =
executor == null ? new TaskExecutor(Runnable::run) : new
TaskExecutor(executor);{code}
I think the idea in SOLR-13350 was create an indexSearcherExector, but only use
it when multiThreaded=true as a query param (default=false). The fact we
initialised indexSearcherExector means all KNN queries use that thread pool and
there is no way to disable that.
*Now the Lucene Bug:*
[~benwtrent] noted on
[https://lists.apache.org/thread/ftm6f32f2nsoys6noxx8p2ygpwnlfjc9] that
Multi-threaded vector search over multiple segments can lead to inconsistent
results
So the fact Solr by default always uses multi threaded search 9.7+ triggers the
lucene bug. But I think in Solr we should also have a way to not use
multi-threaded KNN like we do for other queries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17619) Generate CHANGELOG.md (formerly CHANGES.txt) via logchange
[ https://issues.apache.org/jira/browse/SOLR-17619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922520#comment-17922520 ] David Smiley commented on SOLR-17619: - As this tool forces us to put some structural changes into each change file, it's an opportunity for us to add things there around backwards-compatibility / upgrades that we put in the "major-changes-in-10.adoc" (and other version) file. Imagine running the tool to generate a separate file for populating the 'adoc'? Ultimately there is some editorial work to clean/organize it but I think we'll be more comprehensive with this approach, mostly preventing us from forgetting to add to the 'adoc'. > Generate CHANGELOG.md (formerly CHANGES.txt) via logchange > -- > > Key: SOLR-17619 > URL: https://issues.apache.org/jira/browse/SOLR-17619 > Project: Solr > Issue Type: Task > Components: Build >Reporter: David Smiley >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > The [logchange|https://github.com/logchange/logchange] tool helps projects > maintain a log of changes. Each change (e.g. from a PR) will no longer edit a > CHANGES.txt file; instead it will include a _new_ YAML file in an appropriate > directory with others for the next Solr version. The release process will > execute the tool, which will build a CHANGELOG.md file and will probably also > do something with the YAML files (remove?). > Decide the most convenient way for us to run the tool for change authors. > Could a gradle task do it? See [this > issue|https://github.com/logchange/logchange/issues/397] filed on the > logchange project. > Outcome of this issue: > * a logchange tool configuration file -- logchange-config.yml > * Solr 10's CHANGES.txt entries converted to YAML. (start this issue by > doing only a few before over-investing) > * a dev-docs page > ** for contributors/everyone: basic info explaining how each new change > should be recorded. Explain how to run the tool to generate the YAML file. > What field(s) matter the most; what should be ignored. Link to further > details. > ** for release manager: how to produce CHANGELOG.md. Link to further > details. Ultimately this will probably move to the release wizard in some > fashion. > TBD: CHANGES.txt < 10 and CHANGELOG.md > 10 ? > TBD: changes.html generation in the release process will be removed or will > change. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17642) Knn queries use multi threaded search unintentionally
[
https://issues.apache.org/jira/browse/SOLR-17642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-17642:
-
Description:
First reported on
[https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh|https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh,],
Dr. Andreas Moll reported a failing query.
I was able to reproduce is consistently locally on 9.7 and 9.8 using Dr.
Andreas Moll's repro steps.
There are two bugs in play here
*The Solr side bug:*
* Solr 9.7 added SOLR-13350 but disabled multi-threaded search by default. I
can confirm the failing query follows this code path
{code:java}
if (!MultiThreadedSearcher.allowMT(pf.postFilter, cmd)) {
log.trace("SINGLE THREADED search, skipping collector manager in
getDocListNC");{code}
* However when following that code path the query goes to
AbstractKnnVectorQuery and this is where executor thread pool gets used
{code:java}
TimeLimitingKnnCollectorManager knnCollectorManager =
new TimeLimitingKnnCollectorManager(
getKnnCollectorManager(k, indexSearcher), indexSearcher.getTimeout());
TaskExecutor taskExecutor = indexSearcher.getTaskExecutor();
{code}
* This executor was added in SOLR-13350 in CoreContainer(
indexSearcherExecutor) . If you follow the SolrIndexSearcher code that calls
{code:java}
In SolrIndexSearcher.java
super(wrapReader(core, r),
core.getCoreContainer().getIndexSearcherExecutor());{code}
{code:java}
In IndexSearcher.java
this.executor = executor;
this.taskExecutor =
executor == null ? new TaskExecutor(Runnable::run) : new
TaskExecutor(executor);{code}
I think the idea in SOLR-13350 was create an indexSearcherExector, but only use
it when multiThreaded=true as a query param (default=false). The fact we
initialised indexSearcherExector means all KNN queries use that thread pool and
there is no way to disable that.
*Now the Lucene Bug:*
[~benwtrent] noted on
[https://lists.apache.org/thread/ftm6f32f2nsoys6noxx8p2ygpwnlfjc9] that
Multi-threaded vector search over multiple segments can lead to inconsistent
results
So the fact Solr by default always uses multi threaded search 9.7+ triggers the
lucene bug. But I think in Solr we should also have a way to not use
multi-threaded KNN like we do for other queries.
was:
First reported on
[https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh|https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh,],
Dr. Andreas Moll reported a failing query.
I was able to reproduce is consistently locally on 9.7 and 9.8 using Dr.
Andreas Moll's repro steps.
There are two bugs in play here
*The Solr side bug:*
* Solr 9.7 added https://issues.apache.org/jira/browse/SOLR-13350 but disabled
multi-threaded search by default. I can confirm the failing query follows this
code path
{code:java}
if (!MultiThreadedSearcher.allowMT(pf.postFilter, cmd)) {
log.trace("SINGLE THREADED search, skipping collector manager in
getDocListNC");{code}
* However when following that code path the query goes to
AbstractKnnVectorQuery and this is where executor thread pool gets used
{code:java}
TimeLimitingKnnCollectorManager knnCollectorManager =
new TimeLimitingKnnCollectorManager(
getKnnCollectorManager(k, indexSearcher), indexSearcher.getTimeout());
TaskExecutor taskExecutor = indexSearcher.getTaskExecutor();
{code}
* This executor was added in SOLR-13350 in CoreContainer(
indexSearcherExecutor) . If you follow the SolrIndexSearcher code that calls
{code:java}
In SolrIndexSearcher.java
super(wrapReader(core, r),
core.getCoreContainer().getIndexSearcherExecutor());{code}
{code:java}
In IndexSearcher.java
this.executor = executor;
this.taskExecutor =
executor == null ? new TaskExecutor(Runnable::run) : new
TaskExecutor(executor);{code}
I think the idea in SOLR-13350 was create an indexSearcherExector, but only use
it when multiThreaded=true as a query param (default=false). The fact we
initialised indexSearcherExector means all KNN queries use that thread pool and
there is no way to disable that.
*Now the Lucene Bug:*
[~benwtrent] noted on
[https://lists.apache.org/thread/ftm6f32f2nsoys6noxx8p2ygpwnlfjc9] that
Multi-threaded vector search over multiple segments can lead to inconsistent
results
So the fact Solr by default always uses multi threaded search 9.7+ triggers the
lucene bug. But I think in Solr we should also have a way to not use
multi-threaded KNN like we do for other queries.
> Knn queries use multi threaded search unintentionally
> ---
>
> Key: SOLR-17642
> URL: https://issues.apache.org/jira/browse/SOLR-17642
> Project: Solr
> Issue Type: Bug
>Reporter: Varun Thacker
>Priority: Major
>
> First reported on
> [https://lists.apache.org/thread/xn1xg
[PR] Add NavigableObject.wrap. Deprecate MapWriter.EMPTY and MapWriterMap. [solr]
dsmiley opened a new pull request, #3148: URL: https://github.com/apache/solr/pull/3148 Introduce NavigableObject.wrap to coerce its arg to a NavigableObject. Maybe call this method "of"? Deprecate MapWriter.EMPTY; don't need it anymore (due to the above); was only used once. Enhance Utils to not need MapWriterMap for its functions, and to slightly optimize for SimpleOrderedMap becoming a Map soon. Deprecate MapWriterMap. Ultimately may simply make package scope and maybe rename to orient itself around it's NavigableObject-ness, which is what it's only used for, after the changes here. No JIRA: too internal; low impact. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17638) Some CLI errors not logged when starting prometheus exporter
Philipp Trulson created SOLR-17638:
--
Summary: Some CLI errors not logged when starting prometheus
exporter
Key: SOLR-17638
URL: https://issues.apache.org/jira/browse/SOLR-17638
Project: Solr
Issue Type: Bug
Components: contrib - prometheus-exporter
Affects Versions: 9.8
Reporter: Philipp Trulson
Hey everyone, we are currently upgrading our 8.11 setup to 9.8 and right now
I'm working on the prometheus-exporter.
To my surprise, the program was exiting directly without any output at all,
which was very frustrating. Only after manually putting a log4j2 config in the
classpath the culprit was finally visible:
{noformat}
org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
--zkhost{noformat}
So the program was exiting because the cli option was renamed from zkhost to
zk-host, would have been an easy fix normally, _if_ the program would have told
me. What's funny though is that other messages are being logged to the console
and I can't find the difference atm. My guess is that something got messed up
after switching the cli lib in SOLR-16996.
Reproduction:
{code:java}
$ podman run --rm -it --entrypoint bash solr:9.8.0
/opt/solr/prometheus-exporter/bin/solr-exporter --port=9854
ERROR - 2025-01-30 08:52:53.052;
org.apache.solr.prometheus.exporter.SolrExporter; Must provide either
--base-url or --zk-host
Exception in thread "main" java.lang.NullPointerException: Cannot invoke
"org.apache.solr.prometheus.exporter.SolrScrapeConfiguration.getType()" because
"configuration" is null
at
org.apache.solr.prometheus.exporter.SolrExporter.createScraper(SolrExporter.java:127)
at
org.apache.solr.prometheus.exporter.SolrExporter.(SolrExporter.java:90)
at
org.apache.solr.prometheus.exporter.SolrExporter.main(SolrExporter.java:426)
$ podman run --rm -it --entrypoint bash solr:9.8.0
/opt/solr/prometheus-exporter/bin/solr-exporter --zkhost=lalala
-> no output{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17638) Some CLI errors not logged when starting prometheus exporter
[
https://issues.apache.org/jira/browse/SOLR-17638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Trulson updated SOLR-17638:
---
Description:
Hey everyone, we are currently upgrading our 8.11 setup to 9.8 and right now
I'm working on the prometheus-exporter.
To my surprise, the program was exiting directly without any output at all,
which was very frustrating. Only after manually putting a log4j2 config in the
classpath the culprit was finally visible:
{noformat}
org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
--zkhost{noformat}
So the program was exiting because the cli option was renamed from zkhost to
zk-host, would have been an easy fix normally, _if_ the program would have told
me. What's funny though is that other messages are being logged to the console
and I can't find the difference atm. My guess is that something got messed up
after switching the cli lib in SOLR-16996.
Reproduction:
{code:java}
$ podman run --rm -it --entrypoint bash solr:9.8.0
/opt/solr/prometheus-exporter/bin/solr-exporter --port=9854
ERROR - 2025-01-30 08:52:53.052;
org.apache.solr.prometheus.exporter.SolrExporter; Must provide either
--base-url or --zk-host
Exception in thread "main" java.lang.NullPointerException: Cannot invoke
"org.apache.solr.prometheus.exporter.SolrScrapeConfiguration.getType()" because
"configuration" is null
at
org.apache.solr.prometheus.exporter.SolrExporter.createScraper(SolrExporter.java:127)
at
org.apache.solr.prometheus.exporter.SolrExporter.(SolrExporter.java:90)
at
org.apache.solr.prometheus.exporter.SolrExporter.main(SolrExporter.java:426)
$ podman run --rm -it --entrypoint bash solr:9.8.0
/opt/solr/prometheus-exporter/bin/solr-exporter --zkhost=lalala
-> no output{code}
(Small sidenote: The program errors when not providing --base-url or --zk-host,
even though the help text says {_}If omitted both the -b parameter and the -z
parameter, connect to http://localhost:8983/solr{_}. Not sure if it's worth
opening a separate issue for that, looks like the CLI args are being worked on
already in 9.9)
was:
Hey everyone, we are currently upgrading our 8.11 setup to 9.8 and right now
I'm working on the prometheus-exporter.
To my surprise, the program was exiting directly without any output at all,
which was very frustrating. Only after manually putting a log4j2 config in the
classpath the culprit was finally visible:
{noformat}
org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
--zkhost{noformat}
So the program was exiting because the cli option was renamed from zkhost to
zk-host, would have been an easy fix normally, _if_ the program would have told
me. What's funny though is that other messages are being logged to the console
and I can't find the difference atm. My guess is that something got messed up
after switching the cli lib in SOLR-16996.
Reproduction:
{code:java}
$ podman run --rm -it --entrypoint bash solr:9.8.0
/opt/solr/prometheus-exporter/bin/solr-exporter --port=9854
ERROR - 2025-01-30 08:52:53.052;
org.apache.solr.prometheus.exporter.SolrExporter; Must provide either
--base-url or --zk-host
Exception in thread "main" java.lang.NullPointerException: Cannot invoke
"org.apache.solr.prometheus.exporter.SolrScrapeConfiguration.getType()" because
"configuration" is null
at
org.apache.solr.prometheus.exporter.SolrExporter.createScraper(SolrExporter.java:127)
at
org.apache.solr.prometheus.exporter.SolrExporter.(SolrExporter.java:90)
at
org.apache.solr.prometheus.exporter.SolrExporter.main(SolrExporter.java:426)
$ podman run --rm -it --entrypoint bash solr:9.8.0
/opt/solr/prometheus-exporter/bin/solr-exporter --zkhost=lalala
-> no output{code}
> Some CLI errors not logged when starting prometheus exporter
>
>
> Key: SOLR-17638
> URL: https://issues.apache.org/jira/browse/SOLR-17638
> Project: Solr
> Issue Type: Bug
> Components: contrib - prometheus-exporter
>Affects Versions: 9.8
>Reporter: Philipp Trulson
>Priority: Major
>
> Hey everyone, we are currently upgrading our 8.11 setup to 9.8 and right now
> I'm working on the prometheus-exporter.
> To my surprise, the program was exiting directly without any output at all,
> which was very frustrating. Only after manually putting a log4j2 config in
> the classpath the culprit was finally visible:
>
> {noformat}
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
> --zkhost{noformat}
>
> So the program was exiting because the cli option was renamed from zkhost to
> zk-host, would have been an easy fix normally, _if_ the program would have
> told me. What's funny though is that other messages are being logged to the
> console and I can't find the difference at
[jira] [Created] (SOLR-17639) Upgrade Prometheus Java client to v1
Philipp Trulson created SOLR-17639: -- Summary: Upgrade Prometheus Java client to v1 Key: SOLR-17639 URL: https://issues.apache.org/jira/browse/SOLR-17639 Project: Solr Issue Type: Improvement Components: metrics Affects Versions: 9.8 Reporter: Philipp Trulson The prometheus-simpleclient that Solr uses at the moment is kind of EOL and wasn't updated since June '22. Instead a new v1 client was released with different artifacts and some major changes in the core architecture: [https://prometheus.github.io/client_java/migration/simpleclient/] Also note that some JVM metrics were renamed, which may require changes to Grafana dashboards. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[PR] Create 2025-01-30-how-to-use-apache-solr-request-parameters-api.md [solr-site]
lizbiella opened a new pull request, #142: URL: https://github.com/apache/solr-site/pull/142 added a blog post about request parameters api -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
