[PR] SOLR-17574: Move host allow list cache to AllowListUrlChecker. [solr]
bruno-roustant opened a new pull request, #2892: URL: https://github.com/apache/solr/pull/2892 Remove the host allow list cache from ClusterState and move it to it's only caller AllowListUrlChecker. There it is cached based on live nodes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17574) ClusterState getHostAllowList isn't always updated when live nodes change
[
https://issues.apache.org/jira/browse/SOLR-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-17574:
--
Labels: pull-request-available (was: )
> ClusterState getHostAllowList isn't always updated when live nodes change
> -
>
> Key: SOLR-17574
> URL: https://issues.apache.org/jira/browse/SOLR-17574
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Affects Versions: 8.8.2
>Reporter: David Smiley
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> ClusterState.getHostAllowList was introduced in SOLR-15217 to validate a host
> may be reached for distributed-search. But it's a cached value computed from
> the live nodes, and the live nodes may change. Thus it can happen that a new
> live node arrives, gets replicas, and suddenly, distributed-search fails with
> a 403 ("Forbidden"), with an error string containing this substring: "is
> neither a live node of the cluster nor in the configured".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17581) Less Zookeeper watches in tests
Pierre Salagnac created SOLR-17581:
--
Summary: Less Zookeeper watches in tests
Key: SOLR-17581
URL: https://issues.apache.org/jira/browse/SOLR-17581
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Pierre Salagnac
This is a follow-up item of SOLR-17453.
Production code has two variants of {{waitForState()}} method. First one only
checks a collection {{{}state.json{}}}, while the second also checks the list
of live nodes in addition to the collection state. When expected condition is
not found immediately, Zookeeper watches are set on both collection state and
live node list.
In test utilities, only a variant that checks both the collection state and
list of live nodes exists. Quite often, this method is used when the test wants
to check a condition on the collection state only. It means a useless watch is
set on the list of live nodes.
The proposal is to introduce a test variant of {{waitForState()}} that accepts
only a condition on collection state, and uses it in all tests that don't want
to check live nodes. Goal is to reduce complexity of test for slightly faster
execution and maybe catch some issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17577) solr.indexfetcher.sotimeout property in tests non used
[ https://issues.apache.org/jira/browse/SOLR-17577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17902311#comment-17902311 ] ASF subversion and git services commented on SOLR-17577: Commit 5f888e548ad77da40bb1881392a6f06b6c800018 in solr's branch refs/heads/main from Eric Pugh [ https://gitbox.apache.org/repos/asf?p=solr.git;h=5f888e548ad ] SOLR-17577: Remove dead system property (#2888) > solr.indexfetcher.sotimeout property in tests non used > -- > > Key: SOLR-17577 > URL: https://issues.apache.org/jira/browse/SOLR-17577 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: replication (java) >Affects Versions: 9.7 >Reporter: Eric Pugh >Priority: Trivial > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Looks like we had a solr.indexfetcher.sotimeout system property to let tests > change timing, but at some point it was renamed to > solr.indexfetcher.sotimeout2, by tests setting the variable, but no one reads > it... I think it's dead code. > https://github.com/epugh/solr/commit/2196663156d2d72bc61e72c296a2c778754f881f#diff-262dfc9c91ab8b173e62d4f32934b6a03af424411fdd6e5aff4a4196bdf2a2fe -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17574: Move host allow list cache to AllowListUrlChecker. [solr]
dsmiley commented on code in PR #2892:
URL: https://github.com/apache/solr/pull/2892#discussion_r1865912289
##
solr/core/src/test/org/apache/solr/security/AllowListUrlCheckerTest.java:
##
@@ -196,6 +200,46 @@ public void testHostParsingNoProtocol() throws Exception {
equalTo(AllowListUrlChecker.parseHostPorts(urls("https://abc-1.com:8983/solr";;
}
+ @Test
+ public void testLiveNodesToHostUrlCache() throws Exception {
+// Given some live nodes defined in the cluster state.
+Set liveNodes =
+new HashSet<>(Arrays.asList("1.2.3.4:8983_solr", "1.2.3.4:9000_",
"1.2.3.4:9001_solr-2"));
Review Comment:
`Set.of` instead?
##
solr/core/src/java/org/apache/solr/security/AllowListUrlChecker.java:
##
@@ -154,6 +157,29 @@ public void checkAllowList(List urls, ClusterState
clusterState)
}
}
+ /**
+ * Gets the set of live hosts urls (host:port) built from the set of live
nodes.
+ * The set is cached to be reused until the live nodes change.
+ */
+ private Set getLiveHostUrls(ClusterState clusterState) {
+if (clusterState == null) {
+ return Set.of();
+}
+Set liveNodes = clusterState.getLiveNodes();
+if (liveHostUrlsCache == null || liveNodes != liveNodesCache) {
+ liveHostUrlsCache = buildLiveHostUrls(liveNodes);
+ liveNodesCache = liveNodes;
+}
Review Comment:
I'm a bit nervous of some race... maybe a synchronized is simple and in
practice shouldn't bottleneck as live nodes doesn't change?
##
solr/core/src/test/org/apache/solr/security/AllowListUrlCheckerTest.java:
##
@@ -196,6 +200,46 @@ public void testHostParsingNoProtocol() throws Exception {
equalTo(AllowListUrlChecker.parseHostPorts(urls("https://abc-1.com:8983/solr";;
}
+ @Test
+ public void testLiveNodesToHostUrlCache() throws Exception {
+// Given some live nodes defined in the cluster state.
+Set liveNodes =
+new HashSet<>(Arrays.asList("1.2.3.4:8983_solr", "1.2.3.4:9000_",
"1.2.3.4:9001_solr-2"));
+ClusterState clusterState1 = new ClusterState(liveNodes, new HashMap<>());
+
+// When we call the AllowListUrlChecker.checkAllowList method on both
valid and invalid urls.
+AtomicInteger callCount = new AtomicInteger();
+AllowListUrlChecker checker = new AllowListUrlChecker(List.of()) {
+ @Override
+ Set buildLiveHostUrls(Set liveNodes) {
+callCount.incrementAndGet();
+return super.buildLiveHostUrls(liveNodes);
+ }
+};
+for (int i = 0; i < 3; i++) {
+ checker.checkAllowList(List.of("1.2.3.4:8983", "1.2.3.4:9000",
"1.2.3.4:9001"), clusterState1);
+ SolrException exception = expectThrows(
+ SolrException.class,
+ () -> checker.checkAllowList(List.of("1.1.3.4:8983"),
clusterState1));
+ assertThat(exception.code(),
equalTo(SolrException.ErrorCode.FORBIDDEN.code));
+}
+// Then we verify that the AllowListUrlChecker caches the live host urls
and only builds them once.
+assertThat(callCount.get(), equalTo(1));
+
+// And when the ClusterState live nodes change.
+liveNodes = new HashSet<>(Arrays.asList("2.3.4.5:8983_solr",
"2.3.4.5:9000_", "2.3.4.5:9001_solr-2"));
Review Comment:
Set.of?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17576: Remove deprecated master/slave support in ReplicationHandler [solr]
epugh merged PR #2887: URL: https://github.com/apache/solr/pull/2887 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17577) solr.indexfetcher.sotimeout property in tests non used
[ https://issues.apache.org/jira/browse/SOLR-17577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17902319#comment-17902319 ] ASF subversion and git services commented on SOLR-17577: Commit 91f1e9517316d5bbc919a961bb1178d6bf26c67d in solr's branch refs/heads/branch_9x from Eric Pugh [ https://gitbox.apache.org/repos/asf?p=solr.git;h=91f1e951731 ] SOLR-17577: Remove dead system property (#2888) (cherry picked from commit 5f888e548ad77da40bb1881392a6f06b6c800018) > solr.indexfetcher.sotimeout property in tests non used > -- > > Key: SOLR-17577 > URL: https://issues.apache.org/jira/browse/SOLR-17577 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: replication (java) >Affects Versions: 9.7 >Reporter: Eric Pugh >Priority: Trivial > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Looks like we had a solr.indexfetcher.sotimeout system property to let tests > change timing, but at some point it was renamed to > solr.indexfetcher.sotimeout2, by tests setting the variable, but no one reads > it... I think it's dead code. > https://github.com/epugh/solr/commit/2196663156d2d72bc61e72c296a2c778754f881f#diff-262dfc9c91ab8b173e62d4f32934b6a03af424411fdd6e5aff4a4196bdf2a2fe -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17579) Remove unused methods in IndexFetcher and ReplicationHandler
[ https://issues.apache.org/jira/browse/SOLR-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SOLR-17579: -- Labels: pull-request-available (was: ) > Remove unused methods in IndexFetcher and ReplicationHandler > > > Key: SOLR-17579 > URL: https://issues.apache.org/jira/browse/SOLR-17579 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: replication (java) >Affects Versions: 9.8 >Reporter: Eric Pugh >Assignee: Eric Pugh >Priority: Minor > Labels: pull-request-available > Attachments: f7418bcb82a9668128c7bf72c1e25bbe31026fd9.patch > > Time Spent: 10m > Remaining Estimate: 0h > > There are dead methods and typos and unused exceptions! > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17577: Remove dead system property [solr]
epugh merged PR #2888: URL: https://github.com/apache/solr/pull/2888 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[PR] SOLR-16781: Disable by default on 9.x [solr]
gerlowskija opened a new pull request, #2894: URL: https://github.com/apache/solr/pull/2894 https://issues.apache.org/jira/browse/SOLR-16781 # Description `` usage will now log a warning by default for future 9.x releases. Wary users can re-enabled the feature by specifying a sysprop: `solr.config.lib.enabled=true`. This PR is a companion to https://github.com/apache/solr/pull/2875, which makes more aggressive changes (removing `` altogether) on the 'main' branch. As such, this PR is only intended for branch_9x, and #2875 is only intended for 'main'. # Tests Still need a test to validate that `` tags are ignored if the feature is not enabled. # Checklist Please review the following and check all that apply: - [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [x] I have created a Jira issue and added the issue ID to my pull request title. - [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation) - [ ] I have run `./gradlew check`. - [ ] I have added tests for my changes. - [x] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Resolved] (SOLR-17576) Remove deprecated master/slave configuration support from Replication
[ https://issues.apache.org/jira/browse/SOLR-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Pugh resolved SOLR-17576. -- Fix Version/s: main (10.0) Assignee: Eric Pugh Resolution: Fixed > Remove deprecated master/slave configuration support from Replication > - > > Key: SOLR-17576 > URL: https://issues.apache.org/jira/browse/SOLR-17576 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) > Components: replication (java) >Affects Versions: main (10.0) >Reporter: Eric Pugh >Assignee: Eric Pugh >Priority: Major > Labels: pull-request-available > Fix For: main (10.0) > > Time Spent: 1h > Remaining Estimate: 0h > > Picking up from the work that Marcus started, we can now remove the > deprecated keywords master/slave. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16781: Remove solrconfig.xml directives [solr]
HoustonPutman commented on code in PR #2875:
URL: https://github.com/apache/solr/pull/2875#discussion_r1866702118
##
solr/core/src/test-files/solr/configsets/upload/with-lib-directive/solrconfig.xml:
##
@@ -37,8 +37,6 @@
${tests.luceneMatchVersion:LATEST}
-
Review Comment:
Maybe this file doesn't need to exist at all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16781: Disable by default on 9.x [solr]
HoustonPutman commented on code in PR #2894: URL: https://github.com/apache/solr/pull/2894#discussion_r1866712826 ## solr/core/src/java/org/apache/solr/cli/RunExampleTool.java: ## @@ -620,10 +620,15 @@ protected Map startSolr( if (!isWindows && cwdPath.length() > 1 && solrHome.startsWith(cwdPath)) solrHome = solrHome.substring(cwdPath.length() + 1); +final var syspropArg = Review Comment: If you are adding this, should the lib directive be removed from the example configSet? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16781: Remove solrconfig.xml directives [solr]
gerlowskija commented on PR #2875: URL: https://github.com/apache/solr/pull/2875#issuecomment-2512047561 Alright - I've fixed merge conflicts, and tested out the examples, so this should be good to merge shortly. Thanks for the reviews all! I've created a corresponding change for branch_9x that will disable `` by default, but still leave the feature available for users that really want it. Would appreciate any feedback there as well! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16781: Disable by default on 9.x [solr]
HoustonPutman commented on code in PR #2894:
URL: https://github.com/apache/solr/pull/2894#discussion_r1866722807
##
solr/core/src/test/org/apache/solr/cli/TestSolrCLIRunExample.java:
##
@@ -70,6 +72,7 @@ public static void beforeClass() {
Paths.get(".").toAbsolutePath().toString().contains(" "));
// to be true
System.setProperty("solr.directoryFactory",
"solr.NRTCachingDirectoryFactory");
+System.setProperty(LIB_ENABLED_SYSPROP, "true");
Review Comment:
If we remove the lib directives in the example ConfigSet, these should be
able to be removed, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17568: SolrCloud shouldn't proxy/route a core specific request [solr]
dsmiley merged PR #2885: URL: https://github.com/apache/solr/pull/2885 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Resolved] (SOLR-17568) Remove remote *core* query in SolrCloud mode
[
https://issues.apache.org/jira/browse/SOLR-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley resolved SOLR-17568.
-
Resolution: Fixed
> Remove remote *core* query in SolrCloud mode
>
>
> Key: SOLR-17568
> URL: https://issues.apache.org/jira/browse/SOLR-17568
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Labels: pull-request-available
> Fix For: main (10.0)
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> If a SolrCloud node receives a request with a local core name (not a
> collection name), it will be handled by the local core. But it even works
> with a remote core -- it'll be forwarded to the node that has a core by that
> name. I think there's no use-case for that -- users should only reference
> collections (any node) or a core on a node where it's known to exist (on that
> node!) since it's a core not a cluster concept. This got my attention
> because the functionality requires iterating all collections. This would
> happen for any request when a collection/core is not known to the node (and
> may not exist at all). I shall remove this in Solr 10.
> There is exactly one test for this functionality --
> {{org.apache.solr.cloud.AbstractBasicDistributedZk2TestBase#addAndQueryDocs}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17310: Configurable LeafSorter to customize segment search order [solr]
github-actions[bot] commented on PR #2477: URL: https://github.com/apache/solr/pull/2477#issuecomment-2513226182 This PR has had no activity for 60 days and is now labeled as stale. Any new activity will remove the stale label. To attract more reviewers, please tag people who might be familiar with the code area and/or notify the d...@solr.apache.org mailing list. To exempt this PR from being marked as stale, make it a draft PR or add the label "exempt-stale". If left unattended, this PR will be closed after another 60 days of inactivity. Thank you for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17574: Move host allow list cache to AllowListUrlChecker. [solr]
bruno-roustant commented on PR #2892: URL: https://github.com/apache/solr/pull/2892#issuecomment-2513761279 This is already changed. ClusterState.setLiveNodes sets the immutable copy and getLiveNodes gets its reference. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
