Re: [I] Mounting a volume to an init container is not clear [solr-operator]
jamesla commented on issue #736: URL: https://github.com/apache/solr-operator/issues/736#issuecomment-2553089526 Thanks @mcarroll1 I did build that example by reading the full spec and I've read it a second time and I still can't seem to see what I'm doing wrong. Does that example work for you? I'm wondering whether the feature is broken? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Deprecate MapWriter.append [solr]
dsmiley commented on code in PR #2919:
URL: https://github.com/apache/solr/pull/2919#discussion_r1892659835
##
solr/solrj/src/java/org/apache/solr/common/MapWriter.java:
##
@@ -70,8 +74,7 @@ public MapWriter.EntryWriter put(CharSequence k, Object v) {
writer.endObject();
}
- void writeMap(EntryWriter ew) throws IOException;
-
+ @Deprecated
Review Comment:
for an obscure method; I think we shouldn't burden ourselves with overly
helping likely non-existent users of such. Personally I do plenty of code
sleuthing to identify when lines of code are added/removed like a Deprecate or
whatever.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17290) Update SyncStrategy and PeerSyncWithLeader to use the recovery Http2SolrClient
[ https://issues.apache.org/jira/browse/SOLR-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17907105#comment-17907105 ] ASF subversion and git services commented on SOLR-17290: Commit 4b3e1ef942d4edebd28452ac8e82e8f8106d226c in solr's branch refs/heads/branch_9x from David Smiley [ https://gitbox.apache.org/repos/asf?p=solr.git;h=4b3e1ef942d ] CHANGES.txt: SOLR-17592 and SOLR-17290 (cherry picked from commit 864c5e4c0f7258b033ee94f8d03b3f07f40c32ac) (cherry picked from commit 920928b39d572b864e679eda428786f66e1360e5) > Update SyncStrategy and PeerSyncWithLeader to use the recovery Http2SolrClient > -- > > Key: SOLR-17290 > URL: https://issues.apache.org/jira/browse/SOLR-17290 > Project: Solr > Issue Type: Sub-task >Affects Versions: main (10.0) >Reporter: Sanjay Dutt >Assignee: Sanjay Dutt >Priority: Major > Fix For: 9.7 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > PeerSyncWithLeader and SyncStrategy both were using the default HttpClient. > Because both of these classes are meant for "recovery" operations, It has > been suggested to switch their client to use recovery Http2SolrClient. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17306) Solr Repeater or Slave loses data after restart when replication is not enabled on leader
[
https://issues.apache.org/jira/browse/SOLR-17306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17907106#comment-17907106
]
ASF subversion and git services commented on SOLR-17306:
Commit bb7546375c5a9166da3383d60adb2815cb468908 in solr's branch
refs/heads/branch_9x from Martin Anzinger
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=bb7546375c5 ]
SOLR-17306: fix replication problem on follower restart (#2918)
(cherry picked from commit 9cef6e390719cbd7b55085cfef98fcb053785f77)
> Solr Repeater or Slave loses data after restart when replication is not
> enabled on leader
> -
>
> Key: SOLR-17306
> URL: https://issues.apache.org/jira/browse/SOLR-17306
> Project: Solr
> Issue Type: Bug
>Affects Versions: 9.2, 9.3, 9.4, 9.5, 9.6
>Reporter: Peter Kroiss
>Assignee: Eric Pugh
>Priority: Major
> Labels: pull-request-available
> Attachments: solr-replication-test.txt
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We are testing Solr 9.6.2 in a leader - repeater - follower configuration. We
> have times where we write the leader heavily, in that time replication is
> disabled to save bandwidth.
> In the time, when replication is disabled on leader, the repeater restarts
> for some reason, the repeater loses all documents and doesn't recover when
> the leader is opened for replication.
> The documents are deleted but indexVersion and generation properties are set
> to the value of the leader, so the repeater or follower doesn't recover when
> the leader is opened for replication again.
> It recovers only when there are commits on the leader after opening the
> replication.
> Log:
> 2024-05-22 06:18:42.186 INFO (qtp16373883-27-null-23) [c: s: r: x:mycore
> t:null-23] o.a.s.c.S.Request webapp=/solr path=/replication
> params=\{wt=json&command=details} status=0 QTime=10
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Leader's generation: 0
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Leader's version: 0
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Follower's generation: 2913
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Follower's version: 1716300697144
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher New index in Leader. Deleting mine...
>
> --> there is no new Index in Leader it is only closed for replication
>
>
> We think the problem is in IndexFetcher
> old: if (IndexDeletionPolicyWrapper.getCommitTimestamp(commit) != 0L) {
> forceReplication - will probably fix the problem
> new : if (forceReplication &&
> IndexDeletionPolicyWrapper.getCommitTimestamp(commit) != 0L) {
>
>
>
>
> When investigation the problem we also found some inconsistencies in the
> details request. There are two fragments leader. When the leader is closed
> for replication the property leader. replicationEnabled is set to true, the
> property follower. leaderDetails. Leader. replicationEnabled is correct.
>
> Example
> curl -s
> "https://solr9-repeater:8983/solr/mycore/replication?wt=json&command=details";
> | jq '.details |
> { indexSize: .indexSize, indexVersion: .indexVersion, generation:
> .generation, indexPath: .indexPath, leader: \\{ replicableVersion:
> .leader.replicableVersion, replicableGeneration:
> .leader.replicableGeneration, replicationEnabled: .leader.replicationEnabled }
> ,
> follower: { leaderDetails: { indexSize: .follower.leaderDetails.indexSize,
> generation: .follower.leaderDetails.generation,
> indexVersion: .follower.leaderDetails.indexVersion, indexPath:
> .follower.leaderDetails.indexPath,
> leader:
> { replicableVersion: .follower.leaderDetails.leader.replicableVersion ,
> replicableGeneration: .follower.leaderDetails.leader.replicableGeneration,
> replicationEnabled: .follower.leaderDetails.leader.replicationEnabled }
> }}
> }'
>
> {
> "indexSize": "10.34 GB",
> "indexVersion": 1716358708159,
> "generation": 2913,
> "indexPath": "/var/solr/data/mycore/data/index.20240522061946262",
> "leader":
> { "replicableVersion": 1716358708159, "replicableGeneration": 2913,
> "replicationEnabled": "true" }
> ,
> "follower": {
> "leaderDetails": {
> "indexSize": "10.34 GB",
> "generation": 2913,
> "indexVersion": 1716358708159,
> "indexPath": "/var/solr/data/mycore/data/restore.20240508131046932",
> "leader":
> { "replicableVersion": 1716358708159, "replicableGeneration":
> 2913,
[jira] [Commented] (SOLR-17592) Apache->Jetty HttpClient for SolrCloudManager
[ https://issues.apache.org/jira/browse/SOLR-17592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17907104#comment-17907104 ] ASF subversion and git services commented on SOLR-17592: Commit 4b3e1ef942d4edebd28452ac8e82e8f8106d226c in solr's branch refs/heads/branch_9x from David Smiley [ https://gitbox.apache.org/repos/asf?p=solr.git;h=4b3e1ef942d ] CHANGES.txt: SOLR-17592 and SOLR-17290 (cherry picked from commit 864c5e4c0f7258b033ee94f8d03b3f07f40c32ac) (cherry picked from commit 920928b39d572b864e679eda428786f66e1360e5) > Apache->Jetty HttpClient for SolrCloudManager > - > > Key: SOLR-17592 > URL: https://issues.apache.org/jira/browse/SOLR-17592 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: David Smiley >Assignee: Sanjay Dutt >Priority: Major > Fix For: 9.8 > > > Migrating from Apache HttpClient to Jetty HttpClient for SolrCloudManager. > That's used server-side for a variety of things in SolrCloud. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Resolved] (SOLR-17306) Solr Repeater or Slave loses data after restart when replication is not enabled on leader
[
https://issues.apache.org/jira/browse/SOLR-17306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Pugh resolved SOLR-17306.
--
Fix Version/s: 9.8
Resolution: Fixed
Thank you for the fix!
> Solr Repeater or Slave loses data after restart when replication is not
> enabled on leader
> -
>
> Key: SOLR-17306
> URL: https://issues.apache.org/jira/browse/SOLR-17306
> Project: Solr
> Issue Type: Bug
>Affects Versions: 9.2, 9.3, 9.4, 9.5, 9.6
>Reporter: Peter Kroiss
>Assignee: Eric Pugh
>Priority: Major
> Labels: pull-request-available
> Fix For: 9.8
>
> Attachments: solr-replication-test.txt
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We are testing Solr 9.6.2 in a leader - repeater - follower configuration. We
> have times where we write the leader heavily, in that time replication is
> disabled to save bandwidth.
> In the time, when replication is disabled on leader, the repeater restarts
> for some reason, the repeater loses all documents and doesn't recover when
> the leader is opened for replication.
> The documents are deleted but indexVersion and generation properties are set
> to the value of the leader, so the repeater or follower doesn't recover when
> the leader is opened for replication again.
> It recovers only when there are commits on the leader after opening the
> replication.
> Log:
> 2024-05-22 06:18:42.186 INFO (qtp16373883-27-null-23) [c: s: r: x:mycore
> t:null-23] o.a.s.c.S.Request webapp=/solr path=/replication
> params=\{wt=json&command=details} status=0 QTime=10
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Leader's generation: 0
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Leader's version: 0
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Follower's generation: 2913
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Follower's version: 1716300697144
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher New index in Leader. Deleting mine...
>
> --> there is no new Index in Leader it is only closed for replication
>
>
> We think the problem is in IndexFetcher
> old: if (IndexDeletionPolicyWrapper.getCommitTimestamp(commit) != 0L) {
> forceReplication - will probably fix the problem
> new : if (forceReplication &&
> IndexDeletionPolicyWrapper.getCommitTimestamp(commit) != 0L) {
>
>
>
>
> When investigation the problem we also found some inconsistencies in the
> details request. There are two fragments leader. When the leader is closed
> for replication the property leader. replicationEnabled is set to true, the
> property follower. leaderDetails. Leader. replicationEnabled is correct.
>
> Example
> curl -s
> "https://solr9-repeater:8983/solr/mycore/replication?wt=json&command=details";
> | jq '.details |
> { indexSize: .indexSize, indexVersion: .indexVersion, generation:
> .generation, indexPath: .indexPath, leader: \\{ replicableVersion:
> .leader.replicableVersion, replicableGeneration:
> .leader.replicableGeneration, replicationEnabled: .leader.replicationEnabled }
> ,
> follower: { leaderDetails: { indexSize: .follower.leaderDetails.indexSize,
> generation: .follower.leaderDetails.generation,
> indexVersion: .follower.leaderDetails.indexVersion, indexPath:
> .follower.leaderDetails.indexPath,
> leader:
> { replicableVersion: .follower.leaderDetails.leader.replicableVersion ,
> replicableGeneration: .follower.leaderDetails.leader.replicableGeneration,
> replicationEnabled: .follower.leaderDetails.leader.replicationEnabled }
> }}
> }'
>
> {
> "indexSize": "10.34 GB",
> "indexVersion": 1716358708159,
> "generation": 2913,
> "indexPath": "/var/solr/data/mycore/data/index.20240522061946262",
> "leader":
> { "replicableVersion": 1716358708159, "replicableGeneration": 2913,
> "replicationEnabled": "true" }
> ,
> "follower": {
> "leaderDetails": {
> "indexSize": "10.34 GB",
> "generation": 2913,
> "indexVersion": 1716358708159,
> "indexPath": "/var/solr/data/mycore/data/restore.20240508131046932",
> "leader":
> { "replicableVersion": 1716358708159, "replicableGeneration":
> 2913, "replicationEnabled": "false" }
> }
> }
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.ap
Re: [PR] SOLR-17541: LBSolrClient implementations should agree on 'getClient()' semantics [solr]
jdyer1 commented on PR #2899: URL: https://github.com/apache/solr/pull/2899#issuecomment-2554671937 I added a CHANGES.txt entry to this PR and plan to merge this early next week. I also will separately add a deprecation on `CloudHttp2SolrClient#withHttpClient` in `branch_9x`, to be released with Solr 9.9. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Deprecate MapWriter.append [solr]
epugh commented on code in PR #2919:
URL: https://github.com/apache/solr/pull/2919#discussion_r1892383853
##
solr/solrj/src/java/org/apache/solr/common/MapWriter.java:
##
@@ -70,8 +74,7 @@ public MapWriter.EntryWriter put(CharSequence k, Object v) {
writer.endObject();
}
- void writeMap(EntryWriter ew) throws IOException;
-
+ @Deprecated
Review Comment:
I kind of wish we always provide a bit more detail on these `@Deprecate`
tags. I like when I see a "since" as it let's me know when the code can be
deleted. I like when there is a "use X instead" comment.
I did a pass the other week around deprecated code and a lot it wasn't clear
what would trigger the deleting of it.. Or how long it had been deprecated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17306: fix replication problem on follower restart [solr]
epugh commented on PR #2918: URL: https://github.com/apache/solr/pull/2918#issuecomment-2554513658 waiting for checks to complete and then will merge! let's hope my nemesis `CHANGES.txt` doesn't get me! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17601) HttpSolrCall: INVALID_STATE: use an HTTP response header instead
David Smiley created SOLR-17601: --- Summary: HttpSolrCall: INVALID_STATE: use an HTTP response header instead Key: SOLR-17601 URL: https://issues.apache.org/jira/browse/SOLR-17601 Project: Solr Issue Type: Improvement Components: SolrCloud, SolrJ Reporter: David Smiley CloudSolrClient & Solr (HttpSolrCall, specifically) work together so that CloudSolrClient knows the collection's client-side cached state is out-of-date. But the state version is appended to the payload of the response from Solr, which CloudSolrClient removes (awkward). Furthermore if HttpSolrCall realizes it needs to proxy the request to another node, it can't (using this strategy) so it fails the request with HTTP 510 which CloudSolrClient understands. If that request was retry-able, the client won't know but requests like indexing are not and so the client sees the error. I propose using an HTTP response header to have the state version, and to always supply it from SolrCloud instead of only when a request parameter is passed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[PR] SOLR-17541: Deprecate `CloudHttp2SolrClient.Builder#withHttpClient` (Solr 9.9) [solr]
jdyer1 opened a new pull request, #2920: URL: https://github.com/apache/solr/pull/2920 For Solr 9.9, deprecate `CloudHttp2SolrClient.Builder#withHttpClient` in favor of `CloudHttp2SolrClient.Builder#withInternalClientBuilder`. PR #2899 removes this in Solr 10.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17541: Deprecate `CloudHttp2SolrClient.Builder#withHttpClient` (Solr 9.9) [solr]
jdyer1 closed pull request #2920: SOLR-17541: Deprecate `CloudHttp2SolrClient.Builder#withHttpClient` (Solr 9.9) URL: https://github.com/apache/solr/pull/2920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17306) Solr Repeater or Slave loses data after restart when replication is not enabled on leader
[
https://issues.apache.org/jira/browse/SOLR-17306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17907095#comment-17907095
]
ASF subversion and git services commented on SOLR-17306:
Commit 9cef6e390719cbd7b55085cfef98fcb053785f77 in solr's branch
refs/heads/main from Martin Anzinger
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=9cef6e39071 ]
SOLR-17306: fix replication problem on follower restart (#2918)
> Solr Repeater or Slave loses data after restart when replication is not
> enabled on leader
> -
>
> Key: SOLR-17306
> URL: https://issues.apache.org/jira/browse/SOLR-17306
> Project: Solr
> Issue Type: Bug
>Affects Versions: 9.2, 9.3, 9.4, 9.5, 9.6
>Reporter: Peter Kroiss
>Assignee: Eric Pugh
>Priority: Major
> Labels: pull-request-available
> Attachments: solr-replication-test.txt
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We are testing Solr 9.6.2 in a leader - repeater - follower configuration. We
> have times where we write the leader heavily, in that time replication is
> disabled to save bandwidth.
> In the time, when replication is disabled on leader, the repeater restarts
> for some reason, the repeater loses all documents and doesn't recover when
> the leader is opened for replication.
> The documents are deleted but indexVersion and generation properties are set
> to the value of the leader, so the repeater or follower doesn't recover when
> the leader is opened for replication again.
> It recovers only when there are commits on the leader after opening the
> replication.
> Log:
> 2024-05-22 06:18:42.186 INFO (qtp16373883-27-null-23) [c: s: r: x:mycore
> t:null-23] o.a.s.c.S.Request webapp=/solr path=/replication
> params=\{wt=json&command=details} status=0 QTime=10
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Leader's generation: 0
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Leader's version: 0
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Follower's generation: 2913
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Follower's version: 1716300697144
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher New index in Leader. Deleting mine...
>
> --> there is no new Index in Leader it is only closed for replication
>
>
> We think the problem is in IndexFetcher
> old: if (IndexDeletionPolicyWrapper.getCommitTimestamp(commit) != 0L) {
> forceReplication - will probably fix the problem
> new : if (forceReplication &&
> IndexDeletionPolicyWrapper.getCommitTimestamp(commit) != 0L) {
>
>
>
>
> When investigation the problem we also found some inconsistencies in the
> details request. There are two fragments leader. When the leader is closed
> for replication the property leader. replicationEnabled is set to true, the
> property follower. leaderDetails. Leader. replicationEnabled is correct.
>
> Example
> curl -s
> "https://solr9-repeater:8983/solr/mycore/replication?wt=json&command=details";
> | jq '.details |
> { indexSize: .indexSize, indexVersion: .indexVersion, generation:
> .generation, indexPath: .indexPath, leader: \\{ replicableVersion:
> .leader.replicableVersion, replicableGeneration:
> .leader.replicableGeneration, replicationEnabled: .leader.replicationEnabled }
> ,
> follower: { leaderDetails: { indexSize: .follower.leaderDetails.indexSize,
> generation: .follower.leaderDetails.generation,
> indexVersion: .follower.leaderDetails.indexVersion, indexPath:
> .follower.leaderDetails.indexPath,
> leader:
> { replicableVersion: .follower.leaderDetails.leader.replicableVersion ,
> replicableGeneration: .follower.leaderDetails.leader.replicableGeneration,
> replicationEnabled: .follower.leaderDetails.leader.replicationEnabled }
> }}
> }'
>
> {
> "indexSize": "10.34 GB",
> "indexVersion": 1716358708159,
> "generation": 2913,
> "indexPath": "/var/solr/data/mycore/data/index.20240522061946262",
> "leader":
> { "replicableVersion": 1716358708159, "replicableGeneration": 2913,
> "replicationEnabled": "true" }
> ,
> "follower": {
> "leaderDetails": {
> "indexSize": "10.34 GB",
> "generation": 2913,
> "indexVersion": 1716358708159,
> "indexPath": "/var/solr/data/mycore/data/restore.20240508131046932",
> "leader":
> { "replicableVersion": 1716358708159, "replicableGeneration":
> 2913, "replicationEnabled": "false" }
> }
> }
> }
--
This m
Re: [PR] SOLR-17306: fix replication problem on follower restart [solr]
epugh merged PR #2918: URL: https://github.com/apache/solr/pull/2918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17601) HttpSolrCall: INVALID_STATE: use an HTTP response header instead
[
https://issues.apache.org/jira/browse/SOLR-17601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley updated SOLR-17601:
Description:
*Background & problem:* CloudSolrClient & Solr (HttpSolrCall, specifically)
work together so that CloudSolrClient knows its collection's client-side cached
state is out-of-date. HttpSolrCall appends the state version to the payload of
the response, assuming a NamedList, which CloudSolrClient removes on receipt.
This is awkward. If HttpSolrCall realizes it needs to proxy the request to
another node (likely due to an out-of-date state in the client for talking to
the "wrong" node), it can't (using this strategy) so it fails the request with
HTTP 510 which CloudSolrClient understands. Awkward again. If that request
was retry-able, the user of SolrJ won't know but requests like indexing (HTTP
POST) are not and so the user sees the error.
*Proposal:* I propose using an HTTP response header to return the state
version. _Always_ return it from SolrCloud; can be done for proxied requests
too, letting CloudSolrClient know when to expire its cache entry.
The _{{{}stateVer_{}}} parameter and potentially failing with HTTP 510 may be
obsolete?
was:
CloudSolrClient & Solr (HttpSolrCall, specifically) work together so that
CloudSolrClient knows the collection's client-side cached state is out-of-date.
But the state version is appended to the payload of the response from Solr,
which CloudSolrClient removes (awkward). Furthermore if HttpSolrCall realizes
it needs to proxy the request to another node, it can't (using this strategy)
so it fails the request with HTTP 510 which CloudSolrClient understands. If
that request was retry-able, the client won't know but requests like indexing
are not and so the client sees the error.
I propose using an HTTP response header to have the state version, and to
always supply it from SolrCloud instead of only when a request parameter is
passed.
> HttpSolrCall: INVALID_STATE: use an HTTP response header instead
>
>
> Key: SOLR-17601
> URL: https://issues.apache.org/jira/browse/SOLR-17601
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud, SolrJ
>Reporter: David Smiley
>Priority: Major
>
> *Background & problem:* CloudSolrClient & Solr (HttpSolrCall, specifically)
> work together so that CloudSolrClient knows its collection's client-side
> cached state is out-of-date. HttpSolrCall appends the state version to the
> payload of the response, assuming a NamedList, which CloudSolrClient removes
> on receipt. This is awkward. If HttpSolrCall realizes it needs to proxy the
> request to another node (likely due to an out-of-date state in the client for
> talking to the "wrong" node), it can't (using this strategy) so it fails the
> request with HTTP 510 which CloudSolrClient understands. Awkward again. If
> that request was retry-able, the user of SolrJ won't know but requests like
> indexing (HTTP POST) are not and so the user sees the error.
> *Proposal:* I propose using an HTTP response header to return the state
> version. _Always_ return it from SolrCloud; can be done for proxied requests
> too, letting CloudSolrClient know when to expire its cache entry.
> The _{{{}stateVer_{}}} parameter and potentially failing with HTTP 510 may be
> obsolete?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17306) Solr Repeater or Slave loses data after restart when replication is not enabled on leader
[
https://issues.apache.org/jira/browse/SOLR-17306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17907116#comment-17907116
]
ASF subversion and git services commented on SOLR-17306:
Commit 0cad3f7ebf881ea25becf0f759b3fe59626e41fa in solr's branch
refs/heads/branch_9_8 from Martin Anzinger
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=0cad3f7ebf8 ]
SOLR-17306: fix replication problem on follower restart (#2918)
(cherry picked from commit 9cef6e390719cbd7b55085cfef98fcb053785f77)
> Solr Repeater or Slave loses data after restart when replication is not
> enabled on leader
> -
>
> Key: SOLR-17306
> URL: https://issues.apache.org/jira/browse/SOLR-17306
> Project: Solr
> Issue Type: Bug
>Affects Versions: 9.2, 9.3, 9.4, 9.5, 9.6
>Reporter: Peter Kroiss
>Assignee: Eric Pugh
>Priority: Major
> Labels: pull-request-available
> Fix For: 9.8
>
> Attachments: solr-replication-test.txt
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We are testing Solr 9.6.2 in a leader - repeater - follower configuration. We
> have times where we write the leader heavily, in that time replication is
> disabled to save bandwidth.
> In the time, when replication is disabled on leader, the repeater restarts
> for some reason, the repeater loses all documents and doesn't recover when
> the leader is opened for replication.
> The documents are deleted but indexVersion and generation properties are set
> to the value of the leader, so the repeater or follower doesn't recover when
> the leader is opened for replication again.
> It recovers only when there are commits on the leader after opening the
> replication.
> Log:
> 2024-05-22 06:18:42.186 INFO (qtp16373883-27-null-23) [c: s: r: x:mycore
> t:null-23] o.a.s.c.S.Request webapp=/solr path=/replication
> params=\{wt=json&command=details} status=0 QTime=10
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Leader's generation: 0
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Leader's version: 0
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Follower's generation: 2913
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher Follower's version: 1716300697144
> 2024-05-22 06:18:46.195 INFO (indexFetcher-43-thread-1) [c: s: r: x:mycore
> t:] o.a.s.h.IndexFetcher New index in Leader. Deleting mine...
>
> --> there is no new Index in Leader it is only closed for replication
>
>
> We think the problem is in IndexFetcher
> old: if (IndexDeletionPolicyWrapper.getCommitTimestamp(commit) != 0L) {
> forceReplication - will probably fix the problem
> new : if (forceReplication &&
> IndexDeletionPolicyWrapper.getCommitTimestamp(commit) != 0L) {
>
>
>
>
> When investigation the problem we also found some inconsistencies in the
> details request. There are two fragments leader. When the leader is closed
> for replication the property leader. replicationEnabled is set to true, the
> property follower. leaderDetails. Leader. replicationEnabled is correct.
>
> Example
> curl -s
> "https://solr9-repeater:8983/solr/mycore/replication?wt=json&command=details";
> | jq '.details |
> { indexSize: .indexSize, indexVersion: .indexVersion, generation:
> .generation, indexPath: .indexPath, leader: \\{ replicableVersion:
> .leader.replicableVersion, replicableGeneration:
> .leader.replicableGeneration, replicationEnabled: .leader.replicationEnabled }
> ,
> follower: { leaderDetails: { indexSize: .follower.leaderDetails.indexSize,
> generation: .follower.leaderDetails.generation,
> indexVersion: .follower.leaderDetails.indexVersion, indexPath:
> .follower.leaderDetails.indexPath,
> leader:
> { replicableVersion: .follower.leaderDetails.leader.replicableVersion ,
> replicableGeneration: .follower.leaderDetails.leader.replicableGeneration,
> replicationEnabled: .follower.leaderDetails.leader.replicationEnabled }
> }}
> }'
>
> {
> "indexSize": "10.34 GB",
> "indexVersion": 1716358708159,
> "generation": 2913,
> "indexPath": "/var/solr/data/mycore/data/index.20240522061946262",
> "leader":
> { "replicableVersion": 1716358708159, "replicableGeneration": 2913,
> "replicationEnabled": "true" }
> ,
> "follower": {
> "leaderDetails": {
> "indexSize": "10.34 GB",
> "generation": 2913,
> "indexVersion": 1716358708159,
> "indexPath": "/var/solr/data/mycore/data/restore.20240508131046932",
> "leader":
> { "replicableVersion": 1716358708159, "repl
Re: [PR] SOLR-17548: Switch all public Java APIs from File to Path [solr]
epugh merged PR #2907: URL: https://github.com/apache/solr/pull/2907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17587: Prometheus Writer duplicate TYPE information in exposition format [solr]
dsmiley commented on PR #2902: URL: https://github.com/apache/solr/pull/2902#issuecomment-2555764301 I could say the same thing for dependencies. People add/remove dependencies in build files (Maven/Gradle/Ivy) but ultimately what we want to know is, what JARs changed in the ultimate distribution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17605) Umbrella: CloudSolrClient: HTTP ClusterStateProvider
[ https://issues.apache.org/jira/browse/SOLR-17605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-17605: Description: Vision: make the HTTP ClusterStateProvider (Solr URLs) be the primary (only?) way that users use CloudSolrClient. The solrj-zookeeper module should eventually be absorbed into Solr. This will track some related improvements as well, albeit may not be strictly required for this vision. [dev-list thread|https://lists.apache.org/thread/tvpvokb7xnnnzc02ksmph4jy1qyl3mcc] Why: * Principled — ZooKeeper is conceptually behind Solr; clients shouldn’t talk to it. * Fewer dependencies for clients (no ZooKeeper or Netty). * Better security — only Solr should talk to ZooKeeper! Security settings and key configuration files are stored in ZooKeeper. * Eliminate the impact of ZK storage on clients. The change of where the configSet name was stored in ZK is an example. PRS is another. And other changes I’ve seen in a fork. * Reduce complexity of SolrJ from an operational standpoint and bug risks (e.g. no ZkStateReader there). No Zookeeper related configuration (jute.maxbuffer, etc.) * Reduce complexity of SolrCloud by limiting the range of use of key classes like ZkStateReader to only be in Solr instead of also existing in SolrJ. For example it’s not clear if/when LazyCollectionRef’s are used in SolrJ but with this separation, it’d be clearer that it couldn’t exist in SolrJ. * Increase our options for classes in solrj-zookeeper, like adding more dependencies (traces & metrics) without concern of burdening any user/client. * Reliably working with a collection after collection creation. If you’ve seen waitForActiveCollection after creating a collection in our tests, this is what I mean (and it’s not strictly a test issue!). It's sad; make them go away! was:Vision: make the HTTP ClusterStateProvider (Solr URLs) be the primary (only?) way that users use CloudSolrClient. This will track some related improvements as well, albeit mayn't be strictly required for this vision. > Umbrella: CloudSolrClient: HTTP ClusterStateProvider > > > Key: SOLR-17605 > URL: https://issues.apache.org/jira/browse/SOLR-17605 > Project: Solr > Issue Type: Improvement >Reporter: David Smiley >Priority: Major > > Vision: make the HTTP ClusterStateProvider (Solr URLs) be the primary (only?) > way that users use CloudSolrClient. The solrj-zookeeper module should > eventually be absorbed into Solr. This will track some related improvements > as well, albeit may not be strictly required for this vision. > [dev-list > thread|https://lists.apache.org/thread/tvpvokb7xnnnzc02ksmph4jy1qyl3mcc] > Why: > * Principled — ZooKeeper is conceptually behind Solr; clients shouldn’t talk > to it. > * Fewer dependencies for clients (no ZooKeeper or Netty). > * Better security — only Solr should talk to ZooKeeper! Security settings > and key configuration files are stored in ZooKeeper. > * Eliminate the impact of ZK storage on clients. The change of where the > configSet name was stored in ZK is an example. PRS is another. And other > changes I’ve seen in a fork. > * Reduce complexity of SolrJ from an operational standpoint and bug risks > (e.g. no ZkStateReader there). No Zookeeper related configuration > (jute.maxbuffer, etc.) > * Reduce complexity of SolrCloud by limiting the range of use of key classes > like ZkStateReader to only be in Solr instead of also existing in SolrJ. For > example it’s not clear if/when LazyCollectionRef’s are used in SolrJ but with > this separation, it’d be clearer that it couldn’t exist in SolrJ. > * Increase our options for classes in solrj-zookeeper, like adding more > dependencies (traces & metrics) without concern of burdening any user/client. > * Reliably working with a collection after collection creation. If you’ve > seen waitForActiveCollection after creating a collection in our tests, this > is what I mean (and it’s not strictly a test issue!). It's sad; make them go > away! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17153) CloudSolrClient should not throw "Collection not found" with an out-dated ClusterState
[ https://issues.apache.org/jira/browse/SOLR-17153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17907229#comment-17907229 ] David Smiley commented on SOLR-17153: - The state of this issue is misleading. It was Fixed yet later mostly reverted, rendering the title of this issue and its description something that was *not* achieved. I'll see if I can change the resolution to something else. > CloudSolrClient should not throw "Collection not found" with an out-dated > ClusterState > -- > > Key: SOLR-17153 > URL: https://issues.apache.org/jira/browse/SOLR-17153 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Reporter: David Smiley >Priority: Major > Fix For: 9.6 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > Today, CloudSolrClient will locally fail if it's asked to send a request to a > collection that it thinks does not exist due to its local ClusterState view > being out-of-date. We shouldn't fail! And most SolrCloud tests should then > remove their waitForState calls that follow collection creation! Other stale > state matters are out-of-scope. > Proposal: CloudSolrClient shouldn't try and be too smart. Always route a > request to Solr (any node); don't presume its state is up-to-date. Maybe, > after a response is received, it can check if its state has been updated and > if not then explicitly get a new state. Or not if that's too complicated. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17582 Stream CLUSTERSTATUS API response [solr]
mlbiscoc commented on code in PR #2916:
URL: https://github.com/apache/solr/pull/2916#discussion_r1892947508
##
solr/solrj/src/java/org/apache/solr/client/solrj/impl/BaseHttpClusterStateProvider.java:
##
@@ -185,7 +184,9 @@ private DocCollection fetchCollectionState(SolrClient
client, String collection)
SimpleOrderedMap cluster =
submitClusterStateRequest(client, collection,
ClusterStateRequestType.FETCH_COLLECTION);
-var collStateMap = (Map)
cluster.findRecursive("collections", collection);
+var collState = (Map) cluster.findRecursive("collections");
Review Comment:
Made the small enhancement to NamedList and added a test to make sure it
works. Lmk what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17582 Stream CLUSTERSTATUS API response [solr]
mlbiscoc commented on code in PR #2916:
URL: https://github.com/apache/solr/pull/2916#discussion_r1892947219
##
solr/core/src/java/org/apache/solr/handler/admin/ClusterStatus.java:
##
@@ -368,4 +341,77 @@ public static Map
postProcessCollectionJSON(Map
collection.put("health", Health.combine(healthStates).toString());
return collection;
}
+
+ private class SolrCollectionProperiesIterator implements
Iterator> {
+
+final Iterator it;
+Map> collectionVsAliases;
+String routeKey;
+List liveNodes;
+String shard;
+
+public SolrCollectionProperiesIterator(
+Iterator it,
+Map> collectionVsAliases,
+String routeKey,
+List liveNodes,
+String shard) {
+ this.it = it;
+ this.collectionVsAliases = collectionVsAliases;
+ this.routeKey = routeKey;
+ this.liveNodes = liveNodes;
+ this.shard = shard;
+}
+
+@Override
+public boolean hasNext() {
+ return it.hasNext();
+}
+
+@Override
+public NamedList next() {
+ NamedList collectionProps = new SimpleOrderedMap<>();
+ DocCollection clusterStateCollection = it.next();
+ Map collectionStatus;
+ String name = clusterStateCollection.getName();
+
+ Set requestedShards = new HashSet<>();
+ if (routeKey != null) {
+DocRouter router = clusterStateCollection.getRouter();
+Collection slices = router.getSearchSlices(routeKey, null,
clusterStateCollection);
+for (Slice slice : slices) {
+ requestedShards.add(slice.getName());
+}
+ }
+ if (shard != null) {
+String[] paramShards = shard.split(",");
+requestedShards.addAll(Arrays.asList(paramShards));
+ }
Review Comment:
Moved it out so its not per iteration.
##
solr/core/src/java/org/apache/solr/handler/admin/ClusterStatus.java:
##
@@ -368,4 +341,77 @@ public static Map
postProcessCollectionJSON(Map
collection.put("health", Health.combine(healthStates).toString());
return collection;
}
+
+ private class SolrCollectionProperiesIterator implements
Iterator> {
+
+final Iterator it;
+Map> collectionVsAliases;
+String routeKey;
+List liveNodes;
+String shard;
+
+public SolrCollectionProperiesIterator(
+Iterator it,
+Map> collectionVsAliases,
+String routeKey,
+List liveNodes,
+String shard) {
+ this.it = it;
+ this.collectionVsAliases = collectionVsAliases;
+ this.routeKey = routeKey;
+ this.liveNodes = liveNodes;
+ this.shard = shard;
+}
+
+@Override
+public boolean hasNext() {
+ return it.hasNext();
+}
+
+@Override
+public NamedList next() {
+ NamedList collectionProps = new SimpleOrderedMap<>();
+ DocCollection clusterStateCollection = it.next();
+ Map collectionStatus;
+ String name = clusterStateCollection.getName();
+
+ Set requestedShards = new HashSet<>();
+ if (routeKey != null) {
+DocRouter router = clusterStateCollection.getRouter();
+Collection slices = router.getSearchSlices(routeKey, null,
clusterStateCollection);
+for (Slice slice : slices) {
+ requestedShards.add(slice.getName());
+}
+ }
+ if (shard != null) {
+String[] paramShards = shard.split(",");
+requestedShards.addAll(Arrays.asList(paramShards));
+ }
+
+ byte[] bytes = Utils.toJSON(clusterStateCollection);
+ @SuppressWarnings("unchecked")
+ Map docCollection = (Map)
Utils.fromJSON(bytes);
Review Comment:
So looked at this and I understand now why it was done this way. It wants to
just write this out to the `TextWriter` as a normal POJO but that doesn't seem
to be what the `DocCollection` class is. So instead what was done was to just
write it to JSON `byte[]` then back to a `Map` which was the "easiest" way.
Was looking to avoid this back and forth, but there were a few options. I
tried using an `ObjectMapper` to `Map.class` but there is an error for unable
to cast Instant due to a missing dependency for Jackson we need to introduce.
`Java 8 date/time type `java.time.Instant` not supported by default Issue`
Other way is to introduce a some kind of `toMap` method so that the
TextWriter can write this as a generic `Map`.
Another option which actually looks like the way we should go is I found
that the `DocCollection` class extends ZkNodeProps which implements
`MapWriter`. `DocCollection` already overrides
`[writeMap](https://github.com/apache/solr/blob/9cef6e390719cbd7b55085cfef98fcb053785f77/solr/solrj/src/java/org/apache/solr/common/cloud/DocCollection.java#L427)`
so we could just return this to the TextWriter! Unfortunately the
ClusterStatus class does a bunch of JSON post processing such as `Health` added
to the `Map` that the output is missing some things because of this
`[postProcessCollectionJSON()](http
Re: [PR] SOLR-15751: Modify v2 `GET /collections/collName` to support full ColStatus response [solr]
gerlowskija commented on PR #2912: URL: https://github.com/apache/solr/pull/2912#issuecomment-257975 > Will aim to merge this in a few days pending any more feedback Actually - thinking on it a bit - the v1 `/admin/segments` API is used under the hood by COLSTATUS, and has a very similar response format, so it probably makes sense to address within this PR as well. I'll look to add that shortly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17582 Stream CLUSTERSTATUS API response [solr]
mlbiscoc commented on code in PR #2916:
URL: https://github.com/apache/solr/pull/2916#discussion_r1893059557
##
solr/core/src/java/org/apache/solr/handler/admin/ClusterStatus.java:
##
@@ -205,54 +214,32 @@ private void fetchClusterStatusForCollOrAlias(
}
}
-// TODO use an Iterable to stream the data to the client instead of
gathering it all in mem
-
-NamedList collectionProps = new SimpleOrderedMap<>();
-
-collectionStream.forEach(
-clusterStateCollection -> {
- Map collectionStatus;
- String name = clusterStateCollection.getName();
-
- Set requestedShards = new HashSet<>();
- if (routeKey != null) {
-DocRouter router = clusterStateCollection.getRouter();
-Collection slices =
-router.getSearchSlices(routeKey, null, clusterStateCollection);
-for (Slice slice : slices) {
- requestedShards.add(slice.getName());
-}
- }
- if (shard != null) {
-String[] paramShards = shard.split(",");
-requestedShards.addAll(Arrays.asList(paramShards));
- }
-
- byte[] bytes = Utils.toJSON(clusterStateCollection);
- @SuppressWarnings("unchecked")
- Map docCollection = (Map)
Utils.fromJSON(bytes);
- collectionStatus = getCollectionStatus(docCollection, name,
requestedShards);
-
- collectionStatus.put("znodeVersion",
clusterStateCollection.getZNodeVersion());
- collectionStatus.put(
- "creationTimeMillis",
clusterStateCollection.getCreationTime().toEpochMilli());
-
- if (collectionVsAliases.containsKey(name) &&
!collectionVsAliases.get(name).isEmpty()) {
-collectionStatus.put("aliases", collectionVsAliases.get(name));
+MapWriter collectionPropsWriter =
+ew -> {
+ Iterator> it =
+ collectionStream
+ .map(
+ (collectionState) ->
+ collectionPropsResponse(
+ collectionState,
+ collectionVsAliases,
+ routeKey,
+ liveNodes,
+ requestedShards))
+ .iterator();
+ while (it.hasNext()) {
+Map props = it.next();
+props.forEach(
+(key, value) -> {
+ try {
+ew.put(key, value);
+ } catch (IOException e) {
+throw new RuntimeException(e);
+ }
+});
Review Comment:
I am thinking, if we can change up DocCollection in another PR, we can
delete all those post processing functions and condense the response to this.
```java
MapWriter collectionPropsWriter =
ew -> {
Iterator it = collectionStream.iterator();
while (it.hasNext()) {
DocCollection collState = it.next();
ew.put(collState.getName(), collState);
}
};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17582 Stream CLUSTERSTATUS API response [solr]
dsmiley commented on code in PR #2916:
URL: https://github.com/apache/solr/pull/2916#discussion_r1893056849
##
solr/core/src/java/org/apache/solr/handler/admin/ClusterStatus.java:
##
@@ -342,76 +356,47 @@ public static Map
postProcessCollectionJSON(Map
return collection;
}
- private class SolrCollectionProperiesIterator implements
Iterator> {
-
-final Iterator it;
-Map> collectionVsAliases;
-String routeKey;
-List liveNodes;
-String shard;
-
-public SolrCollectionProperiesIterator(
-Iterator it,
-Map> collectionVsAliases,
-String routeKey,
-List liveNodes,
-String shard) {
- this.it = it;
- this.collectionVsAliases = collectionVsAliases;
- this.routeKey = routeKey;
- this.liveNodes = liveNodes;
- this.shard = shard;
-}
-
-@Override
-public boolean hasNext() {
- return it.hasNext();
-}
-
-@Override
-public NamedList next() {
- NamedList collectionProps = new SimpleOrderedMap<>();
- DocCollection clusterStateCollection = it.next();
- Map collectionStatus;
- String name = clusterStateCollection.getName();
-
- Set requestedShards = new HashSet<>();
- if (routeKey != null) {
-DocRouter router = clusterStateCollection.getRouter();
-Collection slices = router.getSearchSlices(routeKey, null,
clusterStateCollection);
-for (Slice slice : slices) {
- requestedShards.add(slice.getName());
-}
- }
- if (shard != null) {
-String[] paramShards = shard.split(",");
-requestedShards.addAll(Arrays.asList(paramShards));
+ private Map collectionPropsResponse(
+ DocCollection clusterStateCollection,
+ Map> collectionVsAliases,
+ String routeKey,
+ List liveNodes,
+ Set requestedShards) {
+Map collectionProps = new HashMap<>();
+Map collectionStatus;
+String name = clusterStateCollection.getName();
+
+if (routeKey != null) {
+ DocRouter router = clusterStateCollection.getRouter();
+ Collection slices = router.getSearchSlices(routeKey, null,
clusterStateCollection);
+ for (Slice slice : slices) {
+requestedShards.add(slice.getName());
}
+}
- byte[] bytes = Utils.toJSON(clusterStateCollection);
- @SuppressWarnings("unchecked")
- Map docCollection = (Map)
Utils.fromJSON(bytes);
- collectionStatus = getCollectionStatus(docCollection, name,
requestedShards);
+byte[] bytes = Utils.toJSON(clusterStateCollection);
+@SuppressWarnings("unchecked")
+Map docCollection = (Map)
Utils.fromJSON(bytes);
+collectionStatus = getCollectionStatus(docCollection, name,
requestedShards);
- collectionStatus.put("znodeVersion",
clusterStateCollection.getZNodeVersion());
- collectionStatus.put(
- "creationTimeMillis",
clusterStateCollection.getCreationTime().toEpochMilli());
+collectionStatus.put("znodeVersion",
clusterStateCollection.getZNodeVersion());
+collectionStatus.put(
+"creationTimeMillis",
clusterStateCollection.getCreationTime().toEpochMilli());
- if (collectionVsAliases.containsKey(name) &&
!collectionVsAliases.get(name).isEmpty()) {
-collectionStatus.put("aliases", collectionVsAliases.get(name));
- }
- String configName = clusterStateCollection.getConfigName();
- collectionStatus.put("configName", configName);
- if (solrParams.getBool("prs", false) &&
clusterStateCollection.isPerReplicaState()) {
-PerReplicaStates prs = clusterStateCollection.getPerReplicaStates();
-collectionStatus.put("PRS", prs);
- }
- collectionProps.add(name, collectionStatus);
-
- // now we need to walk the collectionProps tree to cross-check replica
state with live
- // nodes
- crossCheckReplicaStateWithLiveNodes(liveNodes, collectionProps);
- return collectionProps;
+if (collectionVsAliases.containsKey(name) &&
!collectionVsAliases.get(name).isEmpty()) {
+ collectionStatus.put("aliases", collectionVsAliases.get(name));
+}
+String configName = clusterStateCollection.getConfigName();
+collectionStatus.put("configName", configName);
+if (solrParams.getBool("prs", false) &&
clusterStateCollection.isPerReplicaState()) {
+ PerReplicaStates prs = clusterStateCollection.getPerReplicaStates();
+ collectionStatus.put("PRS", prs);
}
+collectionProps.put(name, collectionStatus);
Review Comment:
it seems that we're returning an extra map here? Shouldn't we be returning
collectionStatus (that which holds the keys-value pairs describing this
collection), not collectionProps, which holds lists of collections (or did but
now has just one)
##
solr/core/src/java/org/apache/solr/handler/admin/ClusterStatus.java:
##
@@ -342,76 +356,47 @@ public static Map
postProcessCollectionJSON(Map
return collection;
}
- private class S
[jira] [Created] (SOLR-17602) Maintain final JAR dependencies in source control to track changes
David Smiley created SOLR-17602: --- Summary: Maintain final JAR dependencies in source control to track changes Key: SOLR-17602 URL: https://issues.apache.org/jira/browse/SOLR-17602 Project: Solr Issue Type: Task Components: Build Reporter: David Smiley When we make changes to Solr's dependencies (add/remove/change), we edit our build files, and the code review process shows these changes to corresponding build files. However, what we all *really* want to know is the impact the change has on the artifacts our users consume. Almost nobody validates the impact; we hope for the best and find out of problems long later. This issue tracks one artifact: Solr's final assembly (any of the zip, tar.gz, or Docker). I propose committing to source control a machine generated file listing of the dependencies in a text file. This file shall be updated based on executing a gradle task TBD. When gradle "check" is run, it will henceforth ensure that this file hasn't been modified or doesn't match the output of the script's generation (details TBD). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17603) Maintain Solrj transitive dependencies in source control to track changes
David Smiley created SOLR-17603: --- Summary: Maintain Solrj transitive dependencies in source control to track changes Key: SOLR-17603 URL: https://issues.apache.org/jira/browse/SOLR-17603 Project: Solr Issue Type: Task Reporter: David Smiley When we make changes to Solr's dependencies (add/remove/change), we edit our build files, and the code review process shows these changes to corresponding build files. However, what we all *really* want to know is the impact the change has on the artifacts our users consume. Almost nobody validates the impact; we hope for the best and find out of problems long later. This issue tracks one artifact: SolrJ's transitive dependency list. I propose committing to source control a machine generated file listing of the dependencies in a text file. This file shall be updated based on executing a gradle task TBD. When gradle "check" is run, it will henceforth ensure that this file hasn't been modified or doesn't match the output of the script's generation (details TBD). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17587: Prometheus Writer duplicate TYPE information in exposition format [solr]
dsmiley commented on code in PR #2902:
URL: https://github.com/apache/solr/pull/2902#discussion_r1893230595
##
solr/core/src/java/org/apache/solr/metrics/prometheus/core/SolrCoreCacheMetric.java:
##
@@ -38,7 +37,7 @@ public SolrCoreCacheMetric(
public SolrCoreMetric parseLabels() {
String[] parsedMetric = metricName.split("\\.");
if (dropwizardMetric instanceof Gauge) {
- labels.put("cacheType", parsedMetric[2]);
+ labels.put("cacheType", parsedMetric[3]);
Review Comment:
Previously index 2 lined up with the sample above on line 33. But now; I
dunno. I suppose your data example string is now out of date.
Same comment as other source files you edited similarly.
##
solr/core/src/java/org/apache/solr/handler/admin/MetricsHandler.java:
##
@@ -186,10 +187,22 @@ private NamedList
handlePrometheusExport(SolrParams params) {
List metricTypes = parseMetricTypes(params);
List metricFilters =
metricTypes.stream().map(MetricType::asMetricFilter).collect(Collectors.toList());
+
Set requestedRegistries = parseRegistries(params);
+MetricRegistry mergedCoreRegistries = new MetricRegistry();
for (String registryName : requestedRegistries) {
MetricRegistry dropwizardRegistry = metricManager.registry(registryName);
+
+ // Merge all core registries into a single registry and
+ // append the core name to the metric to avoid duplicate metrics name
+ if (registryName.startsWith("solr.core")) {
+mergedCoreRegistries.registerAll(
+
Arrays.stream(registryName.split("\\.")).skip(1).collect(Collectors.joining("_")),
Review Comment:
It'd be faster to do a substring on the index after the first period, and
then with the result of that, replace all periods with underscores. Having a
little utility method to do this with comment would ultimately be clearer IMO
##
solr/core/src/java/org/apache/solr/metrics/prometheus/core/SolrCoreMetric.java:
##
@@ -25,28 +26,26 @@
/** Base class is a wrapper to export a solr.core {@link
com.codahale.metrics.Metric} */
public abstract class SolrCoreMetric extends SolrMetric {
+
public String coreName;
- public SolrCoreMetric(
- Metric dropwizardMetric, String coreName, String metricName, boolean
cloudMode) {
+ public SolrCoreMetric(Metric dropwizardMetric, String metricName) {
super(dropwizardMetric, metricName);
-this.coreName = coreName;
-labels.put("core", coreName);
-if (cloudMode) {
- appendCloudModeLabels();
-}
- }
-
- private void appendCloudModeLabels() {
-Matcher m = CLOUD_CORE_PATTERN.matcher(coreName);
-if (m.find()) {
- labels.put("collection", m.group(1));
- labels.put("shard", m.group(2));
- labels.put("replica", m.group(3));
+Matcher cloudPattern = CLOUD_CORE_PATTERN.matcher(metricName);
+Matcher standalonePattern = STANDALONE_CORE_PATTERN.matcher(metricName);
+if (cloudPattern.find()) {
+ coreName = cloudPattern.group(1);
+ labels.put("core", cloudPattern.group(1));
+ labels.put("collection", cloudPattern.group(2));
+ labels.put("shard", cloudPattern.group(3));
+ labels.put("replica", cloudPattern.group(4));
Review Comment:
I believe it's possible to have named groups in a regular expression. If
you enhance your regular expressions as such, you could generically apply the
pattern with Matcher and extract all named groups without hard-coding the
expectations separate from the regular expression, which is annoying and
error-prone to maintain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17582 Stream CLUSTERSTATUS API response [solr]
mlbiscoc commented on code in PR #2916:
URL: https://github.com/apache/solr/pull/2916#discussion_r1892947285
##
solr/core/src/java/org/apache/solr/handler/admin/ClusterStatus.java:
##
@@ -368,4 +341,77 @@ public static Map
postProcessCollectionJSON(Map
collection.put("health", Health.combine(healthStates).toString());
return collection;
}
+
+ private class SolrCollectionProperiesIterator implements
Iterator> {
+
+final Iterator it;
+Map> collectionVsAliases;
+String routeKey;
+List liveNodes;
+String shard;
+
+public SolrCollectionProperiesIterator(
+Iterator it,
+Map> collectionVsAliases,
+String routeKey,
+List liveNodes,
+String shard) {
+ this.it = it;
+ this.collectionVsAliases = collectionVsAliases;
+ this.routeKey = routeKey;
+ this.liveNodes = liveNodes;
+ this.shard = shard;
+}
+
+@Override
+public boolean hasNext() {
+ return it.hasNext();
+}
+
+@Override
+public NamedList next() {
+ NamedList collectionProps = new SimpleOrderedMap<>();
+ DocCollection clusterStateCollection = it.next();
+ Map collectionStatus;
+ String name = clusterStateCollection.getName();
+
+ Set requestedShards = new HashSet<>();
+ if (routeKey != null) {
+DocRouter router = clusterStateCollection.getRouter();
+Collection slices = router.getSearchSlices(routeKey, null,
clusterStateCollection);
+for (Slice slice : slices) {
+ requestedShards.add(slice.getName());
+}
+ }
+ if (shard != null) {
+String[] paramShards = shard.split(",");
+requestedShards.addAll(Arrays.asList(paramShards));
+ }
+
+ byte[] bytes = Utils.toJSON(clusterStateCollection);
+ @SuppressWarnings("unchecked")
+ Map docCollection = (Map)
Utils.fromJSON(bytes);
Review Comment:
So looked at this and I understand now why it was done this way. It wants to
just write this out to the `TextWriter` as a normal POJO but that doesn't seem
to be what the `DocCollection` class is. So instead what was done was to just
write it to JSON `byte[]` then back to a `Map` which was the "easiest" way.
Was looking to avoid this back and forth, but there were a few options. I
tried using an `ObjectMapper` to `Map.class` but there is an error for unable
to cast Instant due to a missing dependency for Jackson we need to introduce.
`Java 8 date/time type `java.time.Instant` not supported by default Issue`
Other way is to introduce a some kind of `toMap` method so that the
TextWriter can write this as a generic `Map`.
Another option which actually looks like the way we should go is I found
that the `DocCollection` class extends ZkNodeProps which implements
`MapWriter`. `DocCollection` already overrides
[writeMap](https://github.com/apache/solr/blob/9cef6e390719cbd7b55085cfef98fcb053785f77/solr/solrj/src/java/org/apache/solr/common/cloud/DocCollection.java#L427)
so we could just return this to the TextWriter! Unfortunately the
ClusterStatus class does a bunch of JSON post processing such as `Health` added
to the `Map` that the output is missing some things because of this
[postProcessCollectionJSON()](https://github.com/apache/solr/blob/9cef6e390719cbd7b55085cfef98fcb053785f77/solr/core/src/java/org/apache/solr/handler/admin/ClusterStatus.java#L334)
method.
I am thinking we should refactor DocCollection to so we can just return this
but the changes were much more drastic but may be worth it. Maybe in a
different JIRA? This scope continues to creep with me adding improvement to
`NamedList`. Would be happy to refactor this and pick it up if you agree. Would
clean up much more code and avoid this JSON processing for every collection
iteration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[I] Recommended approach for loading a schema and configset using the operator [solr-operator]
jamesla opened a new issue, #740: URL: https://github.com/apache/solr-operator/issues/740 I'm struggling to load a schema and a configset using the operator. In my docker-compose I do it like this: ```yaml config-loader: image: alpine/curl:latest container_name: solr-init volumes: - ./solrconfig.xml:/solrconfig.xml - ./managed-schema.xml:/managed-schema.xml depends_on: solr-cloud: condition: service_healthy entrypoint: > /bin/sh -c ' curl -s -X PUT --header "Content-Type:application/octet-stream" --data-binary @solrconfig.xml "http://solr-cloud:8983/api/cluster/configs/llm_papers_configset/solrconfig.xml";; curl -X PUT --header "Content-Type:application/octet-stream" --data-binary @managed-schema.xml "http://solr-cloud:8983/api/cluster/configs/llm_papers_configset/managed-schema.xml";; curl "solr-cloud:8983/solr/admin/collections?action=CREATE&name=llm_papers&numShards=1&replicationFactor=1&collection.configName=llm_papers_configset&wt=json&zkHost=zookeeper:2181"; ' ``` I have tried using a sidecarContainer however I feel like that is the wrong approach since solrcloud api needs to be ready in order to receive the requests which creates a race condition. I've also read through the CRD options but I can't find anything related to configsets or schemas. What is the recommended way to approach this using the operator? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17582 Stream CLUSTERSTATUS API response [solr]
dsmiley commented on code in PR #2916:
URL: https://github.com/apache/solr/pull/2916#discussion_r1893072837
##
solr/core/src/java/org/apache/solr/handler/admin/ClusterStatus.java:
##
@@ -205,54 +214,32 @@ private void fetchClusterStatusForCollOrAlias(
}
}
-// TODO use an Iterable to stream the data to the client instead of
gathering it all in mem
-
-NamedList collectionProps = new SimpleOrderedMap<>();
-
-collectionStream.forEach(
-clusterStateCollection -> {
- Map collectionStatus;
- String name = clusterStateCollection.getName();
-
- Set requestedShards = new HashSet<>();
- if (routeKey != null) {
-DocRouter router = clusterStateCollection.getRouter();
-Collection slices =
-router.getSearchSlices(routeKey, null, clusterStateCollection);
-for (Slice slice : slices) {
- requestedShards.add(slice.getName());
-}
- }
- if (shard != null) {
-String[] paramShards = shard.split(",");
-requestedShards.addAll(Arrays.asList(paramShards));
- }
-
- byte[] bytes = Utils.toJSON(clusterStateCollection);
- @SuppressWarnings("unchecked")
- Map docCollection = (Map)
Utils.fromJSON(bytes);
- collectionStatus = getCollectionStatus(docCollection, name,
requestedShards);
-
- collectionStatus.put("znodeVersion",
clusterStateCollection.getZNodeVersion());
- collectionStatus.put(
- "creationTimeMillis",
clusterStateCollection.getCreationTime().toEpochMilli());
-
- if (collectionVsAliases.containsKey(name) &&
!collectionVsAliases.get(name).isEmpty()) {
-collectionStatus.put("aliases", collectionVsAliases.get(name));
+MapWriter collectionPropsWriter =
+ew -> {
+ Iterator> it =
+ collectionStream
+ .map(
+ (collectionState) ->
+ collectionPropsResponse(
+ collectionState,
+ collectionVsAliases,
+ routeKey,
+ liveNodes,
+ requestedShards))
+ .iterator();
+ while (it.hasNext()) {
+Map props = it.next();
+props.forEach(
+(key, value) -> {
+ try {
+ew.put(key, value);
+ } catch (IOException e) {
+throw new RuntimeException(e);
+ }
+});
Review Comment:
I should mention Solr has two similar APIs, ClusterStatus & ColStatus. See
org.apache.solr.handler.admin.ColStatus#getColStatus. It'd be awesome if there
was a single method that takes a DocCollection (and some other params as
needed) to produce a NamedList. At least ColStatus's code doesn't have the sad
JSON round-trip. Maybe it's own PR or not as you wish.
##
solr/core/src/java/org/apache/solr/handler/admin/ClusterStatus.java:
##
@@ -205,54 +214,32 @@ private void fetchClusterStatusForCollOrAlias(
}
}
-// TODO use an Iterable to stream the data to the client instead of
gathering it all in mem
-
-NamedList collectionProps = new SimpleOrderedMap<>();
-
-collectionStream.forEach(
-clusterStateCollection -> {
- Map collectionStatus;
- String name = clusterStateCollection.getName();
-
- Set requestedShards = new HashSet<>();
- if (routeKey != null) {
-DocRouter router = clusterStateCollection.getRouter();
-Collection slices =
-router.getSearchSlices(routeKey, null, clusterStateCollection);
-for (Slice slice : slices) {
- requestedShards.add(slice.getName());
-}
- }
- if (shard != null) {
-String[] paramShards = shard.split(",");
-requestedShards.addAll(Arrays.asList(paramShards));
- }
-
- byte[] bytes = Utils.toJSON(clusterStateCollection);
- @SuppressWarnings("unchecked")
- Map docCollection = (Map)
Utils.fromJSON(bytes);
- collectionStatus = getCollectionStatus(docCollection, name,
requestedShards);
-
- collectionStatus.put("znodeVersion",
clusterStateCollection.getZNodeVersion());
- collectionStatus.put(
- "creationTimeMillis",
clusterStateCollection.getCreationTime().toEpochMilli());
-
- if (collectionVsAliases.containsKey(name) &&
!collectionVsAliases.get(name).isEmpty()) {
-collectionStatus.put("aliases", collectionVsAliases.get(name));
+MapWriter collectionPropsWriter =
+ew -> {
+ Iterator> it =
+ collectionStream
+ .map(
+ (collectionState) ->
+ collectionPr
Re: [PR] Deprecate MapWriter.append [solr]
dsmiley commented on PR #2919: URL: https://github.com/apache/solr/pull/2919#issuecomment-2555703092 I'm thinking of merging this with removal of the deprecated method, and in 9x & branch_9_8 simply deprecating the method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17587: Prometheus Writer duplicate TYPE information in exposition format [solr]
mlbiscoc commented on PR #2902: URL: https://github.com/apache/solr/pull/2902#issuecomment-2555793583 > Correct me if I'm wrong, but had you kept the test that you had removed because hossman didn't like it (it was imperfect), a reviewer (like me but you too) would be able to see in this PR a diff against the output format to understand the impact (or lack of impact). In theory, yes you would see the difference that is happening here. But since you can't see the output, here is a basically the change to the output: Before: ``` curl 'localhost:8983/solr/admin/metrics?wt=prometheus' | grep "#" | sort # TYPE solr_metrics_core_average_request_time gauge # TYPE solr_metrics_core_average_request_time gauge # TYPE solr_metrics_core_average_searcher_warmup_time gauge # TYPE solr_metrics_core_average_searcher_warmup_time gauge # TYPE solr_metrics_core_cache gauge # TYPE solr_metrics_core_cache gauge # TYPE solr_metrics_core_highlighter_requests_total counter # TYPE solr_metrics_core_highlighter_requests_total counter # TYPE solr_metrics_core_index_size_bytes gauge # TYPE solr_metrics_core_index_size_bytes gauge # TYPE solr_metrics_core_requests_time_total counter # TYPE solr_metrics_core_requests_time_total counter # TYPE solr_metrics_core_requests_total counter # TYPE solr_metrics_core_requests_total counter # TYPE solr_metrics_core_searcher_documents gauge # TYPE solr_metrics_core_searcher_documents gauge # TYPE solr_metrics_core_tlog_total counter # TYPE solr_metrics_core_tlog_total counter # TYPE solr_metrics_core_update_handler gauge # TYPE solr_metrics_core_update_handler gauge # TYPE solr_metrics_jetty_dispatches_total counter # TYPE solr_metrics_jetty_requests_total counter # TYPE solr_metrics_jetty_response_total counter # TYPE solr_metrics_jvm_buffers gauge # TYPE solr_metrics_jvm_buffers_bytes gauge # TYPE solr_metrics_jvm_gc gauge # TYPE solr_metrics_jvm_gc_seconds gauge # TYPE solr_metrics_jvm_heap gauge # TYPE solr_metrics_jvm_memory_pools_bytes gauge # TYPE solr_metrics_jvm_threads gauge # TYPE solr_metrics_node_connections gauge # TYPE solr_metrics_node_core_root_fs_bytes gauge # TYPE solr_metrics_node_cores gauge # TYPE solr_metrics_node_requests_time_total counter # TYPE solr_metrics_node_requests_total counter # TYPE solr_metrics_node_thread_pool_total counter # TYPE solr_metrics_os gauge ``` After: ``` curl 'localhost:8983/solr/admin/metrics?wt=prometheus' | grep "#" | sort # TYPE solr_metrics_core_average_request_time gauge # TYPE solr_metrics_core_average_searcher_warmup_time gauge # TYPE solr_metrics_core_cache gauge # TYPE solr_metrics_core_highlighter_requests_total counter # TYPE solr_metrics_core_index_size_bytes gauge # TYPE solr_metrics_core_requests_time_total counter # TYPE solr_metrics_core_requests_total counter # TYPE solr_metrics_core_searcher_documents gauge # TYPE solr_metrics_core_tlog_total counter # TYPE solr_metrics_core_update_handler gauge # TYPE solr_metrics_jetty_dispatches_total counter # TYPE solr_metrics_jetty_requests_total counter # TYPE solr_metrics_jetty_response_total counter # TYPE solr_metrics_jvm_buffers gauge # TYPE solr_metrics_jvm_buffers_bytes gauge # TYPE solr_metrics_jvm_gc gauge # TYPE solr_metrics_jvm_gc_seconds gauge # TYPE solr_metrics_jvm_heap gauge # TYPE solr_metrics_jvm_memory_pools_bytes gauge # TYPE solr_metrics_jvm_threads gauge # TYPE solr_metrics_node_connections gauge # TYPE solr_metrics_node_core_root_fs_bytes gauge # TYPE solr_metrics_node_cores gauge # TYPE solr_metrics_node_requests_time_total counter # TYPE solr_metrics_node_requests_total counter # TYPE solr_metrics_node_thread_pool_total counter # TYPE solr_metrics_os gauge ``` `#` is just a comment in Prometheus but `# TYPE` is special. Some technologies with verify there are dupes while others don't. This just removes the duplicates and merged the metrics under a single `# TYPE` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17604) Move state caching from CloudSolrClient to ClusterStateProvider
David Smiley created SOLR-17604: --- Summary: Move state caching from CloudSolrClient to ClusterStateProvider Key: SOLR-17604 URL: https://issues.apache.org/jira/browse/SOLR-17604 Project: Solr Issue Type: Improvement Components: SolrCloud, SolrJ Reporter: David Smiley This is a refactoring but not a hidden one and it impacts performance positively for both ZK and HTTP ClusterStateProviders. Propose that collection state caching responsibilities move from CloudSolrClient to the ClusterStateProvider. CSC needs to obtain a DocCollection from CSP (exists) and needs to inform CSP on occasion that there's a version of a DocCollection at a higher version (new method – informNewVersion(collName,collVersion), so that the CSP can invalidate/update it's cache accordingly (details TBD and may evolve). The ZK CSP would do nothing since it watches everything (for better or worse) and it'll get up to date on its own. Also, hopefully can remove ClusterState.CollectionRef concept from CSP's API. Separately: deprecating ClusterState access altogether by CSC & CSP (albeit will exist specifically on the ZK CSP). CSP can have methods to obtain whatever someone may want and/or users can use standard Solr admin APIs to obtain info. Probably needs a collection-list method. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17605) Umbrella: CloudSolrClient & ClusterStateProvider
David Smiley created SOLR-17605: --- Summary: Umbrella: CloudSolrClient & ClusterStateProvider Key: SOLR-17605 URL: https://issues.apache.org/jira/browse/SOLR-17605 Project: Solr Issue Type: Improvement Reporter: David Smiley Vision: make the HTTP ClusterStateProvider (Solr URLs) be the primary (only?) way that users use CloudSolrClient. This will track some related improvements as well, albeit mayn't be strictly required for this vision. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17605) Umbrella: CloudSolrClient: HTTP ClusterStateProvider
[ https://issues.apache.org/jira/browse/SOLR-17605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-17605: Summary: Umbrella: CloudSolrClient: HTTP ClusterStateProvider (was: Umbrella: CloudSolrClient & ClusterStateProvider) > Umbrella: CloudSolrClient: HTTP ClusterStateProvider > > > Key: SOLR-17605 > URL: https://issues.apache.org/jira/browse/SOLR-17605 > Project: Solr > Issue Type: Improvement >Reporter: David Smiley >Priority: Major > > Vision: make the HTTP ClusterStateProvider (Solr URLs) be the primary (only?) > way that users use CloudSolrClient. This will track some related > improvements as well, albeit mayn't be strictly required for this vision. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17153) CloudSolrClient should not throw "Collection not found" with an out-dated ClusterState
[ https://issues.apache.org/jira/browse/SOLR-17153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SOLR-17153: -- Labels: pull-request-available (was: ) > CloudSolrClient should not throw "Collection not found" with an out-dated > ClusterState > -- > > Key: SOLR-17153 > URL: https://issues.apache.org/jira/browse/SOLR-17153 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Reporter: David Smiley >Priority: Major > Labels: pull-request-available > Fix For: 9.6 > > Time Spent: 6h > Remaining Estimate: 0h > > Today, CloudSolrClient will locally fail if it's asked to send a request to a > collection that it thinks does not exist due to its local ClusterState view > being out-of-date. We shouldn't fail! And most SolrCloud tests should then > remove their waitForState calls that follow collection creation! Other stale > state matters are out-of-scope. > Proposal: CloudSolrClient shouldn't try and be too smart. Always route a > request to Solr (any node); don't presume its state is up-to-date. Maybe, > after a response is received, it can check if its state has been updated and > if not then explicitly get a new state. Or not if that's too complicated. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17153) CloudSolrClient should not throw "Collection not found" with an out-dated ClusterState
[ https://issues.apache.org/jira/browse/SOLR-17153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-17153: Description: _(Present status: reverted / not accomplished. New issues needed to continue)_ Today, CloudSolrClient will locally fail if it's asked to send a request to a collection that it thinks does not exist due to its local ClusterState view being out-of-date. We shouldn't fail! And most SolrCloud tests should then remove their waitForState calls that follow collection creation! Other stale state matters are out-of-scope. Proposal: CloudSolrClient shouldn't try and be too smart. Always route a request to Solr (any node); don't presume its state is up-to-date. Maybe, after a response is received, it can check if its state has been updated and if not then explicitly get a new state. Or not if that's too complicated. was: Today, CloudSolrClient will locally fail if it's asked to send a request to a collection that it thinks does not exist due to its local ClusterState view being out-of-date. We shouldn't fail! And most SolrCloud tests should then remove their waitForState calls that follow collection creation! Other stale state matters are out-of-scope. Proposal: CloudSolrClient shouldn't try and be too smart. Always route a request to Solr (any node); don't presume its state is up-to-date. Maybe, after a response is received, it can check if its state has been updated and if not then explicitly get a new state. Or not if that's too complicated. > CloudSolrClient should not throw "Collection not found" with an out-dated > ClusterState > -- > > Key: SOLR-17153 > URL: https://issues.apache.org/jira/browse/SOLR-17153 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Reporter: David Smiley >Priority: Major > Labels: pull-request-available > Fix For: 9.6 > > Time Spent: 6h > Remaining Estimate: 0h > > _(Present status: reverted / not accomplished. New issues needed to > continue)_ > Today, CloudSolrClient will locally fail if it's asked to send a request to a > collection that it thinks does not exist due to its local ClusterState view > being out-of-date. We shouldn't fail! And most SolrCloud tests should then > remove their waitForState calls that follow collection creation! Other stale > state matters are out-of-scope. > Proposal: CloudSolrClient shouldn't try and be too smart. Always route a > request to Solr (any node); don't presume its state is up-to-date. Maybe, > after a response is received, it can check if its state has been updated and > if not then explicitly get a new state. Or not if that's too complicated. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17606) ClusterStateProvider.collectionStream
David Smiley created SOLR-17606: --- Summary: ClusterStateProvider.collectionStream Key: SOLR-17606 URL: https://issues.apache.org/jira/browse/SOLR-17606 Project: Solr Issue Type: Improvement Components: SolrCloud, SolrJ Reporter: David Smiley Provide a collectionStream() method to ClusterStateProvider with the same semantics as ClusterState.collectionStream. The point is for Solr users of CloudSolrClient (like the Prometheus Exporter) to not refer to ClusterState. The work is primarily in the HTTP CSP. It could have a naive impl that loads from ClusterState (fine for most users) but it'd be better to try to _stream_ the response directly from Solr somehow so that all the DocCollections aren't materialized in memory. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17520) CloudSolrClient with HTTP ClusterState shouldn't time-expire recently used cache entries
[ https://issues.apache.org/jira/browse/SOLR-17520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-17520: Priority: Minor (was: Major) > CloudSolrClient with HTTP ClusterState shouldn't time-expire recently used > cache entries > > > Key: SOLR-17520 > URL: https://issues.apache.org/jira/browse/SOLR-17520 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud, SolrJ >Reporter: David Smiley >Priority: Minor > > When using CloudSolrClient with HTTP URLs to Solr for the cluster state: > CloudSolrClient caches a collection's state (DocCollection) for a limited > time in ExpiringCachedDocCollection. It doesn't matter how > frequently/recently this collection is used; it'll expire anyway. I think > instead, the expiration should keep getting pushed back on each use so that > we don't needlessly ask Solr for the DocCollection every ~minute. The > possible stale-ness of a DocCollection shall be handled by existing > "stateVer" mechanisms. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17606) ClusterStateProvider.collectionStream
[ https://issues.apache.org/jira/browse/SOLR-17606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-17606: Priority: Minor (was: Major) This is a nice-to-have. A potential caller of this (like the Prometheus Exporter) could instead work directly with Solr's admin APIs to get the state of all collections. > ClusterStateProvider.collectionStream > - > > Key: SOLR-17606 > URL: https://issues.apache.org/jira/browse/SOLR-17606 > Project: Solr > Issue Type: Improvement > Components: SolrCloud, SolrJ >Reporter: David Smiley >Priority: Minor > > Provide a collectionStream() method to ClusterStateProvider with the same > semantics as ClusterState.collectionStream. The point is for Solr users of > CloudSolrClient (like the Prometheus Exporter) to not refer to ClusterState. > The work is primarily in the HTTP CSP. It could have a naive impl that loads > from ClusterState (fine for most users) but it'd be better to try to _stream_ > the response directly from Solr somehow so that all the DocCollections aren't > materialized in memory. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Resolved] (SOLR-17153) CloudSolrClient should not throw "Collection not found" with an out-dated ClusterState
[ https://issues.apache.org/jira/browse/SOLR-17153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley resolved SOLR-17153. - Resolution: Incomplete (was: Fixed) > CloudSolrClient should not throw "Collection not found" with an out-dated > ClusterState > -- > > Key: SOLR-17153 > URL: https://issues.apache.org/jira/browse/SOLR-17153 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Reporter: David Smiley >Priority: Major > Fix For: 9.6 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > Today, CloudSolrClient will locally fail if it's asked to send a request to a > collection that it thinks does not exist due to its local ClusterState view > being out-of-date. We shouldn't fail! And most SolrCloud tests should then > remove their waitForState calls that follow collection creation! Other stale > state matters are out-of-scope. > Proposal: CloudSolrClient shouldn't try and be too smart. Always route a > request to Solr (any node); don't presume its state is up-to-date. Maybe, > after a response is received, it can check if its state has been updated and > if not then explicitly get a new state. Or not if that's too complicated. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17153) CloudSolrClient should not throw "Collection not found" with an out-dated ClusterState
[ https://issues.apache.org/jira/browse/SOLR-17153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-17153: Status: Reopened (was: Closed) > CloudSolrClient should not throw "Collection not found" with an out-dated > ClusterState > -- > > Key: SOLR-17153 > URL: https://issues.apache.org/jira/browse/SOLR-17153 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Reporter: David Smiley >Priority: Major > Fix For: 9.6 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > Today, CloudSolrClient will locally fail if it's asked to send a request to a > collection that it thinks does not exist due to its local ClusterState view > being out-of-date. We shouldn't fail! And most SolrCloud tests should then > remove their waitForState calls that follow collection creation! Other stale > state matters are out-of-scope. > Proposal: CloudSolrClient shouldn't try and be too smart. Always route a > request to Solr (any node); don't presume its state is up-to-date. Maybe, > after a response is received, it can check if its state has been updated and > if not then explicitly get a new state. Or not if that's too complicated. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17153: CloudSolrClient should not throw "Collection not found" with an out-dated ClusterState [solr]
dsmiley commented on PR #2363: URL: https://github.com/apache/solr/pull/2363#issuecomment-2556137265 @markrmiller , thinking back to your feedback, I have an idea of a solution in several steps: (A) In https://issues.apache.org/jira/browse/SOLR-17601 I propose using HTTP response headers to inform the client of the state version of a collection that the client is interacting with. Imagine if collection admin commands participated in this (not just handlers on the core) like create collection. Thus when CREATE returns, the header indicates the version of the final created state. Let's also assume waitForState=true[1]. (B) Even though the CloudSolrClient's cache[2] wouldn't yet have a DocCollection, let's imagine an enhancement to this cache in which it can hold a cache entry without a DocCollection but with a version to indicate we now want this version (or greater). At least for HTTP ClusterStateProvider, it will fetch the CLUSTERSTATE from Solr via HTTP. What's new -- imagine it adds the \_stateVer\_ parameter. (C) HttpSolrCall will see the \_stateVer\_ param. If it's DocCollection isn't as high as this version, it forces an update. I think this part works as-is but perhaps some work may be needed. [1] why the heck is waitForState not true by default? [2] the CSC's cache should move to ClusterStateProvider -- https://issues.apache.org/jira/browse/SOLR-17604 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17153: CloudSolrClient should not throw "Collection not found" with an out-dated ClusterState [solr]
markrmiller commented on PR #2363: URL: https://github.com/apache/solr/pull/2363#issuecomment-2556330775 Yeah, that’s how I would do it. For any command that alters the state, return the version of the modified state and have the client make sure that’s at least the version it’s using for the next command. The Cloud client does *something* like this for some case(s), (if I remember right, as something that kicks in on retries), but I don’t recall it properly covering this stuff. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
