Re: [PR] SOLR-17309: Enhance certificate based authentication plugin with flexible cert principal resolution [solr]
epugh commented on PR #3029: URL: https://github.com/apache/solr/pull/3029#issuecomment-2589605275 Could you explain a bit more what the general use case is? I read the code, and I see lots of complex words I don't understand ;-).Maybe the Ref Guide docs would make it all clear! I do like having Solr support more types of authentications! One concern is that is this a place we should leverage another project that would insulate us from the specifics of various auth tools? (maybe something for a larger discussion). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Move DOMUtil ConfigNode methods to ConfigNode, with some name changes. [solr]
epugh commented on code in PR #3027:
URL: https://github.com/apache/solr/pull/3027#discussion_r1914614723
##
solr/solrj/src/java/org/apache/solr/common/ConfigNode.java:
##
@@ -93,9 +107,16 @@ default String attr(String name) {
return attributes().get(name);
}
- default String requiredStrAttr(String name, Supplier err) {
+ /**
+ * Like {@link #attr(String)} but throws an error (incorporating {@code
missing_err}) if not
+ * found.
+ */
+ default String attrRequired(String name, String missing_err) {
Review Comment:
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Move DOMUtil ConfigNode methods to ConfigNode, with some name changes. [solr]
epugh commented on code in PR #3027:
URL: https://github.com/apache/solr/pull/3027#discussion_r1914615978
##
solr/core/src/java/org/apache/solr/core/PluginInfo.java:
##
@@ -100,39 +100,23 @@ public String toString() {
}
}
+ /** From XML. */
Review Comment:
i think it's fine. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[PR] SOLR-17562: Unify JAX-RS "raw response" endpoints [solr]
gerlowskija opened a new pull request, #3032: URL: https://github.com/apache/solr/pull/3032 https://issues.apache.org/jira/browse/SOLR-17562 # Description Several v2 APIs return raw files or streams of data, including: ZooKeeperReadAPI, NodeFileStore, and CoreReplication.fetchFile. But the APIs vary slightly in how they support this: ZooKeeperReadAPI uses the deprecated "ContentStream" with "RawResponseWriter", NodeFileStore directly attaches a "SolrCore.RawWriter" to the underlying SolrQueryResponse, and CoreReplication follows the JAX-RS best practice of using the "StreamingOutput" interface. The biggest hurdle to aligning on a single approach is modifying the templates used by our SolrJ code generation to handle the new approach (and "big stream of data" style responses in general). # Solution This PR tackles this by introducing a new SolrResponse implementation, `InputStreamResponse`, to wrap the NamedList produced by the existing `InputStreamResponseParser`. (InputStreamResponseParser creates a NamedList with two entries, "responseStatus" (an int) and "stream" (an InputStream)). The new SolrResponse type is used as the response type by our code-gen template for any "raw" responses annotated with a "rawOutput" tag introduced by this PR. Docs are added in `dev-docs/v2-api-conventions.adoc` to highlight this setup. NOTE: this PR provides a unified approach for "raw" API responses (e.g. index files returned by replication APIs, blobs stored in the filestore, etc.). Callers of these APIs essentially get a glorified InputStream to work with on the client side. The PR doesn't aim to handle other APIs that stream more structured data, e.g. the doc-stream returned by `/export`. We can likely do better than a "glorified InputStream" in these cases, but this PR makes no attempt to provide that. # Tests Additional unit tests in `InputStreamResponseTest`. Existing tests for the ZK and replication APIs continue to pass. # Checklist Please review the following and check all that apply: - [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [x] I have created a Jira issue and added the issue ID to my pull request title. - [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation) - [x] I have developed this patch against the `main` branch. - [ ] I have run `./gradlew check`. - [x] I have added tests for my changes. - [x] I have added documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17562) Unify v2 API streaming support
[
https://issues.apache.org/jira/browse/SOLR-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-17562:
--
Labels: pull-request-available (was: )
> Unify v2 API streaming support
> --
>
> Key: SOLR-17562
> URL: https://issues.apache.org/jira/browse/SOLR-17562
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: v2 API
>Reporter: Jason Gerlowski
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Several v2 APIs return raw files or streams of data, including:
> {{ZooKeeperReadAPI}}, {{NodeFileStore}}, and {{CoreReplication.fetchFile}}.
> But the APIs vary slightly in how they support this: ZooKeeperReadAPI uses
> the deprecated "ContentStream" with "RawResponseWriter", NodeFileStore
> directly attaches a "SolrCore.RawWriter" to the underlying SolrQueryResponse,
> and CoreReplication follows the JAX-RS best practice of using the
> "StreamingOutput" interface.
> This ticket aims to align all of these approaches and document our approach
> in {{dev-docs/apis.adoc}} or a similar file.
> The preferred approach ([see discussion
> here|https://github.com/apache/solr/pull/2734]) at the time of writing is to
> use StreamingOutput. If this doesn't change, this ticket will need to:
> * modify our Java codegen template and related code to support this new
> response type. (Java codegen currently requires that all responses subclass
> SolrJerseyResponse)
> * remove the "x-omitFromCodegen" tag from any APIs using StreamingOutput (see
> ReplicationApis.fetchFile for an example)
> * Switch other raw-file/streaming APIs over to using StreamingOutput.
> Validate v1 and v2 responses.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Migrate schema designer to filestoreapi part deux [solr]
epugh commented on code in PR #3031:
URL: https://github.com/apache/solr/pull/3031#discussion_r1914833957
##
solr/core/src/java/org/apache/solr/filestore/DistribFileStore.java:
##
@@ -534,7 +534,6 @@ public void refresh(String path) {
@SuppressWarnings({"rawtypes"})
List myFiles = list(path, s -> true);
for (Object f : l) {
- // TODO: https://issues.apache.org/jira/browse/SOLR-15426
Review Comment:
@madrob you opened up this JIRA, and I *think* this TODO is incorrect or
otherwise overcome by events. Regardless, it doesn't appear connected to
SOLR-15426... Can you confirm?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16391: Convert "modify-coll" API to JAX-RS [solr]
gerlowskija commented on code in PR #2930:
URL: https://github.com/apache/solr/pull/2930#discussion_r1915243956
##
solr/api/src/java/org/apache/solr/client/api/endpoint/CollectionApis.java:
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.api.endpoint;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.Parameter;
+import jakarta.ws.rs.DELETE;
+import jakarta.ws.rs.GET;
+import jakarta.ws.rs.POST;
+import jakarta.ws.rs.PUT;
+import jakarta.ws.rs.Path;
+import jakarta.ws.rs.PathParam;
+import jakarta.ws.rs.QueryParam;
+import org.apache.solr.client.api.model.CreateCollectionRequestBody;
+import org.apache.solr.client.api.model.ListCollectionsResponse;
+import org.apache.solr.client.api.model.ModifyCollectionRequestBody;
+import org.apache.solr.client.api.model.SubResponseAccumulatingJerseyResponse;
+
+/** V2 API definitions for manipulating Solr collections */
+public interface CollectionApis {
+
+ @Path("/collections")
+ interface List {
+@GET
+@Operation(
+summary = "List all collections in this Solr cluster",
+tags = {"collections"})
+ListCollectionsResponse listCollections();
+ }
+
+ /** V2 API definition for creating a SolrCloud collection */
+ @Path("/collections")
+ interface Create {
+@POST
+@Operation(
+summary = "Creates a new SolrCloud collection.",
+tags = {"collections"})
+SubResponseAccumulatingJerseyResponse
createCollection(CreateCollectionRequestBody requestBody)
+throws Exception;
+ }
+
+ @Path("/collections/{collectionName}")
+ interface Modify {
+@PUT
Review Comment:
> I think that if we embrace PATCH only here, then yeah, it will be kind of
obsurcure. But if we use it on multiple APIs, then I think it's perfectly fine.
That seems like a reasonable line to walk here. Anyone have other APIs in
mind that use similar partial-update semantics? If so, I'll switch this to
'PATCH'. Otherwise I'll stick with 'PUT' for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16391: Convert "modify-coll" API to JAX-RS [solr]
gerlowskija commented on PR #2930: URL: https://github.com/apache/solr/pull/2930#issuecomment-2590552508 > Perhaps HTTP DELETE for deleting them one at a time and we see if this is good enough for folks? OK, good call - I'll pursue that route for now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Migrate schema designer to filestoreapi part deux [solr]
madrob commented on code in PR #3031:
URL: https://github.com/apache/solr/pull/3031#discussion_r1915252465
##
solr/core/src/java/org/apache/solr/filestore/DistribFileStore.java:
##
@@ -534,7 +534,6 @@ public void refresh(String path) {
@SuppressWarnings({"rawtypes"})
List myFiles = list(path, s -> true);
for (Object f : l) {
- // TODO: https://issues.apache.org/jira/browse/SOLR-15426
Review Comment:
I think it was actually related to SOLR-15385, I must have made a copy/paste
error here. Reading my comments and surface-level tracing the code suggests
that this is still an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[PR] SOLR-6122 POC new cancel handler for SOLR-6122 [solr]
QFREEE opened a new pull request, #3033: URL: https://github.com/apache/solr/pull/3033 https://issues.apache.org/jira/browse/SOLR-6122 # Description The PR implemented a new overseer level collection handler to cancel submitted & in-progress collections api calls. As an example, this PR only opt-in backup collection api for in-progress cancel. in-progress collections api calls are not complete, as core level cancel is not implemented. Current implemention removes tracking of in-progress collections api from overseers, which can be useful in some situation like backup. # Solution General flow: -> Client request cancle with async ID through ```http://localhost:8983/solr/admin/collections?action=CANCEL&requestid=1``` -> Cancel message handled by `OverseerCancelMessageHandler` to determine if this asyncId exist and if it's currently in submitted or in-progress state : - if submitted: remove from `workQueue` and clear `blockedTasks` in case submitted task is in blocked tasks - if in-progress: check if is `isInProgressCancelable` -> remove in-progress tasks from tracking queues in overseer `workQueue` & `runningMap` & `runningTasks` Api: New V2 api interface for cancel handler Overseer: - stored asyncId on to ZK's running map to keep track of in-process async task easier - Adjust `OverseerTaskQueue` to handle remove / get item from queue given zk path OverseerCancelMessageHandler: - a new message handler specifically to handle cancel. Since cancel collections api involovs removing queue event from overseer's queue, added this part of logic as message handler instead of a new collection api # Tests Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem. # Checklist Please review the following and check all that apply: - [ ] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [ ] I have created a Jira issue and added the issue ID to my pull request title. - [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation) - [ ] I have developed this patch against the `main` branch. - [ ] I have run `./gradlew check`. - [ ] I have added tests for my changes. - [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[PR] Migrate schema designer to filestoreapi part deux [solr]
epugh opened a new pull request, #3031: URL: https://github.com/apache/solr/pull/3031 https://issues.apache.org/jira/browse/SOLR-X # Description Migrate Schema Designer calls from using Blob store to FileStore # Solution Swap implementation methods. # Tests So far, just using existing tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Migrate schema designer to filestoreapi part deux [solr]
epugh commented on PR #3031: URL: https://github.com/apache/solr/pull/3031#issuecomment-2589942671 Progress so far @gerlowskija -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Migrate schema designer to filestoreapi part deux [solr]
epugh commented on code in PR #3031:
URL: https://github.com/apache/solr/pull/3031#discussion_r1914833957
##
solr/core/src/java/org/apache/solr/filestore/DistribFileStore.java:
##
@@ -534,7 +534,6 @@ public void refresh(String path) {
@SuppressWarnings({"rawtypes"})
List myFiles = list(path, s -> true);
for (Object f : l) {
- // TODO: https://issues.apache.org/jira/browse/SOLR-15426
Review Comment:
@madrob you opened up this JIRA, and I *think* this TODO is incorrect or
otherwise overcome by events TODO. Regardless, it doesn't appear connected to
SOLR-15426... Can you confirm?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Demonstrate roundtrip export/import works [solr]
gerlowskija commented on code in PR #2940: URL: https://github.com/apache/solr/pull/2940#discussion_r1915077934 ## solr/solr-ref-guide/modules/indexing-guide/pages/reindexing.adoc: ## @@ -165,6 +165,20 @@ Once the alias is in place and you are satisfied you no longer need the old data One advantage of this option is that you can switch back to the old collection if you discover problems our testing did not uncover. Of course this option can require more resources until the old collection can be deleted. +=== Exporting/Importing Data from Solr + +Sometimes you don't want to run your full ETL pipeline to reindex into another collection, you just want to take the data in your existing collection, export it, and then import it back. Review Comment: [-0] AFAIK `bin/solr export` uses the `/export` handler and [can only return fields that have docValues enabled](https://solr.apache.org/guide/solr/latest/query-guide/exporting-result-sets.html#field-requirements). That's a huge limitation that we should probably mention here! Imagine someone's confusion when they follow these docs and somehow lose all of the text fields they were searching on! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Demonstrate roundtrip export/import works [solr]
gerlowskija commented on code in PR #2940: URL: https://github.com/apache/solr/pull/2940#discussion_r1915089283 ## solr/solr-ref-guide/modules/indexing-guide/pages/reindexing.adoc: ## @@ -165,6 +165,20 @@ Once the alias is in place and you are satisfied you no longer need the old data One advantage of this option is that you can switch back to the old collection if you discover problems our testing did not uncover. Of course this option can require more resources until the old collection can be deleted. +=== Exporting/Importing Data from Solr + +Sometimes you don't want to run your full ETL pipeline to reindex into another collection, you just want to take the data in your existing collection, export it, and then import it back. + +There are a number of third party tools that do this, see https://solr.cool/ for more information. However, if you want to use what ships with Solr then we have some options: + +1. Use `bin/solr export` with the JSON output format (`.json`), and the `bin/solr post` tool to post that data back. Review Comment: [Q] How would a user choose between these three options? Or put differently - why would they choose one over the others? If there's no strong differentiator between the three, is there value in mentioning them all individually? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Demonstrate roundtrip export/import works [solr]
gerlowskija commented on code in PR #2940: URL: https://github.com/apache/solr/pull/2940#discussion_r1915077934 ## solr/solr-ref-guide/modules/indexing-guide/pages/reindexing.adoc: ## @@ -165,6 +165,20 @@ Once the alias is in place and you are satisfied you no longer need the old data One advantage of this option is that you can switch back to the old collection if you discover problems our testing did not uncover. Of course this option can require more resources until the old collection can be deleted. +=== Exporting/Importing Data from Solr + +Sometimes you don't want to run your full ETL pipeline to reindex into another collection, you just want to take the data in your existing collection, export it, and then import it back. Review Comment: [-0] AFAIK `bin/solr export` uses the `/export` handler and [can only return fields that have docValues enabled](https://solr.apache.org/guide/solr/latest/query-guide/exporting-result-sets.html#field-requirements). So it'd drop most text fields, etc. That's a huge limitation that we should probably mention here! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Migrate schema designer to filestoreapi part deux [solr]
gerlowskija commented on PR #3031: URL: https://github.com/apache/solr/pull/3031#issuecomment-2590329199 Looks good to me so far. I notice this is still "Draft"; curious what else is remaining here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Migrate schema designer to filestoreapi part deux [solr]
gerlowskija commented on code in PR #3031:
URL: https://github.com/apache/solr/pull/3031#discussion_r1915099312
##
solr/core/src/java/org/apache/solr/handler/designer/SchemaDesignerConfigSetHelper.java:
##
@@ -494,62 +498,59 @@ protected void validateTypeChange(String configSet,
SchemaField field, FieldType
SolrException.ErrorCode.BAD_REQUEST,
"Cannot change type of the _version_ field; it must be a plong.");
}
-List docs = getStoredSampleDocs(configSet);
+List docs = retrieveSampleDocs(configSet);
if (!docs.isEmpty()) {
schemaSuggester.validateTypeChange(field, toType, docs);
}
}
void deleteStoredSampleDocs(String configSet) {
-try {
- cloudClient().deleteByQuery(BLOB_STORE_ID, "id:" + configSet +
"_sample/*", 10);
-} catch (IOException | SolrServerException | SolrException exc) {
- final String excStr = exc.toString();
- log.warn("Failed to delete sample docs from blob store for {} due to:
{}", configSet, excStr);
-}
+String path =
+"blob" + "/" + configSet
++ "_sample"; // needs to be made unique to support multiple
uploads. Maybe hash the
+// docs?
+// why do I have to do this in two stages?
+DistribFileStore.deleteZKFileEntry(cc.getZkController().getZkClient(),
path);
+cc.getFileStore().delete(path);
}
+ // I don't like this guy just hanging out here to support retrieveSampleDocs.
+ List docs = Collections.emptyList();
+
@SuppressWarnings("unchecked")
- List getStoredSampleDocs(final String configSet) throws
IOException {
-List docs = null;
+ List retrieveSampleDocs(final String configSet) throws
IOException {
-final URI uri;
-try {
- uri =
- collectionApiEndpoint(BLOB_STORE_ID, "blob", configSet + "_sample")
- .setParameter(CommonParams.WT, "filestream")
- .build();
-} catch (URISyntaxException e) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
-}
+String path =
+"blob" + "/" + configSet
Review Comment:
[0] Might be nice to make the "blob/" bit here and other similar path
segments constants, that can be reused on both the storage and retrieval side
of this helper code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Fix smoketest for Solr 10 [solr]
HoustonPutman commented on PR #3024: URL: https://github.com/apache/solr/pull/3024#issuecomment-2590708982 > LGTM, assuming this change is for `main` only and won't be backported to branch_9x? > > Also, assuming you picked Java25 in some places as it's the next LTS after Java 21? Yes to both questions! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-6122: POC cancel submitted not started async task [solr]
QFREEE closed pull request #2790: SOLR-6122: POC cancel submitted not started async task URL: https://github.com/apache/solr/pull/2790 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] chore(deps): update dependency com.diffplug.spotless to v7 [solr]
solrbot commented on PR #3005: URL: https://github.com/apache/solr/pull/3005#issuecomment-2591452513 ### ⚠️ Artifact update problem Renovate failed to update an artifact related to this branch. You probably do not want to merge this PR as-is. ♻ Renovate will retry this branch, including artifacts, only when one of the following happens: - any of the package files in this branch needs updating, or - the branch becomes conflicted, or - you click the rebase/retry checkbox if found above, or - you rename this PR's title to start with "rebase!" to trigger it manually The artifact failure details are included below: # File name: gradle/libs.versions.toml ``` Command failed: ./gradlew --console=plain --dependency-verification lenient -q :dependencies :solr:dependencies :solr:api:dependencies :solr:benchmark:dependencies :solr:core:dependencies :solr:cross-dc-manager:dependencies :solr:distribution:dependencies :solr:docker:dependencies :solr:documentation:dependencies :solr:example:dependencies :solr:modules:dependencies :solr:packaging:dependencies :solr:prometheus-exporter:dependencies :solr:server:dependencies :solr:solr-ref-guide:dependencies :solr:solrj:dependencies :solr:solrj-streaming:dependencies :solr:solrj-zookeeper:dependencies :solr:test-framework:dependencies :solr:webapp:dependencies :solr:modules:analysis-extras:dependencies :solr:modules:clustering:dependencies :solr:modules:cross-dc:dependencies :solr:modules:extraction:dependencies :solr:modules:gcs-repository:dependencies :solr:modules:hdfs:dependencies :solr:modules:jwt-auth:dependencies :solr:modules:langid:dependencies :solr:modules:llm:dependencies :solr:modules :ltr:dependencies :solr:modules:opentelemetry:dependencies :solr:modules:s3-repository:dependencies :solr:modules:scripting:dependencies :solr:modules:sql:dependencies --update-locks com.diffplug.spotless:com.diffplug.spotless.gradle.plugin FAILURE: Build failed with an exception. * What went wrong: Failed to notify task execution graph listener. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the lock file, '--write-locks' along is not sufficient. > Use the ':writeLocks' task to write the
Re: [PR] SOLR-17611 SolrJ User-Agent, pass Solr version [solr]
dsmiley commented on code in PR #2927:
URL: https://github.com/apache/solr/pull/2927#discussion_r1915917903
##
solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java:
##
@@ -1170,6 +1171,21 @@ protected Object _getHandler() {
return handler;
}
+ /** Gets the client (user-agent) SolrJ version, or null if isn't SolrJ. */
+ public SolrVersion getUserAgentSolrVersion() {
+String header = req.getHeader("User-Agent");
+if (header == null || !header.startsWith("Solr")) {
+ return null;
+}
+try {
+ return SolrVersion.valueOf(header.substring(header.lastIndexOf(' ') +
1));
Review Comment:
I added this to the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17486) Potentially unused code in SolrDispatchFilter - abortErrorMessage
[ https://issues.apache.org/jira/browse/SOLR-17486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913165#comment-17913165 ] calelin commented on SOLR-17486: Anyone has bandwidth to create a PR for this? If not please let me know whether I could help create a PR. Thanks! > Potentially unused code in SolrDispatchFilter - abortErrorMessage > - > > Key: SOLR-17486 > URL: https://issues.apache.org/jira/browse/SOLR-17486 > Project: Solr > Issue Type: Bug >Affects Versions: main (10.0) >Reporter: Gus Heck >Priority: Minor > Attachments: image-2024-10-09-10-01-33-503.png > > > I was reading code on main, and noticed that SolrDispatchFilter has a string > field called abortErrorMessage. This is checked in HttpSolrCall, and the code > purports to exit if there is an abort message, but I see no place in code > where this is ever set. > It is protected access so maybe there's some intent to set it in an > overridden version of SolrDispatchFilter somewhere, that has become obsolete? > This code looks like it can be deleted. Raising this issue to find out if > anyone has a notion of why it should be retained. > !image-2024-10-09-10-01-33-503.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17611 SolrJ User-Agent, pass Solr version [solr]
dsmiley commented on code in PR #2927:
URL: https://github.com/apache/solr/pull/2927#discussion_r1915918653
##
solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java:
##
@@ -1170,6 +1171,21 @@ protected Object _getHandler() {
return handler;
}
+ /** Gets the client (user-agent) SolrJ version, or null if isn't SolrJ. */
+ public SolrVersion getUserAgentSolrVersion() {
+String header = req.getHeader("User-Agent");
+if (header == null || !header.startsWith("Solr")) {
+ return null;
+}
+try {
+ return SolrVersion.valueOf(header.substring(header.lastIndexOf(' ') +
1));
Review Comment:
BTW that method has no callers; isn't tested. I did manually test it.
Obviously I intend to have it be used soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] build: remove gradle/ant-compat [solr]
dsmiley merged PR #3030: URL: https://github.com/apache/solr/pull/3030 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-6122) API to cancel an already submitted/running Collections API call
[ https://issues.apache.org/jira/browse/SOLR-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913167#comment-17913167 ] Yuntong Qu commented on SOLR-6122: -- Hi folks, raised another [PR|https://github.com/apache/solr/pull/3033] for POC implementation of in-progress cancel. Explained my approach in the PR, however, would love to get some ideas: - In order for core level to truly cancel in-progress collections API, I believe this will have to be collection cmd specific. One idea might be to add different check points within core-level cmd code. Maybe there could be other approaches? > API to cancel an already submitted/running Collections API call > --- > > Key: SOLR-6122 > URL: https://issues.apache.org/jira/browse/SOLR-6122 > Project: Solr > Issue Type: Wish > Components: SolrCloud >Reporter: Anshum Gupta >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Right now we can trigger a long running task with no way to cancel it > cleanly. > We should have an API that interrupts the already running/submitted > collections API call. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] build: remove gradle/ant-compat [solr]
dsmiley commented on PR #3030: URL: https://github.com/apache/solr/pull/3030#issuecomment-2591621730 No delta to the POM, before & after. I see no signing info in the POM though; not sure what to make of that. I'll merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Migrate schema designer to filestoreapi part deux [solr]
dsmiley commented on code in PR #3031:
URL: https://github.com/apache/solr/pull/3031#discussion_r1915959864
##
solr/core/src/java/org/apache/solr/filestore/FileStore.java:
##
@@ -38,8 +42,13 @@ public interface FileStore {
*/
void put(FileEntry fileEntry) throws IOException;
- /** read file content from a given path */
- void get(String path, Consumer filecontent, boolean getMissing)
throws IOException;
+ /**
+ * Read file content from a given path.
+ *
+ * TODO: Is fetchMissing actually used? I don't see it being used, but
the IDE doesn't flag it
+ * not being used!
Review Comment:
Lucene/Solr devs use a "nocommit" comment for something like this.
Precommit will fail so we remember to address it. I also configure IntelliJ's
"TODO" feature to consider "nocommit" and thus it shows in blood red (gets my
attention)
##
solr/core/src/java/org/apache/solr/handler/designer/SchemaDesignerConfigSetHelper.java:
##
@@ -494,62 +498,59 @@ protected void validateTypeChange(String configSet,
SchemaField field, FieldType
SolrException.ErrorCode.BAD_REQUEST,
"Cannot change type of the _version_ field; it must be a plong.");
}
-List docs = getStoredSampleDocs(configSet);
+List docs = retrieveSampleDocs(configSet);
if (!docs.isEmpty()) {
schemaSuggester.validateTypeChange(field, toType, docs);
}
}
void deleteStoredSampleDocs(String configSet) {
-try {
- cloudClient().deleteByQuery(BLOB_STORE_ID, "id:" + configSet +
"_sample/*", 10);
-} catch (IOException | SolrServerException | SolrException exc) {
- final String excStr = exc.toString();
- log.warn("Failed to delete sample docs from blob store for {} due to:
{}", configSet, excStr);
-}
+String path =
+"blob" + "/" + configSet
++ "_sample"; // needs to be made unique to support multiple
uploads. Maybe hash the
+// docs?
+// why do I have to do this in two stages?
+DistribFileStore.deleteZKFileEntry(cc.getZkController().getZkClient(),
path);
+cc.getFileStore().delete(path);
}
+ // I don't like this guy just hanging out here to support retrieveSampleDocs.
+ List docs = Collections.emptyList();
+
@SuppressWarnings("unchecked")
- List getStoredSampleDocs(final String configSet) throws
IOException {
-List docs = null;
+ List retrieveSampleDocs(final String configSet) throws
IOException {
-final URI uri;
-try {
- uri =
- collectionApiEndpoint(BLOB_STORE_ID, "blob", configSet + "_sample")
- .setParameter(CommonParams.WT, "filestream")
- .build();
-} catch (URISyntaxException e) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
-}
+String path =
+"blob" + "/" + configSet
++ "_sample"; // needs to be made unique to support multiple
uploads. Maybe hash the
+// docs?
-HttpGet httpGet = new HttpGet(uri);
try {
- HttpResponse entity =
- ((CloudLegacySolrClient)
cloudClient()).getHttpClient().execute(httpGet);
- int statusCode = entity.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
-byte[] bytes = readAllBytes(() -> entity.getEntity().getContent());
-if (bytes.length > 0) {
- docs = (List) Utils.fromJavabin(bytes);
-}
- } else if (statusCode != HttpStatus.SC_NOT_FOUND) {
-byte[] bytes = readAllBytes(() -> entity.getEntity().getContent());
-throw new IOException(
-"Failed to lookup stored docs for "
-+ configSet
-+ " due to: "
-+ new String(bytes, StandardCharsets.UTF_8));
- } // else not found is ok
-} finally {
- httpGet.releaseConnection();
+ cc.getFileStore()
+ .get(
+ path,
+ entry -> {
+try (InputStream is = entry.getInputStream()) {
+ byte[] bytes = is.readAllBytes();
+ if (bytes.length > 0) {
+docs = (List) Utils.fromJavabin(bytes);
+ }
+ // Do something with content...
+} catch (IOException e) {
+ log.error("Error reading file content", e);
+}
+ },
+ true);
+} catch (java.io.FileNotFoundException e) {
Review Comment:
not sure why fully qualified
##
solr/core/src/java/org/apache/solr/handler/designer/SchemaDesignerConfigSetHelper.java:
##
@@ -494,62 +498,59 @@ protected void validateTypeChange(String configSet,
SchemaField field, FieldType
SolrException.ErrorCode.BAD_REQUEST,
"Cannot change type of the _version_ field; it must be a plong.");
}
-List docs = getStoredSampleDocs(configSet);
+List docs = retrieveSampleDocs(configSet);
if (!docs.isEmpty()) {
schemaSuggester.validateTy
Re: [PR] SOLR-17562: Unify JAX-RS "raw response" endpoints [solr]
dsmiley commented on code in PR #3032:
URL: https://github.com/apache/solr/pull/3032#discussion_r1915979535
##
solr/solrj/src/java/org/apache/solr/client/solrj/InputStreamResponse.java:
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj;
+
+import static
org.apache.solr.client.solrj.impl.InputStreamResponseParser.HTTP_STATUS_KEY;
+import static
org.apache.solr.client.solrj.impl.InputStreamResponseParser.STREAM_KEY;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.util.function.Function;
+import org.apache.solr.client.solrj.impl.InputStreamResponseParser;
+import org.apache.solr.client.solrj.response.SimpleSolrResponse;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.IOUtils;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.ObjectReleaseTracker;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Represents the NamedList response format created by {@link
InputStreamResponseParser}.
+ *
+ * Particularly useful when targeting APIs that return arbitrary or binary
data (e.g. replication
+ * APIs for fetching index files)
+ */
+public class InputStreamResponse extends SimpleSolrResponse {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ // True if the HTTP status is '200 OK', false otherwise
+ public static final Function HTTP_OK_VALIDATOR = (status)
-> 200 == status;
+ // True if the HTTP status is neither a 4xx or a 5xx error.
+ public static final Function NON_ERROR_CODE_VALIDATOR =
+ (status) -> status < 399;
+
+ @Override
+ public void setResponse(NamedList rsp) {
+if (rsp.get(STREAM_KEY) == null) {
+ throw new IllegalArgumentException(
+ "Missing key '"
+ + STREAM_KEY
+ + "'; "
+ + getClass().getSimpleName()
+ + " can only be used with requests or clients configured to use "
+ + InputStreamResponseParser.class.getSimpleName());
+}
+super.setResponse(rsp);
+ }
+
+ public int getHttpStatus() {
+return (int) getResponse().get(HTTP_STATUS_KEY);
+ }
+
+ /**
+ * Access the server response as an {@link InputStream}, regardless of the
HTTP status code
+ *
+ * Caller is responsible for consuming and closing the stream, and
releasing it from the
+ * tracking done by {@link ObjectReleaseTracker}. No validation is done on
the HTTP status code.
+ */
+ public InputStream getResponseStream() {
+final NamedList resp = getResponse();
+
+return (InputStream) resp.get(STREAM_KEY);
+ }
+
+ /**
+ * Access the server response as an {@link InputStream}, after ensuring that
the HTTP status code
+ * is 200 ('OK')
+ *
+ * Caller is responsible for consuming and closing the stream, and
releasing it from the
+ * tracking done by {@link ObjectReleaseTracker}.
+ */
+ public InputStream getResponseStreamIfSuccessful() {
+return getResponseStreamIfSuccessful(HTTP_OK_VALIDATOR);
+ }
+
+ /**
+ * Access the server response as an {@link InputStream}, after ensuring the
HTTP status code
+ * passes a provided validator.
+ *
+ * @param statusValidator a function that returns true iff the response body
should be returned
+ */
+ public InputStream getResponseStreamIfSuccessful(Function
statusValidator) {
+validateExpectedStatus(statusValidator);
+return getResponseStream();
+ }
+
+ private void validateExpectedStatus(Function
statusChecker) {
+final var httpStatus = getHttpStatus();
+if (!statusChecker.apply(httpStatus)) {
+ try {
+log.error(
+"Request returned unexpected HTTP status code {}; response
content: {}",
+httpStatus,
+consumeAndStringifyForLogging(getResponseStream()));
+ } catch (IOException e) {
+log.error("could not print error", e);
+ }
+ throw new SolrException(
+ SolrException.ErrorCode.getErrorCode(httpStatus),
+ "Unexpected status code [" + httpS
[PR] Publish build scans to develocity.apache.org [solr]
clayburn opened a new pull request, #3035: URL: https://github.com/apache/solr/pull/3035 # Description This PR migrates the Solr project to publish Build Scans to the the new Develocity instance at develocity.apache.org. # Solution N/A # Tests N/A # Checklist Please review the following and check all that apply: - [ ] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [ ] I have created a Jira issue and added the issue ID to my pull request title. - [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation) - [ ] I have developed this patch against the `main` branch. - [ ] I have run `./gradlew check`. - [ ] I have added tests for my changes. - [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17519) CloudSolrClient with HTTP ClusterState can forget live nodes and then fail
[
https://issues.apache.org/jira/browse/SOLR-17519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913086#comment-17913086
]
Houston Putman commented on SOLR-17519:
---
{quote}Another reason to exclusively use the configured Solr URLs (s) is that
the operator might want to use a load balancer (may even _already_ be using
one), not actually directly a Solr node itself. It might be a single dependable
URL.
{quote}
This is absolutely true, but then the user shouldn't be using the
"CloudSolrClient" correct? I think the naming is bad here. If I'm not mistaken,
the "CloudSolrClient"s act more like "LiveNodeSolrClients". Instead of the
LoadBalancingSolrClient which just load balances between a set of URLs. So in
this case, the operator or people using a single load balancer shouldn't use
the "CloudSolrClient", correct? Maybe this requires a better name or better
documentation. I'm not sure.
While I think this is true, I also don't think that the
HttpClusterStateProvider should be locked into this pattern, even if this is
the only use case for it (Not sure what else it's used for). The
HttpClusterStateProvider should have the option to use dynamic node discovery,
or just stick with its initial set of urls (as David is pushing for). The
CloudSolrClient should probably then use the dynamic reloading, for the reasons
listed above.
To summarize, I like the PR, but I think the dynamic part should be
opt-out-able in the HttpClusterStateProvider
> CloudSolrClient with HTTP ClusterState can forget live nodes and then fail
> --
>
> Key: SOLR-17519
> URL: https://issues.apache.org/jira/browse/SOLR-17519
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud, SolrJ
>Reporter: David Smiley
>Priority: Major
> Labels: newdev, pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> When using CloudSolrClient with HTTP URLs to Solr for the cluster state:
> If all live nodes disappear temporarily (hard cluster restart?), the client
> can permanently fail to talk to the cluster, and thus would need to be
> restarted to recover.
> Credit [~ilan] on the dev list:
> {quote}The current implementation removes non live nodes from the set of
> nodes to connect to. Getting the live nodes requires connecting to a specific
> node in the cluster that is therefore live when that happens. Worst case, if
> there is a single node up in the cluster, the client ends with a single node
> in its connection candidates list. For the issue to manifest, that Solr node
> then has to go down. Subsequently, even if other nodes are up, the client
> only has the address of a down node and can't connect.
> The fix is not a big deal. Nodes initially passed as configuration to the
> client should never be removed from the set of candidate nodes to connect to,
> even if they are not live. Other live nodes could be added to that set (and
> removed from it if we so desire when they are no longer live) to increase
> resiliency in case the cluster does have live nodes but all initially
> configured nodes are not live. The design issue is treating the configured
> set of nodes to connect to and the set of live nodes as one thing.
> {quote}
> See org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16391: Convert "modify-coll" API to JAX-RS [solr]
HoustonPutman commented on PR #2930: URL: https://github.com/apache/solr/pull/2930#issuecomment-2591063358 > Personally, I'm worried the API design issues highlight deeper confusion in the underlying functionality. Why do we support arbitrary properties in "modify" that are somehow different from "collectionprops". When would a user choose to use one or the other? I don't see an answer in our docs, and I don't know myself 😕 ) Yeah, this is very confusing to me. If the two have to be separate, maybe metadata is a better name for it, and we go with "Collection Properties" and "Collection Metadata", though I do not think the two are as separate as we would like them to be. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16391: Convert "modify-coll" API to JAX-RS [solr]
gerlowskija commented on PR #2930: URL: https://github.com/apache/solr/pull/2930#issuecomment-2591493830 I like "metadata" from a naming perspective. > I do not think the two are as separate as we would like them to be. Yeah, agreed. Worth some digging there to see what the original intention was, and whether it's worth maintaining both options (at least in terms of the arbitrary/user-specified "modify" properties). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16391: Convert "modify-coll" API to JAX-RS [solr]
dsmiley commented on PR #2930: URL: https://github.com/apache/solr/pull/2930#issuecomment-2591540203 +1 on "metadata" vs "properties" for nomenclature. I was thinking about the earlier conundrum a bit. Let's just do PATCH to `/api/collections/myCollName` to update anything updatable. To unset a value, why not just use null? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16391: Convert "modify-coll" API to JAX-RS [solr]
gerlowskija commented on PR #2930:
URL: https://github.com/apache/solr/pull/2930#issuecomment-2590915285
Ugh, actually, using an HTTP DELETE would conflict with existing paths.
- `DELETE /api/collections/someCollName/properties/somePropName` is my first
choice, but that conflicts with an existing API for deleting
collection-properties.
- `DELETE /api/collections/someCollName/somePropName` was my next thought,
but this also conflicts since "properties" is a valid prop name ("modify"
allows users to specify arbitrary props)
We could come up with some new path-segment to differentiate "modify"
properties from "collection props", say, `DELETE
/api/collections/someCollName/collStateProperties/somePropName`, but that feels
hacky? idk.
Personally, I'm worried the API design issues highlight deeper confusion in
the underlying functionality. Why do we support arbitrary properties in
"modify" that are somehow different from "collectionprops". When would a user
choose to use one or the other? I don't see an answer in our docs, and I don't
know myself 😕 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17568) Remove remote *core* query in SolrCloud mode
[
https://issues.apache.org/jira/browse/SOLR-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913101#comment-17913101
]
David Smiley commented on SOLR-17568:
-
A scenario where this can happen is when a replica moves (technically added and
deleted) and the client (CloudSolrClient) doesn't know about it yet. It
self-heals as the client gets an updated state and routes correctly.
> Remove remote *core* query in SolrCloud mode
>
>
> Key: SOLR-17568
> URL: https://issues.apache.org/jira/browse/SOLR-17568
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Labels: pull-request-available
> Fix For: main (10.0)
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> If a SolrCloud node receives a request with a local core name (not a
> collection name), it will be handled by the local core. But it even works
> with a remote core -- it'll be forwarded to the node that has a core by that
> name. I think there's no use-case for that -- users should only reference
> collections (any node) or a core on a node where it's known to exist (on that
> node!) since it's a core not a cluster concept. This got my attention
> because the functionality requires iterating all collections. This would
> happen for any request when a collection/core is not known to the node (and
> may not exist at all). I shall remove this in Solr 10.
> There is exactly one test for this functionality --
> {{org.apache.solr.cloud.AbstractBasicDistributedZk2TestBase#addAndQueryDocs}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] build: remove gradle/ant-compat [solr]
dsmiley commented on PR #3030: URL: https://github.com/apache/solr/pull/3030#issuecomment-2591304848 The only thing I want to do before merging is spot-check some generated Maven POMs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17519) CloudSolrClient with HTTP ClusterState can forget live nodes and then fail
[
https://issues.apache.org/jira/browse/SOLR-17519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913105#comment-17913105
]
David Smiley commented on SOLR-17519:
-
Houston:
When I said "the operator might want to use a load balancer" I meant the
user/deployer, not _necessarily_ the k8s Solr Operator. You imply a load
balancer invalidates the utility of CloudSolrClient but CSC is designed to
reduce network hops. You don't know what HttpClusterStateProvider is for...
See the parent issue for an explanation but I suspect I misunderstand you. You
are making a distinction between CSC and the provider; the provider's sole job
is to provide the ClusterState (including live nodes). CSC's job is to route
user requests to the optimal node.
{quote}I think the dynamic part should be opt-out-able in the
HttpClusterStateProvider
{quote}
But since it exists, still need to deal with the accompanying implementation
details & test. So we don't get a simplicity win. Ah well; shrug.
> CloudSolrClient with HTTP ClusterState can forget live nodes and then fail
> --
>
> Key: SOLR-17519
> URL: https://issues.apache.org/jira/browse/SOLR-17519
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud, SolrJ
>Reporter: David Smiley
>Priority: Major
> Labels: newdev, pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> When using CloudSolrClient with HTTP URLs to Solr for the cluster state:
> If all live nodes disappear temporarily (hard cluster restart?), the client
> can permanently fail to talk to the cluster, and thus would need to be
> restarted to recover.
> Credit [~ilan] on the dev list:
> {quote}The current implementation removes non live nodes from the set of
> nodes to connect to. Getting the live nodes requires connecting to a specific
> node in the cluster that is therefore live when that happens. Worst case, if
> there is a single node up in the cluster, the client ends with a single node
> in its connection candidates list. For the issue to manifest, that Solr node
> then has to go down. Subsequently, even if other nodes are up, the client
> only has the address of a down node and can't connect.
> The fix is not a big deal. Nodes initially passed as configuration to the
> client should never be removed from the set of candidate nodes to connect to,
> even if they are not live. Other live nodes could be added to that set (and
> removed from it if we so desire when they are no longer live) to increase
> resiliency in case the cluster does have live nodes but all initially
> configured nodes are not live. The design issue is treating the configured
> set of nodes to connect to and the set of live nodes as one thing.
> {quote}
> See org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[PR] build: Fix Out of space in CodeCache error [solr]
HoustonPutman opened a new pull request, #3034: URL: https://github.com/apache/solr/pull/3034 Fixing the same problem as https://github.com/apache/lucene/issues/14090 The full error is "VirtualMachineError: Out of space in CodeCache for adapters in gradle" The fix is to increase the reservedCodeCacheSize from the default to 256m, which is what Lucene uses now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] build: Fix Out of space in CodeCache error [solr]
epugh commented on PR #3034: URL: https://github.com/apache/solr/pull/3034#issuecomment-2591048494 Thanks for this! I've been seeing this recently too... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Fix smoketest for Solr 10 [solr]
HoustonPutman merged PR #3024: URL: https://github.com/apache/solr/pull/3024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Move DOMUtil ConfigNode methods to ConfigNode, with some name changes. [solr]
dsmiley merged PR #3027: URL: https://github.com/apache/solr/pull/3027 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17611 SolrJ User-Agent, pass Solr version [solr]
dsmiley commented on PR #2927: URL: https://github.com/apache/solr/pull/2927#issuecomment-2591610680 Ready to merge; will do within a couple days if I hear nothing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
