Re: [PR] SOLR-17151 Check limits between calls to components in SearchHandler [solr]
fsparv commented on code in PR #2801:
URL: https://github.com/apache/solr/pull/2801#discussion_r1834560447
##
solr/core/src/test/org/apache/solr/core/ExitableDirectoryReaderTest.java:
##
@@ -70,9 +70,12 @@ public void testPrefixQuery() throws Exception {
// this time we should get a query cache hit and hopefully no exception?
this may change in the
// future if time checks are put into other places.
-assertJQ(req("q", q, "timeAllowed", "1", "sleep", sleep), assertionString);
+// 2024-4-15: it did change..., and now this fails with 1 or 2 ms and
passes with 3ms... I see
+// no way this won't be terribly brittle. Maybe TestInjection of some sort
to bring this back?
Review Comment:
I don't think that helps. The test wants to verify that the result was
cached and is hoping that the request **_won't_** time out because of that, but
that only worked because timeAllowed was only measured/triggered in a small
area of code sensitive to the availability of the cache. Now we check it all
over the place. If we raise the limit too much it's not a test, and if this
test gets starved for resources it will fail spuriously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17151 Check limits between calls to components in SearchHandler [solr]
fsparv commented on code in PR #2801:
URL: https://github.com/apache/solr/pull/2801#discussion_r1834565410
##
solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java:
##
@@ -626,44 +697,97 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
rsp.addToLog(ThreadCpuTimer.CPU_TIME, totalShardCpuTime);
}
}
+ }
-// SOLR-5550: still provide shards.info if requested even for a short
circuited distrib request
-if (!rb.isDistrib
-&& req.getParams().getBool(ShardParams.SHARDS_INFO, false)
-&& rb.shortCircuitedURL != null) {
- NamedList shardInfo = new SimpleOrderedMap<>();
- SimpleOrderedMap nl = new SimpleOrderedMap<>();
- if (rsp.getException() != null) {
-Throwable cause = rsp.getException();
-if (cause instanceof SolrServerException) {
- cause = ((SolrServerException) cause).getRootCause();
-} else {
- if (cause.getCause() != null) {
-cause = cause.getCause();
- }
+ private static boolean prepareComponents(
+ SolrQueryRequest req, ResponseBuilder rb, RTimerTree timer,
List components)
+ throws IOException {
+if (timer == null) {
+ // non-debugging prepare phase
+ for (SearchComponent component : components) {
+if (checkLimitsBefore(component, "prepare", rb.req, rb.rsp,
components)) {
+ shortCircuitedResults(req, rb);
+ return false;
}
-nl.add("error", cause.toString());
-if (!core.getCoreContainer().hideStackTrace()) {
- StringWriter trace = new StringWriter();
- cause.printStackTrace(new PrintWriter(trace));
- nl.add("trace", trace.toString());
+component.prepare(rb);
+ }
+} else {
+ // debugging prepare phase
+ RTimerTree subt = timer.sub("prepare");
+ for (SearchComponent c : components) {
+if (checkLimitsBefore(c, "prepare debug", rb.req, rb.rsp, components))
{
+ shortCircuitedResults(req, rb);
+ return false;
}
- } else if (rb.getResults() != null) {
-nl.add("numFound", rb.getResults().docList.matches());
-nl.add(
-"numFoundExact",
-rb.getResults().docList.hitCountRelation() ==
TotalHits.Relation.EQUAL_TO);
-nl.add("maxScore", rb.getResults().docList.maxScore());
+rb.setTimer(subt.sub(c.getName()));
+c.prepare(rb);
+rb.getTimer().stop();
}
- nl.add("shardAddress", rb.shortCircuitedURL);
- nl.add("time", req.getRequestTimer().getTime()); // elapsed time of this
request so far
+ subt.stop();
+}
+return true;
+ }
- int pos = rb.shortCircuitedURL.indexOf("://");
- String shardInfoName =
- pos != -1 ? rb.shortCircuitedURL.substring(pos + 3) :
rb.shortCircuitedURL;
- shardInfo.add(shardInfoName, nl);
- rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ private static String stageInEnglish(int nextStage) {
+// This should probably be a enum, but that change should be its own
ticket.
+switch (nextStage) {
+ case STAGE_START:
+return "START";
+ case STAGE_PARSE_QUERY:
+return "PARSE_QUERY";
+ case STAGE_TOP_GROUPS:
+return "TOP_GROUPS";
+ case STAGE_EXECUTE_QUERY:
+return "EXECUTE_QUERY";
+ case STAGE_GET_FIELDS:
+return "GET_FIELDS";
+// nobody wants to think it was DONE and canceled after it completed...
+ case STAGE_DONE:
+return "FINISHING";
+ default:
+throw new SolrException(
+SolrException.ErrorCode.SERVER_ERROR, "Unrecognized stage:" +
nextStage);
+}
+ }
+
+ private static void shortCircuitedResults(SolrQueryRequest req,
ResponseBuilder rb) {
+
+if (rb.rsp.getResponse() == null) {
+ rb.rsp.addResponse(new SolrDocumentList());
+
+ // If a cursorMark was passed, and we didn't progress, set
+ // the nextCursorMark to the same position
+ String cursorStr =
rb.req.getParams().get(CursorMarkParams.CURSOR_MARK_PARAM);
+ if (null != cursorStr) {
+rb.rsp.add(CursorMarkParams.CURSOR_MARK_NEXT, cursorStr);
+ }
}
+if (rb.isDebug()) {
+ NamedList debug = new NamedList<>();
+ debug.add("explain", new NamedList<>());
+ rb.rsp.add("debug", debug);
+}
+rb.rsp.setPartialResults(rb.req);
+ }
+
+ private static boolean checkLimitsBefore(
+ SearchComponent c,
+ String when,
+ SolrQueryRequest req,
+ SolrQueryResponse resp,
+ List components) {
+
+return getQueryLimits(req, resp)
+.maybeExitWithPartialResults(
+() ->
+"["
++ when
++ "] Limit(s) exceeded prior to "
++ c.getName()
++ " in "
++ components.stream()
+.map(SearchComponent::
Re: [PR] SOLR-17535: Deprecate ClusterState.forEachCollection [solr]
dsmiley commented on code in PR #2854:
URL: https://github.com/apache/solr/pull/2854#discussion_r1834868691
##
solr/solrj/src/java/org/apache/solr/common/cloud/ClusterState.java:
##
@@ -425,7 +425,10 @@ public Stream collectionStream() {
/**
* Calls {@code consumer} with a resolved {@link DocCollection}s for all
collections. Use this
* sparingly in case there are many collections.
+ *
+ * @deprecated see {@link #collectionStream()}
Review Comment:
It has existed for some releases, thus I deprecate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17535: Deprecate ClusterState.forEachCollection [solr]
dsmiley commented on PR #2854: URL: https://github.com/apache/solr/pull/2854#issuecomment-2465480585 I'll merge this Monday in case there are any further comments, but I wouldn't anticipate any for this simple one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Resolved] (SOLR-17453) Replace CloudUtil.waitForState and some TimeOut with ZkStateReader.waitForState
[ https://issues.apache.org/jira/browse/SOLR-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley resolved SOLR-17453. - Fix Version/s: 9.8 Resolution: Fixed Merged – thanks for contributing Pierre! > Replace CloudUtil.waitForState and some TimeOut with > ZkStateReader.waitForState > --- > > Key: SOLR-17453 > URL: https://issues.apache.org/jira/browse/SOLR-17453 > Project: Solr > Issue Type: Improvement >Reporter: David Smiley >Priority: Minor > Labels: newdev, pull-request-available > Fix For: 9.8 > > Time Spent: 2.5h > Remaining Estimate: 0h > > We should universally use ZkStateReader.waitForState when waiting for the > ClusterState to change based on a predicate. SolrCloudTestCase.waitForState > is fine since it calls the former. But CloudUtil.waitForState does not; it > should be replaced. Additionally, TimeOut is used in some places wait > waitForState ought to be used, like CreateCollectionCmd. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17151 Check limits between calls to components in SearchHandler [solr]
fsparv commented on code in PR #2801:
URL: https://github.com/apache/solr/pull/2801#discussion_r1834603826
##
solr/core/src/test/org/apache/solr/core/ExitableDirectoryReaderTest.java:
##
@@ -70,9 +70,12 @@ public void testPrefixQuery() throws Exception {
// this time we should get a query cache hit and hopefully no exception?
this may change in the
// future if time checks are put into other places.
-assertJQ(req("q", q, "timeAllowed", "1", "sleep", sleep), assertionString);
+// 2024-4-15: it did change..., and now this fails with 1 or 2 ms and
passes with 3ms... I see
+// no way this won't be terribly brittle. Maybe TestInjection of some sort
to bring this back?
Review Comment:
What we really need for this test is a cache admin command that can list the
contents of the a cache (and one would expect a manual clear capability too,
though that's not needed here) One might be able to get by with a metrics call,
looking for an increment in cache hits, but that's pretty heavy reliance on
coincidence rather than explicit testing of the actual desired behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17151 Check limits between calls to components in SearchHandler [solr]
fsparv commented on code in PR #2801:
URL: https://github.com/apache/solr/pull/2801#discussion_r1834603826
##
solr/core/src/test/org/apache/solr/core/ExitableDirectoryReaderTest.java:
##
@@ -70,9 +70,12 @@ public void testPrefixQuery() throws Exception {
// this time we should get a query cache hit and hopefully no exception?
this may change in the
// future if time checks are put into other places.
-assertJQ(req("q", q, "timeAllowed", "1", "sleep", sleep), assertionString);
+// 2024-4-15: it did change..., and now this fails with 1 or 2 ms and
passes with 3ms... I see
+// no way this won't be terribly brittle. Maybe TestInjection of some sort
to bring this back?
Review Comment:
What we really need for this test is a cache admin command that can list the
contents of the cache (and one would expect a manual clear capability too,
though that's not needed here) One might be able to get by with a metrics call,
looking for an increment in cache hits, but that's pretty heavy reliance on
coincidence rather than explicit testing of the actual desired behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Assigned] (SOLR-17406) Introduce Gradle Version Catalogs
[ https://issues.apache.org/jira/browse/SOLR-17406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christos Malliaridis reassigned SOLR-17406: --- Assignee: Christos Malliaridis > Introduce Gradle Version Catalogs > - > > Key: SOLR-17406 > URL: https://issues.apache.org/jira/browse/SOLR-17406 > Project: Solr > Issue Type: Improvement > Components: Gradle >Affects Versions: main (10.0) >Reporter: Christos Malliaridis >Assignee: Christos Malliaridis >Priority: Minor > Labels: dependencies, gradle, pull-request-available > Time Spent: 4h 10m > Remaining Estimate: 0h > > With Gradle it is possible to manage versions and dependencies in gradle > files via version catalogs. This allows us to cleanup dependency resolution > and move various versions from across the project to a single file. > Dawid Weiss demonstrated in [Lucene PR > #13484|https://github.com/apache/lucene/pull/13484/] such a migration > together with a few improvements in the gradle build files that could be > addressed in this ticket aswell. > The migration include the removal of palantir's consistent versions plugin. > Dev list thread: > [https://lists.apache.org/thread/wh6c0x2ncbpd61pwwddtv3l256lv9kr5] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17406) Introduce Gradle Version Catalogs
[ https://issues.apache.org/jira/browse/SOLR-17406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christos Malliaridis updated SOLR-17406: Status: Patch Available (was: Open) > Introduce Gradle Version Catalogs > - > > Key: SOLR-17406 > URL: https://issues.apache.org/jira/browse/SOLR-17406 > Project: Solr > Issue Type: Improvement > Components: Gradle >Affects Versions: main (10.0) >Reporter: Christos Malliaridis >Assignee: Christos Malliaridis >Priority: Minor > Labels: dependencies, gradle, pull-request-available > Time Spent: 4h 10m > Remaining Estimate: 0h > > With Gradle it is possible to manage versions and dependencies in gradle > files via version catalogs. This allows us to cleanup dependency resolution > and move various versions from across the project to a single file. > Dawid Weiss demonstrated in [Lucene PR > #13484|https://github.com/apache/lucene/pull/13484/] such a migration > together with a few improvements in the gradle build files that could be > addressed in this ticket aswell. > The migration include the removal of palantir's consistent versions plugin. > Dev list thread: > [https://lists.apache.org/thread/wh6c0x2ncbpd61pwwddtv3l256lv9kr5] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17151 Check limits between calls to components in SearchHandler [solr]
fsparv commented on code in PR #2801:
URL: https://github.com/apache/solr/pull/2801#discussion_r1834649523
##
solr/core/src/test/org/apache/solr/core/ExitableDirectoryReaderTest.java:
##
@@ -70,9 +70,12 @@ public void testPrefixQuery() throws Exception {
// this time we should get a query cache hit and hopefully no exception?
this may change in the
// future if time checks are put into other places.
-assertJQ(req("q", q, "timeAllowed", "1", "sleep", sleep), assertionString);
+// 2024-4-15: it did change..., and now this fails with 1 or 2 ms and
passes with 3ms... I see
+// no way this won't be terribly brittle. Maybe TestInjection of some sort
to bring this back?
Review Comment:
I just looked at the ability to check directly. There is some hope since we
could
```
SolrCores solrCores =
ExitableDirectoryReaderTest.h.getCoreContainer().solrCores;
List cores = solrCores.getCores();
for (SolrCore core : cores) {
// identify core we need here...
}
```
but assuming we can identify the appropriate core, there isn't even an
internal api to check caches. Would look for something steming from
`core.getSearcher().get().???` but we would need to add an accessor for the
queryCache and then on SolrCache a way to inspect the keys of the cache...
that seems way beyond what this change should attempt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17453) Replace CloudUtil.waitForState and some TimeOut with ZkStateReader.waitForState
[ https://issues.apache.org/jira/browse/SOLR-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896774#comment-17896774 ] ASF subversion and git services commented on SOLR-17453: Commit 8ee2fe280cf84b8f8780a18c4b52c404e7cfaa81 in solr's branch refs/heads/branch_9x from Pierre Salagnac [ https://gitbox.apache.org/repos/asf?p=solr.git;h=8ee2fe280cf ] SOLR-17453: Leverage waitForState() instead of busy waiting (#2737) Leverage waitForState() instead of busy waiting in CREATE, MIGRATE, REINDEXCOLLECTION, MOVEREPLICA commands, and in some tests. (cherry picked from commit b657ec0c2176ef58c224147b71a91fa5a8267e51) > Replace CloudUtil.waitForState and some TimeOut with > ZkStateReader.waitForState > --- > > Key: SOLR-17453 > URL: https://issues.apache.org/jira/browse/SOLR-17453 > Project: Solr > Issue Type: Improvement >Reporter: David Smiley >Priority: Minor > Labels: newdev, pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > We should universally use ZkStateReader.waitForState when waiting for the > ClusterState to change based on a predicate. SolrCloudTestCase.waitForState > is fine since it calls the former. But CloudUtil.waitForState does not; it > should be replaced. Additionally, TimeOut is used in some places wait > waitForState ought to be used, like CreateCollectionCmd. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-14673: Add bin/solr stream CLI [solr]
malliaridis commented on PR #2479: URL: https://github.com/apache/solr/pull/2479#issuecomment-2465603881 A few more notes: - [ ] `bin/solr stream --help` does not include any reference to the stream expression argument required at position 0 (or any other position that is valid) - [ ] When providing an invalid stream expression, error message is not helpful ```sh bin\solr.cmd stream "search(q=*:*)" -e local -s http://localhost:8983 -c techproducts ``` (missing collection in expression) returns "Unable to construct instance of org.apache.solr.client.solrj.io.stream.SearchFacadeStream" - I would like a message that at least says something like "double check your stream expression" if it is one of the causes - [ ] For my understanding, is it a technical limitation that `-e local` does not require a collection (`-c`), but `-e solr` requires one? - [ ] I also noticed that providing `-s` requires a URI scheme, but `-z` does not allow one. Is this inconsistency for all commands? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-14673: Add bin/solr stream CLI [solr]
malliaridis commented on code in PR #2479:
URL: https://github.com/apache/solr/pull/2479#discussion_r1834961429
##
solr/core/src/java/org/apache/solr/cli/StreamTool.java:
##
@@ -0,0 +1,507 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cli;
+
+import java.io.BufferedReader;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.LineNumberReader;
+import java.io.PrintStream;
+import java.io.Reader;
+import java.io.StringReader;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.solr.client.solrj.io.Lang;
+import org.apache.solr.client.solrj.io.SolrClientCache;
+import org.apache.solr.client.solrj.io.Tuple;
+import org.apache.solr.client.solrj.io.comp.StreamComparator;
+import org.apache.solr.client.solrj.io.stream.PushBackStream;
+import org.apache.solr.client.solrj.io.stream.SolrStream;
+import org.apache.solr.client.solrj.io.stream.StreamContext;
+import org.apache.solr.client.solrj.io.stream.TupleStream;
+import org.apache.solr.client.solrj.io.stream.expr.Explanation;
+import org.apache.solr.client.solrj.io.stream.expr.Expressible;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpressionParser;
+import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.handler.CatStream;
+
+/** Supports stream command in the bin/solr script. */
+public class StreamTool extends ToolBase {
+
+ public StreamTool() {
+this(CLIO.getOutStream());
+ }
+
+ public StreamTool(PrintStream stdout) {
+super(stdout);
+ }
+
+ private final SolrClientCache solrClientCache = new SolrClientCache();
+
+ @Override
+ public String getName() {
+return "stream";
+ }
+
+ private static final Option EXECUTION_OPTION =
+ Option.builder("e")
+ .longOpt("execution")
+ .hasArg()
+ .argName("CONTEXT")
+ .required(false)
+ .desc(
+ "Execution context is either 'local' (i.e CLI
process) or 'solr'. Default is 'solr'")
+ .build();
+
+ private static final Option COLLECTION_OPTION = Option.builder("c")
+.longOpt("name")
+.argName("NAME")
+.hasArg()
+.desc(
+"Name of the collection to execute on if workers are 'solr'.
Required for 'solr' worker.")
+.build();
+
+ private static final Option FIELDS_OPTION = Option.builder("f")
+.longOpt("fields")
+.argName("FIELDS")
+.hasArg()
+.required(false)
+.desc(
+"The fields in the tuples to output. (defaults to fields in
the first tuple of result set).")
+.build();
+
+ private static final Option HEADER_OPTION =
+ Option.builder()
+ .longOpt("header")
+ .required(false)
+ .desc("Whether or not to include a header line.
(default=false)")
+ .build();
+ private static final Option DELIMITER_OPTION = Option.builder()
+ .longOpt("delimiter")
+.argName("CHARACTER")
+.hasArg()
+.required(false)
+.desc("The output delimiter. (default=tab).")
+.build();
+ private static final Option ARRAY_DELIMITER_OPTION = Option.builder()
+.longOpt("array-delimiter")
+.argName("CHARACTER")
+.hasArg()
+.required(false)
+.desc("The delimiter multi-valued fields. (default=|)")
+.build();
+
+ @O
Re: [PR] SOLR-14673: Add bin/solr stream CLI [solr]
malliaridis commented on code in PR #2479:
URL: https://github.com/apache/solr/pull/2479#discussion_r1834961429
##
solr/core/src/java/org/apache/solr/cli/StreamTool.java:
##
@@ -0,0 +1,507 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cli;
+
+import java.io.BufferedReader;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.LineNumberReader;
+import java.io.PrintStream;
+import java.io.Reader;
+import java.io.StringReader;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.solr.client.solrj.io.Lang;
+import org.apache.solr.client.solrj.io.SolrClientCache;
+import org.apache.solr.client.solrj.io.Tuple;
+import org.apache.solr.client.solrj.io.comp.StreamComparator;
+import org.apache.solr.client.solrj.io.stream.PushBackStream;
+import org.apache.solr.client.solrj.io.stream.SolrStream;
+import org.apache.solr.client.solrj.io.stream.StreamContext;
+import org.apache.solr.client.solrj.io.stream.TupleStream;
+import org.apache.solr.client.solrj.io.stream.expr.Explanation;
+import org.apache.solr.client.solrj.io.stream.expr.Expressible;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpressionParser;
+import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.handler.CatStream;
+
+/** Supports stream command in the bin/solr script. */
+public class StreamTool extends ToolBase {
+
+ public StreamTool() {
+this(CLIO.getOutStream());
+ }
+
+ public StreamTool(PrintStream stdout) {
+super(stdout);
+ }
+
+ private final SolrClientCache solrClientCache = new SolrClientCache();
+
+ @Override
+ public String getName() {
+return "stream";
+ }
+
+ private static final Option EXECUTION_OPTION =
+ Option.builder("e")
+ .longOpt("execution")
+ .hasArg()
+ .argName("CONTEXT")
+ .required(false)
+ .desc(
+ "Execution context is either 'local' (i.e CLI
process) or 'solr'. Default is 'solr'")
+ .build();
+
+ private static final Option COLLECTION_OPTION = Option.builder("c")
+.longOpt("name")
+.argName("NAME")
+.hasArg()
+.desc(
+"Name of the collection to execute on if workers are 'solr'.
Required for 'solr' worker.")
+.build();
+
+ private static final Option FIELDS_OPTION = Option.builder("f")
+.longOpt("fields")
+.argName("FIELDS")
+.hasArg()
+.required(false)
+.desc(
+"The fields in the tuples to output. (defaults to fields in
the first tuple of result set).")
+.build();
+
+ private static final Option HEADER_OPTION =
+ Option.builder()
+ .longOpt("header")
+ .required(false)
+ .desc("Whether or not to include a header line.
(default=false)")
+ .build();
+ private static final Option DELIMITER_OPTION = Option.builder()
+ .longOpt("delimiter")
+.argName("CHARACTER")
+.hasArg()
+.required(false)
+.desc("The output delimiter. (default=tab).")
+.build();
+ private static final Option ARRAY_DELIMITER_OPTION = Option.builder()
+.longOpt("array-delimiter")
+.argName("CHARACTER")
+.hasArg()
+.required(false)
+.desc("The delimiter multi-valued fields. (default=|)")
+.build();
+
+ @O
Re: [PR] SOLR-14673: Add bin/solr stream CLI [solr]
malliaridis commented on code in PR #2479:
URL: https://github.com/apache/solr/pull/2479#discussion_r1834874458
##
solr/core/src/java/org/apache/solr/cli/StreamTool.java:
##
@@ -0,0 +1,507 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cli;
+
+import java.io.BufferedReader;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.LineNumberReader;
+import java.io.PrintStream;
+import java.io.Reader;
+import java.io.StringReader;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.solr.client.solrj.io.Lang;
+import org.apache.solr.client.solrj.io.SolrClientCache;
+import org.apache.solr.client.solrj.io.Tuple;
+import org.apache.solr.client.solrj.io.comp.StreamComparator;
+import org.apache.solr.client.solrj.io.stream.PushBackStream;
+import org.apache.solr.client.solrj.io.stream.SolrStream;
+import org.apache.solr.client.solrj.io.stream.StreamContext;
+import org.apache.solr.client.solrj.io.stream.TupleStream;
+import org.apache.solr.client.solrj.io.stream.expr.Explanation;
+import org.apache.solr.client.solrj.io.stream.expr.Expressible;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpressionParser;
+import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.handler.CatStream;
+
+/** Supports stream command in the bin/solr script. */
+public class StreamTool extends ToolBase {
+
+ public StreamTool() {
+this(CLIO.getOutStream());
+ }
+
+ public StreamTool(PrintStream stdout) {
+super(stdout);
+ }
+
+ private final SolrClientCache solrClientCache = new SolrClientCache();
+
+ @Override
+ public String getName() {
+return "stream";
+ }
+
+ private static final Option EXECUTION_OPTION =
+ Option.builder("e")
+ .longOpt("execution")
+ .hasArg()
+ .argName("CONTEXT")
+ .required(false)
+ .desc(
+ "Execution context is either 'local' (i.e CLI
process) or 'solr'. Default is 'solr'")
+ .build();
+
+ private static final Option COLLECTION_OPTION = Option.builder("c")
+.longOpt("name")
+.argName("NAME")
+.hasArg()
+.desc(
+"Name of the collection to execute on if workers are 'solr'.
Required for 'solr' worker.")
+.build();
+
+ private static final Option FIELDS_OPTION = Option.builder("f")
+.longOpt("fields")
+.argName("FIELDS")
+.hasArg()
+.required(false)
+.desc(
+"The fields in the tuples to output. (defaults to fields in
the first tuple of result set).")
+.build();
+
+ private static final Option HEADER_OPTION =
+ Option.builder()
+ .longOpt("header")
+ .required(false)
Review Comment:
Any `.required(false)` is obsolete and could be removed.
##
solr/core/src/java/org/apache/solr/cli/StreamTool.java:
##
@@ -0,0 +1,507 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by appl
[jira] [Created] (SOLR-17550) preferred Overseer might not work
David Smiley created SOLR-17550: --- Summary: preferred Overseer might not work Key: SOLR-17550 URL: https://issues.apache.org/jira/browse/SOLR-17550 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: SolrCloud Reporter: David Smiley By code inspection, ZkController.setPreferredOverseer creates a command message improperly. The key "node" should exist with the node name as value but instead the key is itself the node name – a bug introduced by a refactoring in 2022. Ideally we understand the ramifications; like shouldn't this be found by a test? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-14680: Remove SimpleMap (affects NamedList) [solr]
dsmiley commented on code in PR #2856:
URL: https://github.com/apache/solr/pull/2856#discussion_r1835195200
##
solr/core/src/java/org/apache/solr/util/DataConfigNode.java:
##
@@ -54,31 +53,17 @@ public DataConfigNode(ConfigNode root) {
e.setValue(List.copyOf(e.getValue()));
}
}
-this.kids = kids.isEmpty() ? EMPTY : new
WrappedSimpleMap<>(Map.copyOf(kids));
+this.kids = Map.copyOf(kids);
}
- public String subtituteVal(String s) {
+ private static String substituteVal(String s) {
return PropertiesUtil.substitute(s, SUBSTITUTES.get());
Review Comment:
This particular aspect of the design of ConfigNode framework is, um, ...
really concerning to me: a ThreadLocal to hold the substitution rules.
**Wow**, so depending on which thread is *looking*(using) the map, they get
different answers?! @noblepaul can you offer an explanation as to why it's
this way instead of a normal field on the ConfigNode? This design has me
concerned with respect to this PR since there were some Map conversions (via
`asMap` which surprisingly creates a new Map; isn't a view) that maybe are now
a view; I need to check. The difference is that previously the value would be
materialized using the ThreadLocal of whoever called asMap but now would be
whoever is looking at the values. Subtle!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-17551) zk upconfig command broken on Windows
[
https://issues.apache.org/jira/browse/SOLR-17551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingchun Zhao updated SOLR-17551:
-
Attachment: SOLR-17551.patch
> zk upconfig command broken on Windows
> -
>
> Key: SOLR-17551
> URL: https://issues.apache.org/jira/browse/SOLR-17551
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Affects Versions: 9.7
> Environment: Windows Server 2022, Windows 10
>Reporter: Mingchun Zhao
>Priority: Critical
> Attachments: SOLR-17551.patch
>
>
> When running the `solr zk upconfig` command on Windows, the following error
> occurs.
> {code:java}
> C:\Program Files\solr-9.7.0\bin>solr zk upconfig -d sitecore_configs -n
> sitecoreconfigset -z 10.1.0.11:2181,10.1.0.12:2181,10.1.0.13:2181
> 'ELSE' is not recognized as an internal or external command,
> operable program or batch file.
> Failed to parse command-line arguments due to: Unrecognized option: --zk-host
> ZK_HOSTusage: bin/solr upconfig -d -n [-url ] [-z
> ]List of options:
> -d,--conf-dir Local directory with configs.
> -n,--conf-name Configset name in ZooKeeper.
> -url,--solr-url Base Solr URL, which can be used to determine the
> zk-host if that's not known; defaults to:
> http://Solrcloud-test-Solr01:8983.
> -z,--zk-host Zookeeper connection string; unnecessary if ZK_HOST
> is defined in solr.in.sh; otherwise,
> defaults to localhost:9983.Please see the Reference
> Guide for more tools documentation:
> https://solr.apache.org/guide/solr/latest/deployment-guide/solr-control-script-reference.html
> WARN - 2024-11-07 11:29:32.428; org.apache.solr.common.cloud.SolrZkClient;
> Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it
> creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to
> Zookeeper nodes
> WARN - 2024-11-07 11:29:33.666; org.apache.solr.common.cloud.SolrZkClient;
> Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates
> 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
> Uploading C:\Program
> Files\solr-9.7.0\server\solr\configsets\sitecore_configs\conf for config
> sitecoreconfigset to ZooKeeper at 10.1.0.11:2181 {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17552) NamedList.asShallowMap improvements
David Smiley created SOLR-17552: --- Summary: NamedList.asShallowMap improvements Key: SOLR-17552 URL: https://issues.apache.org/jira/browse/SOLR-17552 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: David Smiley NamedList.asShallowMap could use some improvements. * javadocs * putAll doesn't honor allowDups to add new pairs for existing keys * several // TODO implement more efficiently -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Created] (SOLR-17551) zk upconfig command broken on Windows
Mingchun Zhao created SOLR-17551:
Summary: zk upconfig command broken on Windows
Key: SOLR-17551
URL: https://issues.apache.org/jira/browse/SOLR-17551
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: SolrCloud
Affects Versions: 9.7
Environment: Windows Server 2022, Windows 10
Reporter: Mingchun Zhao
When running the `solr zk upconfig` command on Windows, the following error
occurs.
{code:java}
C:\Program Files\solr-9.7.0\bin>solr zk upconfig -d sitecore_configs -n
sitecoreconfigset -z 10.1.0.11:2181,10.1.0.12:2181,10.1.0.13:2181
'ELSE' is not recognized as an internal or external command,
operable program or batch file.
Failed to parse command-line arguments due to: Unrecognized option: --zk-host
ZK_HOSTusage: bin/solr upconfig -d -n [-url ] [-z
]List of options:
-d,--conf-dir Local directory with configs.
-n,--conf-name Configset name in ZooKeeper.
-url,--solr-url Base Solr URL, which can be used to determine the
zk-host if that's not known; defaults to:
http://Solrcloud-test-Solr01:8983.
-z,--zk-host Zookeeper connection string; unnecessary if ZK_HOST
is defined in solr.in.sh; otherwise,
defaults to localhost:9983.Please see the Reference
Guide for more tools documentation:
https://solr.apache.org/guide/solr/latest/deployment-guide/solr-control-script-reference.html
WARN - 2024-11-07 11:29:32.428; org.apache.solr.common.cloud.SolrZkClient;
Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it
creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to
Zookeeper nodes
WARN - 2024-11-07 11:29:33.666; org.apache.solr.common.cloud.SolrZkClient;
Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates
'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
Uploading C:\Program
Files\solr-9.7.0\server\solr\configsets\sitecore_configs\conf for config
sitecoreconfigset to ZooKeeper at 10.1.0.11:2181 {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] chore(deps): update io.netty:* to v4.1.114.final [solr]
malliaridis commented on PR #2702: URL: https://github.com/apache/solr/pull/2702#issuecomment-2465468387 We could introduce in this PR netty BOM https://mvnrepository.com/artifact/io.netty/netty-bom/4.1.114.Final to sync all netty libs based on the BOM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[PR] SOLR-16116: Fix various issues with Curator [solr]
HoustonPutman opened a new pull request, #2855: URL: https://github.com/apache/solr/pull/2855 https://issues.apache.org/jira/browse/SOLR-16116 Will eventually post a list of fixed tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17525: Text Embedder Query Parser [solr]
cpoerschke commented on code in PR #2809:
URL: https://github.com/apache/solr/pull/2809#discussion_r1834066298
##
solr/modules/llm/src/java/org/apache/solr/llm/embedding/SolrEmbeddingModel.java:
##
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.llm.embedding;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.model.embedding.DimensionAwareEmbeddingModel;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.lucene.util.Accountable;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.solr.llm.store.EmbeddingModelException;
+
+public class SolrEmbeddingModel implements Accountable {
+ private static final long BASE_RAM_BYTES =
+ RamUsageEstimator.shallowSizeOfInstance(SolrEmbeddingModel.class);
+ public static final String TIMEOUT_PARAM = "timeout";
+ public static final String MAX_SEGMENTS_PER_BATCH_PARAM =
"maxSegmentsPerBatch";
+ public static final String MAX_RETRIES_PARAM = "maxRetries";
+
+ protected final String name;
+ private final Map params;
+ private final EmbeddingModel embedder;
+ private Integer hashCode;
+
+ public static SolrEmbeddingModel getInstance(
+ String className, String name, Map params) throws
EmbeddingModelException {
+try {
+ EmbeddingModel embedder;
+ Class modelClass = Class.forName(className);
+ var builder = modelClass.getMethod("builder").invoke(null);
+ if (params != null) {
+for (String paramName : params.keySet()) {
+ switch (paramName) {
+case TIMEOUT_PARAM:
+ Duration timeOut = Duration.ofSeconds((Long)
params.get(paramName));
+ builder.getClass().getMethod(paramName,
Duration.class).invoke(builder, timeOut);
+ break;
+case MAX_SEGMENTS_PER_BATCH_PARAM:
+ builder
+ .getClass()
+ .getMethod(paramName, Integer.class)
+ .invoke(builder, ((Long) params.get(paramName)).intValue());
+ break;
+case MAX_RETRIES_PARAM:
+ builder
+ .getClass()
+ .getMethod(paramName, Integer.class)
+ .invoke(builder, ((Long) params.get(paramName)).intValue());
+ break;
+default:
+ ArrayList methods = new ArrayList<>();
+ for (var method : builder.getClass().getMethods()) {
+if (paramName.equals(method.getName()) &&
method.getParameterCount() == 1) {
+ methods.add(method);
+}
+ }
+ if (methods.size() == 1) {
+methods.get(0).invoke(builder, params.get(paramName));
+ } else {
+builder
+.getClass()
+.getMethod(paramName, String.class)
+.invoke(builder, params.get(paramName));
+ }
+ }
+}
+ }
+ embedder = (EmbeddingModel)
builder.getClass().getMethod("build").invoke(builder);
+ return new SolrEmbeddingModel(name, embedder, params);
+} catch (final Exception e) {
+ throw new EmbeddingModelException("Model loading failed for " +
className, e);
+}
+ }
+
+ public SolrEmbeddingModel(String name, EmbeddingModel embedder, Map params) {
+this.name = name;
+this.embedder = embedder;
+this.params = params;
+this.hashCode = calculateHashCode();
+ }
+
+ public float[] vectorise(String text) {
+Embedding vector = embedder.embed(text).content();
+return vector.vector();
+ }
+
+ @Override
+ public String toString() {
+return getClass().getSimpleName() + "(name=" + getName() + ")";
+ }
+
+ @Override
+ public long ramBytesUsed() {
+return BASE_RAM_BYTES
++ RamUsageEstimator.sizeOfObject(name)
++ RamUsageEstimator.sizeOfObject(embedder);
+ }
+
+ @Override
+ public int hashCode() {
+if (hashCode == null) {
+ hashCode = calculateHashCode();
+}
Review Comment:
compute
Re: [PR] SOLR-17525: Text Embedder Query Parser [solr]
cpoerschke commented on code in PR #2809:
URL: https://github.com/apache/solr/pull/2809#discussion_r183404
##
solr/modules/llm/src/java/org/apache/solr/llm/embedding/SolrEmbeddingModel.java:
##
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.llm.embedding;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.model.embedding.DimensionAwareEmbeddingModel;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.lucene.util.Accountable;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.solr.llm.store.EmbeddingModelException;
+
+public class SolrEmbeddingModel implements Accountable {
+ private static final long BASE_RAM_BYTES =
+ RamUsageEstimator.shallowSizeOfInstance(SolrEmbeddingModel.class);
+ public static final String TIMEOUT_PARAM = "timeout";
+ public static final String MAX_SEGMENTS_PER_BATCH_PARAM =
"maxSegmentsPerBatch";
+ public static final String MAX_RETRIES_PARAM = "maxRetries";
+
+ protected final String name;
+ private final Map params;
+ private final EmbeddingModel embedder;
+ private Integer hashCode;
Review Comment:
```suggestion
private final Integer hashCode;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17525: Text Embedder Query Parser [solr]
cpoerschke commented on code in PR #2809:
URL: https://github.com/apache/solr/pull/2809#discussion_r1834068088
##
solr/modules/llm/src/java/org/apache/solr/llm/embedding/SolrEmbeddingModel.java:
##
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.llm.embedding;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.model.embedding.DimensionAwareEmbeddingModel;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.lucene.util.Accountable;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.solr.llm.store.EmbeddingModelException;
+
+public class SolrEmbeddingModel implements Accountable {
+ private static final long BASE_RAM_BYTES =
+ RamUsageEstimator.shallowSizeOfInstance(SolrEmbeddingModel.class);
+ public static final String TIMEOUT_PARAM = "timeout";
+ public static final String MAX_SEGMENTS_PER_BATCH_PARAM =
"maxSegmentsPerBatch";
+ public static final String MAX_RETRIES_PARAM = "maxRetries";
Review Comment:
```suggestion
private static final String TIMEOUT_PARAM = "timeout";
private static final String MAX_SEGMENTS_PER_BATCH_PARAM =
"maxSegmentsPerBatch";
private static final String MAX_RETRIES_PARAM = "maxRetries";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17525: Text Embedder Query Parser [solr]
cpoerschke commented on code in PR #2809:
URL: https://github.com/apache/solr/pull/2809#discussion_r1834068534
##
solr/modules/llm/src/java/org/apache/solr/llm/embedding/SolrEmbeddingModel.java:
##
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.llm.embedding;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.model.embedding.DimensionAwareEmbeddingModel;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.lucene.util.Accountable;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.solr.llm.store.EmbeddingModelException;
+
+public class SolrEmbeddingModel implements Accountable {
+ private static final long BASE_RAM_BYTES =
+ RamUsageEstimator.shallowSizeOfInstance(SolrEmbeddingModel.class);
+ public static final String TIMEOUT_PARAM = "timeout";
+ public static final String MAX_SEGMENTS_PER_BATCH_PARAM =
"maxSegmentsPerBatch";
+ public static final String MAX_RETRIES_PARAM = "maxRetries";
+
+ protected final String name;
Review Comment:
maybe, haven't checked
```suggestion
private final String name;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17525: Text Embedder Query Parser [solr]
cpoerschke commented on code in PR #2809:
URL: https://github.com/apache/solr/pull/2809#discussion_r1834068534
##
solr/modules/llm/src/java/org/apache/solr/llm/embedding/SolrEmbeddingModel.java:
##
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.llm.embedding;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.model.embedding.DimensionAwareEmbeddingModel;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.lucene.util.Accountable;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.solr.llm.store.EmbeddingModelException;
+
+public class SolrEmbeddingModel implements Accountable {
+ private static final long BASE_RAM_BYTES =
+ RamUsageEstimator.shallowSizeOfInstance(SolrEmbeddingModel.class);
+ public static final String TIMEOUT_PARAM = "timeout";
+ public static final String MAX_SEGMENTS_PER_BATCH_PARAM =
"maxSegmentsPerBatch";
+ public static final String MAX_RETRIES_PARAM = "maxRetries";
+
+ protected final String name;
Review Comment:
minor: getter is provided i.e. can make this private
```suggestion
private final String name;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17525: Text Embedder Query Parser [solr]
cpoerschke commented on code in PR #2809:
URL: https://github.com/apache/solr/pull/2809#discussion_r1834073353
##
solr/modules/llm/src/java/org/apache/solr/llm/embedding/SolrEmbeddingModel.java:
##
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.llm.embedding;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.model.embedding.DimensionAwareEmbeddingModel;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.lucene.util.Accountable;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.solr.llm.store.EmbeddingModelException;
+
+public class SolrEmbeddingModel implements Accountable {
+ private static final long BASE_RAM_BYTES =
+ RamUsageEstimator.shallowSizeOfInstance(SolrEmbeddingModel.class);
+ public static final String TIMEOUT_PARAM = "timeout";
+ public static final String MAX_SEGMENTS_PER_BATCH_PARAM =
"maxSegmentsPerBatch";
+ public static final String MAX_RETRIES_PARAM = "maxRetries";
+
+ protected final String name;
+ private final Map params;
+ private final EmbeddingModel embedder;
+ private Integer hashCode;
+
+ public static SolrEmbeddingModel getInstance(
+ String className, String name, Map params) throws
EmbeddingModelException {
+try {
+ EmbeddingModel embedder;
+ Class modelClass = Class.forName(className);
+ var builder = modelClass.getMethod("builder").invoke(null);
+ if (params != null) {
+for (String paramName : params.keySet()) {
+ switch (paramName) {
+case TIMEOUT_PARAM:
+ Duration timeOut = Duration.ofSeconds((Long)
params.get(paramName));
+ builder.getClass().getMethod(paramName,
Duration.class).invoke(builder, timeOut);
+ break;
+case MAX_SEGMENTS_PER_BATCH_PARAM:
+ builder
+ .getClass()
+ .getMethod(paramName, Integer.class)
+ .invoke(builder, ((Long) params.get(paramName)).intValue());
+ break;
+case MAX_RETRIES_PARAM:
+ builder
+ .getClass()
+ .getMethod(paramName, Integer.class)
+ .invoke(builder, ((Long) params.get(paramName)).intValue());
+ break;
+default:
+ ArrayList methods = new ArrayList<>();
+ for (var method : builder.getClass().getMethods()) {
+if (paramName.equals(method.getName()) &&
method.getParameterCount() == 1) {
+ methods.add(method);
+}
+ }
+ if (methods.size() == 1) {
+methods.get(0).invoke(builder, params.get(paramName));
+ } else {
+builder
+.getClass()
+.getMethod(paramName, String.class)
+.invoke(builder, params.get(paramName));
+ }
+ }
+}
+ }
+ embedder = (EmbeddingModel)
builder.getClass().getMethod("build").invoke(builder);
+ return new SolrEmbeddingModel(name, embedder, params);
+} catch (final Exception e) {
+ throw new EmbeddingModelException("Model loading failed for " +
className, e);
+}
+ }
+
+ public SolrEmbeddingModel(String name, EmbeddingModel embedder, Map params) {
+this.name = name;
+this.embedder = embedder;
+this.params = params;
+this.hashCode = calculateHashCode();
+ }
+
+ public float[] vectorise(String text) {
+Embedding vector = embedder.embed(text).content();
+return vector.vector();
+ }
+
+ @Override
+ public String toString() {
+return getClass().getSimpleName() + "(name=" + getName() + ")";
+ }
+
+ @Override
+ public long ramBytesUsed() {
+return BASE_RAM_BYTES
++ RamUsageEstimator.sizeOfObject(name)
++ RamUsageEstimator.sizeOfObject(embedder);
+ }
+
+ @Override
+ public int hashCode() {
+if (hashCode == null) {
+ hashCode = calculateHashCode();
+}
+return hashCode;
+ }
Re: [PR] SOLR-17525: Text Embedder Query Parser [solr]
cpoerschke commented on code in PR #2809: URL: https://github.com/apache/solr/pull/2809#discussion_r1834162601 ## solr/modules/llm/src/java/org/apache/solr/llm/embedding/SolrEmbeddingModel.java: ## @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.solr.llm.embedding; + +import dev.langchain4j.data.embedding.Embedding; +import dev.langchain4j.model.embedding.DimensionAwareEmbeddingModel; Review Comment: ```suggestion ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17525: Text Embedder Query Parser [solr]
cpoerschke commented on code in PR #2809:
URL: https://github.com/apache/solr/pull/2809#discussion_r1834169995
##
solr/modules/llm/src/test/org/apache/solr/llm/embedding/DummyEmbeddingModel.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.llm.embedding;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.data.segment.TextSegment;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.output.Response;
+import java.util.ArrayList;
+import java.util.List;
+
+public class DummyEmbeddingModel implements EmbeddingModel {
+ final float[] embedding;
+
+ public DummyEmbeddingModel(int[] embedding) {
+this.embedding = new float[] {embedding[0], embedding[1], embedding[2],
embedding[3]};
Review Comment:
https://github.com/SeaseLtd/solr/pull/6/commits/e7d9e63651d33256c0dc62ed18f38bbab72b0332
suggests to support float values and dimensions other than 4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-4587: integrate lucene-monitor into solr [solr]
cpoerschke commented on code in PR #2382: URL: https://github.com/apache/solr/pull/2382#discussion_r1834211925 ## solr/modules/monitor/src/java/org/apache/solr/monitor/search/SolrMonitorQueryCollector.java: ## @@ -0,0 +1,106 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one or more + * * contributor license agreements. See the NOTICE file distributed with + * * this work for additional information regarding copyright ownership. + * * The ASF licenses this file to You under the Apache License, Version 2.0 + * * (the "License"); you may not use this file except in compliance with + * * the License. You may obtain a copy of the License at + * * + * * http://www.apache.org/licenses/LICENSE-2.0 + * * + * * Unless required by applicable law or agreed to in writing, software + * * distributed under the License is distributed on an "AS IS" BASIS, + * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * * See the License for the specific language governing permissions and + * * limitations under the License. + * + */ + +package org.apache.solr.monitor.search; + +import java.io.IOException; +import org.apache.lucene.index.LeafReaderContext; +import org.apache.lucene.monitor.QCEVisitor; Review Comment: ```suggestion import org.apache.lucene.monitor.QCEVisitor; import org.apache.lucene.search.ConstantScoreQuery; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-4587: integrate lucene-monitor into solr [solr]
cpoerschke commented on code in PR #2382:
URL: https://github.com/apache/solr/pull/2382#discussion_r1834333404
##
solr/modules/monitor/build.gradle:
##
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply plugin: 'java-library'
+
+description = 'Apache Solr Monitor'
+
+dependencies {
+
+implementation project(":solr:core")
+implementation project(":solr:solrj")
+implementation "org.apache.lucene:lucene-core"
+implementation "org.apache.lucene:lucene-monitor"
+implementation 'com.github.ben-manes.caffeine:caffeine'
Review Comment:
the CI says
```
Execution failed for task ':solr:modules:monitor:analyzeClassesDependencies'.
> Dependency analysis found issues.
usedUndeclaredArtifacts
- io.dropwizard.metrics:metrics-core:4.2.26@jar
```
and a remedy might be (something like)
```suggestion
implementation 'com.github.ben-manes.caffeine:caffeine'
implementation 'io.dropwizard.metrics:metrics-core'
```
however with that and `./gradlew clean ; ./gradlew
:solr:modules:monitor:analyzeClassesDependencies` locally it then instead says
```
Execution failed for task ':solr:modules:monitor:analyzeClassesDependencies'.
> Dependency analysis found issues.
unusedDeclaredArtifacts
- io.dropwizard.metrics:metrics-core:4.2.25@jar
```
instead i.e. the opposite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Resolved] (SOLR-17495) Change CLI DeleteTool deleting of configs to opt in instead of default behavior
[ https://issues.apache.org/jira/browse/SOLR-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Pugh resolved SOLR-17495. -- Fix Version/s: main (10.0) Resolution: Fixed > Change CLI DeleteTool deleting of configs to opt in instead of default > behavior > --- > > Key: SOLR-17495 > URL: https://issues.apache.org/jira/browse/SOLR-17495 > Project: Solr > Issue Type: Sub-task > Components: cli >Affects Versions: main (10.0) >Reporter: Eric Pugh >Assignee: Eric Pugh >Priority: Major > Labels: pull-request-available > Fix For: main (10.0) > > Time Spent: 1h 10m > Remaining Estimate: 0h > > In discussion on SOLR-17488 we identified that the --delete-config option, > which by default is true, is an odd user experience. You need to do a > --delete-config false to not delete a config. > > However, i think deleting a collection and deleting a config are two seperate > conceptual things in reality. > > So in 10, lets make --delete-config something you opt into otherwise configs > are NOT deleted. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17495) Change CLI DeleteTool deleting of configs to opt in instead of default behavior
[ https://issues.apache.org/jira/browse/SOLR-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896660#comment-17896660 ] ASF subversion and git services commented on SOLR-17495: Commit b8cbe41b86a6aac4f93dbbbe972af845a66d in solr's branch refs/heads/main from Eric Pugh [ https://gitbox.apache.org/repos/asf?p=solr.git;h=b8cbe41b86a ] SOLR-17495: Change CLI delete command to not delete configs by default. (#2761) Decouples the lifecycle of a collection and a configset when using bin/solr delete. > Change CLI DeleteTool deleting of configs to opt in instead of default > behavior > --- > > Key: SOLR-17495 > URL: https://issues.apache.org/jira/browse/SOLR-17495 > Project: Solr > Issue Type: Sub-task > Components: cli >Affects Versions: main (10.0) >Reporter: Eric Pugh >Assignee: Eric Pugh >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > In discussion on SOLR-17488 we identified that the --delete-config option, > which by default is true, is an odd user experience. You need to do a > --delete-config false to not delete a config. > > However, i think deleting a collection and deleting a config are two seperate > conceptual things in reality. > > So in 10, lets make --delete-config something you opt into otherwise configs > are NOT deleted. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17495: Change CLI delete command to not delete configs by default. [solr]
epugh merged PR #2761: URL: https://github.com/apache/solr/pull/2761 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Update creation of URL to avoid deprecation message [solr]
epugh commented on PR #2851: URL: https://github.com/apache/solr/pull/2851#issuecomment-2464671567 I tested by checking the build output messages on github running this PR against others, and I no longer see the message. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] Update creation of URL to avoid deprecation message [solr]
epugh merged PR #2851: URL: https://github.com/apache/solr/pull/2851 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17151 Check limits between calls to components in SearchHandler [solr]
sigram commented on code in PR #2801:
URL: https://github.com/apache/solr/pull/2801#discussion_r1834075014
##
solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java:
##
@@ -626,44 +697,97 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
rsp.addToLog(ThreadCpuTimer.CPU_TIME, totalShardCpuTime);
}
}
+ }
-// SOLR-5550: still provide shards.info if requested even for a short
circuited distrib request
-if (!rb.isDistrib
-&& req.getParams().getBool(ShardParams.SHARDS_INFO, false)
-&& rb.shortCircuitedURL != null) {
- NamedList shardInfo = new SimpleOrderedMap<>();
- SimpleOrderedMap nl = new SimpleOrderedMap<>();
- if (rsp.getException() != null) {
-Throwable cause = rsp.getException();
-if (cause instanceof SolrServerException) {
- cause = ((SolrServerException) cause).getRootCause();
-} else {
- if (cause.getCause() != null) {
-cause = cause.getCause();
- }
+ private static boolean prepareComponents(
+ SolrQueryRequest req, ResponseBuilder rb, RTimerTree timer,
List components)
+ throws IOException {
+if (timer == null) {
+ // non-debugging prepare phase
+ for (SearchComponent component : components) {
+if (checkLimitsBefore(component, "prepare", rb.req, rb.rsp,
components)) {
+ shortCircuitedResults(req, rb);
+ return false;
}
-nl.add("error", cause.toString());
-if (!core.getCoreContainer().hideStackTrace()) {
- StringWriter trace = new StringWriter();
- cause.printStackTrace(new PrintWriter(trace));
- nl.add("trace", trace.toString());
+component.prepare(rb);
+ }
+} else {
+ // debugging prepare phase
+ RTimerTree subt = timer.sub("prepare");
+ for (SearchComponent c : components) {
+if (checkLimitsBefore(c, "prepare debug", rb.req, rb.rsp, components))
{
+ shortCircuitedResults(req, rb);
+ return false;
}
- } else if (rb.getResults() != null) {
-nl.add("numFound", rb.getResults().docList.matches());
-nl.add(
-"numFoundExact",
-rb.getResults().docList.hitCountRelation() ==
TotalHits.Relation.EQUAL_TO);
-nl.add("maxScore", rb.getResults().docList.maxScore());
+rb.setTimer(subt.sub(c.getName()));
+c.prepare(rb);
+rb.getTimer().stop();
}
- nl.add("shardAddress", rb.shortCircuitedURL);
- nl.add("time", req.getRequestTimer().getTime()); // elapsed time of this
request so far
+ subt.stop();
+}
+return true;
+ }
- int pos = rb.shortCircuitedURL.indexOf("://");
- String shardInfoName =
- pos != -1 ? rb.shortCircuitedURL.substring(pos + 3) :
rb.shortCircuitedURL;
- shardInfo.add(shardInfoName, nl);
- rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ private static String stageInEnglish(int nextStage) {
+// This should probably be a enum, but that change should be its own
ticket.
+switch (nextStage) {
+ case STAGE_START:
+return "START";
+ case STAGE_PARSE_QUERY:
+return "PARSE_QUERY";
+ case STAGE_TOP_GROUPS:
+return "TOP_GROUPS";
+ case STAGE_EXECUTE_QUERY:
+return "EXECUTE_QUERY";
+ case STAGE_GET_FIELDS:
+return "GET_FIELDS";
+// nobody wants to think it was DONE and canceled after it completed...
+ case STAGE_DONE:
+return "FINISHING";
+ default:
+throw new SolrException(
+SolrException.ErrorCode.SERVER_ERROR, "Unrecognized stage:" +
nextStage);
+}
+ }
+
+ private static void shortCircuitedResults(SolrQueryRequest req,
ResponseBuilder rb) {
+
+if (rb.rsp.getResponse() == null) {
+ rb.rsp.addResponse(new SolrDocumentList());
+
+ // If a cursorMark was passed, and we didn't progress, set
+ // the nextCursorMark to the same position
+ String cursorStr =
rb.req.getParams().get(CursorMarkParams.CURSOR_MARK_PARAM);
+ if (null != cursorStr) {
+rb.rsp.add(CursorMarkParams.CURSOR_MARK_NEXT, cursorStr);
+ }
}
+if (rb.isDebug()) {
+ NamedList debug = new NamedList<>();
+ debug.add("explain", new NamedList<>());
+ rb.rsp.add("debug", debug);
+}
+rb.rsp.setPartialResults(rb.req);
+ }
+
+ private static boolean checkLimitsBefore(
+ SearchComponent c,
+ String when,
+ SolrQueryRequest req,
+ SolrQueryResponse resp,
+ List components) {
+
+return getQueryLimits(req, resp)
+.maybeExitWithPartialResults(
+() ->
+"["
++ when
++ "] Limit(s) exceeded prior to "
++ c.getName()
++ " in "
++ components.stream()
+.map(SearchComponent::
Re: [PR] SOLR-17535: Deprecate ClusterState.forEachCollection [solr]
murblanc commented on code in PR #2854:
URL: https://github.com/apache/solr/pull/2854#discussion_r1834511103
##
solr/solrj/src/java/org/apache/solr/common/cloud/ClusterState.java:
##
@@ -425,7 +425,10 @@ public Stream collectionStream() {
/**
* Calls {@code consumer} with a resolved {@link DocCollection}s for all
collections. Use this
* sparingly in case there are many collections.
+ *
+ * @deprecated see {@link #collectionStream()}
Review Comment:
If this method was introduced a few days ago (can't remember), I think it
can be removed rather than deprecated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-14680) Provide simple interfaces to our concrete SolrCloud classes
[ https://issues.apache.org/jira/browse/SOLR-14680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SOLR-14680: -- Labels: clean-api pull-request-available (was: clean-api) > Provide simple interfaces to our concrete SolrCloud classes > --- > > Key: SOLR-14680 > URL: https://issues.apache.org/jira/browse/SOLR-14680 > Project: Solr > Issue Type: Improvement >Reporter: Noble Paul >Assignee: Noble Paul >Priority: Minor > Labels: clean-api, pull-request-available > Time Spent: 11h 10m > Remaining Estimate: 0h > > All our current implementations of SolrCloud such as > # ClusterState > # DocCollection > # Slice > # Replica > etc are concrete classes. Providing alternate implementations or wrappers is > extremely difficult. > SOLR-14613 is attempting to create such interfaces to make their sdk simpler > The objective is not to have a comprehensive set of methods in these > interfaces. We will start out with a subset of required interfaces. We > guarantee is that signatures of methods in these interfaces will not be > deleted/changed . But we may add more methods as and when it suits us -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
