Re: Remove duplicates in destination of copy field

2023-10-24 Thread Mónica Marrero
Thank you, Hoss. It works well.

Best,

Mónica

On Fri, 20 Oct 2023 at 20:55, Chris Hostetter 
wrote:

>
> copyField -- at a schema level -- is a very low level operation that
> happens at the moment the documents are being added to the index (long
> after he update processor chains are run)
>
> More complicated logic and/around copying values from one field to another
> as part of an update processor chain can be done using the
> CloneFieldUpdateProcessorFactory
>
>
>
> -Hoss
> http://www.lucidworks.com/
>

-- 
Disclaimer: This email and any files transmitted with it are confidential 
and intended solely for the use of the individual or entity to whom they 
are
addressed. If you have received this email in error please notify the 
system manager. If you are not the named addressee you should not 
disseminate,
distribute or copy this email. Please notify the sender 
immediately by email if you have received this email by mistake and delete 
this email from your
system.


issue with f..qf in solr 9.4

2023-10-24 Thread Noah Torp-Smith
Hello,

When I spin up the techproducts example in solr 9.1.1, I am able to send this 
to the /query endpoint and get a reasonable response:

{
"query": "+all:belkin",
"fields": "id compName_s",
"offset": 0,
"limit": 10,
"params": {
"defType": "edismax",
"f.all.qf": "id compName_s address_s"
   }
}

The point is that "all" then specifies a list of fields to look in, "all" is 
just a name, it could be anything.

When I send the same to the /query endpoint in 9.4, I get a message stating 
that the "all" field does not exist. We use the f.<>.qf  construction for a 
variety of things, so it'd be sad for us if that was discontinued.

Is this a bug or intentional?

Thanks,

/Noah



--

Noah Torp-Smith (n...@dbc.dk)


Solr 9.2.1 custom plugin read values in solrconfig.xml

2023-10-24 Thread elisabeth benoit
Hello,

We have a solr custom plugin that we are actually migrating from solr 7.3.1
to solr 9.2.1.

In our solrconfig.xml we have the following line:




The values stopwords and startswith are defined in solrconfig and we read
those values in the newSearcher function of a class implementing
SolrEventListener.

public void newSearcher(SolrIndexSearcher newSearcher,
SolrIndexSearcher currentSearcher) {
SolrConfig config = newSearcher.getCore().getSolrConfig();
String functionName = "//valueSourceParser[@name='" + this.name + "']";
String[] stopWords = config.get(functionName +
"/@stopwords").txt().split(";") ;
StopWordSubtypeLoader.startsWithValue = Integer.parseInt(config.get(
  functionName + "/@startswith").txt());
StopWordSubtypeLoader.stopWords = new
HashSet(Arrays.asList(stopWords));
LOGGER.info("startsWithValue = " + StopWordSubtypeLoader.startsWithValue);
LOGGER.info("stopwords = " + String.join(",",
StopWordSubtypeLoader.stopWords));
}

This was working in solr 7.3.1 but is not working in solr 9.2.1. When I put
a break point in newSearcher function in a unit test, it never gets there,
as if newSearcher is not called.

Did something change? I cant find an exemple of a similar case with solr 9.

Best regards,
Elisabeth


Re: Solr 9.2.1 custom plugin read values in solrconfig.xml

2023-10-24 Thread elisabeth benoit
I forgot to add we have this line in solrconfig

 


Le mar. 24 oct. 2023 à 14:04, elisabeth benoit 
a écrit :

>
> Hello,
>
> We have a solr custom plugin that we are actually migrating from solr
> 7.3.1 to solr 9.2.1.
>
> In our solrconfig.xml we have the following line:
>
>
>  class="com.cie.lbs.solr.search.function.StopWordSubtypeValueSourceParser"
> stopwords="le;la;les;"
> startswith="3"
> />
>
> The values stopwords and startswith are defined in solrconfig and we read
> those values in the newSearcher function of a class implementing
> SolrEventListener.
>
> public void newSearcher(SolrIndexSearcher newSearcher, SolrIndexSearcher 
> currentSearcher) {
> SolrConfig config = newSearcher.getCore().getSolrConfig();
> String functionName = "//valueSourceParser[@name='" + this.name + "']";
> String[] stopWords = config.get(functionName + 
> "/@stopwords").txt().split(";") ;
> StopWordSubtypeLoader.startsWithValue = Integer.parseInt(config.get(
>   functionName + "/@startswith").txt());
> StopWordSubtypeLoader.stopWords = new 
> HashSet(Arrays.asList(stopWords));
> LOGGER.info("startsWithValue = " + StopWordSubtypeLoader.startsWithValue);
> LOGGER.info("stopwords = " + String.join(",", 
> StopWordSubtypeLoader.stopWords));
> }
>
> This was working in solr 7.3.1 but is not working in solr 9.2.1. When I
> put a break point in newSearcher function in a unit test, it never gets
> there, as if newSearcher is not called.
>
> Did something change? I cant find an exemple of a similar case with solr 9.
>
> Best regards,
> Elisabeth
>
>
>
>


Re: TruncateFieldUpdateProcessorFactor isn't being applied

2023-10-24 Thread Walter Underwood
Thanks for confirming. Yes, we’ll use the CloneFieldUpdateProcessor Factory.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Oct 23, 2023, at 11:36 PM, Mikhail Khludnev  wrote:
> 
> Hello Walter.
> I'm afraid the copyField directive is handled quite after update
> processor chain.
> May you try with CloneFieldUpdateProcessorFactory before truncating?
> 
> 
> On Mon, Oct 23, 2023 at 8:29 PM Walter Underwood 
> wrote:
> 
>> I’ve defined TruncateFieldUpdateProcessorFactor in our update request
>> processor chain, targeted at solr.StrField, but it isn’t truncating the
>> content. That field is populated by a copyField directive. Is the copyField
>> evaluated after the update chain? Could that cause this to not work?
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>> 
> 
> -- 
> Sincerely yours
> Mikhail Khludnev



Re: Num Found of docs containing vector fields

2023-10-24 Thread rajani m
Hi Alessandro,

 The range query type does not work, it returns 400 with the response as
"Range Queries are not supported for Dense Vector fields. Please use the
{!knn} query parser to run K nearest neighbors search queries."


On Mon, Oct 23, 2023 at 2:36 AM Alessandro Benedetti 
wrote:

> Hi,
> Using the knn query parser is not going to cut it as it's driven by the
> topK parameter (and you do all distance similarity and stuff).
>
> Normally in Solr you just do: field:[* TO *] to retrieve docs that have at
> least one value in a field.
> I can't try right now, can you check if that's implemented correctly?
>
> Cheers
>
> On Mon, 23 Oct 2023, 01:35 rajani m,  wrote:
>
> > Hi Solr Users,
> >
> >   How do you query Solr to get a count of the number of docs consisting
> of
> > vector fields?  Would you make use of knn qp and populate a query vector
> of
> > the same length with 1's? For example,
> > localhost:8983/solr/v9/select?q={!knn
> > f=vector_field topK=10}[512 dim vector].  Also, this http request with
> knn
> > parser usually returns a "414 Request-URI Too Large", what are your
> > alternatives?
> >
> > Thank you,
> > Rajani
> >
>


Re: Solr 9.2.1 custom plugin read values in solrconfig.xml

2023-10-24 Thread Mikhail Khludnev
I suppose a test may bypass some configs or stages.
I think the default config shipped with distro has a listener configured
and we can see that it works in bootstrap logs.

On Tue, Oct 24, 2023 at 3:03 PM elisabeth benoit 
wrote:

> Hello,
>
> We have a solr custom plugin that we are actually migrating from solr 7.3.1
> to solr 9.2.1.
>
> In our solrconfig.xml we have the following line:
>
>
>  class="com.cie.lbs.solr.search.function.StopWordSubtypeValueSourceParser"
> stopwords="le;la;les;"
> startswith="3"
> />
>
> The values stopwords and startswith are defined in solrconfig and we read
> those values in the newSearcher function of a class implementing
> SolrEventListener.
>
> public void newSearcher(SolrIndexSearcher newSearcher,
> SolrIndexSearcher currentSearcher) {
> SolrConfig config = newSearcher.getCore().getSolrConfig();
> String functionName = "//valueSourceParser[@name='" + this.name +
> "']";
> String[] stopWords = config.get(functionName +
> "/@stopwords").txt().split(";") ;
> StopWordSubtypeLoader.startsWithValue = Integer.parseInt(config.get(
>   functionName + "/@startswith").txt());
> StopWordSubtypeLoader.stopWords = new
> HashSet(Arrays.asList(stopWords));
> LOGGER.info("startsWithValue = " +
> StopWordSubtypeLoader.startsWithValue);
> LOGGER.info("stopwords = " + String.join(",",
> StopWordSubtypeLoader.stopWords));
> }
>
> This was working in solr 7.3.1 but is not working in solr 9.2.1. When I put
> a break point in newSearcher function in a unit test, it never gets there,
> as if newSearcher is not called.
>
> Did something change? I cant find an exemple of a similar case with solr 9.
>
> Best regards,
> Elisabeth
>


-- 
Sincerely yours
Mikhail Khludnev


Re: issue with f..qf in solr 9.4

2023-10-24 Thread Chris Hostetter


(thank you for using an example query that works against the techproducts 
example! .. makes it very easy to reproduce)

At the *qparser* level, what you are doing is still working in 9.4..

hossman@slate:~/lucene/solr [j11] [tags/releases/solr/9.4.0] $ curl -sS 
'http://localhost:8983/solr/techproducts/select?omitHeader=true&fl=id&defType=edismax&q=all:belkin&f.all.qf=compName_s+id+address_s'
{
  "response":{
"numFound":1,
"start":0,
"numFoundExact":true,
"docs":[{
  "id":"belkin"
}]
  }
}

...the error you are getting only seems to happen when using the JSON 
Request API. (as in your email)

Below is the ERROR log w/stacktrace that i get when i try your request 
(FWIW: including solr error log messages in email questions about request 
errors is always a great way to help devs answer your questions) 

The main thing that jumps out at me is that the edismax parser isn't 
involved -- it appears to have decided the LuceneQParser should be used?

Which makes me speculate that something broke in how the "params block" is 
parsed when using the JSON Request API?

Skimming CHANGES.txt looking for mentions of JSON Request API changes led 
me to this...

* SOLR-16916: Use of the JSON Query DSL should ignore the defType parameter
  (Christina Chortaria, Max Kadel, Ryan Laddusaw, Jane Sandberg, David Smiley)

...i'm having a hard time wrapping my head around the jira comments ... 

the CHANGES.txt entry is written like the point of the issue was to 
intentionally 'ignore' defType in the JSON Query API, but the comments in 
the Jira read like using 'defType' was broken in 7.2 and this issue 
"fixed" it so it would work again starting in 9.4? ... the commit itself 
only shows testing what happens if 'defType=edismax' is defined as a 
request handler default.

I'm not really sure what's going on here or what the intent was ... i've 
posted a comment in the jira...

https://issues.apache.org/jira/browse/SOLR-16916





2023-10-24 16:57:07.705 ERROR (qtp1535026957-24) [ x:techproducts 
t:localhost-24] o.a.s.h.RequestHandlerBase Client exception => 
org.apache.solr.common.SolrException: undefined field all
at 
org.apache.solr.schema.IndexSchema.getDynamicFieldType(IndexSchema.java:1478)
org.apache.solr.common.SolrException: undefined field all
at 
org.apache.solr.schema.IndexSchema.getDynamicFieldType(IndexSchema.java:1478) 
~[?:?]
at 
org.apache.solr.schema.IndexSchema$SolrQueryAnalyzer.getWrappedAnalyzer(IndexSchema.java:500)
 
~[?:?]
at 
org.apache.lucene.analysis.DelegatingAnalyzerWrapper$DelegatingReuseStrategy.getReusableComponents(DelegatingAnalyzerWrapper.java:83)
 
~[?:?]
at 
org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:184) ~[?:?]
at 
org.apache.lucene.util.QueryBuilder.createFieldQuery(QueryBuilder.java:256) 
~[?:?]
at 
org.apache.solr.parser.SolrQueryParserBase.newFieldQuery(SolrQueryParserBase.java:527)
 
~[?:?]
at 
org.apache.solr.parser.QueryParser.newFieldQuery(QueryParser.java:68) 
~[?:?]
at 
org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:1140)
 
~[?:?]
at 
org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:856)
 
~[?:?]
at org.apache.solr.parser.QueryParser.Term(QueryParser.java:454) 
~[?:?]
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:293) 
~[?:?]
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:173) 
~[?:?]
at 
org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:143) 
~[?:?]
at 
org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:274) 
~[?:?]
at 
org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:51) ~[?:?]
at org.apache.solr.search.QParser.getQuery(QParser.java:188) 
~[?:?]
at 
org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:172)
 
~[?:?]
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:431)
 
~[?:?]
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:226)
 
~[?:?]
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2901) 
~[?:?]





: Date: Tue, 24 Oct 2023 12:01:20 +
: From: Noah Torp-Smith 
: Reply-To: users@solr.apache.org
: To: "users@solr.apache.org" 
: Subject: issue with f..qf in solr 9.4
: 
: Hello,
: 
: When I spin up the techproducts example in solr 9.1.1, I am able to send this 
to the /query endpoint and get a reasonable response:
: 
: {
: "query": "+all:belkin",
: "fields": "id compName_s",
: "offset": 0,
: "limit": 10,
: "params": {
: "defType": "edismax",
: "f.all.qf": "id compName_s address_s"
:}
: }
: 
: The point is that "all" then specifies a list of fields to look in, "all" is 
just a name, it could be anything.
: 
: When I send the same to the /query endpoint in 9.4, I get a mes

Re: Performance issue in Wildcard Query in Solr 8.9.0

2023-10-24 Thread Mikhail Khludnev
Hello Vishal,
I've done some research earlier https://www.youtube.com/watch?v=FQPKAmh0s_I
but haven't got an elegant solution for this problem.

ReverseWildcard hardly helps here, but just blows up an index size, so drop
it first. NGram Tokenizer blows ups indexes much more.

Then, make an experiment reducing the number of segments, via 
and requesting fewer segments as possible.
The reason behind it is that segmentation repeats almost the same terms in
every segment.

Second, try the following range query [0 TO z] - it should hit many terms
and almost all docs.
It gives you the estimate for a heavy wildcard expansion query. I suppose
the wildcard query will run somewhat about runtime of that range query.
If the range is running slow you can only add hardware and slice more
shards (but it hardly scales linearly).

Another measure, which is worth taking is to limit Solr heap leaving enough
RAM to mmap index files.

On Mon, Oct 23, 2023 at 2:00 PM Vishal Patel 
wrote:

> We are using Solr 8.9.0. We have configured Solr cloud like 2 shards and
> each shard has one replica. We have used 5 zoo keepers for Solr cloud.
>
>  We have created collection name documents and index size of one shard is
> 21GB. Schema fields like here
>  required="true" multiValued="false" omitNorms="true" termVectors="false"
> termPositions="false" termOffsets="false" docValues="true"/>
>  multiValued="false" omitNorms="true" termVectors="false"
> termPositions="false" termOffsets="false" omitTermFreqAndPositions="true"/>
>  positionIncrementGap="100">
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
>
>
> We want to search data which contains test. So, we are making our query
> doc_ref:*test*. I think wildcard query is taking high memory and CPU.
> Sometimes we faced issue that collection goes into recovery mode due to
> usage of wildcard query.
> Fo better performance, We have implemented ReversedWildcardFilterFactory:
> https://risdenk.github.io/2018/10/25/apache-solr-leading-wildcard-queries-reversedwildcardfilterfactory.html
>
> How can we search after the applying ReversedWildcardFilterFactory? We are
> not getting benefits in term of query execution time if we search in same
> manner doc_ref_rev:*test*
>
> Can you please suggest best approach when we want to search wildcard
> string(*test*) when index size is large?
>
> Regards,
>
> Vishal
>
>

-- 
Sincerely yours
Mikhail Khludnev


Re: Performance issue in Wildcard Query in Solr 8.9.0

2023-10-24 Thread Shawn Heisey

On 10/23/23 05:00, Vishal Patel wrote:

We want to search data which contains test. So, we are making our query 
doc_ref:*test*. I think wildcard query is taking high memory and CPU. Sometimes 
we faced issue that collection goes into recovery mode due to usage of wildcard 
query.
Fo better performance, We have implemented ReversedWildcardFilterFactory: 
https://risdenk.github.io/2018/10/25/apache-solr-leading-wildcard-queries-reversedwildcardfilterfactory.html


As Mikhail indicated, ReversedWildcardFilterFactory is not designed to 
help with this.  It is for leading wildcards, and your query has both 
leading and trailing wildcards.


Wildcard queries are particularly resource intensive.

Let's say that doc_ref:*test* matches one million different terms in the 
doc_ref field.  I am not talking about documents, I am talking about terms.


Internally, Solr will do this in two steps:  First it will expand the 
wildcard to retrieve all one million matching terms, and then it will 
execute the query, which will literally contain one million terms.  This 
is going to consume a lot of CPU and memory.


Will "test" be a distinct word in the doc_ref field, or would you also 
need it to match a value of abctestxyz?  If it's a distinctive word, you 
might be better off with a relatively standard analysis chain on a 
fieldType of TextField and no wildcards.


Thanks,
Shawn



How many replicas are queried per shard in one SolrCloud request?

2023-10-24 Thread Doug Turnbull
This may be a silly question, but I can't seem to find an answer. Perhaps
just my google-fu is weak.

If I query a SolrCloud cluster, with debug=true, In the tracking output, I
will see during GET_TOP_IDS a list of N replicas per shard.

shards.url=
http://solr-node-1:8983/solr/my_collectiion_shard1_replica_1234|http://solr-node-1:8983/solr/my_collectiion_shard1_replica_5678

Does this imply ALL of these replicas are queried, and the first response
is aggregated back into the response? Or is EXACTLY ONE replica queried?

AND

To reduce tail latency - ie waiting for the slowest core - is there a way
to control how many replicas per shard are requested?

Thanks
-Doug


Re: How many replicas are queried per shard in one SolrCloud request?

2023-10-24 Thread Doug Turnbull
Maybe this verbiage implies exactly one replica? Though the debug output
seems to imply otherwise?

> When a Solr node receives a search request, the request is routed behind
the scenes to a replica of a shard that is part of the collection being
searched.

https://solr.apache.org/guide/8_4/distributed-requests.html

On Tue, Oct 24, 2023 at 4:44 PM Doug Turnbull 
wrote:

> This may be a silly question, but I can't seem to find an answer. Perhaps
> just my google-fu is weak.
>
> If I query a SolrCloud cluster, with debug=true, In the tracking output, I
> will see during GET_TOP_IDS a list of N replicas per shard.
>
> shards.url=
> http://solr-node-1:8983/solr/my_collectiion_shard1_replica_1234|http://solr-node-1:8983/solr/my_collectiion_shard1_replica_5678
>
> Does this imply ALL of these replicas are queried, and the first response
> is aggregated back into the response? Or is EXACTLY ONE replica queried?
>
> AND
>
> To reduce tail latency - ie waiting for the slowest core - is there a way
> to control how many replicas per shard are requested?
>
> Thanks
> -Doug
>


Re: How many replicas are queried per shard in one SolrCloud request?

2023-10-24 Thread Chris Hostetter

: Maybe this verbiage implies exactly one replica? Though the debug output
: seems to imply otherwise?
: 
: > When a Solr node receives a search request, the request is routed behind
: the scenes to a replica of a shard that is part of the collection being
: searched.
: 
: https://solr.apache.org/guide/8_4/distributed-requests.html

Only one replica (per shard) gets any given "stage" request.

what you're seeing lower on that page is examples of the older (pre-cloud) 
syntax for requesting a request be routed to multiple shards -- using a 
COMMA (',') seperated list of "shards" and the ability 
ask solr to pick a random replica of each shard by futher specifying a 
PIPE ('|') delimiated list of URLs.

basically the idea was in a mutlti shard, multi-replica request, you could 
either send the request to a specific list of shards (',') where you had 
already picked which replica URL it shoudl send to for each shard, or you 
could have solr pick the replicas for you by sending a list ('|') of 
replica urls.

...but that's with the 'shards' param indented for users...

The structure of those "lists of lists" was/is used in a various debugging 
formats and made it's way into the 'shard.url' param that you are asking 
about regarding requests to individual replicas -- which is relic of a 
bygone era that existed to support the ShardAugmenterFactory (aka: 
fl=[shard])

Full historical analysis in the jira where we removed it in 9.3...

https://issues.apache.org/jira/browse/SOLR-9378


: 
: On Tue, Oct 24, 2023 at 4:44 PM Doug Turnbull 
: wrote:
: 
: > This may be a silly question, but I can't seem to find an answer. Perhaps
: > just my google-fu is weak.
: >
: > If I query a SolrCloud cluster, with debug=true, In the tracking output, I
: > will see during GET_TOP_IDS a list of N replicas per shard.
: >
: > shards.url=
: > 
http://solr-node-1:8983/solr/my_collectiion_shard1_replica_1234|http://solr-node-1:8983/solr/my_collectiion_shard1_replica_5678
: >
: > Does this imply ALL of these replicas are queried, and the first response
: > is aggregated back into the response? Or is EXACTLY ONE replica queried?
: >
: > AND
: >
: > To reduce tail latency - ie waiting for the slowest core - is there a way
: > to control how many replicas per shard are requested?
: >
: > Thanks
: > -Doug
: >
: 

-Hoss
http://www.lucidworks.com/

Re: Solr Heap usage

2023-10-24 Thread rajani m
When you see memory filling up on the visualvm gui, request a heap dump.
The heap dump file gets stored on the node's /tmp location. Use the eclipse
memory analyzer tool (MAT) to load the downloaded ".dmp" file, the MAT
analyzes the heap and generates a report of objects on the heap.

On Mon, Oct 23, 2023 at 1:48 PM Surya R  wrote:

> Hi. We have a solr index with around 2.5M documents , running solr 8.11 .
> We have bunch of fields that we faceet/sort/highlight on. Heap is 8GB, out
> of 12 GB RAM
>
> These fields were not defined with docValues=true due to some legacy design
> issues , but if we were to rewrite the app, we would definitely have the
> docvalues enabled for those fields that participate in
> facet/sort/highlight.
>
> Problem:
> I do see garbage collection ( G1GC) kicking in when the traffic is moderate
> to high, with a  "pause full" event that takes around 2seconds to complete.
> This event happens more often when the traffic is high.
>
> I see this GC event happening when the humongous regions go beyond 1000.
> The humongous region size is 4M
>
> I took a heap dump and analyzed with VisualVM, but I couldnt pinpoint what
> objects are using the 'humougous region' .
>
> My question is, using the VisualVM, is it possible to pinpoint which / what
> objects are using the 'humongous region' ?
>
> If for example I have an solr indexed field called  "answerSnippet" , if
> that is the culprit, Is it possible to see the field answerSnippet in the
> VisualVM heap dump output?
>
> All i see is byte[] with 55% heap occupancy, but dont know what objects
> constitute that byte[]
>
> Can anybody throw some light on this ? I really appreciate the help.
>
>
> thanks
> -- Surya
>


Re: Solr Heap usage

2023-10-24 Thread Arrieta, Alejandro
Hello,

this link may help you, it has examples:
https://risdenk.github.io/2017/12/18/ambari-infra-solr-ranger.html

Kind Regards,
Alejandro Arrieta

On Tue, Oct 24, 2023 at 9:18 PM rajani m  wrote:

> When you see memory filling up on the visualvm gui, request a heap dump.
> The heap dump file gets stored on the node's /tmp location. Use the eclipse
> memory analyzer tool (MAT) to load the downloaded ".dmp" file, the MAT
> analyzes the heap and generates a report of objects on the heap.
>
> On Mon, Oct 23, 2023 at 1:48 PM Surya R  wrote:
>
> > Hi. We have a solr index with around 2.5M documents , running solr 8.11 .
> > We have bunch of fields that we faceet/sort/highlight on. Heap is 8GB,
> out
> > of 12 GB RAM
> >
> > These fields were not defined with docValues=true due to some legacy
> design
> > issues , but if we were to rewrite the app, we would definitely have the
> > docvalues enabled for those fields that participate in
> > facet/sort/highlight.
> >
> > Problem:
> > I do see garbage collection ( G1GC) kicking in when the traffic is
> moderate
> > to high, with a  "pause full" event that takes around 2seconds to
> complete.
> > This event happens more often when the traffic is high.
> >
> > I see this GC event happening when the humongous regions go beyond 1000.
> > The humongous region size is 4M
> >
> > I took a heap dump and analyzed with VisualVM, but I couldnt pinpoint
> what
> > objects are using the 'humougous region' .
> >
> > My question is, using the VisualVM, is it possible to pinpoint which /
> what
> > objects are using the 'humongous region' ?
> >
> > If for example I have an solr indexed field called  "answerSnippet" , if
> > that is the culprit, Is it possible to see the field answerSnippet in the
> > VisualVM heap dump output?
> >
> > All i see is byte[] with 55% heap occupancy, but dont know what objects
> > constitute that byte[]
> >
> > Can anybody throw some light on this ? I really appreciate the help.
> >
> >
> > thanks
> > -- Surya
> >
>


Re: SOLR 8.11.2 Cloud issue

2023-10-24 Thread Ishan Chattopadhyaya
I tried to reproduce this issue on 8.11.3 (unreleased) and it seems to work
well: https://imgur.com/a/lTO4sim

On Mon, 23 Oct 2023 at 12:51, Scott Q.  wrote:

> Oops, here we go:
>
> https://imgur.com/a/jSVKgkZ
>
> On Monday, 23/10/2023 at 02:24 Ishan Chattopadhyaya wrote:
>
>
>
> Unfortunately, the mailing list doesn't allow images to be attached. I
> think it is time to retire this mailing list, move to a modern
> discussion
> forum. Thankfully, some community members are taking a lead on that
> front.
>
> On Mon, 23 Oct, 2023, 11:47 am Scott Q.,  wrote:
>
> > Hmm...yes.
> >
> > Attaching image - this is when I access the ~cloud URL via direct
> link.
> > But notice there's no Cloud menu on the left side...
> >
> >
> > If I access the direct links
> > On Sunday, 22/10/2023 at 21:38 Ishan Chattopadhyaya wrote:
> >
> > Are you sure you're running Solr in SolrCloud mode?
> >
> > On Sun, 22 Oct, 2023, 3:36 pm Scott Q.,  wrote:
> >
> > > I have SOLR 8.11.2 running with 5 nodes in Cloud
> > > configuration and it works great. In fact it was an upgrade from
> an
> > > earlier version that's been stable for a few years.
> > >
> > >
> > > Since upgrading to 8.11.2 the WEB UI no longer displays the Cloud
> menu
> > > on the left. I can still access it via direct URL because I know
> the
> > > links and I can see the graph & nodes. But the nodes for example
> don't
> > > show any status such as uptime, used space, etc. I don't see any
> > > errors in the logs.
> > >
> > >
> > > In the browser console I see it tries to access two URLs which
> always
> > > fail and show up in red:
> > >
> > >
> > >
> http://192.168.1.10:8983/solr/admin/info/system?_=1697493975084&wt=json
> > >
> > >
> > >
> > > and this one:
> > >
> > >
> > >
> > >
> >
>
> http://192.168.1.10:8983/solr/admin/info/system?_=1697493975084&nodes=192.168.1.10:8983_solr,192.168.1.11:8983_solr,192.168.1.13:8983_solr,192.168.1.4:8983_solr,192.168.1.9:8983_solr&wt=json
> > >
> > >
> > >
> > > with payload:
> > >
> > >
> > >
> > > *
> > > _:
> > > 1697493975084
> > > *
> > > nodes:
> > > 192.168.1.10:8983_solr,192.168.1.11:8983_solr,192.168.1.13:8983
> > > _solr,192.168.1.4:8983_solr,192.168.1.9:8983_solr
> > > *
> > > wt:
> > > json
> > >
> > >
> > >
> > > Again, the system works fine but it's the Cloud menu that's wonky.
> > >
> > >
> > > Is this a known bug or does anyone know what else might be
> happening ?
> > >
> > >
> > > Thanks!
> > >
> >
> >
>