Intercept atomic indexing when it indexes other fields from their stored values

2021-04-08 Thread gnandre
Hi, IIRC the way atomic indexing works under the hood is by indexing stored values of all fields again. Is there a way to intercept that and change the value of some field that I want?

weightExpression operator support

2021-04-19 Thread gnandre
Which operators are supported in weightExpression field for Solr suggester. e.g. if I want to use power function, should I use pow? First I thought, it uses function queries. But it does not seem like that. Following example shows that it is using the ln function. https://solr.apache.org/guide/8_

Invalid reference for suggester

2021-04-20 Thread gnandre
I am using solr 8.5.2. I am trying to configure multiple suggester definition in Solr search component and got error information like: { "responseHeader":{ "status":500, "QTime":1718343}, "error":{ "msg":"Invalid reference 'link_score'", "trace":"java.lang.IllegalArgumentExcept

WordDelimiter does not generate expected token

2021-04-21 Thread gnandre
Hi, I have a field value as bim.ClassUnderlying and a search query as classunderlying does not return any results. If I search for classUnderlying, it works.What can I change so that it works for classunderlying query too? If I change splitOnCaseChange value from 1 to 0 in index time analyzer chai

SecureRandom algorithm 'NativePRNG' is in use

2021-04-28 Thread gnandre
Hi, I intermittently face this issue sometimes while running the unit tests. SecureRandom algorithm 'NativePRNG' is in use by your JVM, which is a potentially blocking algorithm on some environments. Please report the details of this failure (and your JVM vendor/version) to solr-u...@lucene.apach

Re: SecureRandom algorithm 'NativePRNG' is in use

2021-04-28 Thread gnandre
One more thing, -Dtest.solr.allowed.securerandom=NativePRNG doesn't seem to help and I haven't tried the other option yet. On Wed, Apr 28, 2021 at 8:41 PM gnandre wrote: > Hi, > > I intermittently face this issue sometimes while running the unit tests. > > SecureRan

Issue with expand feature in collapse/expand result set workflow

2021-05-20 Thread gnandre
Hi, I am using Solr 8.5.2 (non-SolrCloud) mode. It is a non-sharded scenario. I am trying to collapse the resultset based on one of the string fields (say X). Collapse query works as expected. It collapses the result set by excluding the documents that have the same field X and retains only the to

Re: Issue with expand feature in collapse/expand result set workflow

2021-05-20 Thread gnandre
> Did you try expand.rows parameter for how many items to include in each > group? > > https://solr.apache.org/guide/8_7/collapse-and-expand-results.html > > > > Regards, > >Alex. > > > > On Thu, 20 May 2021 at 13:02, gnandre wrote: > > > > > >

Invalid query error when using rq param with a shards query

2021-06-09 Thread gnandre
Hi, I am using Solr 8.5.2 with the LTR feature enabled. I have successfully uploaded a trained model to Solr. When I add rq={!ltr model=model efi.query=$q reRankDocs=100} param to my regular search query, it works. But when I pass a shards param with a list of shards then I run into the following

Re: Invalid query error when using rq param with a shards query

2021-06-09 Thread gnandre
Please ignore, this was an issue with the incorrect values I was passing to shards param. On Wed, Jun 9, 2021 at 12:52 PM gnandre wrote: > Hi, > > I am using Solr 8.5.2 with the LTR feature enabled. I have successfully > uploaded a trained model to Solr. When I add rq={!ltr

Re: Is the Solr-User Nabble Forum dead?

2021-08-06 Thread gnandre
Usually, I would end my solr related search query with keyword nabble to see lot of useful results from this mailing list in Google. What alternative do you suggest now for similar search workflow through Google. On Thu, Jul 1, 2021, 8:22 AM Jan Høydahl wrote: > Nabble is not managed by Apache.

Suggester index replication

2022-02-15 Thread gnandre
Is there a way to get suggester index replicated to all search nodes from index node? Do I need to build suggester index for each search node separately?

How to sort Solr search results by hour/day/week/month/year views

2022-03-28 Thread gnandre
I have a search interface that searches for videos. There is a filter there that can sort the results based on views for last hour, last day, last week, last month and last year. Is there a way to achieve this sorting without always needing to update the index? E.g. After each specific duration exp

Re: How to sort Solr search results by hour/day/week/month/year views

2022-03-30 Thread gnandre
> > > https://solr.apache.org/guide/8_11/working-with-external-files-and-processes.html > > > > > > wunder > > > Walter Underwood > > > wun...@wunderwood.org > > > http://observer.wunderwood.org/ (my blog) > > > > > > > On

Atomic indexing without whole document getting indexed again

2022-03-30 Thread gnandre
IIRC, under the hood, atomic indexing indexes the whole document again even if you might be updating just one field of that document. This costs hugely in terms of indexing performance because the other fields might be requiring some significant heavy tokenization. Is there any way around this?

Re: Atomic indexing without whole document getting indexed again

2022-03-31 Thread gnandre
hich is not going to be > fast. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > On Mar 30, 2022, at 9:34 AM, Shawn Heisey wrote: > > > > On 3/30/22 10:27, gnandre wrote: > >> IIRC, under the ho

Re: Atomic indexing without whole document getting indexed again

2022-03-31 Thread gnandre
-place updates still not working? On Thu, Mar 31, 2022 at 12:33 PM gnandre wrote: > Thanks, this is what I was looking for. Although, when I am experimenting > with them now, I see no performance improvement. I suspect that it is still > doing atomic updates and not in-place updates. > H

Re: Atomic indexing without whole document getting indexed again

2022-04-05 Thread gnandre
ass","org.apache.solr.common.SolrException"], "msg":"ERROR: [doc=answers:question:8029] unknown field 'asset_type'", "code":400}} So, I believe that it is still trying to index other fields as well from their stored values and it is not a

Re: Atomic indexing without whole document getting indexed again

2022-04-05 Thread gnandre
:53, gnandre wrote: > > Hi, here are the relevant fields from the schema. > > > > > > docValues > > ="true" multiValued="false" /> > > > docValues="true" multiValued="false"/> > > > > There are n

Re: Atomic indexing without whole document getting indexed again

2022-04-05 Thread gnandre
our Solr > version did not yet have the fix for SOLR-13081 > <https://issues.apache.org/jira/browse/SOLR-13081>. > > Matthew > > On Tue, Apr 5, 2022 at 2:39 PM gnandre wrote: > > > Thanks, Shawn. > > > > I conducted the test that you mentioned. > >

Re: Atomic indexing without whole document getting indexed again

2022-04-05 Thread gnandre
w > > On Tue, Apr 5, 2022 at 3:43 PM gnandre wrote: > > > Thanks, Matthew. > > > > I tried debugging as you suggested. It seems that it is still doing > atomic > > update instead of in-place update. > > I am not using SolrCloud, so I don't think

Does Solr not support parallel suggester index build requests?

2022-05-02 Thread gnandre
To illustrate the issue I am running into: In the following bash program, I execute 2 suggester index building requests simultaneously with 1 process assigned to each call for a same Solr node: for i in {0..1} do curl " http://solr:8983/solr/books/books_suggest_en?wt=json&q=DUMMY_SUGGESTER_TERM&

Delete API for suggester index

2022-05-07 Thread gnandre
Documentation lists only building and reloading parameters for suggest request handler. How to delete suggester index through API?

AddUpdateCommand isInPlaceUpdate function returns false even if incoming request in an in-place update request

2022-06-15 Thread gnandre
Hi, I am using Solr 8.5.2 I made a simple in-place update request to Solr. This request has two fields, one is the unique field and the other one is a field that returns true for AtomicUpdateDocumentMerger.isSupportedFieldForInPlaceUpdate. When I intercept this request during the debugging proce

mod function applied to ms function not working correctly

2022-08-31 Thread gnandre
mod function is not returning correct values when applied to ms(NOW) or _version_ fields. "_version_":1697770046865014784, "ms(NOW)":1661979881038, "mod(ms(NOW),10)": 0.0, "mod(_version_,10)":6.0 It should return 8 and 4 respectively. Am I missing something?

Re: mod function applied to ms function not working correctly

2022-08-31 Thread gnandre
t; Mike > > On Wed, Aug 31, 2022 at 4:07 PM gnandre wrote: > > > mod function is not returning correct values when applied to ms(NOW) or > > _version_ fields. > > > > "_version_":1697770046865014784, "ms(NOW)":1661979881038, > > "m

Atomic indexing as default indexing

2022-09-23 Thread gnandre
Is there a way to make atomic indexing default? Say, even if some clients send non-atomic indexing requests, it should get converted to atomic indexing requests on Solr end, is that possible? I am asking because we usually run into the following issue: 1. Client A is the major contributor of almo

Understanding LTR debug query output

2022-10-07 Thread gnandre
Hi, I have implemented LTR (LambdaRank) functionality but there are some search cases where the relevancy is actually getting worse. I am trying to understand why some results are ranked over the others. Naturally, I am using a debug query to understand what is going on. e.g. here is the explain

How to write custom Solr plugin that can be called core container level instead of core level

2022-11-13 Thread gnandre
e.g. /solr/admin/cores request works without specifying specific core in the request. I took a look at the code of such request handlers (e.g. LoggingHandler) but they simply extend to RequestHandlerBase just like any other core-level request handler. What makes them special so that they do not re

waitForLoadingCoresToFinish does not wait for loading cores to finish

2022-11-13 Thread gnandre
Hi, I am writing a custom Solr plugin and I need to add a check to make sure that all the cores in the node are loaded before proceeding with later code execution. Following is what the code looks like. What I am observing though is that the later code gets executed even if all the cores are not

Re: How to write custom Solr plugin that can be called core container level instead of core level

2022-11-13 Thread gnandre
Thanks, Shawn. I think what you propose there will be very helpful. There are definitely usecases where we want to work at the solr node level and not core level. Fieldcache is one other example. In my case, I am trying to write a simple health check request handler that makes sure that all cores

Re: How to write custom Solr plugin that can be called core container level instead of core level

2022-11-15 Thread gnandre
question to consider when choosing is whether you want the end point > to > >> be protected by authentication. If so then you're going to want to do it > >> Shawn's way. If you want something publicly visible (say to > infrastructure > >> systems without log

Re: How to write custom Solr plugin that can be called core container level instead of core level

2022-11-15 Thread gnandre
Also, is this supported only in SolrCloud mode? On Tue, Nov 15, 2022 at 6:42 PM gnandre wrote: > Thanks, Ishan. Is this available only in Solr 8.6+? I am using 8.5.0 :( > > On Tue, Nov 15, 2022 at 2:19 PM Ishan Chattopadhyaya < > ichattopadhy...@gmail.com> wrote: > >>

How to make /api/cores or /solr/admin/cores wait for all cores to load and then only return?

2022-11-16 Thread gnandre
/api/cores or /solr/admin/cores return very useful information about the current status of all the cores for the solr instance against which they are invoked. However, if Solr instance has just rebooted then these calls come back with empty responses like below or contain only the cores loaded so

Re: How to make /api/cores or /solr/admin/cores wait for all cores to load and then only return?

2022-11-17 Thread gnandre
veloper > http://www.linkedin.com/in/davidwsmiley > > > On Wed, Nov 16, 2022 at 2:02 PM gnandre wrote: > > > /api/cores or /solr/admin/cores return very useful information about the > > current status of all the cores for the solr instance against which they > > are in

Re: How to write custom Solr plugin that can be called core container level instead of core level

2022-11-17 Thread gnandre
6/11/2022 11:52, Jan Høydahl wrote: > > Yes > > > > Seehttps:// > solr.apache.org/guide/solr/latest/configuration-guide/cluster-plugins.html > for the docs. > > > > Jan > > > >> 16. nov. 2022 kl. 00:43 skrev gnandre: > >> > &g

Re: How to write custom Solr plugin that can be called core container level instead of core level

2022-11-19 Thread gnandre
00 AM Shawn Heisey wrote: > On 11/17/22 22:05, gnandre wrote: > > Unfortunately, I am still on legacy mode and not cloud mode. > > So, I can't take advantage of this feature :( > > > > In the meantime, I have implemented the Solr plugin and it is working as > > e

Re: How to write custom Solr plugin that can be called core container level instead of core level

2022-11-20 Thread gnandre
Thanks, Shawn. This is very helpful. I will use this solution! On Sun, Nov 20, 2022 at 12:56 PM Shawn Heisey wrote: > On 11/19/22 21:38, gnandre wrote: > > Thanks, Shawn. I am a bit wary of creating a total core just for this > > health check purpose. > > Isn't it a

Solr is restarting automatically

2022-11-21 Thread gnandre
Hi, I am using Solr 8.5.2 in legacy mode (non-cloud). Some of the Solr nodes are automatically getting restarted after a few days. There is no clear pattern to the rebooting time. Also, no pattern in number of incoming queries or nature of those queries. No particular pattern in errors found in S

Re: Solr is restarting automatically

2022-11-22 Thread gnandre
it just feels like we are not getting to the root of the problem and it might return again at some point. On Tue, Nov 22, 2022 at 8:04 AM matthew sporleder wrote: > On Mon, Nov 21, 2022 at 7:02 PM Shawn Heisey wrote: > > > > On 11/21/22 15:01, gnandre wrote: > > > I am us

FileBasedSpellChecker always returns "correctlySpelled": false

2023-01-11 Thread gnandre
Hi, I am using Solr 8.5.2 and when I use FileBasedSpellChecker, I always get "correctlySpelled": false even if the spellcheck.q value exists in the file. Apart from that it is working as expected. When I give slightly misspelled term to spellcheck.q param, it returns the valid suggestion present

Any way around this atomic update scenario?

2023-01-24 Thread gnandre
Say I have 2 fields. x and y. Following are their schema definitions: Also, there is this copy field, that copies the values from y to x. If I make the following indexing request (non-atomic regular indexing request) { "id": "doc", "x": ["a", "b"], "y": ["c", "d"] } then I end u

Re: Suggester index replication

2023-03-02 Thread gnandre
Can anybody please answer this? Many thanks in advance! On Wed, Feb 16, 2022 at 12:52 AM gnandre wrote: > Is there a way to get suggester index replicated to all search nodes from > index node? Do I need to build suggester index for each search node > separately? >

Donating to Solr

2023-03-02 Thread gnandre
I find this open source project very useful. Is there any way to donate money for it?

Re: Donating to Solr

2023-03-02 Thread gnandre
Thanks! On Thu, Mar 2, 2023, 1:02 PM Doug Turnbull wrote: > Not sure about Solr, but you can donate to the Apache Software Foundation: > > https://www.apache.org/foundation/contributing.html > > On Thu, Mar 2, 2023 at 12:04 PM gnandre wrote: > > > I find this open so

Re: Suggester index replication

2023-03-02 Thread gnandre
of the JSON. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > On Mar 2, 2023, at 9:03 AM, gnandre wrote: > > > > Can anybody please answer this? Many thanks in advance! > > > > On Wed, Feb 16, 2022

Re: Donating to Solr

2023-03-20 Thread gnandre
roject for coordination of how > to > > channel solr-labeled donations. > > > > Jan > > > > > 2. mar. 2023 kl. 19:36 skrev gnandre : > > > > > > Thanks! > > > > > > On Thu, Mar 2, 2023, 1:02 PM Doug Turnbull > > > wrot

How to specify multiples values as default for a multivalued field?

2023-03-20 Thread gnandre
This seems very trivial but it is not working for me and I am not able to figure out why. If I have multivalued field like below, When I index a document, instead of creating an array of strings, it creates just a string like "en,jp"? How can I define the default values such that they show up a

Get a list of all search nodes from index nodes for non-cloud mode

2023-03-28 Thread gnandre
Hi, Is it possible to get a list of all the search instances that are polling a particular index node in non-cloud mode?

Re: Get a list of all search nodes from index nodes for non-cloud mode

2023-03-28 Thread gnandre
I tried to use a call like below but it does not help in getting the above information. http://solr:8983/solr/collection1/replication?command=filecontent&file=replication.properties On Tue, Mar 28, 2023 at 2:58 PM gnandre wrote: > Hi, > > Is it possible to get a list of a

Not able to create custom Solr docker image on top of the base image (Solr 9.2.1)

2023-06-01 Thread gnandre
Hi, I am running into the following issue while creating a custom docker image on top of the official Solr docker image (9.2.1). The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key Because of t

Re: Not able to create custom Solr docker image on top of the base image (Solr 9.2.1)

2023-06-01 Thread gnandre
e public key is not available: NO_PUBKEY 871920D1991BC93C E: The repository 'http://archive.ubuntu.com/ubuntu jammy-backports InRelease' is not signed. E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var

Re: Not able to create custom Solr docker image on top of the base image (Solr 9.2.1)

2023-06-02 Thread gnandre
Thanks for all the replies. FWIW, if I just replace FROM solr:9.2.1 with FROM solr:8.5.2, I do not run into these issues. 8.5.2 is our current prod version. That is why I was thinking maybe there is something in the base image that is causing this. Few things I will try next are: 1. Try to build

Re: Not able to create custom Solr docker image on top of the base image (Solr 9.2.1)

2023-06-02 Thread gnandre
Ok, I tried direct docker build and am still running into the same issue. 01:58:26 |base|gnandre@ah-gnandre-l docker ±|solr_9-2-1 ✗|→ docker build -t test/test:9.2.1-SNAPSHOT -f Dockerfile . [+] Building 33.1s (6/23) => [internal] load build definition from Dockerf

Re: Not able to create custom Solr docker image on top of the base image (Solr 9.2.1)

2023-06-03 Thread gnandre
Thanks, Chris. I tried this workflow with Debian 11 and I can confirm that I am no longer running into this issue anymore On Fri, Jun 2, 2023 at 6:36 PM Chris Hostetter wrote: > > : FWIW, if I just replace FROM solr:9.2.1 with FROM solr:8.5.2, I do not > run > : into these issues. > : 8.5.2 is o

java.lang.NullPointerException: stopWords - solr 9.2.1

2023-06-12 Thread gnandre
Hi, I am using Solr 9.2.1 (official docker image). When I try to index a document, I get the error shown at the bottom of this email. Here is my corresponding filter setting

Re: java.lang.NullPointerException: stopWords - solr 9.2.1

2023-06-12 Thread gnandre
n into \# is it no \# not of on \# or s such t that the their then there these they this to was will with On Mon, Jun 12, 2023 at 11:06 PM gnandre wrote: > Hi, > > I am using Solr 9.2.1 (official docker image). > > When I try to index a document, I get the error shown at th

knn parser not working as expected

2023-07-18 Thread gnandre
Hi, I am trying to use the knn parser in Solr 9.2.1 as follows: curl -X POST -H "Content-Type: application/json" -d '{ "params": { "q": "{!knn f=dense_vector topK=1}[0.06525743007659912,0.015727980062365532,0.003069591475650668,-0.016254400834441185,0.003478930564597249,-0.02475954219698906

Re: knn parser not working as expected

2023-07-20 Thread gnandre
e" } }' http://localhost:8983/solr/doc/select On Wed, Jul 19, 2023 at 3:25 AM Matthias Krüger < mkrue...@opensourceconnections.com> wrote: > The query format looks correct. You can add "debug": true to the posted > JSON to verify whether your suspicion about Solr n

Re: knn parser not working as expected

2023-07-27 Thread gnandre
But the q parameter is still not working. I am stumped. On Fri, Jul 21, 2023 at 1:43 AM gnandre wrote: > Thanks. If I move the knn parser syntax and value to fq param and make q > as *:*, it works and starts giving relevant results instantly. > > curl -X POST -H "Content-Type:

Re: knn parser not working as expected

2023-07-31 Thread gnandre
Any pointers where should I look to resolve this issue? Thanks! On Thu, Jul 27, 2023 at 9:25 PM gnandre wrote: > But the q parameter is still not working. I am stumped. > > On Fri, Jul 21, 2023 at 1:43 AM gnandre wrote: > >> Thanks. If I move the knn parser syntax and val

Re: knn parser not working as expected

2023-08-04 Thread gnandre
3,0.06110243499279022,-0.04396197944879532,0.021046791225671768,-0.04995771124958992,0.09614221751689911,-0.03452020138502121,-0.07397083938121796]"], "parsed_filter_queries":["KnnVectorQuery(KnnVectorQuery:dense_vector[0.06525743,...][1])"]}} On Fri, Aug 4, 2023 at 12:

Re: knn parser not working as expected

2023-08-08 Thread gnandre
> Joel Bernstein > > http://joelsolr.blogspot.com/ > > > > > > On Fri, Aug 4, 2023 at 12:09 PM gnandre wrote: > > > >> Also, here is the debug output for that workaround with fq I mentioned. > >> This debug output is not big. &g

Solr custom query component does not return correct facet counts

2021-03-10 Thread gnandre
I have a simple Solr query component that does some exact match processing by replacing qf and pf params in incoming search requests with new values that point to the fields that do not do stemming, synonymization etc. This works as expected. However in a distributed context (not using SolrCloud,

Conflict between atomic update and highlighting constraints

2021-03-12 Thread gnandre
Hi, I am running into a conflict between two constraints. Atomic updates require copy-field destinations to be stored=false. However, if we want to use these copy-field destination fields in highlighting then they need to be stored=true. How to resolve this conflict?

Re: Solr custom query component does not return correct facet counts

2021-03-12 Thread gnandre
I resolved this issue by extending the class to SearchComponent instead of QueryComponent. It seems that SearchComponent sits at a higher level of abstraction than QueryComponent and is useful when you want to work on a layer above shards. On Wed, Mar 10, 2021 at 8:05 PM gnandre wrote: > I h

Re: Conflict between atomic update and highlighting constraints

2021-03-19 Thread gnandre
inkedin.com/in/davidwsmiley > > > On Fri, Mar 12, 2021 at 12:32 PM gnandre wrote: > > > Hi, > > > > I am running into a conflict between two constraints. > > > > Atomic updates require copy-field destinations to be stored=false. > However, > > if we w

Solr complains about unknown field during atomic indexing

2021-03-19 Thread gnandre
While performing atomic indexing, I run into an error which says 'unknown field X' where X is not a field specified in the schema. It is a discontinued field. After deleting that field from the schema, I have restarted Solr but I have not re-indexed the content back, so the deleted field data stil