Save a new document only when certain fields change

[ufuk yılmaz]

I had a requirement where I needed to index a new document only when there’s a 
change in some of the fields. I implemented it using Solr’s deduplication 
feature : https://solr.apache.org/guide/8_4/de-duplication.html

An example document:
{
  userID: “userid”,
  userName: “usernameA”,
  userLoginCount: 123,
  date: “2020-03-03T13:41:01.104Z”
}

These documents are sent to Solr regularly to be indexed. Requirement was to 
index a new document if user changes his username, otherwise update 
“userLoginCount” and “date” fields. My configuration is:

 
   
 true
 id
 true
 userID,userName
 solr.processor.Lookup3Signature
   
   
   
 

The following document triggers a new index:
{
  userID: “userid”,
  userName: “usernameB”,
  userLoginCount: 128,
  date: “2020-04-03T13:41:01.104Z”
}

This is working nicely with only one problem. If a user changes his username 
from usernameA to usernameB, and later to usernameA again, the older document 
is updated instead. What I was trying to capture was a user’s username changes 
throughout the time, preserving older states, so when this happens there should 
be 3 documents.

Is there a way to achieve this in Solr, or should I find a solution outside 
Solr?

--ufuk yilmaz

Sent from Mail for Windows 10

Re: Solr 8.6 - No live SolrServers available to handle this request when using simple group

[Joel Bernstein]

This certainly looks like a bug. I'm wondering if it's related to the
sortable text field used for the group field, possibly it's not handling
nulls in the same way a string field does. One thing to test is if you're
having problems grouping on a string field without nulls, and then a string
field with nulls.

Joel Bernstein
http://joelsolr.blogspot.com/


On Sat, Apr 17, 2021 at 3:51 AM Nate R  wrote:

> Here is my stacktrace. Looks to be something with the field type. I
> tried with other type fields - Int and ngram text fields. Same error
>
>
> java.lang.NullPointerException
> at org.apache.solr.schema.FieldType.toExternal(FieldType.java:361)
> at
> org.apache.solr.search.grouping.distributed.shardresultserializer.TopGroupsResultTransformer.serializeTopGroups(TopGroupsResultTransformer.java:210)
> at
> org.apache.solr.search.grouping.distributed.shardresultserializer.TopGroupsResultTransformer.transform(TopGroupsResultTransformer.java:77)
> at
> org.apache.solr.search.grouping.distributed.shardresultserializer.TopGroupsResultTransformer.transform(TopGroupsResultTransformer.java:57)
> at
> org.apache.solr.search.grouping.CommandHandler.processResult(CommandHandler.java:214)
> at
> org.apache.solr.handler.component.QueryComponent.doProcessGroupedDistributedSearchSecondPhase(QueryComponent.java:1424)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:386)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:363)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:216)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2646)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:794)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:567)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:357)
> at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
> at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:516)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at
> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
> at
> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection

Performance difference between json.facet and non-json.facet

[Jae Joo]

For the simple facet by field, is there any performance difference between
two?

Jae

BUILD FAILED - Solr 8 on Mac OS Catalina

[Phill Campbell]

This is not an Apache Solr question. This is a developer environment question. 
I apologize for my lack of knowledge with ANT.


$ant common.test

...

-test:
   [junit4]  says jolly good day! Master seed: 3510B76A25B8399C

BUILD FAILED
/development/solr/lucene/common-build.xml:1599: The following error occurred 
while executing this line:
/development/solr//lucene/common-build.xml:1126: Reference junit.classpath not 
found.

Total time: 3 minutes 20 seconds



I have tried adding a CLASSPATH environment variable.
I have tried placing nunit  jars in /Library/Java/Exentions and the “user” 
equivalent location as well.




Common-build.xml

Line 1126 is the “>” on the last line of this snippet:

)



  
  



I don’t know if I have a system configuration problem, or if I am running the 
ANT scripts incorrectly.

Any help appreciated.

Re: Atomic update wrongly deletes child documents

[seez]

Andreas,

Do your nested documents contain only dynamic fields? I am on 8.6.3 and I
ran into this exact same issue. However in my case, even after disabling
catch-all dynamic field, I still see the child documents getting deleted.





--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Atomic update wrongly deletes child documents

[Andreas Hubold]

Hi seez,

no, in my case, nested documents contained only fields as defined in the 
schema. Actually, I didn't even use dynamic fields at all. It was just 
the definition of the dynamic catch-all field that led to the problems. 
It matched the name that was used to set nested documents. Because the 
catch-all field was set to ignore unknown fields, Solr also ignored 
nested documents in the code that performed the atomic update. Maybe 
your problem has a different cause.


Best,
Andreas

seez wrote on 19.04.21 17:48:

Andreas,

Do your nested documents contain only dynamic fields? I am on 8.6.3 and I
ran into this exact same issue. However in my case, even after disabling
catch-all dynamic field, I still see the child documents getting deleted.





--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
.

Re: Atomic update wrongly deletes child documents

[Furkan KAMACI]

Hi Andreas,

Atomic updates are defined within these conditions:

The core functionality of atomically updating a document requires that all
fields in your schema must be configured as stored (stored="true") or
docValues (docValues="true") except for fields which are 
destinations, which must be configured as stored="false". Atomic updates
are applied to the document represented by the existing stored field
values. All data in copyField destinations fields must originate from ONLY
copyField sources.

If not, you lose your field content.

Kind Regards,
Furkan KAMACI

On Mon, Apr 19, 2021 at 8:02 PM Andreas Hubold 
wrote:

> Hi seez,
>
> no, in my case, nested documents contained only fields as defined in the
> schema. Actually, I didn't even use dynamic fields at all. It was just
> the definition of the dynamic catch-all field that led to the problems.
> It matched the name that was used to set nested documents. Because the
> catch-all field was set to ignore unknown fields, Solr also ignored
> nested documents in the code that performed the atomic update. Maybe
> your problem has a different cause.
>
> Best,
> Andreas
>
> seez wrote on 19.04.21 17:48:
> > Andreas,
> >
> > Do your nested documents contain only dynamic fields? I am on 8.6.3 and I
> > ran into this exact same issue. However in my case, even after disabling
> > catch-all dynamic field, I still see the child documents getting deleted.
> >
> >
> >
> >
> >
> > --
> > Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> > .
>
>
>

Re: Regarding internode TLS enablement...

[Jigar Pandya]

Thanks Cassandra.

Here is another question I have:

I don’t know what is the difference between following 2 commands listed on 
https://solr.apache.org/guide/8_8/enabling-ssl.html:

To me, they both are modifying same - clusterprop urlscheme property – one is 
modifying using zkclient and other is using solr API…


$ server/scripts/cloud-scripts/zkcli.sh -zkhost 
server1:2181,server2:2181,server3:2181 -cmd clusterprop -name urlScheme -val 
https

AND

$ 
http://localhost:8983/solr/admin/collections?action=CLUSTERPROP&name=urlScheme&val=https

Are they both needed to be run or 1 will suffice ?

Thanks
Jigar


From: Cassandra Targett 
Date: Wednesday, April 14, 2021 at 12:27 PM
To: users@solr.apache.org , Jigar Pandya 

Subject: Re: Regarding internode TLS enablement...
Ah, right, good point - one node does need to be up in order to run the API 
command. I don’t recall now what I did when I wrote those docs, but if I had to 
guess now I would recommend bring up one node, run the CLUSTERPROP command, 
then bring up the rest of the nodes.
On Apr 13, 2021, 1:53 PM -0500, Jigar Pandya , wrote:

Looks like the urls’ which I sent being adjusted by prrofpoint..

Here is the API, I am talking about – “ 
http://localhost:8983/solr/admin/collections?action=CLUSTERPROP&name=urlScheme&val=https
 
[localhost]
 “

Thanks
Jigar


From: Jigar Pandya 
Date: Tuesday, April 13, 2021 at 11:17 AM
To: users@solr.apache.org , Cassandra Targett 

Subject: Re: Regarding internode TLS enablement...
Thanks Cassandra. This is great that there is an API which can be utilized to 
mark existing collection for https…

I have follow up question, as per the link – this API needs to be run - 
https://urldefense.com/v3/__http://localhost:8983/solr/admin/collections?action=CLUSTERPROP&name=urlScheme&val=https__;!!Ati4tGle!5xcXScwuDrBUAbaCwQA6Pwp_joWVIld6TRTcuRGSnwC3vr2EevEzykxUWjwlnrbUXFY$

above API will require solr to be up and running – what I am not sure what 
should be the order – this is the order which is what I have taken in past:

1. Bring down solr server on each node
2. Configure solr.in.sh with ssl configuration
3. Restart solr

my question is, when this API needs to be done - 
https://urldefense.com/v3/__http://localhost:8983/solr/admin/collections?action=CLUSTERPROP&name=urlScheme&val=https__;!!Ati4tGle!5xcXScwuDrBUAbaCwQA6Pwp_joWVIld6TRTcuRGSnwC3vr2EevEzykxUWjwlnrbUXFY$
  in above listed step.

Can we do above API call after step #3 or before step #1 considering it needs 
solr server to be up and running on atleast one node.
OR
just before step #1, can we just keep one server running and do this step and 
then bring down that server
OR

Bring only one server up after step #4 and run this step.

I really apologize if I have confused you but unfortunately it is not very 
clear…

As per the link – “Once this and all other steps are complete, you can go ahead 
and start Solr” but for above API call to work solr needs to be up on atleast 
one node.

Thanks

Jigar



From: Cassandra Targett 
Date: Monday, April 12, 2021 at 1:29 PM
To: users@solr.apache.org 
Subject: Re: Regarding internode TLS enablement...
At some point after 8.3, we added instructions for what to do with existing 
collections: 
https://urldefense.com/v3/__https://solr.apache.org/guide/8_8/enabling-ssl.html*update-cluster-properties-for-existing-collections__;Iw!!Ati4tGle!7osP7nNU35PQjvJam7Ik8ipo46nc_Y8LsGntQcC55903oTvd2-jlHySqHNdVfEAsUMo$>
 . I don’t see any reason why that wouldn't be valid for 8.3 also.
On Apr 12, 2021, 2:58 PM -0500, Jigar Pandya , wrote:
> Hello,
>
> We use solr cloud 8.3, we have a cluster of 6 VMs.
>
> We are trying to enable SSL for internode communication. I followed the 
> document - 
> https://urldefense.com/v3/__https://

Re: Atomic update wrongly deletes child documents

[Walter Underwood]

Also, the interaction of atomic updates with update request processors is not 
documented.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Apr 19, 2021, at 11:00 AM, Furkan KAMACI  wrote:
> 
> Hi Andreas,
> 
> Atomic updates are defined within these conditions:
> 
> The core functionality of atomically updating a document requires that all
> fields in your schema must be configured as stored (stored="true") or
> docValues (docValues="true") except for fields which are 
> destinations, which must be configured as stored="false". Atomic updates
> are applied to the document represented by the existing stored field
> values. All data in copyField destinations fields must originate from ONLY
> copyField sources.
> 
> If not, you lose your field content.
> 
> Kind Regards,
> Furkan KAMACI
> 
> On Mon, Apr 19, 2021 at 8:02 PM Andreas Hubold 
> wrote:
> 
>> Hi seez,
>> 
>> no, in my case, nested documents contained only fields as defined in the
>> schema. Actually, I didn't even use dynamic fields at all. It was just
>> the definition of the dynamic catch-all field that led to the problems.
>> It matched the name that was used to set nested documents. Because the
>> catch-all field was set to ignore unknown fields, Solr also ignored
>> nested documents in the code that performed the atomic update. Maybe
>> your problem has a different cause.
>> 
>> Best,
>> Andreas
>> 
>> seez wrote on 19.04.21 17:48:
>>> Andreas,
>>> 
>>> Do your nested documents contain only dynamic fields? I am on 8.6.3 and I
>>> ran into this exact same issue. However in my case, even after disabling
>>> catch-all dynamic field, I still see the child documents getting deleted.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> --
>>> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>> .
>> 
>> 
>> 

Re: BUILD FAILED - Solr 8 on Mac OS Catalina

[dmitri maziuk]

On 2021-04-19 10:32 AM, Phill Campbell wrote:


/development/solr//lucene/common-build.xml:1126: Reference junit.classpath not 
found.

...

I don’t know if I have a system configuration problem, or if I am running the 
ANT scripts incorrectly.

Any help appreciated.


Try adding something like
```

  

```
to your top-level build.xml and see what happens.

My guess is junit.classpath would be set somewhere in junit configs but 
I haven't done java in forever and have no idea how lucene build 
environment is set up, so...


Dima

weightExpression operator support

[gnandre]

Which operators are supported in weightExpression field for Solr suggester.
e.g. if I want to use power function, should I use pow?

First I thought, it uses function queries. But it does not seem like that.

Following example shows that it is using the ln function.
https://solr.apache.org/guide/8_5/suggester.html#multiple-dictionaries

However Solr function queries have no such function -
https://solr.apache.org/guide/8_5/function-queries.html