Hello dear Solr Team
In my documents I use fields with field type text_de. Lately I came across
weird behavior with this field type. To reproduce these weird behavior I have
set up a local solr verver on my machine. There I could identify the same
behavior.
To reproduce it yourself, just set up a local solr server as described here:
https://solr.apache.org/guide/solr/latest/deployment-guide/installing-solr.html
Start the solr server: bin/solr start
Create a simple core: bin/solr create -c dario
Now in the browser go to http://localhost:8983/solr/#/dario/schema and add two
fields.
Field 1:
Use name_general for name and text_general for the field type. Leave everything
else as default.
Field 2:
Use name_de for name and text_de for the field type. Leave everything else as
default.
Then go to http://localhost:8983/solr/#/dario/documents to index some documents.
Select CSV for Document Type.
In the textfield Document(s) type in the fallowing text: (hyphens mark start
and end of text, should be excluded from pasted text)
-
name_general,name_de
DARIO,DARIO
DARIOT,DARIOT
DARIOTE,DARIOTE
DARIOTEN,DARIOTEN
DARIOTENU,DARIOTENU
-
Now go to http://localhost:8983/solr/#/dario/query to search documents.
First, I use the query *:* to find all documents. This works as expected.
Then I use the query name_general:* to find all documents with the field
name_general. This again finds all documents as expected. The same for name_de:*
Now I want to find specific documents with the text_general field:
Searching for name_general:DARIO finds one document where name_general is
exactly DARIO. This is what I expected. The same thing happens for all other
names.
Now it gets interesting. Searching for text_de leads to all kind of weird
results. Listed below are the queries and what is found with this query, and if
I expected the result or not:
-
name_de:DARIO -> DARIO (expected)
name_de:DARIOT -> DARIOT (expected), DARIOTE (not expected), DARIOTEN (not
expected)
name_de:DARIOTE -> DARIOT (not expected), DARIOTE (expected), DARIOTEN (not
expected)
name_de:DARIOTEN -> DARIOT (not expected), DARIOTE (not expected), DARIOTEN
(expected)
name_de:DARIOTENU -> DARIOTENU (expected)
-
This probably has something to do with stemming. But why should DARIOTEN be
stemmed to DARIOT? (Sorry for the bad English, as you can see with the usage of
text_de my first language is German)
DARIOTEN is also not defined in the stop words file or anywhere else. So what
is happening here?
Now I also tried enclosing the searched word in quotes. E. g. name_de:"DARIO"
This did not change the results.
So this is one problem that I can not explain, but would very much like to
understand. But this is not my only confusion with this field type. What
fallows is another problem.
Sometimes I want to find documents that contain some text (as opposed to exact
matches)
So I enclose the name with asterisks. I expect these queries to find all the
documents that EITHER exactly match the word OR contain the word.
Again, this works as expected with the name_general field. But again the
name_de behaves weirdly.
I will again list all the queries and the results. This time I will also note
the documents that were missing from the results, but I expected them to be
included. I will note these documents after the pipe |.
As this time no documents are found that should not be found, the expected /
not expected marker is omitted.
-
name_de:*DARIO* -> DARIO, DARIOT, DARIOTE, DARIOTEN, DARIOTENU
name_de:*DARIOT* -> DARIOT, DARIOTE, DARIOTEN, DARIOTENU
name_de:*DARIOTE* -> DARIOTENU | DARIOTE, DARIOTEN
name_de:*DARIOTEN* -> DARIOTENU | DARIOTEN
name_de:*DARIOTENU* -> DARIOTENU
-
As you can see for name_de:*DARIOTE* and name_de:*DARIOTEN* not all documents
are found that I expected to be found.
What are the mechanisms leading to this behavior?
I also tried all queries using regular expressions. E. g. name_de:/.*DARIO.*/
This lead to the same results as before.
I look forward to an easy to understand explanation for why text_de behaves the
way it does.
With kind regards
Dario Viva
