Here are a few discussions https://issues.apache.org/jira/browse/LUCENE-7148
and
https://lucene.apache.org/core/8_11_0/sandbox/org/apache/lucene/search/CoveringQuery.html
On Tue, Aug 2, 2022 at 1:25 PM Mikhail Khludnev <m...@apache.org> wrote:
> Hi, Colvin.
> It reminds me of percolator match logic. I've heard of such plugins for
> Elastic&Solr.
> Think about min_should_match in dismax - mm.
> If one indexes a number of words in a dedicated field, then count every
> term hit via constant score ^=1, sum hits score, then cut off matches with
> a weak coverage via {!frange} (compare sum of scores to a field with a
> number of tokens). It was discussed in comments/list years ago. Not sure if
> we moved toward it already. I also remember that such logic built-in
> you-know-where
> https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-set-query.html
> .
>
> On Mon, Aug 1, 2022 at 6:59 PM Colvin Cowie <colvin.cowie....@gmail.com>
> wrote:
>
>> Hello,
>>
>> Maybe the answer to this is obvious and I'm missing something, but here
>> goes:
>>
>> Suppose I have a field which contains a string of one or more tokens from
>> a
>> set. The set has about 50 possible values, and the values themselves are
>> arbitrary (though they are known ahead of time, and could be ordered
>> alphabetically if it helped). e.g.
>> doc1: "red"
>> doc2: "blip red"
>> doc3: "aardvark blip red"
>> doc4: "aardvark potato"
>>
>> I want to query the field for all documents that contain at least one of
>> the tokens specified in the query *and no tokens that aren't in the
>> query*.
>> What's the best query for that?
>>
>> For example, querying for
>>
>> - "*red*" should *only* match doc1 above
>> - "*blip red*" should match doc1 *and* doc2
>> - "*blip red potato*" should also match doc1 and doc 2.
>> - "*aardvark blip*" would not match any of the documents since neither
>> term appears on its own above, and it would need "*red*" as well to
>> match doc3.
>> - "*aardvark blip red potato*" would match all of the documents.
>>
>>
>> Options?
>>
>> 1. I could formulate the query to include all the required tokens and
>> negate all the other tokens from the set, e.g. "*blip red*" would
>> be "*+(blip
>> red) -(aardvark potato....)*", and "*red*" would be "*+(red) -(aardvark
>> blip potato...)*"... The size of the set is fixed, so the number of
>> terms in the query won't change, just whether they are included or
>> excluded. But having to specify all the negations seems inefficient.
>> 2. I could change the way the data is indexed so that the field is
>> concatenated deterministically and tokenized as a single value, and
>> query
>> for combinations of terms. e.g. "*blip red*" would be "*blip red
>> blip-red*", but with more than a handful of terms the fan-out becomes
>> significant, e.g. "*aardvark* *blip red*" becomes "*aardvark blip red
>> aardvark-blip aardvark-red blip-red aardvark-blip-red *" and so on,
>> with
>> (2^N)-1 combinations.
>>
>> So option 1 should be fairly constant regardless of the number of terms
>> but
>> may be wasteful for low numbers of terms, while option 2 generates > 1000
>> combinations for a query with 10 terms. Is that a problem for Lucene in
>> practice though? For 20 terms it would create >1 million combinations,
>> which does sound like a problem, but a query with that many terms may not
>> be needed.
>>
>> I'm leaning towards 1 - but is it a bad solution? Is there a better option
>> I'm missing?
>>
>> On a related note, does the EnumFieldType enable a more efficient query
>> than other field types, or does it just provide explicit sorting? i.e.
>> would a multivalued EFT be better for this?
>>
>> Thanks,
>> Colvin
>>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
--
Sincerely yours
Mikhail Khludnev
