[
https://issues.apache.org/jira/browse/SOLR-17736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18048278#comment-18048278
]
Chris M. Hostetter commented on SOLR-17736:
-------------------------------------------
We're seeing an epicly high jenkins suite level failure rate from
KnnQParserChildTest that all seem to reproduce w/seeds.
All of the failures i'm seeing are low level asserts in the lucene HNSW merge
logic – but i don't remember ever seeing these types of failures in our
existing Knn tests (or in any lucene layer tests)
_*Does this test index some "invalid" (randomized?) data that is sneaking
through and only causing problems on merge?*_
{noformat}
> java.lang.AssertionError: Nodes are added in the incorrect order!
Comparing NaN to [NaN]
> at
org.apache.lucene.util.hnsw.NeighborArray.addInOrder(NeighborArray.java:80)
> at
org.apache.lucene.util.hnsw.HnswGraphBuilder.popToScratch(HnswGraphBuilder.java:420)
> at
org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNodeInternal(HnswGraphBuilder.java:273)
> at
org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNodeWithEps(HnswGraphBuilder.java:331)
> at
org.apache.lucene.util.hnsw.MergingHnswGraphBuilder.updateGraph(MergingHnswGraphBuilder.java:177)
> at
org.apache.lucene.util.hnsw.MergingHnswGraphBuilder.build(MergingHnswGraphBuilder.java:128)
> at
org.apache.lucene.util.hnsw.IncrementalHnswGraphMerger.merge(IncrementalHnswGraphMerger.java:197)
> at
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.mergeOneField(Lucene99HnswVectorsWriter.java:414)
> at
org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.mergeOneField(PerFieldKnnVectorsFormat.java:126)
> at
org.apache.lucene.codecs.KnnVectorsWriter.merge(KnnVectorsWriter.java:105)
> at
org.apache.lucene.index.SegmentMerger.mergeVectorValues(SegmentMerger.java:271)
> at
org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314)
> at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:158)
> at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
> at
org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:219)
> at
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
> at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:668)
> at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:729)
> java.lang.AssertionError
> at org.apache.solr.util.RTimer.stop(RTimer.java:84)
> at
org.apache.solr.metrics.otel.instruments.AttributedLongTimer$MetricTimer.stop(AttributedLongTimer.java:64)
> at
org.apache.solr.update.SolrIndexWriter.updateMergeMetrics(SolrIndexWriter.java:315)
> at
org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:225)
> at
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
> at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:668)
> at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:729)
2> NOTE: test params are: codec=CheapBastard,
sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=ar-DZ,
timezone=Europe/Kyiv
2> NOTE: Linux 5.4.0-150-generic amd64/Oracle Corporation 21.0.2
(64-bit)/cpus=1,threads=1,free=211861448,total=342360064
2> NOTE: All tests run in this JVM: [KnnQParserChildTest]
2> NOTE: reproduce with: gradlew test --tests KnnQParserChildTest
-Dtests.seed=DE240F2E2F26BBE2 -Dtests.multiplier=3 -Dtests.locale=ar-DZ
-Dtests.timezone=Europe/Kyiv -Dtests.asserts=true -Dtests.file.encoding=UTF-8
{noformat}
{noformat}
2> 9816 INFO (SUITE-KnnQParserChildTest-seed#[629859B6B5C753D0]-worker) []
o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-2 for ERROR logs matching
regex: ignore_exception
> java.lang.AssertionError: Nodes are added in the incorrect order!
Comparing NaN to [0.9870917, 0.9875376, 0.98767954, 0.98845375, 0.98859257,
0.98909235, 0.9895828, 0.99154156, 0.9918212, 0.9930609, 0.9935473, 0.9935473,
0.9936491, 0.99381304, 0.99394286, 0.9941522, 0.9943806, 0.99463594, 0.9949257,
0.99570894, 0.99676853, 0.99692047, 0.99739254, 0.9975331, 0.9976279,
0.9977237, 0.9978772, 0.998008, 0.99819005, 0.9981911, 0.9983059, 0.99832237,
0.9984856, 0.9985875, 0.9989623, 0.999037, 0.999172, 0.9992472, 0.9993324,
0.9993397, 0.9994997, 0.9996587, 0.99969697, 0.9997378, 0.9998307, 0.99986196,
0.99998635]
> at
org.apache.lucene.util.hnsw.NeighborArray.addInOrder(NeighborArray.java:80)
> at
org.apache.lucene.util.hnsw.HnswGraphBuilder.popToScratch(HnswGraphBuilder.java:420)
> at
org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNodeInternal(HnswGraphBuilder.java:273)
> at
org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNodeWithEps(HnswGraphBuilder.java:331)
> at
org.apache.lucene.util.hnsw.MergingHnswGraphBuilder.updateGraph(MergingHnswGraphBuilder.java:177)
> at
org.apache.lucene.util.hnsw.MergingHnswGraphBuilder.build(MergingHnswGraphBuilder.java:128)
> at
org.apache.lucene.util.hnsw.IncrementalHnswGraphMerger.merge(IncrementalHnswGraphMerger.java:197)
> at
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.mergeOneField(Lucene99HnswVectorsWriter.java:414)
> at
org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.mergeOneField(PerFieldKnnVectorsFormat.java:126)
> at
org.apache.lucene.codecs.KnnVectorsWriter.merge(KnnVectorsWriter.java:105)
> at
org.apache.lucene.index.SegmentMerger.mergeVectorValues(SegmentMerger.java:271)
> at
org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314)
> at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:158)
> at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
> at
org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:219)
> at
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
> at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:668)
> at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:729)
> java.lang.AssertionError
> at org.apache.solr.util.RTimer.stop(RTimer.java:84)
> at
org.apache.solr.metrics.otel.instruments.AttributedLongTimer$MetricTimer.stop(AttributedLongTimer.java:64)
> at
org.apache.solr.update.SolrIndexWriter.updateMergeMetrics(SolrIndexWriter.java:315)
> at
org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:225)
> at
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
> at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:668)
> at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:729)
2> NOTE: test params are: codec=Asserting(Lucene103): {}, docValues:{},
maxPointsInLeafNode=1398, maxMBSortInHeap=7.058289543498554,
sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=ar-DZ,
timezone=Europe/Kyiv
2> NOTE: Linux 5.4.0-150-generic amd64/Oracle Corporation 21.0.2
(64-bit)/cpus=1,threads=1,free=213978704,total=311427072
2> NOTE: All tests run in this JVM: [KnnQParserChildTest]
2> NOTE: reproduce with: gradlew test --tests KnnQParserChildTest
-Dtests.seed=629859B6B5C753D0 -Dtests.multiplier=3 -Dtests.locale=ar-DZ
-Dtests.timezone=Europe/Kyiv -Dtests.asserts=true -Dtests.file.encoding=UTF-8
{noformat}
> Introduce support for KNN search on nested vector documents
> -----------------------------------------------------------
>
> Key: SOLR-17736
> URL: https://issues.apache.org/jira/browse/SOLR-17736
> Project: Solr
> Issue Type: New Feature
> Components: vector-search
> Affects Versions: 9.8
> Reporter: Alessandro Benedetti
> Assignee: Alessandro Benedetti
> Priority: Major
> Labels: pull-request-available
> Fix For: 10.1
>
> Attachments: SOLR-17736.patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> This issue tracks the work of introducing the support for KNN search on
> nested vector documents, surfacing the Lucene implementation in here:
> https://github.com/apache/lucene/pull/12434
> This allows both:
> -KNN retrieval of children, applying parent filters with no denormalisation
> needed
> -KNN retrieval of parents (based on children KNN, children level prefiltering
> and parent level prefiltering)
> It's one way of having multi-valued vectors per field, per document in Solr.
> More will come soon
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
