Hello, I've had the same issue a few weeks ago. Turns out it was due to "UniqFiledsUpdateProcessorFactory". You should check out your "solrconfig.xml" and verify if this update processor is involved. It typically removes duplicates (in vectors too !).
https://issues.apache.org/jira/browse/SOLR-17487?focusedCommentId=17888974&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17888974 Guillaume Le lun. 21 oct. 2024 à 23:17, Arun Sudhir <arunsud...@gmail.com> a écrit : > Hello, > I'm using Solr 9.1 to index a dense vector field and running into indexing > errors where i see that if there are values that are exactly the same in my > vector, they are all collapsed into one value in the vector and hence the > length of my vector decreases. My field called "embedding_vec" is defined > in the schema as a knn_vector of length 200 as follows: > > <fieldType name="knn_vector" class="solr.DenseVectorField" vectorDimension= > "200" similarityFunction="cosine"/> > <field name="embedding_vec" type="knn_vector" indexed="true" stored="false" > /> > > > Now, i pass in an input value of length 200 but has 4 exact duplicates and > solr errors out by saying that its length is only 196. I also confirmed > that it is removing the three duplicates with the same value. My input is: > > [0.031153935939073563, 0.0005952338688075542, 0.01049225777387619, > 0.016525305807590485, 0.0027819618117064238, -0.015324749983847141, > 0.09765870869159698, 0.10798954963684082, -0.003730300348252058, > 0.10088709741830826, 0.05581074208021164, -0.0641641914844513, > 0.012328403070569038, -0.02736058086156845, 0.04503599926829338, > 0.05653712898492813, -0.12066096812486649, -0.08571367710828781, > 0.023869886994361877, -0.061863966286182404, 0.019218992441892624, > -0.05242093652486801, -0.029761694371700287, -0.049515388906002045, > -0.00158392742741853, 0.024374322965741158, -0.009200903587043285, > 0.005538701545447111, 0.00043570614070631564, -0.053591225296258926, > 0.028086967766284943, 0.00856531411409378, -0.000391883309930563, > 0.0016558094648644328, 0.009044528007507324, 0.013579403050243855, > -0.0391240157186985, 0.05137170851230621, -0.024616451933979988, > 0.06327638775110245, 0.03119429014623165, -0.15593110024929047, > 0.061823613941669464, 0.021327532827854156, -0.039305612444877625, > -0.03401912748813629, 0.15746457874774933, -0.05714245140552521, > 0.05246128886938095, 0.09346180409193039, 0.0009168115211650729, > 0.050039999186992645, -0.10516470670700073, 0.046690549701452255, > 0.040092531591653824, -0.004201947711408138, 0.058716289699077606, > -0.04834509640932083, 0.056173935532569885, 0.03861958160996437, > 0.0700559988617897, 0.07832874357700348, -0.049555741250514984, > 0.047457288950681686, -0.06892606616020203, 0.0674329325556755, > 0.09572167694568634, -0.08196067810058594, -0.05355087295174599, > 0.08805425465106964, 0.026936855167150497, 0.05782848224043846, > 0.07413183897733688, 0.10274341702461243, 0.04021359607577324, > -0.010653677396476269, 0.04564132168889046, 0.07659348100423813, > -0.05815132334828377, -0.14979715645313263, -0.12348580360412598, > 0.12082238495349884, 0.011168201453983784, 0.046690549701452255, > 0.07453538477420807, 0.022336402907967567, -0.14447031915187836, > -0.234219491481781, 0.021690726280212402, 0.1017749011516571, > 0.011733169667422771, -0.11339709907770157, -0.028773000463843346, > 0.03244528919458389, 0.07869193702936172, -0.06719080358743668, > 0.07538283616304398, -0.027300048619508743, -0.05221915990114212, > 0.011753346771001816, 0.06706973910331726, -0.06355886906385422, > -0.0650116428732872, -0.08805425465106964, 0.21791613101959229, > 0.04447103291749954, 0.01868429034948349, -0.034906934946775436, > -0.0033519738353788853, 0.06779612600803375, 0.04531848430633545, > 0.01140024233609438, 0.004840058274567127, 0.10016071051359177, > -0.044107839465141296, 0.041040871292352676, -0.06666619330644608, > 0.00817185454070568, 0.054640453308820724, -0.06489057838916779, > -0.13922418653964996, -0.11719045042991638, 0.03500782325863838, > -0.02637188695371151, -0.05738458037376404, -0.0132061205804348, > 0.06509235501289368, 0.11767470836639404, 0.042614709585905075, > -0.06719080358743668, -0.11315497010946274, 0.006865366827696562, > -0.02966080605983734, -0.0038639758713543415, -0.009443032555282116, > 0.08442232012748718, 0.08781212568283081, -0.05702138692140579, > 0.06723116338253021, 0.011874411255121231, -0.11033012717962265, > 0.06142006441950798, 0.05169454962015152, -0.005067054647952318, > -0.03137588873505592, -0.028994951397180557, 0.07901477813720703, > -0.06573802977800369, -0.146407350897789, 0.021670548245310783, > -0.1251000016927719, -0.0317189022898674, 0.0012182117206975818, > 0.07429325580596924, 0.07804625481367111, -0.019905025139451027, > 0.06811896711587906, -0.06166219338774681, -0.07134735584259033, > -0.06824003159999847, 0.003558792406693101, 0.02992311306297779, > -0.04535884037613869, 0.018492605537176132, 0.043906062841415405, > -0.07397042214870453, 0.0811939388513565, 0.06093580648303032, > 0.02838962897658348, 0.12211374193429947, 0.068159319460392, > -0.10855451971292496, 0.03607722744345665, -0.1192081943154335, > -0.033171676099300385, 0.0648098737001419, 0.07518106698989868, > 0.026170112192630768, -0.07405112683773041, -0.09144406765699387, > 0.10476116091012955, 0.10645606368780136, -0.11727116256952286, > 0.0375501774251461, -0.047174807637929916, -0.10516470670700073, > -0.012439379468560219, -0.09547954797744751, -0.005967471748590469, > -0.09951503574848175, -0.03500782325863838, -0.041605837643146515, > 0.10903877764940262, -0.08018506318330765, 0.10443831980228424, > 0.04176725819706917, 0.18773071467876434, 0.03194085508584976, > -0.0132061205804348, -0.058756645768880844] > > and i found out that solr makes it into this: [0.031153935939073563, > 0.0005952338688075542, 0.01049225777387619, 0.016525305807590485, > 0.0027819618117064238, -0.015324749983847141, 0.09765870869159698, > 0.10798954963684082, -0.003730300348252058, 0.10088709741830826, > 0.05581074208021164, -0.0641641914844513, 0.012328403070569038, > -0.02736058086156845, 0.04503599926829338, 0.05653712898492813, > -0.12066096812486649, -0.08571367710828781, 0.023869886994361877, > -0.061863966286182404, 0.019218992441892624, -0.05242093652486801, > -0.029761694371700287, -0.049515388906002045, -0.00158392742741853, > 0.024374322965741158, -0.009200903587043285, 0.005538701545447111, > 0.00043570614070631564, -0.053591225296258926, 0.028086967766284943, > 0.00856531411409378, -0.000391883309930563, 0.0016558094648644328, > 0.009044528007507324, 0.013579403050243855, -0.0391240157186985, > 0.05137170851230621, -0.024616451933979988, 0.06327638775110245, > 0.03119429014623165, -0.15593110024929047, 0.061823613941669464, > 0.021327532827854156, -0.039305612444877625, -0.03401912748813629, > 0.15746457874774933, -0.05714245140552521, 0.05246128886938095, > 0.09346180409193039, 0.0009168115211650729, 0.050039999186992645, > -0.10516470670700073, 0.046690549701452255, 0.040092531591653824, > -0.004201947711408138, 0.058716289699077606, -0.04834509640932083, > 0.056173935532569885, 0.03861958160996437, 0.0700559988617897, > 0.07832874357700348, -0.049555741250514984, 0.047457288950681686, > -0.06892606616020203, 0.0674329325556755, 0.09572167694568634, > -0.08196067810058594, -0.05355087295174599, 0.08805425465106964, > 0.026936855167150497, 0.05782848224043846, 0.07413183897733688, > 0.10274341702461243, 0.04021359607577324, -0.010653677396476269, > 0.04564132168889046, 0.07659348100423813, -0.05815132334828377, > -0.14979715645313263, -0.12348580360412598, 0.12082238495349884, > 0.011168201453983784, 0.07453538477420807, 0.022336402907967567, > -0.14447031915187836, -0.234219491481781, 0.021690726280212402, > 0.1017749011516571, 0.011733169667422771, -0.11339709907770157, > -0.028773000463843346, 0.03244528919458389, 0.07869193702936172, > -0.06719080358743668, 0.07538283616304398, -0.027300048619508743, > -0.05221915990114212, 0.011753346771001816, 0.06706973910331726, > -0.06355886906385422, -0.0650116428732872, -0.08805425465106964, > 0.21791613101959229, 0.04447103291749954, 0.01868429034948349, > -0.034906934946775436, -0.0033519738353788853, 0.06779612600803375, > 0.04531848430633545, 0.01140024233609438, 0.004840058274567127, > 0.10016071051359177, -0.044107839465141296, 0.041040871292352676, > -0.06666619330644608, 0.00817185454070568, 0.054640453308820724, > -0.06489057838916779, -0.13922418653964996, -0.11719045042991638, > 0.03500782325863838, -0.02637188695371151, -0.05738458037376404, > -0.0132061205804348, 0.06509235501289368, 0.11767470836639404, > 0.042614709585905075, -0.11315497010946274, 0.006865366827696562, > -0.02966080605983734, -0.0038639758713543415, -0.009443032555282116, > 0.08442232012748718, 0.08781212568283081, -0.05702138692140579, > 0.06723116338253021, 0.011874411255121231, -0.11033012717962265, > 0.06142006441950798, 0.05169454962015152, -0.005067054647952318, > -0.03137588873505592, -0.028994951397180557, 0.07901477813720703, > -0.06573802977800369, -0.146407350897789, 0.021670548245310783, > -0.1251000016927719, -0.0317189022898674, 0.0012182117206975818, > 0.07429325580596924, 0.07804625481367111, -0.019905025139451027, > 0.06811896711587906, -0.06166219338774681, -0.07134735584259033, > -0.06824003159999847, 0.003558792406693101, 0.02992311306297779, > -0.04535884037613869, 0.018492605537176132, 0.043906062841415405, > -0.07397042214870453, 0.0811939388513565, 0.06093580648303032, > 0.02838962897658348, 0.12211374193429947, 0.068159319460392, > -0.10855451971292496, 0.03607722744345665, -0.1192081943154335, > -0.033171676099300385, 0.0648098737001419, 0.07518106698989868, > 0.026170112192630768, -0.07405112683773041, -0.09144406765699387, > 0.10476116091012955, 0.10645606368780136, -0.11727116256952286, > 0.0375501774251461, -0.047174807637929916, -0.012439379468560219, > -0.09547954797744751, -0.005967471748590469, -0.09951503574848175, > -0.03500782325863838, -0.041605837643146515, 0.10903877764940262, > -0.08018506318330765, 0.10443831980228424, 0.04176725819706917, > 0.18773071467876434, 0.03194085508584976, -0.058756645768880844] > The error call satck is: > > org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:246) > at > org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:100) > at > > org.apache.solr.update.AddUpdateCommand.lambda$makeLuceneDocs$0(AddUpdateCommand.java:233) > at > > java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) > at > > java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1602) > at > > java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292) > at > > java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206) > at > > java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161) > at > > java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298) > at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681) > at > > org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:232) > at > > org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1532) > at > org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1521) > at > > org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:1048) > at > > org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:416) > at > > org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:369) > at > > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:300) > at > > org.apache.solr.update.processor.RunUpdateProcessorFactory$RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:76) > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > > org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:535) > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:111) > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:111) > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:111) > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:111) > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:111) > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > > org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:71) > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:111) > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:269) > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:544) > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$0(DistributedUpdateProcessor.java:356) > at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:51) > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:353) > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:235) > at > > org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:111) > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > > org.apache.solr.update.processor.DocBasedVersionConstraintsProcessor.processAdd(DocBasedVersionConstraintsProcessor.java:433) > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > > org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:82) > at > > org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader$1.handle(JsonLoader.java:288) > at > > org.apache.solr.common.util.JsonRecordReader$Node.handleObjectStart(JsonRecordReader.java:386) > at > > org.apache.solr.common.util.JsonRecordReader$Node.parse(JsonRecordReader.java:287) > at > > org.apache.solr.common.util.JsonRecordReader.streamRecords(JsonRecordReader.java:141) > at > > org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.handleSplitMode(JsonLoader.java:266) > at > > org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:176) > at > > org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:160) > at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:86) at > > org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:101) > at > > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:84) > at > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2865) at > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:887) at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:606) at > > org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:250) > at > > org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:218) > at > > org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257) > at > > org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227) > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213) > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) > at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201) > at > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) > at > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) > at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > at > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > at > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) > at > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) > at > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) > at > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) > at > > org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177) > at > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) > at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at > > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322) > at > > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) > at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at org.eclipse.jetty.server.Server.handle(Server.java:516) at > org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) > at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) > at > org.eclipse.jetty.io > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at > org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) > at > > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) > at > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > at > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > at java.base/java.lang.Thread.run(Thread.java:840)\\nCaused by: > org.apache.solr.common.SolrException: Error while creating field > Why is this happening? Can anyone help? > > Arun >
