[
https://issues.apache.org/jira/browse/SOLR-15712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486542#comment-17486542
]
Michael Gibney commented on SOLR-15712:
---------------------------------------
[~tkiseleva], I'm not surprised that setting {{indexed="true"}} and
{{docValues="false"}} caused the error to stop occurring. But I would strongly
encourage you to revert that change and evaluate whether the solution I
proposed (adding {{useDocValuesAsStored="false"}} also fixes the issue -- and
assuming it does, to take that approach instead.
The reason is that based on the contents of ICUCollationField and it's
purpose/use (as a sort key), {{indexed="false" docValues="true"}} is almost
certainly what you want; the configuration change you introduced has the
side-effect of essentially building "docValues"-like data structures in memory.
This increases (depending on index size, perhaps greatly) heap memory usage/GC,
and increases newSearcher warming/initial request latency.
So the proposed configuration would be:
{code:xml}
<dynamicField name="sort_X3b_bg_*" type="collated_bg" stored="false"
indexed="false" docValues="true" useDocValuesAsStored="false" />
{code}
... though it might be preferable to change that setting directly on the
configuration of the {{collated_bg}} fieldType, as opposed to on the
{{dynamicField}} config. (And you'd do well to proactively set
{{useDocValuesAsStored="false"}} on _all_ ICUCollationField fieldTypes. If you
don't use them, it's harmless, and if you _do_ use them, without this change
you'll still have this same problem -- latently -- lurking in your config).
If your index is relatively small, and your heap sufficiently large, you may
never see practical problems from {{docValues="false"}} ... but it's definitely
an anti-pattern. And in fact it's not unlikely that at some point in the future
the behavior you're relying on with {{docValues="false"}} will no longer be
enabled by default, so you may have to explicitly allow building in-memory
"docValues"-like structure via setting {{uninvertible="true"}} ...
> Certain unicode field values throw exceptions for query and backup operations
> -----------------------------------------------------------------------------
>
> Key: SOLR-15712
> URL: https://issues.apache.org/jira/browse/SOLR-15712
> Project: Solr
> Issue Type: Bug
> Components: Backup/Restore, search
> Affects Versions: 7.7.3
> Reporter: Kosala Almeda
> Priority: Major
> Labels: drupal_integration
> Attachments: Selection_119.png, Selection_120.png,
> image-2022-02-02-14-28-58-102.png
>
>
> Following error is thrown when running backup
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: 45
> at org.apache.lucene.util.UnicodeUtil.UTF8toUTF16(UnicodeUtil.java:602)
> at org.apache.lucene.util.BytesRef.utf8ToString(BytesRef.java:137)
> at
> org.apache.solr.search.SolrDocumentFetcher.decodeDVField(SolrDocumentFetcher.java:550)
> at
> org.apache.solr.search.SolrDocumentFetcher.decorateDocValueFields(SolrDocumentFetcher.java:506)
> at
> org.apache.solr.search.SolrDocumentFetcher$RetrieveFieldsOptimizer.getSolrDoc(SolrDocumentFetcher.java:800)
> at
> org.apache.solr.search.SolrDocumentFetcher$RetrieveFieldsOptimizer.access$000(SolrDocumentFetcher.java:672)
> at
> org.apache.solr.search.SolrDocumentFetcher.solrDoc(SolrDocumentFetcher.java:278)
> at org.apache.solr.response.DocsStreamer.next(DocsStreamer.java:95)
> at org.apache.solr.response.DocsStreamer.next(DocsStreamer.java:59)
> at
> org.apache.solr.response.TextResponseWriter.writeDocuments(TextResponseWriter.java:184)
> at
> org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:136)
> at
> org.apache.solr.common.util.JsonTextWriter.writeNamedListAsMapWithDups(JsonTextWriter.java:386)
> at
> org.apache.solr.common.util.JsonTextWriter.writeNamedList(JsonTextWriter.java:292)
> at org.apache.solr.response.JSONWriter.writeResponse(JSONWriter.java:73)
> at
> org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:66)
> at
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
> at
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:811)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:540)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.server.Server.handle(Server.java:502)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> with some debugging found out that select {{\*:\*}} query also throws the
> same error for some of the documents.
> This is identified with search api [solr 4.x
> configset|https://git.drupalcode.org/project/search_api_solr/-/tree/4.1.12/jump-start/solr7/cloud-config-set]
> for Drupal and eastern European language contents.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
