I don't know a direct answer to your questions, but some context of why you are you running a collapse query on 7m documents could help provide insight? What are you trying to achieve? Are the results to be paged in a ui? Is it an analytics workload?
On Wed, Jun 24, 2026, 21:46 Bartosz Fidrysiak <[email protected]> wrote: > We identified a 2–3x performance regression in Solr 9.10.1 compared to > Solr 8.11.2 for collapse > queries that use a string field as a collapse sort field. > > > Test setup > ---------- > > To measure the regression under real production conditions, we configured > both clusters to receive identical traffic simultaneously — every Solr > request is sent to both instances at the same time, making the comparison > direct and unbiased. Both clusters have the same number of nodes, > documents, shards, and shard ranges. The data is sharded by tenant ID, so > each request is served by a single shard with no cross-shard overhead. Solr > schema is the same for both clusters. > > We tested six query variants covering different combinations of collapse > sort fields: no collapse, collapse with date sort, date+long sort, > date+string sort, and string-only sort (see attachments). The results show > that queries with a string field in the collapse sort are consistently and > significantly slower in Solr 9, while queries using only numeric or date > sort fields show no regression. Notably, the string field used in the > collapse sort has very high cardinality, and the worst-case queries process > millions of documents. > > > [image: image.png] > [image: image.png] > [image: image.png] > > Root cause > ---------- > > JFR profiling of the worst-case query (sort="modified_date desc, > document_id asc", ~7M documents) confirmed the root cause. > [image: image.png] > > Lucene 9 changed the internal format for SortedDocValues > (Lucene90DocValuesProducer). The term dictionary (TermsDict) now stores > string values in LZ4-compressed blocks. In Lucene 8, the same data was held > uncompressed in direct memory — reads were instant. In Lucene 9, every time > the collapse logic needs to materialize a string value for comparison or to > record a new group winner, it must decompress an LZ4 block. For ~7M > documents, this decompression is triggered on nearly every document via the > following call chain: > > SortFieldsCompare > -> TermOrdValLeafComparator.copy() > -> lookupOrd() > -> TermsDict.decompressBlock() > -> LZ4.decompress() > > LZ4 decompression accounts for almost 40% of CPU time in the query-serving > thread in Solr 9, > versus near zero in Solr 8. > > Similar concerns were raised in > https://github.com/apache/lucene/issues/11485 > > Questions > --------- > > Q1: What are your recommendations for improving the performance of > collapse queries that use a string field as a sort tiebreaker in Solr 9? > > Q2: Is it possible to disable LZ4 compression for SortedDocValues term > dictionaries — either via a configuration property or a docValuesFormat > option — or is this something that could be planned for a future release? > > Q3: Would it be feasible to lazily materialize string field values in > CollapsingQParserPlugin for group winners, so that lookupOrd() is only > called when a cross-segment comparison is actually needed? This could > improve performance for queries where most groups contain only one document. > > Kind regards, > Bartosz > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected]
