Hi Prasad, This is expected with "deep paging": https://solr.apache.org/guide/solr/latest/query-guide/pagination-of-results.html#performance-problems-with-deep-paging
Have a look at cursors instead, that should solve your problem: https://solr.apache.org/guide/solr/latest/query-guide/pagination-of-results.html#fetching-a-large-number-of-sorted-results-cursors Thomas Op vr 5 apr 2024 om 10:36 schreef prasad bezavada <prasadbezav...@gmail.com >: > Dear Team, > > I'm currently using Solr version 8.11.3, configured with RAM resources (125 > GB physical memory, 64 GB heap memory). The collection comprises 4 shards > within the same node. Through our Java application ( SolrJ), > indexed approximately 8 million records from an RDBMS table into Solr. > > Presently, my task is to query this indexes and exporting the results (5 > million records fetched with my solr query) to PDF format via our Java > application. To avoid potential heap memory issues, I've implemented > pagination (3 lakhs) in the query using start and setrows parameters. > > However, I've encountered an issue where the response time for subsequent > queries to fetch the next set of results (e.g., 3 to 6 lakhs, 6 to 9 lakhs) > progressively increases, leading to socket timeout exceptions. > Additionally, Solr's physical memory consumption exceeds 90%, without > releasing it. > > I have several queries regarding this situation: > > Why does the query time in Solr increase with each pagination query? > What causes Solr to occupy over 90% of physical memory and fail to release > it? > What would be the optimal approach for retrieving 5 million records from > our Java application and exporting them to PDF or other file formats? > Your insights and suggestions on resolving these issues would be greatly > appreciated. > > > -- > *Thanks&Regards* > > *Prasad Bezavada* >
