Very often I have used solr to return over 30 million documents with no ramifications via a wget call and a LONG timeout. granted it took a while and the resulting file was in the multiple GB's of size, but there isnt any issues with it I ever encountered. I also used about a 31gb JVM head and had a few hundred gb of memory on the server but it never got taxed too high
On Wed, Apr 27, 2022 at 5:23 PM Vincenzo D'Amore <v.dam...@gmail.com> wrote: > Hi, I strongly discourage you from downloading so many documents in one > shot, doing this on a normal basis creates humongous memory allocations in > the JVM and this usually leads to have GC problems. > > https://stackoverflow.com/questions/10039778/how-to-get-all-results-from-solr-query > > There are really so many options to read a great amount of documents. > > Natively you can use the export request handler > > > https://solr.apache.org/guide/8_11/exporting-result-sets.html#the-export-requesthandler > > Here is an app I wrote long ago that uses Solr cursors > > https://github.com/freedev/solr-import-export-json > https://solr.apache.org/guide/6_6/pagination-of-results.html#using-cursors > > But even the simple solr pagination with the parameters start and rows can > do better than use rows alone. > > > On Wed, Apr 27, 2022 at 10:35 PM Neha Gupta <neha.gu...@uni-jena.de> > wrote: > > > Hi Andy, > > > > I have different cores with different number of documents. > > > > 1) Core 1: - 227625 docs and each document having approx 10 String > fields. > > > > 2) Core 2: - Approx 3.5 million documents and each having 3 string > fields. > > > > So my question is if i request in one request lets say approximate 10K > > documents using SOLRJ will that be OK. By safe here i mean approx. > > maximum number of documents that i can request without causing any > > problem in receiving a response from SOLR. > > > > Is that enough to answer the question? > > > > On 27/04/2022 22:26, Andy Lester wrote: > > > > > >> On Apr 27, 2022, at 3:23 PM, Neha Gupta<neha.gu...@uni-jena.de> > wrote: > > >> > > >> Just for information I will be firing queries from Java application to > > SOLR using SOLRJ and would like to know how much maximum documents (i.e > > maximum number of rows that i can request in the query) can be returned > > safely from SOLR. > > > It’s impossible to answer that. First, how do you mean “safe”? How big > > are your documents? > > > > > > Let’s turn it around. Do you have a number in mind where you’re > > wondering if Solr can handle it? Like you’re thinking “Can Solr handle 10 > > million documents averaging 10K each”? That’s much easier to address. > > > > > > Andy > > > > -- > Vincenzo D'Amore >
