Yonik: Hmmm, we may be closer to that than it might appear. I happened to need to do some verification yesterday to determine whether I could limit the number of rows returned with TupleStream variants. /export of course doesn't do that, the close on a TupleStream waits until the entire stream is exhausted and throws the bits on the floor.
Anyway, I was playing around with returning 10M rows with the /query and /export handlers and found out that I could indeed use /query and limit the rows. Fine so far. Then just for yucks I decided to try to use the /query handler with rows=100M and... the total processing time was virtually identical to /export. These weren't very sophisticated tests mind you; they did lend evidence that your idea is probably the way to go though. I guess we'd need some kind of param to only return DV fields when wildcard fields were specified, but that's an implementation detail. We could even re-use the /export handler definition in solrconfig.xml and have it define another invariant property indicating this for back-compat and then remove some code. FWIW, using the /query handler _also_ exhausts the result set when you call close(), it's just a smaller set ;). I guess no JIRA for this in particular then, I'll let somebody _else_ raise the "remove ExportWriter JIRA"..... I will also point out that it was a milestone for me when I could get as excited about _removing_ code as writing it.... Erick On Thu, Nov 17, 2016 at 4:40 PM, Yonik Seeley <[email protected]> wrote: > On Thu, Nov 17, 2016 at 6:40 PM, Erick Erickson <[email protected]> > wrote: >> I looked through the JIRAs and didn't see anything relevant, but >> before raising a JIRA I thought I'd see if there was interest. >> >> In the case where there are dynamic fields, one may not know what all >> the fields that are DV fields. I think there's a use case for /export >> being able to accept wildcards, i.e. fl=* or fl=*_s. The idea would be >> that the handler would get the field list and pull in all the fields >> defined with docValues=true. >> >> I suppose that this would be most useful if I got busy and actually >> worked on SOLR-3191:"field exclusion from fl". >> >> It's not clear to me though how expensive the introspection would be though. >> >> Thoughts? > > At the risk of sounding redundant, I'll point out again that if we > simply fixed document retrieval to be faster for the normal > QueryComponent, then this would be moot (and everyone would get faster > doc retrieval). > > I'm sympathetic to the fact that it is more work of course. Hopefully > no one disagrees that "/select" document retrieval should be as fast > as "/export" document retrieval though? > > -Yonik > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
