itschrispeck commented on PR #12945:
URL: https://github.com/apache/pinot/pull/12945#issuecomment-2080300157
> How about we add 2 configs here
Done. Updated the description with the new configs. Lmk if there's any
concern with naming, I changed it slightly for consistency:
```
"forward": {
"targetMaxChunkSize": "1M",
"targetDocsPerChunk": 1000
}
```
Posting the below just for reference:
>I can imagine people want to disable this and always go with the
maxTargetChunkSize for scan intensive case to reduce decompression
I understand why this should be the case, but could not reproduce it in our
prod env. I ended up being able to reproduce it in a microbench, and it turned
out the scans I initially tested were against very low cardinality data. For
higher cardinality data the microbench showed up to 40% faster performance w/
1M vs 4K chunk size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
