andygrove commented on PR #2106: URL: https://github.com/apache/datafusion-ballista/pull/2106#issuecomment-5024748589
> I'm considering case B here, now that we have multi-partition-per-task, there's some instances where it could be better to hash parition: > > ``` > Case A: 1 input partition per task (4 tasks total) > - Sort-shuffle: 4 tasks × 1 input each = 4 files total. Downstream reader for output k opens 4 files, seeks to index[k] in each. > - Hash-writer (M×K layout): 4 tasks × 4 outputs each = 16 files total. Downstream reader for output k opens 4 files (one per producer task), reads each whole. > > Case B: 4 input partitions per task (1 task total, the multi-partition-tasks payoff) > - Sort-shuffle: 1 task × 4 inputs = 4 files total. Downstream reader for output k still opens 4 files with 4 index seeks — same file count as Case A. > - Hash-writer: 1 task × 4 outputs = 4 files total. Downstream reader for output k opens 1 file, reads whole. > ``` > > But what I was working towards anyway was removing that hashing from `ShuffleWriter` entirely: > > ``` > RepartitionExec(Hash) → ShuffleWriterExec(None) > ``` > > Which I think this PR still allows. If there are valid cases for keeping hash-based partition where there are tiny number of output partitions, we can keep it in. I am targeting use cases that could have very high number of output partitions (hundreds/thousands). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
