Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/5403#issuecomment-90775140
Is it more easier to set spark.local.dir to mem disk and store these
shuffle files to ram disks? It is more easier and less GC overhead from my
understanding :smile: .
Besides, I think shuffle framework could have two layers of abstraction:
1. From implementations, there has sort-based shuffle and hash-based
shuffle.
2. From storing side, we could choose file-based mapping to map each bucket
to one file or one part of file; Or as you implemented like memory-based
mapping which mapping one bucket to memory block, or anythings else.
So basically these two layers could have different combinations, like
sort-based on-memory or sort-based on disk shuffle, hash-based on-memory or
hash-based on disk shuffle... I think here the implementation is hash-based
on-memory shuffle, so what do you think if we want sort-based on-memory shuffle?
Just my thoughts, thanks a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
