On Wed, Mar 16, 2016 at 3:29 PM, Mridul Muralidharan
wrote:
> b) Shuffle manager (to get shuffle reader)
>
What's the use case for shuffle manager/reader? This seems like using super
internal APIs in applications.
We have custom join's that leverage it.
It is used to get to direct shuffle'ed iterator - without needing
sort/aggregate/etc.
IIRC the only way to get to it from ShuffleHandle is via shuffle manager.
Regards,
Mridul
On Wed, Mar 16, 2016 at 3:36 PM, Reynold Xin wrote:
>
> On Wed, Mar 16, 2016 a
We use it in executors to get to :
a) spark conf (for getting to hadoop config in map doing custom
writing of side-files)
b) Shuffle manager (to get shuffle reader)
Not sure if there are alternative ways to get to these.
Regards,
Mridul
On Wed, Mar 16, 2016 at 2:52 PM, Reynold Xin wrote:
> Any