Github user rezazadeh commented on the pull request:
https://github.com/apache/spark/pull/1778#issuecomment-56894583
@mengxr Thanks for the optimizations. I merged the latest master into my
branch and pushed to here. Would you like me to merge your branch into mine?
There is no guarantee on sparsity, only on lower shuffle size, so what you
observed is exactly what is expected. We can however allow the user to promote
sparsity by allowing them to set the threshold to values greater than 1. In
this case however, we can't provably guarantee correctness, but it will be
useful for users who have very dense matrices, because the guarantees are
pessimistic and even after promoting sparsity the result will likely be useful
to them.
With this in mind, I propose allowing the threshold to be above 1, and when
it is, instead of giving an error, give a warning that results are not
guaranteed correct, but the computational savings will be useful. Shall I do
that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
