Github user xwu0226 commented on the pull request:
https://github.com/apache/spark/pull/12147#issuecomment-205541295
@hvanhovell Per your request, I did the benchmark between master and
spark-14361 branch without the exclude case. The following is the results. I
will continue on the exclude cases.
```
+-----+----------------+------+-----------+-----+
| Size| Name|Master|SPARK-14361| Diff|
+-----+----------------+------+-----------+-----+
| 2|Entire Partition| 24093| 19106|0.793|
| 2| Growing| 15166| 15032|0.991|
| 2| Shrinking| 15217| 15687|1.031|
| 2| Sliding| 16191| 16314|1.008|
| 8|Entire Partition| 14599| 14749|1.010|
| 8| Growing| 15382| 14883|0.968|
| 8| Shrinking| 15303| 15655|1.023|
| 8| Sliding| 15232| 15245|1.001|
| 32|Entire Partition| 15361| 14990|0.976|
| 32| Growing| 15922| 15073|0.947|
| 32| Shrinking| 16851| 19129|1.135|
| 32| Sliding| 15589| 15655|1.004|
| 128|Entire Partition| 14959| 14486|0.968|
| 128| Growing| 14659| 15095|1.030|
| 128| Shrinking| 18121| 18319|1.011|
| 128| Sliding| 15495| 15030|0.970|
| 256|Entire Partition| 15373| 14951|0.973|
| 256| Growing| 14857| 14545|0.979|
| 256| Shrinking| 20371| 22640|1.111|
| 256| Sliding| 15228| 14626|0.960|
| 1024|Entire Partition| 14774| 14337|0.970|
| 1024| Growing| 15888| 15413|0.970|
| 1024| Shrinking| 39446| 45678|1.158|
| 1024| Sliding| 15260| 14591|0.956|
| 4096|Entire Partition| 14652| 14839|1.013|
| 4096| Growing| 15064| 15099|1.002|
| 4096| Shrinking|147914| 193215|1.306|
| 4096| Sliding| 15225| 15476|1.016|
|16192|Entire Partition| 14155| 14600|1.031|
|16192| Growing| 14040| 14498|1.033|
|16192| Shrinking|699023| 883307|1.264|
|16192| Sliding| 20765| 17278|0.832|
+-----+----------------+------+-----------+-----+
```
It looks like the Shrinking case gets a bit worse than master while the
partition size is larger, while other cases have some minor improvements. I
will look at the logic in shrinking case. I may know the answer already.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
