Re: Spark 3.2.0 first RC next week

2021-08-10 Thread Min Shen
Hi Gengliang, SPARK-36378 (Switch to using RPCResponse to communicate common block push failures to the client) should be another one. This introduces a slight protocol change to push-based shuffle to improve code robustness and performance, and is almost ready to be committed. Because of the prot

Re: Apache Spark 3.2 Expectation

2021-06-16 Thread Min Shen
ERVAL types > >> ><https://issues.apache.org/jira/browse/SPARK-27790>; > >> >- Support Timestamp without time zone data type > >> ><https://issues.apache.org/jira/browse/SPARK-35662>; > >> >- Aggregate (Min/Max/Count) push down for Pa

Re: Enabling push-based shuffle in Spark

2020-01-27 Thread Min Shen
ng Spark > shuffle system's scalability and efficiency, making Spark an even > better > compute engine. > Would like to explore how we can leverage the shuffle plugin API to > make >