Hi Gengliang, Thanks for volunteering as the release manager for Spark 3.2.0. Regarding the ongoing work of push-based shuffle in SPARK-30602, we are close to having all the patches merged to master to enable push-based shuffle. Currently, there are 2 PRs under SPARK-30602 that are under active review (SPARK-32922 and SPARK-35671), and hopefully can be merged soon. We should be able to post the PRs for the other 2 remaining tickets (SPARK-32923 and SPARK-35546) early next week.
The tickets under SPARK-30602 are the minimum set of patches to enable push-based shuffle. We do have other performance/operability enhancements tickets under SPARK-33235 that are needed to fully contribute what we have internally for push-based shuffle. However, these are optional for enabling push-based shuffle. We do strongly prefer to cut the release for Spark 3.2.0 including all the patches under SPARK-30602. This way, we can backport the other performance/operability enhancements tickets under SPARK-33235 into branch-3.2 to be released in future Spark 3.2.x patch releases. I understand the preference of not postponing the branch cut date. We will check with Dongjoon regarding the soft cut date and the flexibility for including the remaining tickets under SPARK-30602 into branch-3.2. Best, Min On Wed, Jun 16, 2021 at 1:20 PM Liang-Chi Hsieh <vii...@gmail.com> wrote: > > Thanks Dongjoon. I've talked with Dongjoon offline to know more this. > As it is soft cut date, there is no reason to postpone it. > > It sounds good then to keep original branch cut date. > > Thank you. > > > > Dongjoon Hyun-2 wrote > > Thank you for volunteering, Gengliang. > > > > Apache Spark 3.2.0 is the first version enabling AQE by default. I'm also > > watching some on-going improvements on that. > > > > https://issues.apache.org/jira/browse/SPARK-33828 (SQL Adaptive > Query > > Execution QA) > > > > To Liang-Chi, I'm -1 for postponing the branch cut because this is a soft > > cut and the committers still are able to commit to `branch-3.3` according > > to their decisions. > > > > Given that Apache Spark had 115 commits in a week in various areas > > concurrently, we should start QA for Apache Spark 3.2 by creating > > branch-3.3 and allowing only limited backporting. > > > > https://github.com/apache/spark/graphs/commit-activity > > > > Bests, > > Dongjoon. > > > > > > On Wed, Jun 16, 2021 at 9:19 AM Liang-Chi Hsieh < > > > viirya@ > > > > wrote: > > > >> First, thanks for being volunteer as the release manager of Spark 3.2.0, > >> Gengliang! > >> > >> And yes, for the two important Structured Streaming features, RocksDB > >> StateStore and session window, we're working on them and expect to have > >> them > >> in the new release. > >> > >> So I propose to postpone the branch cut date. > >> > >> Thank you! > >> > >> Liang-Chi > >> > >> > >> Gengliang Wang-2 wrote > >> > Thanks, Hyukjin. > >> > > >> > The expected target branch cut date of Spark 3.2 is *July 1st* on > >> > https://spark.apache.org/versioning-policy.html. However, I notice > that > >> > there are still multiple important projects in progress now: > >> > > >> > [Core] > >> > > >> > - SPIP: Support push-based shuffle to improve shuffle efficiency > >> > <https://issues.apache.org/jira/browse/SPARK-30602> > >> > > >> > [SQL] > >> > > >> > - Support ANSI SQL INTERVAL types > >> > <https://issues.apache.org/jira/browse/SPARK-27790> > >> > - Support Timestamp without time zone data type > >> > <https://issues.apache.org/jira/browse/SPARK-35662> > >> > - Aggregate (Min/Max/Count) push down for Parquet > >> > <https://issues.apache.org/jira/browse/SPARK-34952> > >> > > >> > [Streaming] > >> > > >> > - EventTime based sessionization (session window) > >> > <https://issues.apache.org/jira/browse/SPARK-10816> > >> > - Add RocksDB StateStore as external module > >> > <https://issues.apache.org/jira/browse/SPARK-34198> > >> > > >> > > >> > I wonder whether we should postpone the branch cut date. > >> > cc Min Shen, Yi Wu, Max Gekk, Huaxin Gao, Jungtaek Lim, Yuanjian > >> > Li, Liang-Chi Hsieh, who work on the projects above. > >> > > >> > On Tue, Jun 15, 2021 at 4:34 PM Hyukjin Kwon < > >> > >> > gurwls223@ > >> > >> > > wrote: > >> > > >> >> +1, thanks. > >> >> > >> >> On Tue, 15 Jun 2021, 16:17 Gengliang Wang, < > >> > >> > ltnwgl@ > >> > >> > > wrote: > >> >> > >> >>> Hi, > >> >>> > >> >>> As the expected release date is close, I would like to volunteer as > >> the > >> >>> release manager for Apache Spark 3.2.0. > >> >>> > >> >>> Thanks, > >> >>> Gengliang > >> >>> > >> > >> > >> > >> > >> > >> -- > >> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ > >> > >> --------------------------------------------------------------------- > >> To unsubscribe e-mail: > > > dev-unsubscribe@.apache > > >> > >> > > > > > > -- > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
