There is one item on our side that we want to backport to 3.3: - vectorized DELTA_BYTE_ARRAY/DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support (https://github.com/apache/spark/pull/35262)
It's already reviewed and approved. On Wed, Mar 16, 2022 at 9:13 AM Tom Graves <tgraves...@yahoo.com.invalid> wrote: > > It looks like the version hasn't been updated on master and still shows > 3.3.0-SNAPSHOT, can you please update that. > > Tom > > On Wednesday, March 16, 2022, 01:41:00 AM CDT, Maxim Gekk > <maxim.g...@databricks.com.invalid> wrote: > > > Hi All, > > I have created the branch for Spark 3.3: > https://github.com/apache/spark/commits/branch-3.3 > > Please, backport important fixes to it, and if you have some doubts, ping me > in the PR. Regarding new features, we are still building the allow list for > branch-3.3. > > Best regards, > Max Gekk > > > On Wed, Mar 16, 2022 at 5:51 AM Dongjoon Hyun <dongjoon.h...@gmail.com> wrote: > > Yes, I agree with you for your whitelist approach for backporting. :) > Thank you for summarizing. > > Thanks, > Dongjoon. > > > On Tue, Mar 15, 2022 at 4:20 PM Xiao Li <gatorsm...@gmail.com> wrote: > > I think I finally got your point. What you want to keep unchanged is the > branch cut date of Spark 3.3. Today? or this Friday? This is not a big deal. > > My major concern is whether we should keep merging the feature work or the > dependency upgrade after the branch cut. To make our release time more > predictable, I am suggesting we should finalize the exception PR list first, > instead of merging them in an ad hoc way. In the past, we spent a lot of time > on the revert of the PRs that were merged after the branch cut. I hope we can > minimize unnecessary arguments in this release. Do you agree, Dongjoon? > > > > Dongjoon Hyun <dongjoon.h...@gmail.com> 于2022年3月15日周二 15:55写道： > > That is not totally fine, Xiao. It sounds like you are asking a change of > plan without a proper reason. > > Although we cut the branch Today according our plan, you still can collect > the list and make a list of exceptions. I'm not blocking what you want to do. > > Please let the community start to ramp down as we agreed before. > > Dongjoon > > > > On Tue, Mar 15, 2022 at 3:07 PM Xiao Li <gatorsm...@gmail.com> wrote: > > Please do not get me wrong. If we don't cut a branch, we are allowing all > patches to land Apache Spark 3.3. That is totally fine. After we cut the > branch, we should avoid merging the feature work. In the next three days, let > us collect the actively developed PRs that we want to make an exception > (i.e., merged to 3.3 after the upcoming branch cut). Does that make sense? > > Dongjoon Hyun <dongjoon.h...@gmail.com> 于2022年3月15日周二 14:54写道： > > Xiao. You are working against what you are saying. > If you don't cut a branch, it means you are allowing all patches to land > Apache Spark 3.3. No? > > > we need to avoid backporting the feature work that are not being well > > discussed. > > > > On Tue, Mar 15, 2022 at 12:12 PM Xiao Li <gatorsm...@gmail.com> wrote: > > Cutting the branch is simple, but we need to avoid backporting the feature > work that are not being well discussed. Not all the members are actively > following the dev list. I think we should wait 3 more days for collecting the > PR list before cutting the branch. > > BTW, there are very few 3.4-only feature work that will be affected. > > Xiao > > Dongjoon Hyun <dongjoon.h...@gmail.com> 于2022年3月15日周二 11:49写道： > > Hi, Max, Chao, Xiao, Holden and all. > > I have a different idea. > > Given the situation and small patch list, I don't think we need to postpone > the branch cut for those patches. It's easier to cut a branch-3.3 and allow > backporting. > > As of today, we already have an obvious Apache Spark 3.4 patch in the branch > together. This situation only becomes worse and worse because there is no way > to block the other patches from landing unintentionally if we don't cut a > branch. > > [SPARK-38335][SQL] Implement parser support for DEFAULT column values > > Let's cut `branch-3.3` Today for Apache Spark 3.3.0 preparation. > > Best, > Dongjoon. > > > On Tue, Mar 15, 2022 at 10:17 AM Chao Sun <sunc...@apache.org> wrote: > > Cool, thanks for clarifying! > > On Tue, Mar 15, 2022 at 10:11 AM Xiao Li <gatorsm...@gmail.com> wrote: > >> > >> For the following list: > >> #35789 [SPARK-32268][SQL] Row-level Runtime Filtering > >> #34659 [SPARK-34863][SQL] Support complex types for Parquet vectorized > >> reader > >> #35848 [SPARK-38548][SQL] New SQL function: try_sum > >> Do you mean we should include them, or exclude them from 3.3? > > > > > > If possible, I hope these features can be shipped with Spark 3.3. > > > > > > > > Chao Sun <sunc...@apache.org> 于2022年3月15日周二 10:06写道： > >> > >> Hi Xiao, > >> > >> For the following list: > >> > >> #35789 [SPARK-32268][SQL] Row-level Runtime Filtering > >> #34659 [SPARK-34863][SQL] Support complex types for Parquet vectorized > >> reader > >> #35848 [SPARK-38548][SQL] New SQL function: try_sum > >> > >> Do you mean we should include them, or exclude them from 3.3? > >> > >> Thanks, > >> Chao > >> > >> On Tue, Mar 15, 2022 at 9:56 AM Dongjoon Hyun <dongjoon.h...@gmail.com> > >> wrote: > >> > > >> > The following was tested and merged a few minutes ago. So, we can remove > >> > it from the list. > >> > > >> > #35819 [SPARK-38524][SPARK-38553][K8S] Bump Volcano to v1.5.1 > >> > > >> > Thanks, > >> > Dongjoon. > >> > > >> > On Tue, Mar 15, 2022 at 9:48 AM Xiao Li <gatorsm...@gmail.com> wrote: > >> >> > >> >> Let me clarify my above suggestion. Maybe we can wait 3 more days to > >> >> collect the list of actively developed PRs that we want to merge to 3.3 > >> >> after the branch cut? > >> >> > >> >> Please do not rush to merge the PRs that are not fully reviewed. We can > >> >> cut the branch this Friday and continue merging the PRs that have been > >> >> discussed in this thread. Does that make sense? > >> >> > >> >> Xiao > >> >> > >> >> > >> >> > >> >> Holden Karau <hol...@pigscanfly.ca> 于2022年3月15日周二 09:10写道： > >> >>> > >> >>> May I suggest we push out one week (22nd) just to give everyone a bit > >> >>> of breathing space? Rushed software development more often results in > >> >>> bugs. > >> >>> > >> >>> On Tue, Mar 15, 2022 at 6:23 AM Yikun Jiang <yikunk...@gmail.com> > >> >>> wrote: > >> >>>> > >> >>>> > To make our release time more predictable, let us collect the PRs > >> >>>> > and wait three more days before the branch cut? > >> >>>> > >> >>>> For SPIP: Support Customized Kubernetes Schedulers: > >> >>>> #35819 [SPARK-38524][SPARK-38553][K8S] Bump Volcano to v1.5.1 > >> >>>> > >> >>>> Three more days are OK for this from my view. > >> >>>> > >> >>>> Regards, > >> >>>> Yikun > >> >>> > >> >>> -- > >> >>> Twitter: https://twitter.com/holdenkarau > >> >>> Books (Learning Spark, High Performance Spark, etc.): > >> >>> https://amzn.to/2MaRAG9 > >> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
