Re: Apache Spark 3.0.2 Release ?

+1 Happy Lunar New Year! Xiao On Fri, Feb 12, 2021 at 5:33 PM Hyukjin Kwon wrote: > Yeah, +1 too > > 2021년 2월 13일 (토) 오전 4:49, Dongjoon Hyun 님이 작성: > >> Thank you, Sean! >> >> On Fri, Feb 12, 2021 at 11:41 AM Sean Owen wrote: >> >>> Sounds like a fine time to me, sure. >>> >>> On Fri, Feb 12,

Re: [VOTE] Release Spark 3.1.1 (RC3)

-1 Could we extend the voting deadline? A few TPC-DS queries (q17, q18, q39a, q39b) are returning different results between Spark 3.0 and Spark 3.1. We need a few more days to understand whether these changes are expected. Xiao Mridul Muralidharan 于2021年2月24日周三 上午10:41写道： > > Sounds good, tha

Re: [VOTE] Release Spark 3.1.1 (RC3)

indeed cause for concern. >>> +1 on extending the voting deadline until we finish investigation of >>> this. >>> >>> Regards, >>> Mridul >>> >>> >>> On Wed, Feb 24, 2021 at 12:55 PM Xiao Li wrote: >>> >>>>

Re: Apache Spark 3.2 Expectation

Thank you, Dongjoon, for initiating this discussion. Let us keep it open. It might take 1-2 weeks to collect from the community all the features we plan to build and ship in 3.2 since we just finished the 3.1 voting. > 3. +100 for Apache Spark 3.2.0 in July 2021. Maybe, we need `branch-cut` > in

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Thank you, Liang-Chi! Xiao On Thu, Mar 4, 2021 at 6:25 PM Hyukjin Kwon wrote: > Thanks @Liang-Chi Hsieh for driving this. > > 2021년 3월 5일 (금) 오전 5:21, Liang-Chi Hsieh 님이 작성: > >> >> Thanks all for the input. >> >> If there is no objection, I am going to cut the branch next Monday. >> >> Thanks

Re: Apache Spark 3.2 Expectation

Below are some nice-to-have features we can work on in Spark 3.2: Lateral Join support , interval data type, timestamp without time zone, un-nesting arbitrary queries, the returned metrics of DSV2, and error message standardization. Spark 3.2 will

Re: [VOTE] SPIP: Support pandas API layer on PySpark

+1 Xiao Takeshi Yamamuro 于2021年3月26日周五 下午4:14写道： > +1 (non-binding) > > On Sat, Mar 27, 2021 at 4:53 AM Liang-Chi Hsieh wrote: > >> +1 (non-binding) >> >> >> rxin wrote >> > +1. Would open up a huge persona for Spark. >> > >> > On Fri, Mar 26 2021 at 11:30 AM, Bryan Cutler < >> >> > cutlerb@ >

Re: Welcoming six new Apache Spark committers

Congratulations, everyone! Xiao Chao Sun 于2021年3月26日周五 下午6:30写道： > Congrats everyone! > > On Fri, Mar 26, 2021 at 6:23 PM Mridul Muralidharan > wrote: > >> >> Congratulations, looking forward to more exciting contributions ! >> >> Regards, >> Mridul >> >> On Fri, Mar 26, 2021 at 8:21 PM Dongjo

Re: Apache Spark 3.1.2 Release?

+1 Thanks, Dongjoon! Xiao On Mon, May 17, 2021 at 8:45 PM Kent Yao wrote: > +1. thanks Dongjoon > > *Kent Yao * > @ Data Science Center, Hangzhou Research Institute, NetEase Corp. > *a spark enthusiast* > *kyuubi is a unified multi-tenant JDBC > interface f

Re: [ANNOUNCE] Apache Spark 3.1.2 released

Thank you! Xiao On Tue, Jun 1, 2021 at 9:29 PM Hyukjin Kwon wrote: > awesome! > > 2021년 6월 2일 (수) 오전 9:59, Dongjoon Hyun 님이 작성: > >> We are happy to announce the availability of Spark 3.1.2! >> >> Spark 3.1.2 is a maintenance release containing stability fixes. This >> release is based on the b

Re: Apache Spark 3.2 Expectation

> > To Liang-Chi, I'm -1 for postponing the branch cut because this is a soft > cut and the committers still are able to commit to `branch-3.3` according > to their decisions. First, I think you are saying "branch-3.2"; Second, the "so cut" means no "code freeze", although we cut the branch. To

Re: Flaky build in GitHub Actions

Thank you, Liang-chi and Hyukjin! On Sun, Jul 25, 2021 at 6:25 PM Hyukjin Kwon wrote: > This is fixed up via Laingchi's PR: > https://github.com/apache/spark/pull/33447. The issue is almost fixed now > and less flaky. > I'm still interacting w/ GitHub Actions: they are still investigating the >

Re: [build system] half of the jenkins workers are down

Thank you, Shane! Xiao shane knapp ☠ 于2021年8月9日周一 下午1:26写道： > turns out that minikube/k8s and friends were being oom-killed and this was > causing all sorts of weirdnesses. > > i've upped the ram limits on all of the k8s jobs to 8G (from 6G), and > we'll keep an eye on things and see how they g

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi, Chao, How long will it take? Normally, in the RC stage, we always revert the upgrade made in the current release. We did the parquet upgrade multiple times in the previous releases for avoiding the major delay in our Spark release Thanks， Xiao On Tue, Aug 31, 2021 at 11:03 AM Chao Sun wro

Re: [VOTE] Release Spark 3.2.0 (RC7)

+1 Xiao Li Yi Wu 于2021年10月11日周一 上午12:08写道： > +1 (non-binding) > > On Mon, Oct 11, 2021 at 1:57 PM Holden Karau wrote: > >> +1 >> >> On Sun, Oct 10, 2021 at 10:46 PM Wenchen Fan wrote: >> >>> +1 >>> >>> On Sat, Oct

Re: [ANNOUNCE] Apache Spark 3.2.0

Thank you, Gengliang! Congrats to our community and all the contributors! Xiao Henrik Peng 于2021年10月19日周二 上午8:26写道： > Congrats and thanks! > > > Gengliang Wang 于2021年10月19日 周二下午10:16写道： > >> Hi all, >> >> Apache Spark 3.2.0 is the third release of the 3.x line. With tremendous >> contribution

Re: [FYI] Build and run tests on Java 17 for Apache Spark 3.3

Thank you! Great job! Xiao On Fri, Nov 12, 2021 at 7:02 PM Mridul Muralidharan wrote: > > Nice job ! > There are some nice API's which should be interesting to explore with JDK > 17 :-) > > Regards. > Mridul > > On Fri, Nov 12, 2021 at 7:08 PM Yuming Wang wrote: > >> Cool, thank you Dongjoon.

Re: [Apache Spark Jenkins] build system shutting down Dec 23th, 2021

Hi, Shane, Thank you for your work on it! Xiao On Mon, Dec 6, 2021 at 6:20 PM L. C. Hsieh wrote: > Thank you, Shane. > > On Mon, Dec 6, 2021 at 4:27 PM Holden Karau wrote: > > > > Shane you kick ass thank you for everything you’ve done for us :) Keep > on rocking :) > > > > On Mon, Dec 6,

Re: [VOTE] SPIP: Catalog API for view metadata

Can we extend the voting window to next Wednesday? This week is a holiday week for the lunar new year. AFAIK, many members in Asia are taking the whole week off. They might not regularly check the emails. Also how about starting a separate email thread starting with [VOTE] ? Happy Lunar New Year!

Re: Apache Spark 3.3 Release

Could you please list which features we want to finish before the branch cut? How long will they take? Xiao Chao Sun 于2022年3月14日周一 13:30写道： > Hi Max, > > As there are still some ongoing work for the above listed SPIPs, can we > still merge them after the branch cut? > > Thanks, > Chao > > On Mo

Re: Apache Spark 3.3 Release

https://github.com/apache/spark/pull/35395 > - https://github.com/apache/spark/pull/35657 > > are actively being reviewed. It seems there are ongoing PRs for other > SPIPs as well but I'm not involved in those so not quite sure whether > they are intended for 3.3 release. > >

Re: Apache Spark 3.3 Release

Let me clarify my above suggestion. Maybe we can wait 3 more days to collect the list of actively developed PRs that we want to merge to 3.3 after the branch cut? Please do not rush to merge the PRs that are not fully reviewed. We can cut the branch this Friday and continue merging the PRs that ha

Re: Apache Spark 3.3 Release

ew minutes ago. So, we can remove > it from the list. > > > > #35819 [SPARK-38524][SPARK-38553][K8S] Bump Volcano to v1.5.1 > > > > Thanks, > > Dongjoon. > > > > On Tue, Mar 15, 2022 at 9:48 AM Xiao Li wrote: > >> > >> Let me clarify my

Re: Apache Spark 3.3 Release

don't cut a branch. > > [SPARK-38335][SQL] Implement parser support for DEFAULT column values > > Let's cut `branch-3.3` Today for Apache Spark 3.3.0 preparation. > > Best, > Dongjoon. > > > On Tue, Mar 15, 2022 at 10:17 AM Chao Sun wrote: > >> Cool, th

Re: Apache Spark 3.3 Release

we need to avoid backporting the feature work that are not being well > discussed. > > > > On Tue, Mar 15, 2022 at 12:12 PM Xiao Li wrote: > >> Cutting the branch is simple, but we need to avoid backporting the >> feature work that are not being well discussed. Not

Re: Apache Spark 3.3 Release

ot blocking what you want to > do. > > Please let the community start to ramp down as we agreed before. > > Dongjoon > > > > On Tue, Mar 15, 2022 at 3:07 PM Xiao Li wrote: > >> Please do not get me wrong. If we don't cut a branch, we are allowing all >&g

Re: SIGMOD System Award for Apache Spark

Congratulations to everyone! Xiao On Fri, May 13, 2022 at 9:34 AM Dongjoon Hyun wrote: > Ya, it's really great!. Congratulations to the whole community! > > Dongjoon. > > On Fri, May 13, 2022 at 8:12 AM Chao Sun wrote: > >> Huge congrats to the whole community! >> >> On Fri, May 13, 2022 at 1:

Re: 回复： [VOTE] Release Spark 3.3.0 (RC6)

+1 Xiao beliefer 于2022年6月13日周一 20:04写道： > +1 AFAIK, no blocking issues now. > Glad to hear to release 3.3.0 ! > > > 在 2022-06-14 09:38:35，"Ruifeng Zheng" 写道： > > +1 (non-binding) > > Maxim, thank you for driving this release! > > thanks, > ruifeng > > > > -- 原始邮件 --

Stickers and Swag

Hi, all, The ASF has an official store at RedBubble that Apache Community Development (ComDev) runs. If you are interested in buying Spark Swag, 70 products featuring the Spark logo are available: https://www.redbubble.com/shop/ap/113203780 Go Spark!

Re: Re: [VOTE][SPIP] Spark Connect

+1 Xiao beliefer 于2022年6月14日周二 03:35写道： > +1 > Yeah, I tried to use Apache Livy, so as we can runing interactive query. > But the Spark Driver in Livy looks heavy. > > The SPIP may resolve the issue. > > > > At 2022-06-14 18:11:21, "Wenchen Fan" wrote: > > +1 > > On Tue, Jun 14, 2022 at 9:38 A

Re: [PSA] Please rebase and sync your master branch in your forked repository

Thank you, Hyukjin! Xiao On Mon, Jun 20, 2022 at 7:01 PM Yi Wu wrote: > Thanks for the work, Hyukjin. > > On Tue, Jun 21, 2022 at 7:59 AM Yuming Wang wrote: > >> Thank you Hyukjin. >> >> On Tue, Jun 21, 2022 at 7:46 AM Hyukjin Kwon wrote: >> >>> After https://github.com/apache/spark/pull/3692

Re: Apache Spark 3.2.2 Release?

+1 Xiao Cheng Su 于2022年7月6日周三 19:16写道： > +1 (non-binding) > > Thanks, > Cheng Su > > On Wed, Jul 6, 2022 at 6:01 PM Yuming Wang wrote: > >> +1 >> >> On Thu, Jul 7, 2022 at 5:53 AM Maxim Gekk >> wrote: >> >>> +1 >>> >>> On Thu, Jul 7, 2022 at 12:26 AM John Zhuge wrote: >>> +1 Thanks for

Welcoming three new PMC members

Hi all, The Spark PMC recently voted to add three new PMC members. Join me in welcoming them to their new roles! New PMC members: Huaxin Gao, Gengliang Wang and Maxim Gekk The Spark PMC

Re: [DISCUSS] SPIP: Support Docker Official Image for Spark

+1 Yikun Jiang 于2022年9月21日周三 07:22写道： > Thanks for all your inputs! BTW, I also create a JIRA to track related > work: https://issues.apache.org/jira/browse/SPARK-40513 > > > can I be involved in this work? > > @qian Of course! Thanks! > > Regards, > Yikun > > On Wed, Sep 21, 2022 at 7:31 PM Xin

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

+1. Xiao On Wed, Oct 5, 2022 at 12:49 PM Sean Owen wrote: > I'm OK with this. It simplifies maintenance a bit, and specifically may > allow us to finally move off of the ancient version of Guava (?) > > On Mon, Oct 3, 2022 at 10:16 PM Dongjoon Hyun > wrote: > >> Hi, All. >> >> I'm wondering if

Re: Welcome Yikun Jiang as a Spark committer

Congratulations, Yikun! Xiao Yikun Jiang 于2022年10月9日周日 19:34写道： > Thank you all! > > Regards, > Yikun > > > On Mon, Oct 10, 2022 at 3:18 AM Chao Sun wrote: > >> Congratulations Yikun! >> >> On Sun, Oct 9, 2022 at 11:14 AM vaquar khan >> wrote: >> >>> Congratulations. >>> >>> Regards, >>> Vaqu

Re: [ANNOUNCE] Apache Spark 3.3.2 released

Thank you, Liang-Chi! Xiao On Sat, Feb 18, 2023 at 1:07 AM beliefer wrote: > Congratulations ! > > > > At 2023-02-17 16:58:22, "L. C. Hsieh" wrote: > >We are happy to announce the availability of Apache Spark 3.3.2! > > > >Spark 3.3.2 is a maintenance release containing stability fixes. This

Re: Slack for PySpark users

+1 + @dev@spark.apache.org This is a good idea. The other Apache projects (e.g., Pinot, Druid, Flink) have created their own dedicated Slack workspaces for faster communication. We can do the same in Apache Spark. The Slack workspace will be maintained by the Apache Spark PMC. I propose to initi

Re: Slack for PySpark users

g official. > > Bests, > Dongjoon. > > > > On Wed, Mar 29, 2023 at 11:32 PM Xiao Li wrote: > >> +1 >> >> + @dev@spark.apache.org >> >> This is a good idea. The other Apache projects (e.g., Pinot, Druid, >> Flink) have created their own dedicate

Re: [VOTE] Release Apache Spark 3.4.0 (RC5)

Hi, Anton, Could you please provide a complete list of exceptions that are being used in the public connector API? Thanks, Xiao Xinrong Meng 于2023年4月5日周三 12:06写道： > Thank you! > > I created a blocker Jira for that for easier tracking: > https://issues.apache.org/jira/browse/SPARK-43041. > > >

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

Thanks for testing it in your environment! > This is a minor issue itself, and only impacts the metrics for push-based > shuffle, but it will essentially completely eliminate the effort > in SPARK-36620. Based on my understanding, this is not a regression. It only affects the new enhancements h

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1 Xiao Li Emil Ejbyfeldt 于2023年4月12日周三 12:39写道： > +1 (non-binding) > > Ran some tests with the Scala 2.13 build using part of our internal > spark workload. > > On 12/04/2023 19:52, Chris Nauroth wrote: > > +1 (non-binding) > > > > * Verified all chec

Re: [ANNOUNCE] Apache Spark 3.4.0 released

Thank you Xinrong! Congratulations everyone! This is a great release with tons of new features! Gengliang Wang 于2023年4月14日周五 13:04写道： > Congratulations everyone! > Thank you Xinrong for driving the release! > > On Fri, Apr 14, 2023 at 12:47 PM Xinrong Meng > wrote: > >> Hi All, >> >> We are

Re: [ANNOUNCE] Apache Spark 3.4.0 released

gt; docker pull apache/spark-r:v3.4.0 > > Thanks, > Dongjoon > > > On Fri, Apr 14, 2023 at 2:56 PM Dongjoon Hyun > wrote: > >> Thank you, Xinrong! >> >> Dongjoon. >> >> >> On Fri, Apr 14, 2023 at 1:37 PM Xiao Li wrote: >> >>>

Re: Apache Spark 3.4.1 Release?

+1 On Fri, Jun 9, 2023 at 08:30 Wenchen Fan wrote: > +1 > > On Fri, Jun 9, 2023 at 8:52 PM Xinrong Meng wrote: > >> +1. Thank you Doonjoon! >> >> Thanks, >> >> Xinrong Meng >> >> Mridul Muralidharan 于2023年6月9日 周五上午5:22写道： >> >>> >>> +1, thanks Dongjoon ! >>> >>> Regards, >>> Mridul >>> >>> On T

Re: [VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

Thanks for starting the vote. I do have a concern about the target release date of Spark 4.0. L. C. Hsieh 于2023年6月12日周一 11:09写道： > +1 > > On Mon, Jun 12, 2023 at 11:06 AM huaxin gao > wrote: > > > > +1 > > > > On Mon, Jun 12, 2023 at 11:05 AM Dongjoon Hyun > wrote: > >> > >> +1 > >> > >> Dong

Re: [VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

;>>> >>>> +1 >>>> >>>> On Mon, Jun 12, 2023 at 12:50 PM kazuyuki tanimura >>>> wrote: >>>> >>>>> +1 (non-binding) >>>>> >>>>> Thank you! >>>>> Kazu >>&

Re: [VOTE][SPIP] Python Data Source API

+1 Xiao Hyukjin Kwon 于2023年7月5日周三 17:28写道： > +1. > > See https://youtu.be/yj7XlTB1Jvc?t=604 :-). > > On Thu, 6 Jul 2023 at 09:15, Allison Wang > wrote: > >> Hi all, >> >> I'd like to start the vote for SPIP: Python Data Source API. >> >> The high-level summary for the SPIP is that it aims to i

Re: Spark Docker Official Image is now available

Thank you, Yikun! This is great! On Wed, Jul 19, 2023 at 7:55 PM Ruifeng Zheng wrote: > Awesome, thank you YiKun for driving this! > > On Thu, Jul 20, 2023 at 9:12 AM Hyukjin Kwon wrote: > >> This is amazing, finally! >> >> On Thu, 20 Jul 2023 at 10:10, Yikun Jiang wrote: >> >>> The spark Dock

Re: [VOTE] SPIP: XML data source support

+1 On Fri, Jul 28, 2023 at 15:54 Sean Owen wrote: > +1 I think that porting the package 'as is' into Spark is probably > worthwhile. > That's relatively easy; the code is already pretty battle-tested and not > that big and even originally came from Spark code, so is more or less > similar alread

Re: Welcome two new Apache Spark committers

Congratulations, Peter and Xiduo! Debasish Das 于2023年8月6日周日 19:08写道： > Congratulations Peter and Xidou. > > On Sun, Aug 6, 2023, 7:05 PM Wenchen Fan wrote: > >> Hi all, >> >> The Spark PMC recently voted to add two new committers. Please join me in >> welcoming them to their new role! >> >> -

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

+1 Xiao Herman van Hovell 于2023年9月6日周三 22:08写道： > Tested connect, and everything looks good. > > +1 > > On Wed, Sep 6, 2023 at 8:11 AM Yuanjian Li wrote: > >> Please vote on releasing the following candidate(RC4) as Apache Spark >> version 3.5.0. >> >> The vote is open until 11:59pm Pacific ti

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 Xiao Yuanjian Li 于2023年9月11日周一 10:53写道： > @Peter Toth I've looked into the details of this > issue, and it appears that it's neither a regression in version 3.5.0 nor a > correctness issue. It's a bug related to a new feature. I think we can fix > this in 3.5.1 and list it as a known issue

Welcome to Our New Apache Spark Committer and PMCs

Hi all, The Spark PMC is delighted to announce that we have voted to add one new committer and two new PMC members. These individuals have consistently contributed to the project and have clearly demonstrated their expertise. New Committer: - Jiaan Geng (focusing on Spark Connect and Spark SQL)

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 huaxin gao 于2023年11月9日周四 16:53写道： > +1 > > On Thu, Nov 9, 2023 at 3:14 PM DB Tsai wrote: > >> +1 >> >> To be completely transparent, I am employed in the same department as >> Zhou at Apple. >> >> I support this proposal, provided that we witness community adoption >> following the release o

Re: [VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

+1 bo yang 于2023年11月15日周三 05:55写道： > +1 > > On Tue, Nov 14, 2023 at 7:18 PM huaxin gao wrote: > >> +1 >> >> On Tue, Nov 14, 2023 at 10:45 AM Holden Karau >> wrote: >> >>> +1 >>> >>> On Tue, Nov 14, 2023 at 10:21 AM DB Tsai wrote: >>> +1 DB Tsai | https://www.dbtsai.com/ | P

Re: Remove HiveContext from Apache Spark 4.0

Thank you for raising it in the dev list. I do not think we should remove HiveContext based on the cost of break and maintenance. FYI, when releasing Spark 3.0, we had a lot of discussions about the related topics https://lists.apache.org/thread/mrx0y078cf3ozs7czykvv864y6dr55xq Dongjoon Hyun 于2

Re: Re: [DISCUSS] Release Spark 3.5.1?

+1 On Sun, Feb 4, 2024 at 6:07 AM beliefer wrote: > +1 > > > > 在 2024-02-04 15:26:13，"Dongjoon Hyun" 写道： > > +1 > > On Sat, Feb 3, 2024 at 9:18 PM yangjie01 > wrote: > >> +1 >> >> 在 2024/2/4 13:13，“Kent Yao”mailto:y...@apache.org>> 写入: >> >> >> +1 >> >> >> Jungtaek Lim > kabhwan.opensou...@gm

Re: [VOTE] Release Apache Spark 3.5.1 (RC2)

+1 Xiao Cheng Pan 于2024年2月20日周二 04:59写道： > +1 (non-binding) > > - Build successfully from source code. > - Pass integration tests with Spark ClickHouse Connector[1] > > [1] https://github.com/housepower/spark-clickhouse-connector/pull/299 > > Thanks, > Cheng Pan > > > > On Feb 20, 2024, at 10:5

Re: [VOTE] Release Apache Spark 2.4.1 (RC8)

Thanks, DB! The Hive UDAF fix https://github.com/apache/spark/commit/0cfefa7e864f443cfd76cff8c50617a8afd080fb was merged this weekend. Xiao DB Tsai 于2019年3月25日周一 下午9:46写道： > RC9 was just cut. Will send out another thread once the build is finished. > > Sincerely, > > DB Tsai >

[VOTE] Release Apache Spark 2.4.3

Please vote on releasing the following candidate as Apache Spark version 2.4.3. The vote is open until May 5th PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.4.3 [ ] -1 Do not release this package because ... To lea

Re: [VOTE] Release Apache Spark 2.4.3

This vote passes! I'll follow up with a formal release announcement soon. +1: Michael Heuer (non-binding) Gengliang Wang (non-binding) Sean Owen (binding) Felix Cheung (binding) Wenchen Fan (binding) Herman van Hovell (binding) Xiao Li (binding) Cheers, Xiao antonkulaga 于2019年5月6日周一 下午2

[ANNOUNCE] Announcing Apache Spark 2.4.3

been possible without you. Xiao Li

Re: Master maven build failing for 6 days -- may need some more eyes

Thanks! Yuming and Gengliang are working on this. On Thu, May 30, 2019 at 8:21 AM Sean Owen wrote: > I might need some help figuring this out. The master Maven build has > been failing for almost a week, and I'm having trouble diagnosing why. > Of course, the PR builder has been fine. > > > Firs

Re: Filter cannot be pushed via a Join

Hi, William, Thanks for reporting it. Could you open a JIRA? Cheers, Xiao William Wong 于2019年6月18日周二 上午8:57写道： > BTW, I noticed a workaround is creating a custom rule to remove 'empty > local relation' from a union table. However, I am not 100% sure if it is > the right approach. > > On Tue,

Re: Jenkins Jobs for Hadoop-3.2 profile

That sounds good to me! @shane knapp Could you help this? Or Dongjoon can do it by himself since he has the access? Cheers, Xiao On Wed, Jun 19, 2019 at 10:56 AM Dongjoon Hyun wrote: > Hi, All. > > So far, we have only `hadoop-2.7` profile jobs. > > - SBT with hadoop-2.7 > - Maven with hado

Re: Jenkins Jobs for Hadoop-3.2 profile

Thank you, Shane!!! Will do it next time. : ) On Wed, Jun 19, 2019 at 3:15 PM shane knapp wrote: > i will do it later this week. also, in the future, please file jiras for > stuff like this rather than pinging me on the list. ;) > > On Wed, Jun 19, 2019 at 1:39 PM Xiao Li wrot

Re: Spark SQL upgrade / migration guide: discoverability and content organization

Yeah, Josh! All these ideas sound good to me. All the top commercial database products have very detailed guide/document about the version upgrading. You can easily find them. Currently, only SQL and ML modules have the migration or upgrade guides. Since Spark 2.3 release, we strictly require the

Re: [SPARK-23207] Repro

Hi, Tyson, Could you open a new JIRA with correctness label? SPARK-23207 might not cover all the scenarios, especially when you using cache. Cheers, Xiao On Fri, Aug 9, 2019 at 9:26 AM wrote: > Hi Sean, > > To finish the job, I did need to set spark.stage.maxConsecutiveAttempts to > a large n

Re: Release Spark 2.3.4

+1 On Fri, Aug 16, 2019 at 4:11 PM Takeshi Yamamuro wrote: > +1, too > > Bests, > Takeshi > > On Sat, Aug 17, 2019 at 7:25 AM Dongjoon Hyun > wrote: > >> +1 for 2.3.4 release as the last release for `branch-2.3` EOL. >> >> Also, +1 for next week release. >> >> Bests, >> Dongjoon. >> >> >> On Fr

Re: JDK11 Support in Apache Spark

Thank you for your contributions! This is a great feature for Spark 3.0! We finally achieve it! Xiao On Sat, Aug 24, 2019 at 12:18 PM Felix Cheung wrote: > That’s great! > > -- > *From:* ☼ R Nair > *Sent:* Saturday, August 24, 2019 10:57:31 AM > *To:* Dongjoon Hyun

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

+1 Xiao Felix Cheung 于2019年8月30日周五 上午2:03写道： > +1 > > Run tests, R tests, r-hub Debian, Ubuntu, mac, Windows > > -- > *From:* Hyukjin Kwon > *Sent:* Wednesday, August 28, 2019 9:14 PM > *To:* Takeshi Yamamuro > *Cc:* dev; Dongjoon Hyun > *Subject:* Re: [VOTE] Releas

Re: maven 3.6.1 removed from apache maven repo

Hi, Tom, To unblock the build, I merged the upgrade to master. https://github.com/apache/spark/pull/25665 Thanks! Xiao On Tue, Sep 3, 2019 at 10:58 AM Tom Graves wrote: > It looks like maven 3.6.1 was removed from the repo - see SPARK-28960. It > looks like they pushed 3.6.2, but I don't s

Re: Welcoming some new committers and PMC members

Congratulations to all of you! Xiao On Mon, Sep 9, 2019 at 5:32 PM Matei Zaharia wrote: > Hi all, > > The Spark PMC recently voted to add several new committers and one PMC > member. Join me in welcoming them to their new roles! > > New PMC member: Dongjoon Hyun > > New committers: Ryan Blue, L

Re: Thoughts on Spark 3 release, or a preview release

https://issues.apache.org/jira/browse/SPARK-28264 SPARK-28264 Revisiting Python / pandas UDF sounds critical for 3.0 preview Xiao On Mon, Sep 16, 2019 at 12:22 PM Erik Erlandson wrote: > > I'm in favor of adding SPARK-25299 > - Use remote stor

Re: [DISCUSS] Spark 2.5 release

+1 on Jungtaek's point. We can revisit this when we release Spark 3.1? After the release of 3.0, I believe we will get more feedback about DSv2 from the community. The current design is just made by a small group of contributors. DSv2 + catalog APIs are still evolving. It is very likely we will mak

Re: [build system] IMPORTANT! northern california fire danger, potential power outage(s)

Hi, Shane, Thank you for letting us know in advance! Xiao On Tue, Oct 8, 2019 at 12:50 PM Shane Knapp wrote: > here in the lovely bay area, we are currently experiencing some > absolutely lovely weather: temps around 20C, light winds, and not a > drop of moisture anywhere. > > this means that

Re: Spark 3.0 preview release feature list and major changes

SPARK-29345 Add an API that allows a user to define and observe arbitrary metrics on streaming queries Let us add this too. Cheers, Xiao On Tue, Oct 8, 2019 at 10:31 PM Wenchen Fan wrote: > Regarding DS v2, I'd like to remove > SPARK-26785 <

Re: [VOTE][SPARK-28885] Follow ANSI store assignment rules in table insertion by default

+1 On Thu, Oct 10, 2019 at 2:13 AM Hyukjin Kwon wrote: > +1 (binding) > > 2019년 10월 10일 (목) 오후 5:11, Takeshi Yamamuro 님이 작성: > >> Thanks for the great work, Gengliang! >> >> +1 for that. >> As I said before, the behaviour is pretty common in DBMSs, so the change >> helps for DMBS users. >> >> Be

Re: Committing while Jenkins down?

I think we are unable to merge any major PR if we do not know whether the tests can pass. Xiao Xiao Li 于2019年10月10日周四 上午8:36写道： > Please check the note from Shane. > > [build system] IMPORTANT! northern california fire danger, potential power > outage(s) > > Thomas graves

Re: Committing while Jenkins down?

Please check the note from Shane. [build system] IMPORTANT! northern california fire danger, potential power outage(s) Thomas graves 于2019年10月10日周四 上午8:35写道： > This is directed towards committers/PMC members. > > It looks like Jenkins will be down for a while, what is everyone's > thoughts on c

Re: Committing while Jenkins down?

Since the outage could be as long as five days I’d rather not just have PRs >> pile up for that entire period. >> > >> > On Thu, Oct 10, 2019 at 8:38 AM Xiao Li wrote: >> >> >> >> I think we are unable to merge any major PR if we do not know wheth

Re: [build system] IMPORTANT! northern california fire danger, potential power outage(s)

That is great news!!! Shane, have a good trip! Xiao On Fri, Oct 11, 2019 at 1:58 PM Shane Knapp wrote: > finally, some good news! power was just restored to campus. > > i'm about to leave town, but jon (CCed) will be heading down to power > things up soon and we should hopefully be building i

Re: SparkGraph review process

> > 1. On the technical side, my main concern is the runtime dependency on > org.opencypher:okapi-shade. okapi depends on several Scala libraries. We > came out with the solution to shade a few Scala libraries to avoid > pollution. However, I'm not super confident that the approach is > sustainable

Add the Google's Code Review Developer Guide as a reference in our code review guide?

Hi, all, Here, I am proposing to add the Google's Code Review Developer Guide as a reference in our code review guide. The guide looks very reasonable to our Spark development too. We do not need to completely follow each rule but it is a good guide

Re: Unable to resolve dependency of sbt-mima-plugin since yesterday

Thank you, Dongjoon! Xiao On Tue, Oct 22, 2019 at 5:08 PM Dongjoon Hyun wrote: > Hi, All. > > This is fixed in master/branch-2.4. > > Bests, > Dongjoon. > > On Tue, Oct 22, 2019 at 12:19 Sean Owen wrote: > >> Weird. Let's discuss at https://issues.apache.org/jira/browse/SPARK-29560 >> >> On Tu

Happy Diwali everyone!!!

Happy Diwali everyone!!! Xiao

Re: Use Hadoop-3.2 as a default Hadoop profile in 3.0.0?

The stability and quality of Hadoop 3.2 profile are unknown. The changes are massive, including Hive execution and a new version of Hive thriftserver. To reduce the risk, I would like to keep the current default version unchanged. When it becomes stable, we can change the default profile to Hadoop

Re: [VOTE] SPARK 3.0.0-preview (RC2)

Spark 3.0 will still use the Hadoop 2.7 profile by default, I think. Hadoop 2.7 profile is much more stable than Hadoop 3.2 profile. On Thu, Oct 31, 2019 at 3:54 PM Sean Owen wrote: > This isn't a big thing, but I see that the pyspark build includes > Hadoop 2.7 rather than 3.2. Maybe later we c

Re: Use Hadoop-3.2 as a default Hadoop profile in 3.0.0?

om HEAD requests before an object was actually created. > > It would be really good if the spark distributions shipped with later > versions of the hadoop artifacts. > > On Mon, Oct 28, 2019 at 7:53 PM Xiao Li wrote: > >> The stability and quality of Hadoop 3.2 prof

Re: Use Hadoop-3.2 as a default Hadoop profile in 3.0.0?

so. > > Bests, > Dongjoon. > > > > On Fri, Nov 1, 2019 at 5:37 PM Jiaxin Shan wrote: > >> +1 for Hadoop 3.2. Seems lots of cloud integration efforts Steve made is >> only available in 3.2. We see lots of users asking for better S3A support >> in Spark. >&g

Re: [build system] Upgrading pyarrow, builds might be temporarily broken

Hi, Bryan, Thank you for your update! Xiao On Thu, Nov 14, 2019 at 8:48 PM Bryan Cutler wrote: > Update: #26133 has been > merged and builds should be passing now, thanks all! > > On Thu, Nov 14, 2019 at 4:12 PM Bryan Cutler wrote: > >> We are in t

Re: [DISCUSS] PostgreSQL dialect

+1 > One particular negative effect has been that new postgresql tests add well > over an hour to tests, Adding postgresql tests is for improving the test coverage of Spark SQL. We should continue to do this by importing more test cases. The quality of Spark highly depends on the test coverage.

Spark 3.0 preview release 2?

I got many great feedbacks from the community about the recent 3.0 preview release. Since the last 3.0 preview release, we already have 353 commits [https://github.com/apache/spark/compare/v3.0.0-preview...master]. There are various important features and behavior changes we want the community to t

Re: Spark 3.0 preview release 2?

er one now. > How about simply moving to a release candidate? If not now then at > least move to code freeze from the start of 2020. There is also some > downside in pushing out the 3.0 release further with previews. > > On Mon, Dec 9, 2019 at 12:32 AM Xiao Li wrote: > > &

Re: I would like to add JDBCDialect to support Vertica database

How can the dev community test it? Xiao On Wed, Dec 11, 2019 at 6:52 AM Sean Owen wrote: > It's probably OK, IMHO. The overhead of another dialect is small. Are > there differences that require a new dialect? I assume so and might > just be useful to summarize them if you open a PR. > > On Tue,

Re: I would like to add JDBCDialect to support Vertica database

ot sure where in the repo that would go. > If automated testing is required, I can ask our engineers whether there > exists something like a mockito that could be included. > > > > Thanks, Bryan H > > > > *From:* Xiao Li [mailto:lix...@databricks.com] > *Sent:* Wedne

Re: Spark 3.0 preview release 2?

;> +1 for another preview >> >> Tom >> >> On Monday, December 9, 2019, 12:32:29 AM CST, Xiao Li < >> gatorsm...@gmail.com> wrote: >> >> >> I got many great feedbacks from the community about the recent 3.0 >> preview release. Since

Re: [VOTE][RESULT] SPARK 3.0.0-preview2 (RC2)

This is the fastest release! Thank you all for making this happen. Happy Holiday! Xiao On Sun, Dec 22, 2019 at 10:58 AM Dongjoon Hyun wrote: > Thank you all. Especially, Yuming as a release manager! > Happy Holidays! > > Cheers, > Dongjoon. > > > On Sun, Dec 22, 2019 at 12:51 AM Yuming Wang w

Re: Spark 3.0 branch cut and code freeze on Jan 31?

Jan 31 is pretty reasonable. Happy Holidays! Xiao On Tue, Dec 24, 2019 at 5:52 AM Sean Owen wrote: > Yep, always happens. Is earlier realistic, like Jan 15? it's all arbitrary > but indeed this has been in progress for a while, and there's a downside to > not releasing it, to making the gap to

  1   2   3   4   >