It's great. Thank you, Ruifeng!
Bests,
Dongjoon.
On Fri, Sep 11, 2020 at 1:54 AM 郑瑞峰 wrote:
> Hi all,
>
> We are happy to announce the availability of Spark 3.0.1!
> Spark 3.0.1 is a maintenance release containing stability fixes. This
> release is based on the branch-3.0 maintenance branch of
Hi, All.
As of today, master branch (Apache Spark 3.1.0) resolved
852+ JIRA issues and 606+ issues are 3.1.0-only patches.
According to the 3.1.0 release window, branch-3.1 will be
created on November 1st and enters QA period.
Here are some notable updates I've been monitoring.
*Language*
01. SP
create table syntax:
>>https://issues.apache.org/jira/browse/SPARK-31257
>>- Bloom filter join: https://issues.apache.org/jira/browse/SPARK-32268
>>
>> Thanks,
>>
>> Xiao
>>
>>
>> Hyukjin Kwon 于2020年10月3日周六 下午5:41写道：
>>
>>>
, Oct 4, 2020 at 10:53 AM Dongjoon Hyun
wrote:
> Thank you all.
>
> BTW, Xiao and Mridul, I'm wondering what date you have in your mind
> specifically.
>
> Usually, `Christmas and New Year season` doesn't give us much additional
> time.
>
> If you think so, co
(using older spark version to extract
> out of hive, then switch to newer spark version) so i am not too worried
> about this. just making sure i understand.
>
> thanks
>
> On Sat, Oct 3, 2020 at 8:17 PM Dongjoon Hyun
> wrote:
>
>> Hi, All.
>>
>> As of today,
Hi, All.
Apache Spark 3.1.0 Release Window is adjusted like the following today.
Please check the latest information on the official website.
-
https://github.com/apache/spark-website/commit/0cd0bdc80503882b4737db7e77cc8f9d17ec12ca
- https://spark.apache.org/versioning-policy.html
Bests,
.
Dongjoon Hyun
It took a long time. Thank you, Hyukjin and all!
Bests,
Dongjoon.
On Wed, Mar 3, 2021 at 3:23 AM Gabor Somogyi
wrote:
> Good to hear and great work Hyukjin! 👏
>
> On Wed, 3 Mar 2021, 11:15 Jungtaek Lim,
> wrote:
>
>> Thanks Hyukjin for driving the huge release, and thanks everyone for
>> contr
.
Dongjoon Hyun
Hi, Stephen and Steve.
Apache Spark community starts to publish it as a snapshot and Apache Spark
3.2.0 will be the first release has it.
-
https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-hadoop-cloud_2.12/3.2.0-SNAPSHOT/
Please check the snapshot artifacts and fi
Thank you, Yi!
On Thu, Jun 24, 2021 at 10:52 PM Yi Wu wrote:
> We are happy to announce the availability of Spark 3.0.3!
>
> Spark 3.0.3 is a maintenance release containing stability fixes. This
> release is based on the branch-3.0 maintenance branch of Spark. We strongly
> recommend all 3.0 u
Thank you so much, Gengliang and all!
Dongjoon.
On Tue, Oct 19, 2021 at 8:48 AM Xiao Li wrote:
> Thank you, Gengliang!
>
> Congrats to our community and all the contributors!
>
> Xiao
>
> Henrik Peng 于2021年10月19日周二 上午8:26写道：
>
>> Congrats and thanks!
>>
>>
>> Gengliang Wang 于2021年10月19日 周二下午10
Thank you again, Huaxin!
Dongjoon.
On Fri, Jan 28, 2022 at 6:23 PM DB Tsai wrote:
> Thank you, Huaxin for the 3.2.1 release!
>
> Sent from my iPhone
>
> On Jan 28, 2022, at 5:45 PM, Chao Sun wrote:
>
>
> Thanks Huaxin for driving the release!
>
> On Fri, Jan 28, 2022 at 5:37 PM Ruifeng Zheng
.
Dongjoon Hyun
It's great. Thank you so much, Yuming!
Dongjoon
On Tue, Oct 25, 2022 at 11:23 PM Yuming Wang wrote:
> We are happy to announce the availability of Apache Spark 3.3.1!
>
> Spark 3.3.1 is a maintenance release containing stability fixes. This
> release is based on the branch-3.3 maintenance branc
Thank you, Chao!
On Wed, Nov 30, 2022 at 8:16 AM Yang,Jie(INF) wrote:
> Thanks, Chao!
>
>
>
> *发件人**: *Maxim Gekk
> *日期**: *2022年11月30日 星期三 19:40
> *收件人**: *Jungtaek Lim
> *抄送**: *Wenchen Fan , Chao Sun ,
> dev , user
> *主题**: *Re: [ANNOUNCE] Apache Spark 3.2.3 released
>
>
>
> Thank you, Cha
Thank you for considering me, but may I ask what makes you think to put me
there, Mich? I'm curious about your reason.
> I have put dongjoon.hyun as a shepherd.
BTW, unfortunately, I cannot help you with that due to my on-going personal
stuff. I'll adjust the JIRA first.
Thanks,
Dongjoon.
On S
Hi, Xiao and all.
(cc Matei)
Please hold on the vote.
There is a concern expressed by ASF board because recent Slack activities
created an isolated silo outside of ASF mailing list archive.
We need to establish a way to embrace it back to ASF archive before
starting anything official.
Bests,
D
gt; open source communities. TBH, we are kind of late. I think we can do the
> same in our community?
>
> We can follow the guide when the ASF has an official process for ASF
> archiving. Since our PMC are the owner of the slack workspace, we can make
> a change based on the policy. W
don't think that matters. However, I stand
>>>>>> corrected
>>>>>> - To be clear, I intentionally didn't refer to any specific mailing
>>>>>> list because we didn't set up any rule here yet.
>>>>>> fair enough
rom relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Mon, 3 Apr 2023 at 20:59, Dongjoon Hyun
> wrote:
>
>> As Mich T
ASF)
>>> have preference. They are going with the way they are convenient.
>>>
>>> Same applies here - if ASF Slack requires a restricted invitation
>>> mechanism then it won't work. Looks like there is a link for an invitation,
>>> but we are also
to keep this
> active.
>
>
>
> On Mon, Apr 3, 2023 at 16:46 Dongjoon Hyun
> wrote:
>
>> Shall we summarize the discussion so far?
>>
>> To sum up, "ASF Slack" vs "3rd-party Slack" was the real background to
>> initiate this thread instead
.
Dongjoon Hyun
.
Dongjoon Hyun
not have been possible without you.
Dongjoon Hyun
Hi, All.
As a part of Apache Spark 4.0.0 (SPARK-44111), the Apache Spark community
starts to have test coverage for all supported Python versions from Today.
- https://github.com/apache/spark/actions/runs/7061665420
Here is a summary.
1. Main CI: All PRs and commits on `master` branch are teste
would not have been possible without you.
Dongjoon Hyun
Congratulations!
Bests,
Dongjoon.
On Wed, Feb 28, 2024 at 11:43 AM beliefer wrote:
> Congratulations!
>
>
>
> At 2024-02-28 17:43:25, "Jungtaek Lim"
> wrote:
>
> Hi everyone,
>
> We are happy to announce the availability of Spark 3.5.1!
>
> Spark 3.5.1 is a maintenance release containing stabi
master
[image: Screenshot 2024-02-29 at 21.12.24.png]
Could you do the follow-up, please?
Thank you in advance.
Dongjoon.
On Thu, Feb 29, 2024 at 2:48 PM John Zhuge wrote:
> Excellent work, congratulations!
>
> On Wed, Feb 28, 2024 at 10:12 PM Dongjoon Hyun
> wrote:
>
>> C
Hi, Cheng.
Thank you for the suggestion. Your suggestion seems to have at least two
themes.
A. Adding a new Apache Spark community policy (contract) to guarantee MySQL
LTS Versions Support.
B. Dropping the support of non-LTS version support (MySQL 8.3/8.2/8.1)
And, it brings me three questions.
not have been possible without you.
Dongjoon Hyun
+dev
I forget to add @user.
Dongjoon.
-- Forwarded message -
From: Dongjoon Hyun
Date: Thu, Dec 8, 2016 at 16:00
Subject: Question about SPARK-11374 (skip.header.line.count)
To:
Hi, All.
Could you give me some opinion?
There is an old SPARK issue, SPARK-11374, about
Did you follow the guide in `IDE Setup` -> `IntelliJ` section of
http://spark.apache.org/developer-tools.html ?
Bests,
Dongjoon.
On Wed, Jun 28, 2017 at 5:13 PM, satyajit vegesna <
satyajit.apas...@gmail.com> wrote:
> Hi All,
>
> When i try to build source code of apache spark code from
> https:
Hi, All.
Today, Apache Spark starts to use Apache ORC 1.4 as a `native` ORC
implementation.
SPARK-20728 Make OrcFileFormat configurable between `sql/hive` and
`sql/core`.
-
https://github.com/apache/spark/commit/326f1d6728a7734c228d8bfaa69442a1c7b92e9b
Thank you so much for all your supports for
Hi, All.
Vectorized ORC Reader is now supported in Apache Spark 2.3.
https://issues.apache.org/jira/browse/SPARK-16060
It has been a long journey. From now, Spark can read ORC files faster
without feature penalty.
Thank you for all your support, especially Wenchen Fan.
It's done by two com
gt;
> Will this affect both:
> 1) spark.read.format("orc").load("...")
> 2) spark.sql("select ... from my_orc_table_in_hive")
>
> ?
>
>
> Le 10 janv. 2018 à 20:14, Dongjoon Hyun écrivait :
> > Hi, All.
> >
> > Vectorized ORC Reader is now
Hi, Eric.
For me, Spark 2.3 works correctly like the following. Could you give us some
reproducible example?
```
scala> sql("set spark.sql.orc.impl=native")
scala> sql("set spark.sql.orc.compression.codec=zlib")
res1: org.apache.spark.sql.DataFrame = [key: string, value: string]
scala> spark.r
You may hit SPARK-23355 (convertMetastore should not ignore table properties).
Since it's a known Spark issue for all Hive tables (Parquet/ORC), could you
check that too?
Bests,
Dongjoon.
On 2018/03/28 01:00:55, Dongjoon Hyun wrote:
> Hi, Eric.
>
> For me, Spark 2.3 works corr
Hi, Jerry.
There is a JIRA issue for that,
https://issues.apache.org/jira/browse/SPARK-24360 .
So far, it's in progress for Hive 3.1.0 Metastore for Apache Spark 2.5.0.
You can track that issue there.
Bests,
Dongjoon.
On Mon, Sep 17, 2018 at 7:01 PM 白也诗无敌 <445484...@qq.com> wrote:
> Hi, guys
Finally, thank you all. Especially, thanks to the release manager, Wenchen!
Bests,
Dongjoon.
On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan wrote:
> + user list
>
> On Fri, Nov 9, 2018 at 2:20 AM Wenchen Fan wrote:
>
>> resend
>>
>> On Thu, Nov 8, 2018 at 11:02 PM Wenchen Fan wrote:
>>
>>>
>>>
Hi, All.
The vote passes. Thanks to all who helped with this release 2.2.3 (the
final 2.2.x)!
I'll follow up later with a release announcement once everything is
published.
+1 (* = binding):
DB Tsai*
Wenchen Fan*
Dongjoon Hyun
Denny Lee
Sean Owen*
Hyukjin Kwon
John Zhuge
+0: None
-1:
We are happy to announce the availability of Spark 2.2.3!
Apache Spark 2.2.3 is a maintenance release, based on the branch-2.2
maintenance branch of Spark. We strongly recommend all 2.2.x users to
upgrade to this stable release.
To download Spark 2.2.3, head over to the download page:
http://spar
Hi, All.
Since we use both Apache JIRA and GitHub actively for Apache Spark
contributions, we have lots of JIRAs and PRs consequently. One specific
thing I've been longing to see is `Jira Issue Type` in GitHub.
How about exposing JIRA issue types at GitHub PRs as GitHub `Labels`? There
are two ma
it may be updated later: so keeping them in sync may be
> an extra effort..
>
> On Thu, 13 Jun 2019, 08:09 Reynold Xin, wrote:
>
>> Seems like a good idea. Can we test this with a component first?
>>
>> On Thu, Jun 13, 2019 at 6:17 AM Dongjoon Hyun
>> wrote:
>&
Hi, All.
JIRA and PR is ready for reviews.
https://issues.apache.org/jira/browse/SPARK-28051 (Exposing JIRA issue
component types at GitHub PRs)
https://github.com/apache/spark/pull/24871
Bests,
Dongjoon.
On Thu, Jun 13, 2019 at 10:48 AM Dongjoon Hyun
wrote:
> Thank you for the feedba
Now, you can see the exposed component labels (ordered by the number of
PRs) here and click the component to search.
https://github.com/apache/spark/labels?sort=count-desc
Dongjoon.
On Fri, Jun 14, 2019 at 1:15 AM Dongjoon Hyun
wrote:
> Hi, All.
>
> JIRA and PR is ready fo
Thank you, Hyukjin !
On Sun, Jun 16, 2019 at 4:12 PM Hyukjin Kwon wrote:
> Labels look good and useful.
>
> On Sat, 15 Jun 2019, 02:36 Dongjoon Hyun, wrote:
>
>> Now, you can see the exposed component labels (ordered by the number of
>> PRs) here and click
Hi, All.
Spark 2.4.3 was released two months ago (8th May).
As of today (9th July), there exist 45 fixes in `branch-2.4` including the
following correctness or blocker issues.
- SPARK-26038 Decimal toScalaBigInt/toJavaBigInteger not work for
decimals not fitting in long
- SPARK-26045 Err
d to happen before 3.0, but could. Usually maintenance
> releases happen 3-4 months apart and the last one was 2 months ago. If
> these are significant issues, sure. It'll probably be August before
> it's out anyway.
>
> On Tue, Jul 9, 2019 at 11:15 AM Dongjoon Hyun
> wrote
Additionally, one more correctness patch landed yesterday.
- SPARK-28015 Check stringToDate() consumes entire input for the
and -[m]m formats
Bests,
Dongjoon.
On Tue, Jul 9, 2019 at 10:11 AM Dongjoon Hyun
wrote:
> Thank you for the reply, Sean. Sure. 2.4.x should be a
s (if we are on
schedule).
- 2.4.4 at the end of July
- 2.3.4 at the end of August (since 2.3.0 was released at the end of
February 2018)
- 3.0.0 (possibily September?)
- 3.1.0 (January 2020?)
Bests,
Dongjoon.
On Thu, Jul 11, 2019 at 1:30 PM Jacek Laskowski wrote:
> Hi,
>
> Thanks Don
Hi, Apache Spark PMC members.
Can we cut Apache Spark 2.4.4 next Monday (22nd July)?
Bests,
Dongjoon.
On Fri, Jul 12, 2019 at 3:18 PM Dongjoon Hyun
wrote:
> Thank you, Jacek.
>
> BTW, I added `@private` since we need PMC's help to make an Apache Spark
> release.
>
> C
Thank you Dongjoon for being a release manager.
>
> If the assumed dates are ok, I would like to volunteer for an 2.3.4
> release manager.
>
> Best Regards,
> Kazuaki Ishizaki,
>
>
>
> From:Dongjoon Hyun
> To:dev , "user @spark" <
> use
Hi, All.
Spark 2.4.3 was released three months ago (8th May).
As of today (13th August), there are 112 commits (75 JIRAs) in `branch-24`
since 2.4.3.
It would be great if we can have Spark 2.4.4.
Shall we start `2.4.4 RC1` next Monday (19th August)?
Last time, there was a request for K8s issue a
t;>
>>>>> On Tue, Aug 13, 2019 at 5:22 PM Sean Owen wrote:
>>>>>
>>>>>> Seems fine to me if there are enough valuable fixes to justify another
>>>>>> release. If there are any other important fixes imminent, it&
Hi, All.
Thanks to your many many contributions,
Apache Spark master branch starts to pass on JDK11 as of today.
(with `hadoop-3.2` profile: Apache Hadoop 3.2 and Hive 2.3.6)
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11
Hi, All.
The vote passes. Thanks to all who helped with this release 2.4.4!
It was very intensive vote with +11 (including +8 PMC votes) and no -1.
I'll follow up later with a release announcement once everything is
published.
+1 (* = binding):
Dongjoon Hyun
Kazuaki Ishizaki
Sean Owen*
We
all community members for contributing to this
release. This release would not have been possible without you.
Dongjoon Hyun
+1 for Apache ORC 1.4.5 release.
Thank you for making the release.
I'd like to mention some notable changes here.
Apache ORC 1.4.5 is not a drop-in replacement for 1.4.4 because of the
following.
ORC-498: ReaderImpl and RecordReaderImpl open separate file handles.
Applications should be u
Indeed! Thank you again, Yuming and all.
Bests,
Dongjoon.
On Tue, Dec 24, 2019 at 13:38 Takeshi Yamamuro
wrote:
> Great work, Yuming!
>
> Bests,
> Takeshi
>
> On Wed, Dec 25, 2019 at 6:00 AM Xiao Li wrote:
>
>> Thank you all. Happy Holidays!
>>
>> Xiao
>>
>> On Tue, Dec 24, 2019 at 12:53 PM Y
all community members for contributing to this
release. This release would not have been possible without you.
Dongjoon Hyun
There was a typo in one URL. The correct release note URL is here.
https://spark.apache.org/releases/spark-release-2-4-5.html
On Sat, Feb 8, 2020 at 5:22 PM Dongjoon Hyun
wrote:
> We are happy to announce the availability of Spark 2.4.5!
>
> Spark 2.4.5 is a maintenance release c
Hi, All.
Apache Spark has been suffered from a known consistency issue on `CHAR`
type behavior among its usages and configurations. However, the evolution
direction has been gradually moving forward to be consistent inside Apache
Spark because we don't have `CHAR` offically. The following is the s
d code that was working for char(3) would now stop
> working.
>
> For new users, depending on whether the underlying metastore char(3) is
> either supported but different from ansi Sql (which is not that big of a
> deal if we explain it) or not supported.
>
> On Sat, Mar 14, 2020
ar 14, 2020 at 17:54 Reynold Xin wrote:
>>>
>>>> I don’t understand this change. Wouldn’t this “ban” confuse the hell
>>>> out of both new and old users?
>>>>
>>>> For old users, their old code that was working for char(3) would now
&
t it to work consistently everywhere.
>
>
> Cheers,
>
> Steve C
>
> On 17 Mar 2020, at 10:01 am, Dongjoon Hyun
> wrote:
>
> Hi, Reynold.
> (And +Michael Armbrust)
>
> If you think so, do you think it's okay that we change the return value
> silently? Then, I&
ar 16, 2020 at 5:29 PM, Reynold Xin wrote:
>
>> I looked up our usage logs (sorry I can't share this publicly) and trim
>> has at least four orders of magnitude higher usage than char.
>>
>>
>> On Mon, Mar 16, 2020 at 5:27 PM, Dongjoon Hyun
>> wrote:
Thank you so much, Holden! :)
On Wed, Jun 10, 2020 at 6:59 PM Hyukjin Kwon wrote:
> Yay!
>
> 2020년 6월 11일 (목) 오전 10:38, Holden Karau 님이 작성:
>
>> We are happy to announce the availability of Spark 2.4.6!
>>
>> Spark 2.4.6 is a maintenance release containing stability, correctness,
>> and security
would not have been possible without you.
Dongjoon Hyun
thout you.
To download Spark 4.0.0-preview2, head over to the download page:
https://archive.apache.org/dist/spark/spark-4.0.0-preview2
To view the documentation:
https://spark.apache.org/docs/preview/
Dongjoon Hyun
Hi All.
We are happy to announce the availability of Apache Spark Connect Swift
Client 0.1.0!
Release Note
- https://github.com/apache/spark-connect-swift/releases/tag/v0.1.0
- https://s.apache.org/spark-connect-swift-0.1.0
Swift Package Index
- https://swiftpackageindex.com/apache/spark-c
Hi All.
We are happy to announce the availability of Apache Spark Kubernetes
Operator 0.1.0!
- Release Note:
* https://github.com/apache/spark-kubernetes-operator/releases/tag/v0.1.0
* https://s.apache.org/spark-kubernetes-operator-0.1.0
- Published Docker Image:
* apache/spark-kubernetes-o
have been possible without you.
Dongjoon Hyun
Hi All.
We are happy to announce the availability of Apache Spark Connect Swift
Client 0.2.0!
Website
- https://apache.github.io/spark-connect-swift/
Release Note
- https://github.com/apache/spark-connect-swift/releases/tag/0.2.0
- https://s.apache.org/spark-connect-swift-0.2.0
Swift Pack
Hi All.
We are happy to announce the availability of Apache Spark Kubernetes
Operator 0.2.0!
- Website
* https://s.apache.org/spark-kubernetes-operator/
- Artifact Hub
*
https://artifacthub.io/packages/helm/spark-kubernetes-operator/spark-kubernetes-operator/
- Release Note
* https://gith
Hi All.
We are happy to announce the availability of Apache Spark Connect Swift
Client 0.3.0!
This is the first release tested with the official Apache Spark 4.0.0.
Website
- https://apache.github.io/spark-connect-swift/
Release Note
- https://github.com/apache/spark-connect-swift/releases/t
Hi All.
We are happy to announce the availability of Apache Spark Kubernetes
Operator 0.3.0!
- Notable Changes
* Built and tested with Apache Spark 4.0 and Spark Connect Swift Client
* Running on Java 24
* Promoting CRDs to v1beta1 from v1alpha1
- Website
* https://s.apache.org/spark-kub
Hi All.
We are happy to announce the availability of Apache Spark Kubernetes
Operator 0.4.0!
- Website
* https://s.apache.org/spark-kubernetes-operator/
- Artifact Hub
*
https://artifacthub.io/packages/helm/spark-kubernetes-operator/spark-kubernetes-operator/
- Release Note
* https://gith
79 matches
Mail list logo
