Hi All,

+1 for the tickets proposed by Ryan Blue

Any possible chance of this one
https://issues.apache.org/jira/browse/SPARK-23406 getting into 2.3.0? It's
a very important feature for us so if it doesn't make the cut I would have
to cherry-pick this commit and compile from the source for our production
release.

Thanks!

On Wed, Feb 21, 2018 at 9:01 AM, Ryan Blue <rb...@netflix.com.invalid>
wrote:

> What does everyone think about getting some of the newer DataSourceV2
> improvements in? It should be low risk because it is a new code path, and
> v2 isn't very usable without things like support for using the output
> commit coordinator to deconflict writes.
>
> The ones I'd like to get in are:
> * Use the output commit coordinator: https://issues.
> apache.org/jira/browse/SPARK-23323
> * Use immutable trees and the same push-down logic as other read paths:
> https://issues.apache.org/jira/browse/SPARK-23203
> * Don't allow users to supply schemas when they aren't supported:
> https://issues.apache.org/jira/browse/SPARK-23418
>
> I think it would make the 2.3.0 release more usable for anyone interested
> in the v2 read and write paths.
>
> Thanks!
>
> On Tue, Feb 20, 2018 at 7:07 PM, Weichen Xu <weichen...@databricks.com>
> wrote:
>
>> +1
>>
>> On Wed, Feb 21, 2018 at 10:07 AM, Marcelo Vanzin <van...@cloudera.com>
>> wrote:
>>
>>> Done, thanks!
>>>
>>> On Tue, Feb 20, 2018 at 6:05 PM, Sameer Agarwal <samee...@apache.org>
>>> wrote:
>>> > Sure, please feel free to backport.
>>> >
>>> > On 20 February 2018 at 18:02, Marcelo Vanzin <van...@cloudera.com>
>>> wrote:
>>> >>
>>> >> Hey Sameer,
>>> >>
>>> >> Mind including https://github.com/apache/spark/pull/20643
>>> >> (SPARK-23468)  in the new RC? It's a minor bug since I've only hit it
>>> >> with older shuffle services, but it's pretty safe.
>>> >>
>>> >> On Tue, Feb 20, 2018 at 5:58 PM, Sameer Agarwal <samee...@apache.org>
>>> >> wrote:
>>> >> > This RC has failed due to
>>> >> > https://issues.apache.org/jira/browse/SPARK-23470.
>>> >> > Now that the fix has been merged in 2.3 (thanks Marcelo!), I'll
>>> follow
>>> >> > up
>>> >> > with an RC5 soon.
>>> >> >
>>> >> > On 20 February 2018 at 16:49, Ryan Blue <rb...@netflix.com> wrote:
>>> >> >>
>>> >> >> +1
>>> >> >>
>>> >> >> Build & tests look fine, checked signature and checksums for src
>>> >> >> tarball.
>>> >> >>
>>> >> >> On Tue, Feb 20, 2018 at 12:54 PM, Shixiong(Ryan) Zhu
>>> >> >> <shixi...@databricks.com> wrote:
>>> >> >>>
>>> >> >>> I'm -1 because of the UI regression
>>> >> >>> https://issues.apache.org/jira/browse/SPARK-23470 : the All Jobs
>>> page
>>> >> >>> may be
>>> >> >>> too slow and cause "read timeout" when there are lots of jobs and
>>> >> >>> stages.
>>> >> >>> This is one of the most important pages because when it's broken,
>>> it's
>>> >> >>> pretty hard to use Spark Web UI.
>>> >> >>>
>>> >> >>>
>>> >> >>> On Tue, Feb 20, 2018 at 4:37 AM, Marco Gaido <
>>> marcogaid...@gmail.com>
>>> >> >>> wrote:
>>> >> >>>>
>>> >> >>>> +1
>>> >> >>>>
>>> >> >>>> 2018-02-20 12:30 GMT+01:00 Hyukjin Kwon <gurwls...@gmail.com>:
>>> >> >>>>>
>>> >> >>>>> +1 too
>>> >> >>>>>
>>> >> >>>>> 2018-02-20 14:41 GMT+09:00 Takuya UESHIN <
>>> ues...@happy-camper.st>:
>>> >> >>>>>>
>>> >> >>>>>> +1
>>> >> >>>>>>
>>> >> >>>>>>
>>> >> >>>>>> On Tue, Feb 20, 2018 at 2:14 PM, Xingbo Jiang
>>> >> >>>>>> <jiangxb1...@gmail.com>
>>> >> >>>>>> wrote:
>>> >> >>>>>>>
>>> >> >>>>>>> +1
>>> >> >>>>>>>
>>> >> >>>>>>>
>>> >> >>>>>>> Wenchen Fan <cloud0...@gmail.com>于2018年2月20日 周二下午1:09写道:
>>> >> >>>>>>>>
>>> >> >>>>>>>> +1
>>> >> >>>>>>>>
>>> >> >>>>>>>> On Tue, Feb 20, 2018 at 12:53 PM, Reynold Xin
>>> >> >>>>>>>> <r...@databricks.com>
>>> >> >>>>>>>> wrote:
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> +1
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> On Feb 20, 2018, 5:51 PM +1300, Sameer Agarwal
>>> >> >>>>>>>>> <sameer.a...@gmail.com>, wrote:
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> this file shouldn't be included?
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> https://dist.apache.org/repos/
>>> dist/dev/spark/v2.3.0-rc4-bin/spark-parent_2.11.iml
>>> >> >>>>>>>>>
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> I've now deleted this file
>>> >> >>>>>>>>>
>>> >> >>>>>>>>>> From: Sameer Agarwal <sameer.a...@gmail.com>
>>> >> >>>>>>>>>> Sent: Saturday, February 17, 2018 1:43:39 PM
>>> >> >>>>>>>>>> To: Sameer Agarwal
>>> >> >>>>>>>>>> Cc: dev
>>> >> >>>>>>>>>> Subject: Re: [VOTE] Spark 2.3.0 (RC4)
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> I'll start with a +1 once again.
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> All blockers reported against RC3 have been resolved and
>>> the
>>> >> >>>>>>>>>> builds are healthy.
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> On 17 February 2018 at 13:41, Sameer Agarwal
>>> >> >>>>>>>>>> <samee...@apache.org>
>>> >> >>>>>>>>>> wrote:
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> Please vote on releasing the following candidate as Apache
>>> >> >>>>>>>>>>> Spark
>>> >> >>>>>>>>>>> version 2.3.0. The vote is open until Thursday February
>>> 22,
>>> >> >>>>>>>>>>> 2018 at 8:00:00
>>> >> >>>>>>>>>>> am UTC and passes if a majority of at least 3 PMC +1
>>> votes are
>>> >> >>>>>>>>>>> cast.
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> [ ] +1 Release this package as Apache Spark 2.3.0
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> [ ] -1 Do not release this package because ...
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> To learn more about Apache Spark, please see
>>> >> >>>>>>>>>>> https://spark.apache.org/
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> The tag to be voted on is v2.3.0-rc4:
>>> >> >>>>>>>>>>> https://github.com/apache/spark/tree/v2.3.0-rc4
>>> >> >>>>>>>>>>> (44095cb65500739695b0324c177c19dfa1471472)
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> List of JIRA tickets resolved in this release can be found
>>> >> >>>>>>>>>>> here:
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> https://issues.apache.org/jira
>>> /projects/SPARK/versions/12339551
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> The release files, including signatures, digests, etc.
>>> can be
>>> >> >>>>>>>>>>> found at:
>>> >> >>>>>>>>>>> https://dist.apache.org/repos/
>>> dist/dev/spark/v2.3.0-rc4-bin/
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> Release artifacts are signed with the following key:
>>> >> >>>>>>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> The staging repository for this release can be found at:
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> https://repository.apache.org/
>>> content/repositories/orgapachespark-1265/
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> The documentation corresponding to this release can be
>>> found
>>> >> >>>>>>>>>>> at:
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> https://dist.apache.org/repos/
>>> dist/dev/spark/v2.3.0-rc4-docs/_site/index.html
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> FAQ
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> =======================================
>>> >> >>>>>>>>>>> What are the unresolved issues targeted for 2.3.0?
>>> >> >>>>>>>>>>> =======================================
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> Please see https://s.apache.org/oXKi. At the time of
>>> writing,
>>> >> >>>>>>>>>>> there are currently no known release blockers.
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> =========================
>>> >> >>>>>>>>>>> How can I help test this release?
>>> >> >>>>>>>>>>> =========================
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> If you are a Spark user, you can help us test this
>>> release by
>>> >> >>>>>>>>>>> taking an existing Spark workload and running on this
>>> release
>>> >> >>>>>>>>>>> candidate,
>>> >> >>>>>>>>>>> then reporting any regressions.
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> If you're working in PySpark you can set up a virtual env
>>> and
>>> >> >>>>>>>>>>> install the current RC and see if anything important
>>> breaks,
>>> >> >>>>>>>>>>> in the
>>> >> >>>>>>>>>>> Java/Scala you can add the staging repository to your
>>> projects
>>> >> >>>>>>>>>>> resolvers and
>>> >> >>>>>>>>>>> test with the RC (make sure to clean up the artifact cache
>>> >> >>>>>>>>>>> before/after so
>>> >> >>>>>>>>>>> you don't end up building with a out of date RC going
>>> >> >>>>>>>>>>> forward).
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> ===========================================
>>> >> >>>>>>>>>>> What should happen to JIRA tickets still targeting 2.3.0?
>>> >> >>>>>>>>>>> ===========================================
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> Committers should look at those and triage. Extremely
>>> >> >>>>>>>>>>> important
>>> >> >>>>>>>>>>> bug fixes, documentation, and API tweaks that impact
>>> >> >>>>>>>>>>> compatibility should be
>>> >> >>>>>>>>>>> worked on immediately. Everything else please retarget to
>>> >> >>>>>>>>>>> 2.3.1 or 2.4.0 as
>>> >> >>>>>>>>>>> appropriate.
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> ===================
>>> >> >>>>>>>>>>> Why is my bug not fixed?
>>> >> >>>>>>>>>>> ===================
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> In order to make timely releases, we will typically not
>>> hold
>>> >> >>>>>>>>>>> the
>>> >> >>>>>>>>>>> release unless the bug in question is a regression from
>>> 2.2.0.
>>> >> >>>>>>>>>>> That being
>>> >> >>>>>>>>>>> said, if there is something which is a regression from
>>> 2.2.0
>>> >> >>>>>>>>>>> and has not
>>> >> >>>>>>>>>>> been correctly targeted please ping me or a committer to
>>> help
>>> >> >>>>>>>>>>> target the
>>> >> >>>>>>>>>>> issue (you can see the open issues listed as impacting
>>> Spark
>>> >> >>>>>>>>>>> 2.3.0 at
>>> >> >>>>>>>>>>> https://s.apache.org/WmoI).
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> --
>>> >> >>>>>>>>>> Sameer Agarwal
>>> >> >>>>>>>>>> Computer Science | UC Berkeley
>>> >> >>>>>>>>>> http://cs.berkeley.edu/~sameerag
>>> >> >>>>>>>>>
>>> >> >>>>>>>>>
>>> >> >>>>>>>>>
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> --
>>> >> >>>>>>>>> Sameer Agarwal
>>> >> >>>>>>>>> Computer Science | UC Berkeley
>>> >> >>>>>>>>> http://cs.berkeley.edu/~sameerag
>>> >> >>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>>
>>> >> >>>>>>
>>> >> >>>>>>
>>> >> >>>>>> --
>>> >> >>>>>> Takuya UESHIN
>>> >> >>>>>> Tokyo, Japan
>>> >> >>>>>>
>>> >> >>>>>> http://twitter.com/ueshin
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>
>>> >> >>>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> Ryan Blue
>>> >> >> Software Engineer
>>> >> >> Netflix
>>> >> >
>>> >> >
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Marcelo
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Reply via email to