i deployed 2.4.0 RC3 on our dev cluster and ran into issue with spark shell and jline. there is probably a simple workaround so this is not a serious issue but just wanted to let you know.
https://issues.apache.org/jira/browse/SPARK-25783 On Mon, Oct 15, 2018 at 4:59 PM Imran Rashid <iras...@cloudera.com.invalid> wrote: > I just discovered https://issues.apache.org/jira/browse/SPARK-25738 with > some more testing. I only marked it as critical, but seems pretty bad -- > I'll defer to others opinion > > On Sat, Oct 13, 2018 at 4:15 PM Dongjoon Hyun <dongjoon.h...@gmail.com> > wrote: > >> Yes. From my side, it's -1 for RC3. >> >> Bests, >> Dongjoon. >> >> On Sat, Oct 13, 2018 at 1:24 PM Holden Karau <hol...@pigscanfly.ca> >> wrote: >> >>> So if it's a blocker would you think this should be a -1? >>> >>> On Fri, Oct 12, 2018 at 3:52 PM Dongjoon Hyun <dongjoon.h...@gmail.com> >>> wrote: >>> >>>> Hi, Holden. >>>> >>>> Since that's a performance at 2.4.0, I marked as `Blocker` four days >>>> ago. >>>> >>>> Bests, >>>> Dongjoon. >>>> >>>> >>>> On Fri, Oct 12, 2018 at 11:45 AM Holden Karau <hol...@pigscanfly.ca> >>>> wrote: >>>> >>>>> Following up I just wanted to make sure this new blocker that Dongjoon >>>>> designated is surfaced - >>>>> https://jira.apache.org/jira/browse/SPARK-25579?filter=12340409&jql=affectedVersion%20%3D%202.4.0%20AND%20cf%5B12310320%5D%20is%20EMPTY%20AND%20project%20%3D%20spark%20AND%20(status%20%3D%20%22In%20Progress%22%20OR%20resolution%20%3D%20Unresolved)%20AND%20priority%20%3D%20Blocker >>>>> >>>>> On Thu, Oct 11, 2018 at 2:05 PM Xiao Li <gatorsm...@gmail.com> wrote: >>>>> >>>>>> -1. We have two correctness bugs: >>>>>> https://issues.apache.org/jira/browse/SPARK-25714 and >>>>>> https://issues.apache.org/jira/browse/SPARK-25708. >>>>>> >>>>>> Let us fix all the three issues in ScalaUDF, as mentioned by Sean. >>>>>> >>>>>> Xiao >>>>>> >>>>>> >>>>>> Sean Owen <sro...@apache.org> 于2018年10月11日周四 上午9:04写道： >>>>>> >>>>>>> This is a legitimate question about the behavior of ScalaUDF after >>>>>>> the >>>>>>> change to support 2.12: >>>>>>> https://github.com/apache/spark/pull/22259#discussion_r224295469 >>>>>>> Not quite a blocker I think, but a potential gotcha we definitely >>>>>>> need >>>>>>> to highlight in release notes. There may be an argument for changing >>>>>>> ScalaUDF again before the release. Have a look, anyone familiar with >>>>>>> catalyst. >>>>>>> On Wed, Oct 10, 2018 at 3:00 PM Sean Owen <sro...@apache.org> wrote: >>>>>>> > >>>>>>> > +1. I tested the source build against Scala 2.12 and common build >>>>>>> > profiles. License and sigs look OK. >>>>>>> > >>>>>>> > No blockers; one critical: >>>>>>> > >>>>>>> > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4 >>>>>>> > >>>>>>> > I think this one is "won't fix" though? not trying to restore the >>>>>>> behavior? >>>>>>> > >>>>>>> > Other items open for 2.4.0: >>>>>>> > >>>>>>> > SPARK-25347 Document image data source in doc site >>>>>>> > SPARK-25584 Document libsvm data source in doc site >>>>>>> > SPARK-25179 Document the features that require Pyarrow 0.10 >>>>>>> > SPARK-25507 Update documents for the new features in 2.4 release >>>>>>> > SPARK-25346 Document Spark builtin data sources >>>>>>> > SPARK-24464 Unit tests for MLlib's Instrumentation >>>>>>> > SPARK-23197 Flaky test: >>>>>>> spark.streaming.ReceiverSuite."receiver_life_cycle" >>>>>>> > SPARK-22809 pyspark is sensitive to imports with dots >>>>>>> > SPARK-21030 extend hint syntax to support any expression for >>>>>>> Python and R >>>>>>> > >>>>>>> > Anyone know enough to close or retarget them? they don't look >>>>>>> critical >>>>>>> > for 2.4, SPARK-25507 has no content, itself. SPARK-25179 "Document >>>>>>> the >>>>>>> > features that require Pyarrow 0.10" however sounds like it could >>>>>>> have >>>>>>> > been important for 2.4? if not a blocker. >>>>>>> > >>>>>>> > PS I don't think that SPARK-25150 is an issue; see JIRA. At least >>>>>>> > there is some ongoing discussion there. >>>>>>> > >>>>>>> > I am evaluating >>>>>>> > https://github.com/apache/spark/pull/22259#discussion_r224252642 >>>>>>> right >>>>>>> > now. >>>>>>> > >>>>>>> > >>>>>>> > On Wed, Oct 10, 2018 at 9:47 AM Wenchen Fan <cloud0...@gmail.com> >>>>>>> wrote: >>>>>>> > > >>>>>>> > > Please vote on releasing the following candidate as Apache Spark >>>>>>> version 2.4.0. >>>>>>> > > >>>>>>> > > The vote is open until October 1 PST and passes if a majority +1 >>>>>>> PMC votes are cast, with >>>>>>> > > a minimum of 3 +1 votes. >>>>>>> > > >>>>>>> > > [ ] +1 Release this package as Apache Spark 2.4.0 >>>>>>> > > [ ] -1 Do not release this package because ... >>>>>>> > > >>>>>>> > > To learn more about Apache Spark, please see >>>>>>> http://spark.apache.org/ >>>>>>> > > >>>>>>> > > The tag to be voted on is v2.4.0-rc3 (commit >>>>>>> 8e4a99bd201b9204fec52580f19ae70a229ed94e): >>>>>>> > > https://github.com/apache/spark/tree/v2.4.0-rc3 >>>>>>> > > >>>>>>> > > The release files, including signatures, digests, etc. can be >>>>>>> found at: >>>>>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-bin/ >>>>>>> > > >>>>>>> > > Signatures used for Spark RCs can be found in this file: >>>>>>> > > https://dist.apache.org/repos/dist/dev/spark/KEYS >>>>>>> > > >>>>>>> > > The staging repository for this release can be found at: >>>>>>> > > >>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1289 >>>>>>> > > >>>>>>> > > The documentation corresponding to this release can be found at: >>>>>>> > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/ >>>>>>> > > >>>>>>> > > The list of bug fixes going into 2.4.0 can be found at the >>>>>>> following URL: >>>>>>> > > https://issues.apache.org/jira/projects/SPARK/versions/12342385 >>>>>>> > > >>>>>>> > > FAQ >>>>>>> > > >>>>>>> > > ========================= >>>>>>> > > How can I help test this release? >>>>>>> > > ========================= >>>>>>> > > >>>>>>> > > If you are a Spark user, you can help us test this release by >>>>>>> taking >>>>>>> > > an existing Spark workload and running on this release >>>>>>> candidate, then >>>>>>> > > reporting any regressions. >>>>>>> > > >>>>>>> > > If you're working in PySpark you can set up a virtual env and >>>>>>> install >>>>>>> > > the current RC and see if anything important breaks, in the >>>>>>> Java/Scala >>>>>>> > > you can add the staging repository to your projects resolvers >>>>>>> and test >>>>>>> > > with the RC (make sure to clean up the artifact cache >>>>>>> before/after so >>>>>>> > > you don't end up building with a out of date RC going forward). >>>>>>> > > >>>>>>> > > =========================================== >>>>>>> > > What should happen to JIRA tickets still targeting 2.4.0? >>>>>>> > > =========================================== >>>>>>> > > >>>>>>> > > The current list of open tickets targeted at 2.4.0 can be found >>>>>>> at: >>>>>>> > > https://issues.apache.org/jira/projects/SPARK and search for >>>>>>> "Target Version/s" = 2.4.0 >>>>>>> > > >>>>>>> > > Committers should look at those and triage. Extremely important >>>>>>> bug >>>>>>> > > fixes, documentation, and API tweaks that impact compatibility >>>>>>> should >>>>>>> > > be worked on immediately. Everything else please retarget to an >>>>>>> > > appropriate release. >>>>>>> > > >>>>>>> > > ================== >>>>>>> > > But my bug isn't fixed? >>>>>>> > > ================== >>>>>>> > > >>>>>>> > > In order to make timely releases, we will typically not hold the >>>>>>> > > release unless the bug in question is a regression from the >>>>>>> previous >>>>>>> > > release. That being said, if there is something which is a >>>>>>> regression >>>>>>> > > that has not been correctly targeted please ping me or a >>>>>>> committer to >>>>>>> > > help target the issue. >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>> >>>>>>> >>>>> >>>>> -- >>>>> Twitter: https://twitter.com/holdenkarau >>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>> >>>> >>> >>> -- >>> Twitter: https://twitter.com/holdenkarau >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> >>
