Sounds good, I’ll make the JIRAs for tracking then and I can ping the original PR authors in their and based on their feedback either include or not.
On Thu, Apr 23, 2020 at 11:51 AM Xiao Li <lix...@databricks.com> wrote: > Actually, SPARK-26390 https://github.com/apache/spark/pull/23343 is just > a small clean up. I do not think it fixes any correctness bugs. > > I think we should discuss your backport plans one by one with the PR > authors and reviewers, since most of them are not closely following the dev > list. > > Xiao > > > On Thu, Apr 23, 2020 at 11:46 AM Holden Karau <hol...@pigscanfly.ca> > wrote: > >> I included 26390 as a candidate since it sounded like it bordered on a >> correctness/expected behaviour fix (eg columpruning rule doing more than >> column pruning), but if it’s too big a change happy to drop that one. >> >> On Thu, Apr 23, 2020 at 11:43 AM Xiao Li <lix...@databricks.com> wrote: >> >>> Hi, Holden, >>> >>> We are trying to avoid backporting the improvement/cleanup PRs to the >>> maintenance releases, especially the core modules, like Spark Core and >>> SQL. For example, SPARK-26390 is a good example. >>> >>> Xiao >>> >>> On Thu, Apr 23, 2020 at 11:17 AM Holden Karau <hol...@pigscanfly.ca> >>> wrote: >>> >>>> Tentatively I'm planning on this list to start backporting. If no one >>>> sees any issues with those I'll start to make backport JIRAs for them for >>>> tracking this afternoon. >>>> SPARK-26390 ColumnPruning rule should only do column pruning >>>> SPARK-25407 Allow nested access for non-existent field for >>>> Parquet file when nested pruning is enabled >>>> SPARK-25559 Remove the unsupported predicates in Parquet when >>>> possible >>>> SPARK-25860 Replace Literal(null, _) with FalseLiteral whenever >>>> possible >>>> SPARK-27514 Skip collapsing windows with empty window expressions >>>> SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() >>>> in test cases >>>> SPARK-27138 Remove AdminUtils calls (fixes deprecation) >>>> SPARK-27981 Remove `Illegal reflective access` warning for >>>> `java.nio.Bits.unaligned()` in JDK9+ >>>> SPARK-26095 Disable parallelization in make-distibution.sh. >>>> (Avoid build hanging) >>>> SPARK-25692 Remove static initialization of worker eventLoop >>>> handling chunk fetch requests within TransportContext. This fixes >>>> ChunkFetchIntegrationSuite as well >>>> SPARK-26306 More memory to de-flake SorterSuite >>>> SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint >>>> SPARK-27676 InMemoryFileIndex should respect >>>> spark.sql.files.ignoreMissingFiles >>>> SPARK-31047 Improve file listing for ViewFileSystem >>>> SPARK-25595 Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES >>>> enabled >>>> >>>> Maybe: >>>> SPARK-27801 Delegate to ViewFileSystem during file listing >>>> correctly >>>> >>>> Not yet merged: >>>> SPARK-31485 Barrier execution hang if insufficient resources >>>> >>>> On Thu, Apr 23, 2020 at 9:13 AM Holden Karau <hol...@pigscanfly.ca> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Thu, Apr 23, 2020 at 9:07 AM edeesis <edee...@gmail.com> wrote: >>>>> >>>>>> There's other information you can obtain from the Pod metadata on a >>>>>> describe >>>>>> than just from the logs, which are typically what's being printed by >>>>>> the >>>>>> Application itself. >>>>> >>>>> Would get pods -w -o yaml do the trick here or is there going to be >>>>> information that wouldn’t be captured that way? >>>>> >>>>>> >>>>>> >>>>>> I've also found that Spark has some trouble obtaining the reason for >>>>>> a K8S >>>>>> executor death (as evident by the >>>>>> spark.kubernetes.executor.lostCheck.maxAttempts config property) >>>>>> >>>>>> I admittedly don't know what should qualify for a backport, but >>>>>> considering >>>>>> 3.0 is a major upgrade (Scala version, et al), is there any room for >>>>>> for >>>>>> being more generous with backporting to 2.4? >>>>> >>>>> I’d like to revisit the conversation around a Spark 2.5 as a >>>>> transitional release. I know that some people are already effectively >>>>> maintaining 2.4+ Selective new functionality backports internally. Maybe >>>>> I’ll kick off that discussion which we can have and that can help inform >>>>> what we should be putting in 2.4. >>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>> >>>>>> -- >>>>> Twitter: https://twitter.com/holdenkarau >>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>> >>>> >>>> >>>> -- >>>> Twitter: https://twitter.com/holdenkarau >>>> Books (Learning Spark, High Performance Spark, etc.): >>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>> >>> >>> >>> -- >>> <https://databricks.com/sparkaisummit/north-america> >>> >> -- >> Twitter: https://twitter.com/holdenkarau >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> > > > -- > <https://databricks.com/sparkaisummit/north-america> > -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
