Re: Getting the ball started on a 2.4.6 release

On Fri, Apr 24, 2020 at 6:14 PM edeesis wrote: > Yes, watching the pod yaml could work for this. Just gotta set up some kind > of thing to do that, thanks for clueing me into that. Sure thing, Kris Nova was the one who clued me into it so just passing it along :) > > > And sounds great re: Spar

Re: Getting the ball started on a 2.4.6 release

Yes, watching the pod yaml could work for this. Just gotta set up some kind of thing to do that, thanks for clueing me into that. And sounds great re: Spark 2.5. Having a transitional release makes sense I think. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: Getting the ball started on a 2.4.6 release

Sounds good, I’ll make the JIRAs for tracking then and I can ping the original PR authors in their and based on their feedback either include or not. On Thu, Apr 23, 2020 at 11:51 AM Xiao Li wrote: > Actually, SPARK-26390 https://github.com/apache/spark/pull/23343 is just > a small clean up. I d

Re: Getting the ball started on a 2.4.6 release

Actually, SPARK-26390 https://github.com/apache/spark/pull/23343 is just a small clean up. I do not think it fixes any correctness bugs. I think we should discuss your backport plans one by one with the PR authors and reviewers, since most of them are not closely following the dev list. Xiao On

Re: Getting the ball started on a 2.4.6 release

I included 26390 as a candidate since it sounded like it bordered on a correctness/expected behaviour fix (eg columpruning rule doing more than column pruning), but if it’s too big a change happy to drop that one. On Thu, Apr 23, 2020 at 11:43 AM Xiao Li wrote: > Hi, Holden, > > We are trying to

Re: Getting the ball started on a 2.4.6 release

Hi, Holden, We are trying to avoid backporting the improvement/cleanup PRs to the maintenance releases, especially the core modules, like Spark Core and SQL. For example, SPARK-26390 is a good example. Xiao On Thu, Apr 23, 2020 at 11:17 AM Holden Karau wrote: > Tentatively I'm planning on this

Re: Getting the ball started on a 2.4.6 release

Tentatively I'm planning on this list to start backporting. If no one sees any issues with those I'll start to make backport JIRAs for them for tracking this afternoon. SPARK-26390 ColumnPruning rule should only do column pruning SPARK-25407 Allow nested access for non-existent field fo

Re: Getting the ball started on a 2.4.6 release

On Thu, Apr 23, 2020 at 9:07 AM edeesis wrote: > There's other information you can obtain from the Pod metadata on a > describe > than just from the logs, which are typically what's being printed by the > Application itself. Would get pods -w -o yaml do the trick here or is there going to be inf

Re: Getting the ball started on a 2.4.6 release

There's other information you can obtain from the Pod metadata on a describe than just from the logs, which are typically what's being printed by the Application itself. I've also found that Spark has some trouble obtaining the reason for a K8S executor death (as evident by the spark.kubernetes.ex

Re: Getting the ball started on a 2.4.6 release

We have a conclusion now and we decide to include SPARK-31509 in the PR of SPARK-31485. So there actually should be only one candidate(But to be honest, it still depends on committers). Best, Yi Wu -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ ---

Re: Getting the ball started on a 2.4.6 release

Thanks, I agree improving that error message instead of hanging could be a good candidate for backporting to 2.4 On Tue, Apr 21, 2020 at 6:43 PM wuyi wrote: > I have one: https://issues.apache.org/jira/browse/SPARK-31485, which could > cause application hang. > > > And, probably, also https://is

Re: Getting the ball started on a 2.4.6 release

I have one: https://issues.apache.org/jira/browse/SPARK-31485, which could cause application hang. And, probably, also https://issues.apache.org/jira/browse/SPARK-31509, to make better guidance of barrier execution for user. But we do not have conclusion yet. Best, Yi Wu -- Sent from: http:

Re: Getting the ball started on a 2.4.6 release

This seem like not very impactful for end-users on K8s assuming they've got logging of some kind set up. Unless I'm missing something. On Tue, Apr 21, 2020 at 4:51 PM edeesis wrote: > I'd like to advocate for: > > https://issues.apache.org/jira/browse/SPARK-25515 > and > https://issues.apache.or

Re: Getting the ball started on a 2.4.6 release

I'd like to advocate for: https://issues.apache.org/jira/browse/SPARK-25515 and https://issues.apache.org/jira/browse/SPARK-29865 Two small QOL changes that make production use of Spark with Kubernetes much easier. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ -

Re: Getting the ball started on a 2.4.6 release

Yes. This one got merged yesterday. Thanks! Xiao On Mon, Apr 20, 2020 at 10:51 AM Sean Owen wrote: > Looks like we have 1 marked for 2.4.6: > https://issues.apache.org/jira/projects/SPARK/versions/12346781 > > https://issues.apache.org/jira/browse/SPARK-31234 ResetCommand should > not wipe out

Re: Getting the ball started on a 2.4.6 release

Looks like we have 1 marked for 2.4.6: https://issues.apache.org/jira/projects/SPARK/versions/12346781 https://issues.apache.org/jira/browse/SPARK-31234 ResetCommand should not wipe out all configs Xiao might be able to comment on that one. On Mon, Apr 20, 2020 at 11:31 AM Holden Karau wrote: >

Getting the ball started on a 2.4.6 release

Hi folks, I’m going to get started on putting together a 2.4.6 release to come out hopefully around the same time as 3.0. Are there any changes in master folks think we should be considering backporting to a 2.4.6 release? Cheers, Holden :) -- Twitter: https://twitter.com/holdenkarau Books (Lea