[jira] [Commented] (SPARK-46912) Spark-submit in cluster mode with standalone cluster uses wrong JAVA_HOME path

2025-07-24 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18009677#comment-18009677 ] Holden Karau commented on SPARK-46912: -- Doing a bit of thinking: I'd generally expe

[jira] [Created] (SPARK-50857) Mark GraphX as deprecated

2025-01-16 Thread Holden Karau (Jira)
Holden Karau created SPARK-50857: Summary: Mark GraphX as deprecated Key: SPARK-50857 URL: https://issues.apache.org/jira/browse/SPARK-50857 Project: Spark Issue Type: Improvement C

[jira] [Created] (SPARK-50539) Improve Spark Shading/Relocation for JVM users

2024-12-10 Thread Holden Karau (Jira)
Holden Karau created SPARK-50539: Summary: Improve Spark Shading/Relocation for JVM users Key: SPARK-50539 URL: https://issues.apache.org/jira/browse/SPARK-50539 Project: Spark Issue Type: Im

[jira] [Commented] (SPARK-49788) Add spark.cleaner.ttl functionality for long lived jobs

2024-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890172#comment-17890172 ] Holden Karau commented on SPARK-49788: -- So the problem with just saying "Spark is d

[jira] [Commented] (SPARK-49828) Expose Column expressions as developer APIs

2024-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890127#comment-17890127 ] Holden Karau commented on SPARK-49828: -- Looks like the PR linking bot didn't work –

[jira] [Comment Edited] (SPARK-49828) Expose Column expressions as developer APIs

2024-09-30 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17885996#comment-17885996 ] Holden Karau edited comment on SPARK-49828 at 9/30/24 6:56 PM: ---

[jira] [Updated] (SPARK-49828) Expose Column expressions as developer APIs

2024-09-29 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-49828: - Description:  Needed for folks who write custom codegen code.   This will involve making Expre

[jira] [Created] (SPARK-49828) Expose Column expression as developer APIs

2024-09-29 Thread Holden Karau (Jira)
Holden Karau created SPARK-49828: Summary: Expose Column expression as developer APIs Key: SPARK-49828 URL: https://issues.apache.org/jira/browse/SPARK-49828 Project: Spark Issue Type: Improv

[jira] [Commented] (SPARK-49788) Add spark.cleaner.ttl functionality for long lived jobs

2024-09-25 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884785#comment-17884785 ] Holden Karau commented on SPARK-49788: -- Rough thoughts on how we could make a new a

[jira] [Commented] (SPARK-49788) Add spark.cleaner.ttl functionality for long lived jobs

2024-09-25 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884768#comment-17884768 ] Holden Karau commented on SPARK-49788: -- Original functionality was removed in 8e19c

[jira] [Commented] (SPARK-49788) Add spark.cleaner.ttl functionality for long lived jobs

2024-09-25 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884760#comment-17884760 ] Holden Karau commented on SPARK-49788: -- CC [~joshrosen] for thoughts. > Add spark.

[jira] [Updated] (SPARK-49788) Add spark.cleaner.ttl functionality for long lived jobs

2024-09-25 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-49788: - Description: We should add a TTL (and maybe a threshold?) to clean shuffle files which have bee

[jira] [Created] (SPARK-49788) Add spark.cleaner.ttl functionality for long lived jobs

2024-09-25 Thread Holden Karau (Jira)
Holden Karau created SPARK-49788: Summary: Add spark.cleaner.ttl functionality for long lived jobs Key: SPARK-49788 URL: https://issues.apache.org/jira/browse/SPARK-49788 Project: Spark Issue

[jira] [Created] (SPARK-48362) Add CollectSetWIthLimit

2024-05-20 Thread Holden Karau (Jira)
Holden Karau created SPARK-48362: Summary: Add CollectSetWIthLimit Key: SPARK-48362 URL: https://issues.apache.org/jira/browse/SPARK-48362 Project: Spark Issue Type: Improvement Com

[jira] [Assigned] (SPARK-44953) Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism

2024-05-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-44953: Assignee: binjie yang > Log a warning (or automatically disable) when shuffle tracking is

[jira] [Resolved] (SPARK-44953) Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism

2024-05-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-44953. -- Resolution: Fixed > Log a warning (or automatically disable) when shuffle tracking is enabled

[jira] [Updated] (SPARK-48101) When using INSERT OVERWRITE with Spark CTEs they may not be fully resolved

2024-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-48101: - Priority: Minor (was: Major) > When using INSERT OVERWRITE with Spark CTEs they may not be full

[jira] [Created] (SPARK-48101) When using INSERT OVERWRITE with Spark CTEs they may not be fully resolved

2024-05-02 Thread Holden Karau (Jira)
Holden Karau created SPARK-48101: Summary: When using INSERT OVERWRITE with Spark CTEs they may not be fully resolved Key: SPARK-48101 URL: https://issues.apache.org/jira/browse/SPARK-48101 Project: S

[jira] [Created] (SPARK-47672) Avoid double evaluation of non-trivial projected elements from filter pushdown

2024-04-01 Thread Holden Karau (Jira)
Holden Karau created SPARK-47672: Summary: Avoid double evaluation of non-trivial projected elements from filter pushdown Key: SPARK-47672 URL: https://issues.apache.org/jira/browse/SPARK-47672 Projec

[jira] [Created] (SPARK-47220) log4j race condition during shutdown

2024-02-28 Thread Holden Karau (Jira)
Holden Karau created SPARK-47220: Summary: log4j race condition during shutdown Key: SPARK-47220 URL: https://issues.apache.org/jira/browse/SPARK-47220 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-47077) sbt build is broken due to selenium change

2024-02-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-47077. -- Resolution: Cannot Reproduce After blowing away my maven + ivy cache it works fine – should ha

[jira] [Created] (SPARK-47077) sbt build is broken due to selenium change

2024-02-16 Thread Holden Karau (Jira)
Holden Karau created SPARK-47077: Summary: sbt build is broken due to selenium change Key: SPARK-47077 URL: https://issues.apache.org/jira/browse/SPARK-47077 Project: Spark Issue Type: Improv

[jira] [Updated] (SPARK-47001) Pushdown Verification in Optimizer.scala should support changed data types

2024-02-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-47001: - Description: When pushing a filter down in a union the data type may not match exactly if the fi

[jira] [Created] (SPARK-47031) Union of with non-determinstic expression should be non-deterministic

2024-02-12 Thread Holden Karau (Jira)
Holden Karau created SPARK-47031: Summary: Union of with non-determinstic expression should be non-deterministic Key: SPARK-47031 URL: https://issues.apache.org/jira/browse/SPARK-47031 Project: Spark

[jira] [Created] (SPARK-45712) Provide a command line flag to override the log4j properties file

2023-10-27 Thread Holden Karau (Jira)
Holden Karau created SPARK-45712: Summary: Provide a command line flag to override the log4j properties file Key: SPARK-45712 URL: https://issues.apache.org/jira/browse/SPARK-45712 Project: Spark

[jira] [Updated] (SPARK-45563) Spark history files backend currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-45563: - Affects Version/s: 4.0.0 (was: 3.3.0) (was:

[jira] [Updated] (SPARK-45563) Spark history files backend currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-45563: - Issue Type: Improvement (was: Bug) > Spark history files backend currently depend on polling fo

[jira] [Updated] (SPARK-45563) Spark history files backend currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-45563: - Description: The spark history server FS  currently depends on polling for loading history files

[jira] [Updated] (SPARK-45563) Spark history files backend currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-45563: - Summary: Spark history files backend currently depend on polling for loading into the history se

[jira] [Created] (SPARK-45563) Spark rolling history files currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
Holden Karau created SPARK-45563: Summary: Spark rolling history files currently depend on polling for loading into the history server Key: SPARK-45563 URL: https://issues.apache.org/jira/browse/SPARK-45563

[jira] [Resolved] (SPARK-44735) Log a warning when inserting columns with the same name by row that don't match up

2023-10-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-44735. -- Fix Version/s: 4.0.0 Resolution: Fixed > Log a warning when inserting columns with the

[jira] [Assigned] (SPARK-44735) Log a warning when inserting columns with the same name by row that don't match up

2023-10-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-44735: Assignee: Jia Fan > Log a warning when inserting columns with the same name by row that d

[jira] [Created] (SPARK-44992) Add support for rack information from an environment variable

2023-08-28 Thread Holden Karau (Jira)
Holden Karau created SPARK-44992: Summary: Add support for rack information from an environment variable Key: SPARK-44992 URL: https://issues.apache.org/jira/browse/SPARK-44992 Project: Spark

[jira] [Created] (SPARK-44970) Spark History File Uploads Can Fail on S3

2023-08-25 Thread Holden Karau (Jira)
Holden Karau created SPARK-44970: Summary: Spark History File Uploads Can Fail on S3 Key: SPARK-44970 URL: https://issues.apache.org/jira/browse/SPARK-44970 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-44955) Add the option for dynamically marking containers for preemption based data

2023-08-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-44955: Summary: Add the option for dynamically marking containers for preemption based data Key: SPARK-44955 URL: https://issues.apache.org/jira/browse/SPARK-44955 Project:

[jira] [Created] (SPARK-44954) Make DEA algorithms pluggable

2023-08-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-44954: Summary: Make DEA algorithms pluggable Key: SPARK-44954 URL: https://issues.apache.org/jira/browse/SPARK-44954 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-44953) Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism

2023-08-24 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-44953: - Parent: SPARK-44951 Issue Type: Sub-task (was: Improvement) > Log a warning (or automat

[jira] [Created] (SPARK-44953) Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism

2023-08-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-44953: Summary: Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism Key: SPARK-44953 URL: https://issues.apache.org/jira/b

[jira] [Created] (SPARK-44951) Improve Spark Dynamic Allocation

2023-08-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-44951: Summary: Improve Spark Dynamic Allocation Key: SPARK-44951 URL: https://issues.apache.org/jira/browse/SPARK-44951 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-44950) Improve Spark Driver Launch Time

2023-08-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-44950: Summary: Improve Spark Driver Launch Time Key: SPARK-44950 URL: https://issues.apache.org/jira/browse/SPARK-44950 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-44769) Add SQL statement to create an empty array with a type

2023-08-10 Thread Holden Karau (Jira)
Holden Karau created SPARK-44769: Summary: Add SQL statement to create an empty array with a type Key: SPARK-44769 URL: https://issues.apache.org/jira/browse/SPARK-44769 Project: Spark Issue

[jira] [Updated] (SPARK-42035) Add a config flag to force exit on JDK major version mismatch

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-42035: - Target Version/s: 4.0.0 > Add a config flag to force exit on JDK major version mismatch > --

[jira] [Updated] (SPARK-42261) K8s will not allocate more execs if there are any pending execs until next snapshot

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-42261: - Target Version/s: 4.0.0 > K8s will not allocate more execs if there are any pending execs until

[jira] [Updated] (SPARK-44511) Allow insertInto to succeed with partion columns specified when they match those on the target table

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-44511: - Target Version/s: 4.0.0 > Allow insertInto to succeed with partion columns specified when they m

[jira] [Updated] (SPARK-42361) Add an option to use external storage to distribute JAR set in cluster mode on Kube

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-42361: - Target Version/s: 4.0.0 > Add an option to use external storage to distribute JAR set in cluster

[jira] [Updated] (SPARK-42260) Log when the K8s Exec Pods Allocator Stalls

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-42260: - Target Version/s: 4.0.0 > Log when the K8s Exec Pods Allocator Stalls >

[jira] [Commented] (SPARK-44727) Improve the error message for dynamic allocation conditions

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752496#comment-17752496 ] Holden Karau commented on SPARK-44727: -- Do you have more context [~chengpan] ? > I

[jira] [Updated] (SPARK-42035) Add a config flag to force exit on JDK major version mismatch

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-42035: - Description: JRE version mismatches can cause errors which are difficult to debug (potentially

[jira] [Updated] (SPARK-34337) Reject disk blocks when out of disk space

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-34337: - Target Version/s: 4.0.0 > Reject disk blocks when out of disk space > --

[jira] [Created] (SPARK-44735) Log a warning when inserting columns with the same name by row that don't match up

2023-08-08 Thread Holden Karau (Jira)
Holden Karau created SPARK-44735: Summary: Log a warning when inserting columns with the same name by row that don't match up Key: SPARK-44735 URL: https://issues.apache.org/jira/browse/SPARK-44735 Pr

[jira] [Commented] (SPARK-44050) Unable to Mount ConfigMap in Driver Pod - ConfigMap Creation Issue

2023-08-07 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751796#comment-17751796 ] Holden Karau commented on SPARK-44050: -- Ah interesting, it sounds like the fix woul

[jira] [Commented] (SPARK-44508) Add user guide for Python UDTFs

2023-08-07 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751795#comment-17751795 ] Holden Karau commented on SPARK-44508: -- I'm not sure this should be a blocker. > A

[jira] [Commented] (SPARK-24282) Add support for PMML export for the Standard Scaler Stage

2023-08-07 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751758#comment-17751758 ] Holden Karau commented on SPARK-24282: -- I don't think were going to do this anymore

[jira] [Resolved] (SPARK-28740) Add support for building with bloop

2023-08-07 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-28740. -- Resolution: Won't Fix > Add support for building with bloop >

[jira] [Commented] (SPARK-32111) Cleanup locks and docs in CoarseGrainedSchedulerBackend

2023-08-07 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751754#comment-17751754 ] Holden Karau commented on SPARK-32111: -- I think this could be a good target for Spa

[jira] [Updated] (SPARK-32111) Cleanup locks and docs in CoarseGrainedSchedulerBackend

2023-08-07 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-32111: - Target Version/s: 4.0.0 Affects Version/s: 4.0.0 > Cleanup locks and docs in CoarseGrainedS

[jira] [Updated] (SPARK-44578) Support pushing down BoundFunction in DSv2

2023-07-31 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-44578: - Description: See [https://github.com/apache/iceberg/pull/7886#discussion_r1257537662]  (was: We

[jira] [Created] (SPARK-44592) Add history link to previous run on SS HA

2023-07-28 Thread Holden Karau (Jira)
Holden Karau created SPARK-44592: Summary: Add history link to previous run on SS HA Key: SPARK-44592 URL: https://issues.apache.org/jira/browse/SPARK-44592 Project: Spark Issue Type: Improve

[jira] [Created] (SPARK-44578) Support pushing down UDFs in DSv2

2023-07-27 Thread Holden Karau (Jira)
Holden Karau created SPARK-44578: Summary: Support pushing down UDFs in DSv2 Key: SPARK-44578 URL: https://issues.apache.org/jira/browse/SPARK-44578 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-37562) Add Spark History Server Links for Kubernetes & other CMs

2023-07-26 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17747676#comment-17747676 ] Holden Karau commented on SPARK-37562: -- So (in theory) the cluster administrator ha

[jira] [Created] (SPARK-44529) Add a flag to resolve docker tags to hashes at launch time

2023-07-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-44529: Summary: Add a flag to resolve docker tags to hashes at launch time Key: SPARK-44529 URL: https://issues.apache.org/jira/browse/SPARK-44529 Project: Spark Is

[jira] [Created] (SPARK-44511) Allow insertInto to succeed with partion columns specified when they match those on the target table

2023-07-21 Thread Holden Karau (Jira)
Holden Karau created SPARK-44511: Summary: Allow insertInto to succeed with partion columns specified when they match those on the target table Key: SPARK-44511 URL: https://issues.apache.org/jira/browse/SPARK-445

[jira] [Created] (SPARK-42762) Improve Logging for disconnects during exec id request

2023-03-11 Thread Holden Karau (Jira)
Holden Karau created SPARK-42762: Summary: Improve Logging for disconnects during exec id request Key: SPARK-42762 URL: https://issues.apache.org/jira/browse/SPARK-42762 Project: Spark Issue

[jira] [Commented] (SPARK-42411) Better support for Istio service mesh while running Spark on Kubernetes

2023-02-12 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687664#comment-17687664 ] Holden Karau commented on SPARK-42411: -- The other option is something around `spark

[jira] [Created] (SPARK-42361) Add an option to use external storage to distribute JAR set in cluster mode on Kube

2023-02-06 Thread Holden Karau (Jira)
Holden Karau created SPARK-42361: Summary: Add an option to use external storage to distribute JAR set in cluster mode on Kube Key: SPARK-42361 URL: https://issues.apache.org/jira/browse/SPARK-42361 P

[jira] [Created] (SPARK-42261) K8s will not allocate more execs if there are any pending execs until next snapshot

2023-01-31 Thread Holden Karau (Jira)
Holden Karau created SPARK-42261: Summary: K8s will not allocate more execs if there are any pending execs until next snapshot Key: SPARK-42261 URL: https://issues.apache.org/jira/browse/SPARK-42261 P

[jira] [Created] (SPARK-42260) Log when the K8s Exec Pods Allocator Stalls

2023-01-31 Thread Holden Karau (Jira)
Holden Karau created SPARK-42260: Summary: Log when the K8s Exec Pods Allocator Stalls Key: SPARK-42260 URL: https://issues.apache.org/jira/browse/SPARK-42260 Project: Spark Issue Type: Impro

[jira] [Created] (SPARK-42173) IPv6 address mapping can fail with sparse addresses

2023-01-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-42173: Summary: IPv6 address mapping can fail with sparse addresses Key: SPARK-42173 URL: https://issues.apache.org/jira/browse/SPARK-42173 Project: Spark Issue Typ

[jira] [Created] (SPARK-42035) Add a config flag to force exit on JDK major version mismatch

2023-01-12 Thread Holden Karau (Jira)
Holden Karau created SPARK-42035: Summary: Add a config flag to force exit on JDK major version mismatch Key: SPARK-42035 URL: https://issues.apache.org/jira/browse/SPARK-42035 Project: Spark

[jira] [Resolved] (SPARK-33782) Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode

2022-12-12 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-33782. -- Fix Version/s: 3.4.0 Assignee: Pralabh Kumar Resolution: Fixed > Place spark.f

[jira] [Resolved] (SPARK-39799) DataSourceV2: View catalog interface

2022-11-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-39799. -- Fix Version/s: 3.4.0 Resolution: Fixed > DataSourceV2: View catalog interface > ---

[jira] [Assigned] (SPARK-39799) DataSourceV2: View catalog interface

2022-11-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-39799: Assignee: John Zhuge > DataSourceV2: View catalog interface > ---

[jira] [Created] (SPARK-40850) Tests for Spark SQL Intrepetered Queries may execute Codegen

2022-10-19 Thread Holden Karau (Jira)
Holden Karau created SPARK-40850: Summary: Tests for Spark SQL Intrepetered Queries may execute Codegen Key: SPARK-40850 URL: https://issues.apache.org/jira/browse/SPARK-40850 Project: Spark

[jira] [Resolved] (SPARK-40428) Add a shutdownhook to CoarseGrained scheduler to avoid dangling resources during abnormal shutdown

2022-10-04 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-40428. -- Fix Version/s: 3.4.0 Assignee: Holden Karau Resolution: Fixed > Add a shutdown

[jira] [Created] (SPARK-40428) Add a shutdownhook to CoarseGrained scheduler to avoid dangling resources during abnormal shutdown

2022-09-14 Thread Holden Karau (Jira)
Holden Karau created SPARK-40428: Summary: Add a shutdownhook to CoarseGrained scheduler to avoid dangling resources during abnormal shutdown Key: SPARK-40428 URL: https://issues.apache.org/jira/browse/SPARK-40428

[jira] [Created] (SPARK-40379) Propagate decommission executor loss reason during onDisconnect in K8s

2022-09-07 Thread Holden Karau (Jira)
Holden Karau created SPARK-40379: Summary: Propagate decommission executor loss reason during onDisconnect in K8s Key: SPARK-40379 URL: https://issues.apache.org/jira/browse/SPARK-40379 Project: Spark

[jira] [Resolved] (SPARK-38969) Graceful decomissionning on Kubernetes fails / decom script error

2022-08-12 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-38969. -- Fix Version/s: 3.4.0 Assignee: Holden Karau Resolution: Fixed Updated decommis

[jira] [Created] (SPARK-39512) Document the Spark Docker container release process

2022-06-18 Thread Holden Karau (Jira)
Holden Karau created SPARK-39512: Summary: Document the Spark Docker container release process Key: SPARK-39512 URL: https://issues.apache.org/jira/browse/SPARK-39512 Project: Spark Issue Typ

[jira] [Created] (SPARK-39088) Add support for forwarding Spark History requests to live running driver when present

2022-05-02 Thread Holden Karau (Jira)
Holden Karau created SPARK-39088: Summary: Add support for forwarding Spark History requests to live running driver when present Key: SPARK-39088 URL: https://issues.apache.org/jira/browse/SPARK-39088

[jira] [Commented] (SPARK-38969) Graceful decomissionning on Kubernetes fails / decom script error

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17530922#comment-17530922 ] Holden Karau commented on SPARK-38969: -- Most likely it's from the `timeout 60` in t

[jira] [Commented] (SPARK-38158) Rdd and shuffle blocks not migrated to new executors when decommission feature is enabled

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17530916#comment-17530916 ] Holden Karau commented on SPARK-38158: -- Do you have executor logs? > Rdd and shuff

[jira] [Commented] (SPARK-38408) Add option for partial nested field writes with nullable fields.

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17530915#comment-17530915 ] Holden Karau commented on SPARK-38408: -- More general > Add option for partial nest

[jira] [Resolved] (SPARK-33178) Dynamic Scaling In/Scaling out of executors - Kubernetes

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-33178. -- Resolution: Incomplete Resolved since there is not enough info for what the request is for. >

[jira] [Resolved] (SPARK-34104) Allow users to specify a maximum decommissioning time

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-34104. -- Fix Version/s: 3.2.0 Resolution: Fixed > Allow users to specify a maximum decommissioni

[jira] [Resolved] (SPARK-34105) In addition to killing exlcuded/flakey executors which should support decommissioning

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-34105. -- Fix Version/s: 3.2.0 Resolution: Fixed > In addition to killing exlcuded/flakey executo

[jira] [Resolved] (SPARK-34530) logError for interrupting block migrations is too high

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-34530. -- Resolution: Incomplete > logError for interrupting block migrations is too high >

[jira] [Resolved] (SPARK-36664) Log time spent waiting for cluster resources

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-36664. -- Fix Version/s: 3.4.0 Assignee: Holden Karau Resolution: Fixed > Log time spent

[jira] [Resolved] (SPARK-38561) Add doc for "Customized Kubernetes Schedulers"

2022-03-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-38561. -- Fix Version/s: 3.3.0 Assignee: Yikun Jiang Resolution: Fixed > Add doc for "Cu

[jira] [Updated] (SPARK-38503) Add warn for getAdditionalPreKubernetesResources in executor side

2022-03-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-38503: - Target Version/s: 3.4.0 > Add warn for getAdditionalPreKubernetesResources in executor side > --

[jira] [Created] (SPARK-38408) Add option for partial nested field writes with nullable fields.

2022-03-03 Thread Holden Karau (Jira)
Holden Karau created SPARK-38408: Summary: Add option for partial nested field writes with nullable fields. Key: SPARK-38408 URL: https://issues.apache.org/jira/browse/SPARK-38408 Project: Spark

[jira] [Created] (SPARK-38090) Make links for stderr/stdout on Spark on Kube configurable

2022-02-02 Thread Holden Karau (Jira)
Holden Karau created SPARK-38090: Summary: Make links for stderr/stdout on Spark on Kube configurable Key: SPARK-38090 URL: https://issues.apache.org/jira/browse/SPARK-38090 Project: Spark Is

[jira] [Updated] (SPARK-38075) Hive script transform with order by and limit will return fake rows

2022-02-01 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-38075: - Fix Version/s: 3.1.3 (was: 3.1.4) > Hive script transform with order by a

[jira] [Updated] (SPARK-38075) Hive script transform with order by and limit will return fake rows

2022-02-01 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-38075: - Fix Version/s: 3.1.4 (was: 3.1.3) Affects Version/s: 3.1.3 >

[jira] [Updated] (SPARK-36782) Deadlock between map-output-dispatcher and dispatcher-BlockManagerMaster upon migrating shuffle blocks

2022-01-10 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-36782: - Fix Version/s: 3.1.3 > Deadlock between map-output-dispatcher and dispatcher-BlockManagerMaster

[jira] [Commented] (SPARK-37562) Add Spark History Server Links for Kubernetes & other CMs

2021-12-06 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454194#comment-17454194 ] Holden Karau commented on SPARK-37562: -- cc [~dongjoon] any thoughts on the param na

[jira] [Created] (SPARK-37562) Add Spark History Server Links for Kubernetes & other CMs

2021-12-06 Thread Holden Karau (Jira)
Holden Karau created SPARK-37562: Summary: Add Spark History Server Links for Kubernetes & other CMs Key: SPARK-37562 URL: https://issues.apache.org/jira/browse/SPARK-37562 Project: Spark Iss

[jira] [Updated] (SPARK-37359) Spark K8s tests are broken

2021-11-17 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-37359: - Description: For decom: We should look for "Decommission executors" instead of "Remove reason s

[jira] [Updated] (SPARK-37359) Spark K8s tests are broken

2021-11-17 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-37359: - Summary: Spark K8s tests are broken (was: Spark K8s decom test looks for a non longer current s

[jira] [Created] (SPARK-37359) Spark K8s decom test looks for a non longer current string

2021-11-17 Thread Holden Karau (Jira)
Holden Karau created SPARK-37359: Summary: Spark K8s decom test looks for a non longer current string Key: SPARK-37359 URL: https://issues.apache.org/jira/browse/SPARK-37359 Project: Spark Is

[jira] [Updated] (SPARK-33782) Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode

2021-09-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-33782: - Target Version/s: 3.3.0 > Place spark.files, spark.jars and spark.files under the current workin

  1   2   3   4   5   >