[jira] [Created] (SPARK-51537) Failed to run third-party Spark ML library on Spark Connect

2025-04-05 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-51537: -- Summary: Failed to run third-party Spark ML library on Spark Connect Key: SPARK-51537 URL: https://issues.apache.org/jira/browse/SPARK-51537 Project: Spark Iss

[jira] [Commented] (SPARK-46762) Spark Connect 3.5 Classloading issue with external jar

2025-03-31 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17939920#comment-17939920 ] Bobby Wang commented on SPARK-46762: Hi [~tenstriker] [~snowch] , I encountered the

[jira] [Commented] (SPARK-45598) Delta table 3.0.0 not working with Spark Connect 3.5.0

2025-03-31 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17939922#comment-17939922 ] Bobby Wang commented on SPARK-45598: I believe this issue has been fixed by https:/

[jira] [Updated] (SPARK-51451) Regression: [UNSUPPORTED_GENERATOR.NESTED_IN_EXPRESSIONS] The generator is not supported: nested in expressions "unresolvedstarwithcolumns(explode(array(0, 1, 2, 3, 4, 5

2025-03-10 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-51451: --- Summary: Regression: [UNSUPPORTED_GENERATOR.NESTED_IN_EXPRESSIONS] The generator is not supported: n

[jira] [Updated] (SPARK-51451) [UNSUPPORTED_GENERATOR.NESTED_IN_EXPRESSIONS] The generator is not supported: nested in expressions "unresolvedstarwithcolumns(explode(array(0, 1, 2, 3, 4, 5, 6, 7, 8, 9

2025-03-10 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-51451: --- Affects Version/s: 4.0.0 > [UNSUPPORTED_GENERATOR.NESTED_IN_EXPRESSIONS] The generator is not suppor

[jira] [Resolved] (SPARK-51320) Failed to run spark ml on connect with pyspark-connect==4.0.0.dev2 installation

2025-02-26 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang resolved SPARK-51320. Resolution: Won't Fix > Failed to run spark ml on connect with pyspark-connect==4.0.0.dev2 > inst

[jira] [Commented] (SPARK-51320) Failed to run spark ml on connect with pyspark-connect==4.0.0.dev2 installation

2025-02-26 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17930906#comment-17930906 ] Bobby Wang commented on SPARK-51320: Close this task due to pyspark-connect==4.0.0.d

[jira] [Created] (SPARK-51320) Failed to run spark ml on connect with pyspark-connect==4.0.0.dev2 installation

2025-02-26 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-51320: -- Summary: Failed to run spark ml on connect with pyspark-connect==4.0.0.dev2 installation Key: SPARK-51320 URL: https://issues.apache.org/jira/browse/SPARK-51320 Project:

[jira] [Updated] (SPARK-50954) Support client side model path overwrite for CV/PL/TVS

2025-01-30 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-50954: --- Summary: Support client side model path overwrite for CV/PL/TVS (was: Support client side model pat

[jira] [Updated] (SPARK-50974) CrossValidator: support foldCol

2025-01-30 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-50974: --- Summary: CrossValidator: support foldCol (was: CrossValidator: foldCol is not supported) > CrossVa

[jira] [Created] (SPARK-51013) support PolynomialExpansion on Connect

2025-01-27 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-51013: -- Summary: support PolynomialExpansion on Connect Key: SPARK-51013 URL: https://issues.apache.org/jira/browse/SPARK-51013 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-51004) Support IndexToString on connect

2025-01-27 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-51004: -- Summary: Support IndexToString on connect Key: SPARK-51004 URL: https://issues.apache.org/jira/browse/SPARK-51004 Project: Spark Issue Type: Sub-task C

[jira] [Commented] (SPARK-50941) Support TrainValidationSplit on Connect

2025-01-26 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17921269#comment-17921269 ] Bobby Wang commented on SPARK-50941: I will make a PR for this task. Thx [~podongfen

[jira] [Commented] (SPARK-50988) Make model uid consistent with estimator

2025-01-25 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17917026#comment-17917026 ] Bobby Wang commented on SPARK-50988: I will make a PR to fix this issue. > Make mod

[jira] [Resolved] (SPARK-50955) combine the estimator read/write and model read/write together

2025-01-25 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang resolved SPARK-50955. Fix Version/s: 4.0.0 Resolution: Fixed Issue is fixed by PR https://github.com/apache/spark

[jira] [Created] (SPARK-50974) CrossValidator: foldCol is not supported

2025-01-23 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-50974: -- Summary: CrossValidator: foldCol is not supported Key: SPARK-50974 URL: https://issues.apache.org/jira/browse/SPARK-50974 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-50959) ImportError: sys.meta_path is None, Python is likely shutting down Exception ignored in:

2025-01-22 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-50959: -- Summary: ImportError: sys.meta_path is None, Python is likely shutting down Exception ignored in: Key: SPARK-50959 URL: https://issues.apache.org/jira/browse/SPARK-50959

[jira] [Created] (SPARK-50955) combine the estimator read/write and model read/write together

2025-01-22 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-50955: -- Summary: combine the estimator read/write and model read/write together Key: SPARK-50955 URL: https://issues.apache.org/jira/browse/SPARK-50955 Project: Spark I

[jira] [Created] (SPARK-50949) Support from_labels/from_arrays_of_labels in StringIndexModel

2025-01-21 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-50949: -- Summary: Support from_labels/from_arrays_of_labels in StringIndexModel Key: SPARK-50949 URL: https://issues.apache.org/jira/browse/SPARK-50949 Project: Spark Is

[jira] [Updated] (SPARK-50948) Support PCA and StringIndexer

2025-01-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-50948: --- Fix Version/s: (was: 4.1) Affects Version/s: 4.1 > Support PCA and StringIndexer > -

[jira] [Created] (SPARK-50948) Support PCA and StringIndexer

2025-01-21 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-50948: -- Summary: Support PCA and StringIndexer Key: SPARK-50948 URL: https://issues.apache.org/jira/browse/SPARK-50948 Project: Spark Issue Type: Sub-task Comp

[jira] [Created] (SPARK-50901) Support transformers

2025-01-20 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-50901: -- Summary: Support transformers Key: SPARK-50901 URL: https://issues.apache.org/jira/browse/SPARK-50901 Project: Spark Issue Type: Sub-task Components: C

[jira] [Created] (SPARK-50897) Don't instantiate the class when loading by ServiceLoader

2025-01-20 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-50897: -- Summary: Don't instantiate the class when loading by ServiceLoader Key: SPARK-50897 URL: https://issues.apache.org/jira/browse/SPARK-50897 Project: Spark Issue T

[jira] [Created] (SPARK-50884) Evaluator: support isLargerBetter on connect ML

2025-01-17 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-50884: -- Summary: Evaluator: support isLargerBetter on connect ML Key: SPARK-50884 URL: https://issues.apache.org/jira/browse/SPARK-50884 Project: Spark Issue Type: Sub-t

[jira] [Created] (SPARK-50869) Support evaluator

2025-01-16 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-50869: -- Summary: Support evaluator Key: SPARK-50869 URL: https://issues.apache.org/jira/browse/SPARK-50869 Project: Spark Issue Type: Sub-task Components: Conn

[jira] [Updated] (SPARK-50844) make ML model to be loaded by ServiceLoader

2025-01-16 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-50844: --- Description: Currently, Since most of models from Spark.ml doesn't have a public no-arguments const

[jira] [Created] (SPARK-50844) make ML model to be loaded by ServiceLoader

2025-01-16 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-50844: -- Summary: make ML model to be loaded by ServiceLoader Key: SPARK-50844 URL: https://issues.apache.org/jira/browse/SPARK-50844 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-50841) Move pyspark.ml.remote into pyspark.ml.connect

2025-01-15 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-50841: -- Summary: Move pyspark.ml.remote into pyspark.ml.connect Key: SPARK-50841 URL: https://issues.apache.org/jira/browse/SPARK-50841 Project: Spark Issue Type: Sub-ta

[jira] [Created] (SPARK-50827) Support plugin for Spark Connect ML

2025-01-15 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-50827: -- Summary: Support plugin for Spark Connect ML Key: SPARK-50827 URL: https://issues.apache.org/jira/browse/SPARK-50827 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-50168) Connect session is not released if not calling spark.stop() explicitly

2024-10-29 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-50168: -- Summary: Connect session is not released if not calling spark.stop() explicitly Key: SPARK-50168 URL: https://issues.apache.org/jira/browse/SPARK-50168 Project: Spark

[jira] [Created] (SPARK-49907) [Connect] Support spark.ml on Connect

2024-10-08 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-49907: -- Summary: [Connect] Support spark.ml on Connect Key: SPARK-49907 URL: https://issues.apache.org/jira/browse/SPARK-49907 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-47668) Add API to remove the existing ResourceProfile from ResourceProfileManager

2024-04-01 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-47668: -- Summary: Add API to remove the existing ResourceProfile from ResourceProfileManager Key: SPARK-47668 URL: https://issues.apache.org/jira/browse/SPARK-47668 Project: Spark

[jira] [Created] (SPARK-47663) Add an end to end tests for checking if spark task works well with resources

2024-03-31 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-47663: -- Summary: Add an end to end tests for checking if spark task works well with resources Key: SPARK-47663 URL: https://issues.apache.org/jira/browse/SPARK-47663 Project: Spa

[jira] [Created] (SPARK-47498) Refine some fractional GPU resource calculation tests.

2024-03-21 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-47498: -- Summary: Refine some fractional GPU resource calculation tests. Key: SPARK-47498 URL: https://issues.apache.org/jira/browse/SPARK-47498 Project: Spark Issue Type

[jira] [Updated] (SPARK-47458) Incorrect to calculate the concurrent task number

2024-03-18 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-47458: --- Summary: Incorrect to calculate the concurrent task number (was: Wrong to calculate the concurrent

[jira] [Created] (SPARK-47458) Wrong to calculate the concurrent task number

2024-03-18 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-47458: -- Summary: Wrong to calculate the concurrent task number Key: SPARK-47458 URL: https://issues.apache.org/jira/browse/SPARK-47458 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-47088) Utilize BigDecimal to calculate the GPU resource

2024-02-18 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-47088: -- Summary: Utilize BigDecimal to calculate the GPU resource Key: SPARK-47088 URL: https://issues.apache.org/jira/browse/SPARK-47088 Project: Spark Issue Type: Imp

[jira] [Created] (SPARK-46812) Make `mapInPandas` / mapInArrow` support ResourceProfile (Stage-Level scheduling)

2024-01-23 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-46812: -- Summary: Make `mapInPandas` / mapInArrow` support ResourceProfile (Stage-Level scheduling) Key: SPARK-46812 URL: https://issues.apache.org/jira/browse/SPARK-46812 Project

[jira] [Commented] (SPARK-46715) Pin `sphinxcontrib-*`

2024-01-15 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17806754#comment-17806754 ] Bobby Wang commented on SPARK-46715: Hi [~podongfeng], I run into the same issue for

[jira] [Created] (SPARK-46721) Some tests failed when cherry-picking https://github.com/apache/spark/pull/43494 back to branch 3.5

2024-01-15 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-46721: -- Summary: Some tests failed when cherry-picking https://github.com/apache/spark/pull/43494 back to branch 3.5 Key: SPARK-46721 URL: https://issues.apache.org/jira/browse/SPARK-46721

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-12-11 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795594#comment-17795594 ] Bobby Wang commented on SPARK-46032: Thx [~haldefaiz] ,  I tried your patch with an

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-22 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788743#comment-17788743 ] Bobby Wang commented on SPARK-46032: Hi [~gurwls223], BTW, I tried it on the fourth

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-22 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788717#comment-17788717 ] Bobby Wang commented on SPARK-46032: Hi [~gurwls223], Any suggestions? > connect: c

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788595#comment-17788595 ] Bobby Wang commented on SPARK-46032: BTW, This morning, I tried it on the third mach

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788584#comment-17788584 ] Bobby Wang commented on SPARK-46032: Hi [~gurwls223], I've pasted all the logs, plea

[jira] [Comment Edited] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788582#comment-17788582 ] Bobby Wang edited comment on SPARK-46032 at 11/22/23 12:54 AM: ---

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788583#comment-17788583 ] Bobby Wang commented on SPARK-46032: Executor Logs, {code:java} Spark Executor Comma

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788582#comment-17788582 ] Bobby Wang commented on SPARK-46032: h1. *Submit the spark application via pyspark*

[jira] [Comment Edited] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788580#comment-17788580 ] Bobby Wang edited comment on SPARK-46032 at 11/22/23 12:46 AM: ---

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788581#comment-17788581 ] Bobby Wang commented on SPARK-46032: h1. Start spark-connect-server which will start

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788580#comment-17788580 ] Bobby Wang commented on SPARK-46032: h1. *Standalone Cluster* h2. Master Log {code:j

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788384#comment-17788384 ] Bobby Wang commented on SPARK-46032: I tried it on another machine and got the same

[jira] [Comment Edited] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788382#comment-17788382 ] Bobby Wang edited comment on SPARK-46032 at 11/21/23 11:52 AM: ---

[jira] [Comment Edited] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788382#comment-17788382 ] Bobby Wang edited comment on SPARK-46032 at 11/21/23 11:52 AM: ---

[jira] [Comment Edited] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788382#comment-17788382 ] Bobby Wang edited comment on SPARK-46032 at 11/21/23 11:51 AM: ---

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788382#comment-17788382 ] Bobby Wang commented on SPARK-46032: {code:java} ~$ scala Welcome to Scala 2.13.12 (

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788378#comment-17788378 ] Bobby Wang commented on SPARK-46032: I tried it on another machine but still had the

[jira] [Updated] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-46032: --- Description: I downloaded spark 3.5 from the spark official website, and then I started a Spark Sta

[jira] [Updated] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-46032: --- Description: I downloaded spark 3.5 from the spark official website, and then I started a Spark Sta

[jira] [Updated] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-46032: --- Description: I downloaded spark 3.5 from the spark official website, and then I started a Spark Sta

[jira] [Created] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-21 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-46032: -- Summary: connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f Key: SPARK-46032 URL: https://issues.apache.org/jira/br

[jira] [Commented] (SPARK-45527) Task fraction resource request is not expected

2023-10-16 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17775948#comment-17775948 ] Bobby Wang commented on SPARK-45527: Working on a PR to fix it. > Task fraction res

[jira] [Created] (SPARK-45495) Support stage level task resource profile for k8s cluster when dynamic allocation disabled

2023-10-10 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-45495: -- Summary: Support stage level task resource profile for k8s cluster when dynamic allocation disabled Key: SPARK-45495 URL: https://issues.apache.org/jira/browse/SPARK-45495

[jira] [Updated] (SPARK-45495) Support stage level task resource profile for k8s cluster when dynamic allocation disabled

2023-10-10 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-45495: --- Description: [https://github.com/apache/spark/pull/37268] has introduced a new feature that support

[jira] [Commented] (SPARK-45250) Support stage level task resource profile for yarn cluster when dynamic allocation disabled

2023-09-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767527#comment-17767527 ] Bobby Wang commented on SPARK-45250: Hi [~ivoson], May I ask why your previous PR [

[jira] [Commented] (SPARK-45250) Support stage level task resource profile for yarn cluster when dynamic allocation disabled

2023-09-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767522#comment-17767522 ] Bobby Wang commented on SPARK-45250: I made a PR to fix it [https://github.com/apach

[jira] [Created] (SPARK-45250) Support stage level task resource profile for yarn cluster when dynamic allocation disabled

2023-09-21 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-45250: -- Summary: Support stage level task resource profile for yarn cluster when dynamic allocation disabled Key: SPARK-45250 URL: https://issues.apache.org/jira/browse/SPARK-45250

[jira] [Updated] (SPARK-40932) Barrier: messages for allGather will be overridden by the following barrier APIs

2022-10-27 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated SPARK-40932: --- Description: When I was working on an internal project which has not been opened source. I found th

[jira] [Created] (SPARK-40932) Barrier: messages for allGather will be overridden by the following barrier APIs

2022-10-27 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-40932: -- Summary: Barrier: messages for allGather will be overridden by the following barrier APIs Key: SPARK-40932 URL: https://issues.apache.org/jira/browse/SPARK-40932 Project:

[jira] [Created] (SPARK-40407) Repartition of DataFrame can result in severe data skew in some special case

2022-09-12 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-40407: -- Summary: Repartition of DataFrame can result in severe data skew in some special case Key: SPARK-40407 URL: https://issues.apache.org/jira/browse/SPARK-40407 Project: Spa

[jira] [Commented] (SPARK-38911) 'test 1 resource profile' throws exception when running it in IDEA separately

2022-04-14 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522646#comment-17522646 ] Bobby Wang commented on SPARK-38911: [~tgraves]  Could you help to check that > 'te

[jira] [Commented] (SPARK-38911) 'test 1 resource profile' throws exception when running it in IDEA separately

2022-04-14 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522645#comment-17522645 ] Bobby Wang commented on SPARK-38911: Just submited an PR for this issue https://gith

[jira] [Created] (SPARK-38911) 'test 1 resource profile' throws exception when running it in IDEA separately

2022-04-14 Thread Bobby Wang (Jira)
Bobby Wang created SPARK-38911: -- Summary: 'test 1 resource profile' throws exception when running it in IDEA separately Key: SPARK-38911 URL: https://issues.apache.org/jira/browse/SPARK-38911 Project: Sp

[jira] [Closed] (ORC-1075) Failed to read rows from the ORC file without statistics in RowIndex when filter is pushed down for 1.6.11

2022-01-14 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/ORC-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang closed ORC-1075. --- Verified. > Failed to read rows from the ORC file without statistics in RowIndex when > filter is pushed down for

[jira] [Commented] (ORC-1075) Failed to read rows from the ORC file without statistics in RowIndex when filter is pushed down for 1.6.11

2022-01-14 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/ORC-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476010#comment-17476010 ] Bobby Wang commented on ORC-1075: - Verified this issue. Closed this issue. > Failed to rea

[jira] [Commented] (ORC-1075) Failed to read rows from the ORC file without statistics in RowIndex when filter is pushed down for 1.6.11

2022-01-06 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/ORC-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469881#comment-17469881 ] Bobby Wang commented on ORC-1075: - It worked for none-1.orc. [~Guiyankuang]  Thx for your q

[jira] [Comment Edited] (ORC-1075) Failed to read rows from the ORC file without statistics in RowIndex when filter is pushed down for 1.6.11

2022-01-06 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/ORC-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469814#comment-17469814 ] Bobby Wang edited comment on ORC-1075 at 1/6/22, 10:11 AM: --- Yeah,

[jira] [Commented] (ORC-1075) Failed to read rows from the ORC file without statistics in RowIndex when filter is pushed down for 1.6.11

2022-01-06 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/ORC-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469814#comment-17469814 ] Bobby Wang commented on ORC-1075: - Yeah, Seems that's the issue. We can have multi-ways to

[jira] [Commented] (ORC-1075) Failed to read rows from the ORC file without statistics in RowIndex when filter is pushed down for 1.6.11

2022-01-06 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/ORC-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469796#comment-17469796 ] Bobby Wang commented on ORC-1075: - Hi [~Guiyankuang] , Thx for your reply. {quote}It is imp

[jira] [Commented] (ORC-1075) Failed to read rows from the ORC file without statistics in RowIndex when filter is pushed down for 1.6.11

2022-01-06 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/ORC-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469779#comment-17469779 ] Bobby Wang commented on ORC-1075: - Sorry, I just tested your PATCH, it still doesn't work f

[jira] (ORC-1075) Failed to read rows from the ORC file without statistics in RowIndex when filter is pushed down for 1.6.11

2022-01-06 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/ORC-1075 ] Bobby Wang deleted comment on ORC-1075: - was (Author: wbo4958): Thx, I just tested your patch. it works. > Failed to read rows from the ORC file without statistics in RowIndex when > filter is pus

[jira] [Commented] (ORC-1075) Failed to read rows from the ORC file without statistics in RowIndex when filter is pushed down for 1.6.11

2022-01-06 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/ORC-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469771#comment-17469771 ] Bobby Wang commented on ORC-1075: - Thx, I just tested your patch. it works. > Failed to re

[jira] [Commented] (ORC-1075) Failed to read rows from the ORC file without statistics in RowIndex when filter is pushed down for 1.6.11

2022-01-05 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/ORC-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469604#comment-17469604 ] Bobby Wang commented on ORC-1075: - Even `index.getNumberOfValues()` can return the actual n

[jira] [Commented] (ORC-1075) Failed to read rows from the ORC file without statistics in RowIndex when filter is pushed down for 1.6.11

2022-01-05 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/ORC-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469242#comment-17469242 ] Bobby Wang commented on ORC-1075: - Hi [~Guiyankuang], Thx for the analysis.   The file wa

[jira] [Created] (ORC-1075) Failed to read rows from the ORC file without statistics in RowIndex when filter is pushed down for 1.6.11

2022-01-04 Thread Bobby Wang (Jira)
Bobby Wang created ORC-1075: --- Summary: Failed to read rows from the ORC file without statistics in RowIndex when filter is pushed down for 1.6.11 Key: ORC-1075 URL: https://issues.apache.org/jira/browse/ORC-1075

[jira] [Commented] (HADOOP-17812) NPE in S3AInputStream read() after failure to reconnect to store

2021-07-31 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391079#comment-17391079 ] Bobby Wang commented on HADOOP-17812: - Hi [~ste...@apache.org] I just cherry-picke

[jira] [Updated] (HADOOP-17812) NPE in S3AInputStream read() after failure to reconnect to store

2021-07-31 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated HADOOP-17812: Attachment: 3.3-branch-failsafe-report.html.gz > NPE in S3AInputStream read() after failure to re

[jira] [Commented] (HADOOP-17812) NPE in S3AInputStream read() after failure to reconnect to store

2021-07-28 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389239#comment-17389239 ] Bobby Wang commented on HADOOP-17812: - Hi Steve, I just found an aws account from

[jira] [Updated] (HADOOP-17812) NPE in S3AInputStream read() after failure to reconnect to store

2021-07-28 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated HADOOP-17812: Attachment: failsafe-report.html.gz > NPE in S3AInputStream read() after failure to reconnect to

[jira] [Commented] (HADOOP-17812) NPE in S3AInputStream read() after failure to reconnect to store

2021-07-25 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17386993#comment-17386993 ] Bobby Wang commented on HADOOP-17812: - Hi [~ste...@apache.org] Thx for your commen

[jira] [Updated] (HADOOP-17812) NPE in S3AInputStream read() after failure to reconnect to store

2021-07-25 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated HADOOP-17812: Attachment: s3a-test.tar.gz > NPE in S3AInputStream read() after failure to reconnect to store >

[jira] [Commented] (HADOOP-17812) NPE in S3AInputStream read() after failure to reconnect to store

2021-07-22 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17385899#comment-17385899 ] Bobby Wang commented on HADOOP-17812: - Hi Steve, I was using hadoop-3.2.0 and the i

[jira] [Updated] (HADOOP-17812) NPE when reading s3a

2021-07-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated HADOOP-17812: Priority: Major (was: Critical) > NPE when reading s3a > > >

[jira] [Updated] (HADOOP-17812) NPE when reading s3a

2021-07-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated HADOOP-17812: Priority: Critical (was: Major) > NPE when reading s3a > > >

[jira] [Updated] (HADOOP-17812) NPE when reading s3a

2021-07-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated HADOOP-17812: Component/s: (was: tools) fs/s3 > NPE when reading s3a > ---

[jira] [Updated] (HADOOP-17812) NPE when reading s3a

2021-07-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated HADOOP-17812: Description: when [reading from S3a storage|https://github.com/apache/hadoop/blob/rel/release-3.

[jira] [Updated] (HADOOP-17812) NPE when reading s3a

2021-07-21 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bobby Wang updated HADOOP-17812: Description: when [reading from S3a storage|https://github.com/apache/hadoop/blob/rel/release-3.

[jira] [Created] (HADOOP-17812) NPE when reading s3a

2021-07-21 Thread Bobby Wang (Jira)
Bobby Wang created HADOOP-17812: --- Summary: NPE when reading s3a Key: HADOOP-17812 URL: https://issues.apache.org/jira/browse/HADOOP-17812 Project: Hadoop Common Issue Type: Bug Compon

[jira] [Created] (HADOOP-17812) NPE when reading s3a

2021-07-21 Thread Bobby Wang (Jira)
Bobby Wang created HADOOP-17812: --- Summary: NPE when reading s3a Key: HADOOP-17812 URL: https://issues.apache.org/jira/browse/HADOOP-17812 Project: Hadoop Common Issue Type: Bug Compon