Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
beliefer commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1911924441 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala: ## @@ -374,6 +374,7 @@ abstract class JdbcDialect extends Serializable with Logging { ca

Re: [PR] [SPARK-50790][PYTHON] Implement parse json in pyspark [spark]

2025-01-10 Thread via GitHub
harshmotw-db commented on code in PR #49450: URL: https://github.com/apache/spark/pull/49450#discussion_r1911903839 ## python/pyspark/sql/variant_utils.py: ## @@ -496,3 +525,297 @@ def _handle_array(cls, value: bytes, pos: int, func: Callable[[List[int]], Any]) ele

Re: [PR] [SPARK-50403][SQL] Fix parameterized `EXECUTE IMMEDIATE` [spark]

2025-01-10 Thread via GitHub
MaxGekk commented on PR #49442: URL: https://github.com/apache/spark/pull/49442#issuecomment-2585113232 @srielau @cloud-fan Could you take a look at the PR, please. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] [SPARK-50694][SQL] Support withColumns / withColumnsRenamed in subqueries [spark]

2025-01-10 Thread via GitHub
ueshin commented on code in PR #49386: URL: https://github.com/apache/spark/pull/49386#discussion_r1911767455 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala: ## @@ -712,6 +717,109 @@ case class UnresolvedStarExceptOrReplace( } } +/**

Re: [PR] [SPARK-48745][INFRA][PYTHON][TESTS][FOLLOWUP] Fix the `pyspark-error` testing environment in GA [spark]

2025-01-10 Thread via GitHub
panbingkun commented on PR #49441: URL: https://github.com/apache/spark/pull/49441#issuecomment-2584977231 cc @zhengruifeng @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
beliefer commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1911875614 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala: ## @@ -374,6 +374,7 @@ abstract class JdbcDialect extends Serializable with Logging { ca

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
beliefer commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1911877120 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +151,19 @@ private case class MySQLDialect() extends JdbcDialect with SQLConfHel

Re: [PR] [SPARK-50773][Core] Disable structured logging by default [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49421: URL: https://github.com/apache/spark/pull/49421#issuecomment-2585053901 For the record, `StructuredSparkLoggerSuite` seems to fail still. ``` [error] Failed: Total 66, Failed 6, Errors 0, Passed 60 [error] Failed tests: [error] org.apache

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1911880414 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +151,19 @@ private case class MySQLDialect() extends JdbcDialect with SQLCon

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1911879959 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala: ## @@ -374,6 +374,7 @@ abstract class JdbcDialect extends Serializable with Logging {

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1911879959 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala: ## @@ -374,6 +374,7 @@ abstract class JdbcDialect extends Serializable with Logging {

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1911880894 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +151,19 @@ private case class MySQLDialect() extends JdbcDialect with SQLCon

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1911885139 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -259,6 +311,8 @@ private case class MySQLDialect() extends JdbcDialect with SQLConf

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1911885139 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -259,6 +311,8 @@ private case class MySQLDialect() extends JdbcDialect with SQLConf

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1911887194 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala: ## @@ -374,6 +374,7 @@ abstract class JdbcDialect extends Serializable with Logging {

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1911879959 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala: ## @@ -374,6 +374,7 @@ abstract class JdbcDialect extends Serializable with Logging {

Re: [PR] [SPARK-50776][K8S][TESTS][FOLLOWUP] Fix code style [spark]

2025-01-10 Thread via GitHub
panbingkun commented on PR #49444: URL: https://github.com/apache/spark/pull/49444#issuecomment-2584815678 > +1, LGTM. > > Thank you. I manually tested this, @panbingkun . > > Let's merge this because mine is blocked by other parts. > > ``` > $ dev/lint-scala > Usin

Re: [PR] [SPARK-50788][TESTS] Add Benchmark for Large-Row Dataframe [spark]

2025-01-10 Thread via GitHub
yhuang-db commented on PR #49447: URL: https://github.com/apache/spark/pull/49447#issuecomment-2584669740 Hi dongjoon-hyun, Thank you for reviewing. I've added the benchmark results to this PR. -- This is an automated message from the Apache Git Service. To respond to the message, p

[PR] [SPARK-50791] Fix NPE in State Store error handling [spark]

2025-01-10 Thread via GitHub
liviazhu-db opened a new pull request, #49451: URL: https://github.com/apache/spark/pull/49451 ### What changes were proposed in this pull request? Directly calling `contains` on a `SparkException.getCondition` may result in a NullPointerException if `getCondition` returns `nu

Re: [PR] [SPARK-50768][CORE] Introduce TaskContext.createResourceUninterruptibly to avoid stream leak by task interruption [spark]

2025-01-10 Thread via GitHub
xuanyuanking commented on code in PR #49413: URL: https://github.com/apache/spark/pull/49413#discussion_r1911781043 ## core/src/test/scala/org/apache/spark/JobCancellationSuite.scala: ## @@ -712,6 +713,140 @@ class JobCancellationSuite extends SparkFunSuite with Matchers with B

Re: [PR] [SPARK-50788][TESTS] Add Benchmark for Large-Row Dataframe [spark]

2025-01-10 Thread via GitHub
gengliangwang commented on code in PR #49447: URL: https://github.com/apache/spark/pull/49447#discussion_r1911780761 ## sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/LargeRowBenchmark.scala: ## @@ -0,0 +1,73 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] [SPARK-50788][TESTS] Add Benchmark for Large-Row Dataframe [spark]

2025-01-10 Thread via GitHub
gengliangwang commented on PR #49447: URL: https://github.com/apache/spark/pull/49447#issuecomment-2584972024 Thanks for the contribution, @yhuang-db. This will be valuable for future performance regression checks. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] [SPARK-50522][SQL] Support for indeterminate collation [spark]

2025-01-10 Thread via GitHub
cloud-fan commented on code in PR #49103: URL: https://github.com/apache/spark/pull/49103#discussion_r1910006494 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala: ## @@ -295,6 +296,8 @@ case class PreprocessTableCreation(catalog: SessionCatalog)

Re: [PR] [SPARK-50522][SQL] Support for indeterminate collation [spark]

2025-01-10 Thread via GitHub
cloud-fan commented on code in PR #49103: URL: https://github.com/apache/spark/pull/49103#discussion_r1910006967 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala: ## @@ -323,6 +326,9 @@ case class PreprocessTableCreation(catalog: SessionCatalog)

[PR] [SPARK-50633][FOLLOWUP] Turn on options verbose for codecov/codecov-action [spark]

2025-01-10 Thread via GitHub
panbingkun opened a new pull request, #49438: URL: https://github.com/apache/spark/pull/49438 ### What changes were proposed in this pull request? The pr aims to turn on options `verbose` for `codecov/codecov-action` to identify issues. ### Why are the changes needed? The is

Re: [PR] [SPARK-50633][FOLLOWUP] Turn on options `verbose` for `codecov/codecov-action` [spark]

2025-01-10 Thread via GitHub
panbingkun commented on PR #49438: URL: https://github.com/apache/spark/pull/49438#issuecomment-2582082733 cc @HyukjinKwon @LuciferYang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] [SPARK-50633][FOLLOWUP] Turn on options `verbose` for `codecov/codecov-action` [spark]

2025-01-10 Thread via GitHub
panbingkun commented on PR #49438: URL: https://github.com/apache/spark/pull/49438#issuecomment-2582087248 https://issues.apache.org/jira/browse/INFRA-26413 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [SPARK-50781][SQL] Cache `QueryPlan.expressions` [spark]

2025-01-10 Thread via GitHub
zhengruifeng commented on PR #49435: URL: https://github.com/apache/spark/pull/49435#issuecomment-2582108171 thanks, merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-50781][SQL] Cache `QueryPlan.expressions` [spark]

2025-01-10 Thread via GitHub
zhengruifeng closed pull request #49435: [SPARK-50781][SQL] Cache `QueryPlan.expressions` URL: https://github.com/apache/spark/pull/49435 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-50751][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_1305`: `UNSUPPORTED_TABLE_CHANGE_IN_JDBC_CATALOG` [spark]

2025-01-10 Thread via GitHub
MaxGekk commented on code in PR #49395: URL: https://github.com/apache/spark/pull/49395#discussion_r1910152456 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalogSuite.scala: ## @@ -444,9 +444,11 @@ class JDBCTableCatalogSuite extends Q

Re: [PR] [MINOR][DOCS] Fix miss semicolon on describe sql example [spark]

2025-01-10 Thread via GitHub
MaxGekk commented on PR #49316: URL: https://github.com/apache/spark/pull/49316#issuecomment-2582300908 +1, LGTM. Merging to master. Thank you, @camilesing and @HyukjinKwon @the-sakthi for review. -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] [MINOR][DOCS] Fix miss semicolon on describe sql example [spark]

2025-01-10 Thread via GitHub
MaxGekk closed pull request #49316: [MINOR][DOCS] Fix miss semicolon on describe sql example URL: https://github.com/apache/spark/pull/49316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] [SPARK-49960][SQL] Provide extension point for custom AgnosticEncoder serde [spark]

2025-01-10 Thread via GitHub
eejbyfeldt commented on PR #48477: URL: https://github.com/apache/spark/pull/48477#issuecomment-2582007337 > extra black box indirection that will not allow optimisation (e.g. constant folding etc.) I am not sure I am following here. Is the black box you talking about the "code hi

Re: [PR] [SPARK-50728][BUILD] Update ORC to 2.1.0 [spark]

2025-01-10 Thread via GitHub
williamhyun commented on PR #49358: URL: https://github.com/apache/spark/pull/49358#issuecomment-2582008452 This PR is ready. cc @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] [MINOR][DOCS] Add new dataframe methods to API references [spark]

2025-01-10 Thread via GitHub
zhengruifeng commented on code in PR #49439: URL: https://github.com/apache/spark/pull/49439#discussion_r1910097626 ## python/pyspark/sql/dataframe.py: ## @@ -6609,10 +6609,10 @@ def asTable(self) -> TableArg: After obtaining a TableArg from a DataFrame using this metho

[PR] [SPARK-50783][CORE] Canonicalize JVM profiler results file name and layout on DFS [spark]

2025-01-10 Thread via GitHub
pan3793 opened a new pull request, #49440: URL: https://github.com/apache/spark/pull/49440 ### What changes were proposed in this pull request? This PR canonicalizes the JVM profiler added in SPARK-46094 profiling result files on DFS to ``` dfsDir/{{APP_ID}}/profile-{{AP

Re: [PR] [SPARK-50783][CORE] Canonicalize JVM profiler results file name and layout on DFS [spark]

2025-01-10 Thread via GitHub
pan3793 commented on code in PR #49440: URL: https://github.com/apache/spark/pull/49440#discussion_r1910116132 ## connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala: ## @@ -38,15 +38,26 @@ private[spark] class ExecutorJVMProfiler(conf:

Re: [PR] [SPARK-50783][CORE] Canonicalize JVM profiler results file name and layout on DFS [spark]

2025-01-10 Thread via GitHub
pan3793 commented on code in PR #49440: URL: https://github.com/apache/spark/pull/49440#discussion_r1910118560 ## connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala: ## @@ -38,15 +38,26 @@ private[spark] class ExecutorJVMProfiler(conf:

Re: [PR] [SPARK-50783][CORE] Canonicalize JVM profiler results file name and layout on DFS [spark]

2025-01-10 Thread via GitHub
pan3793 commented on code in PR #49440: URL: https://github.com/apache/spark/pull/49440#discussion_r1910121468 ## core/src/main/scala/org/apache/spark/util/Utils.scala: ## @@ -2954,6 +2954,15 @@ private[spark] object Utils str.replaceAll("[ :/]", "-").replaceAll("[.${}'\"]"

Re: [PR] [SPARK-50783][CORE] Canonicalize JVM profiler results file name and layout on DFS [spark]

2025-01-10 Thread via GitHub
pan3793 commented on code in PR #49440: URL: https://github.com/apache/spark/pull/49440#discussion_r1910127015 ## connector/profiler/README.md: ## @@ -54,7 +54,7 @@ Then enable the profiling in the configuration. spark.executor.profiling.dfsDir (none) - An HDFS co

Re: [PR] [SPARK-50783][CORE] Canonicalize JVM profiler results file name and layout on DFS [spark]

2025-01-10 Thread via GitHub
pan3793 commented on PR #49440: URL: https://github.com/apache/spark/pull/49440#issuecomment-2582254378 cc @dongjoon-hyun @parthchandra @mridulm @LuciferYang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] [SPARK-49960][SQL] Provide extension point for custom AgnosticEncoder serde [spark]

2025-01-10 Thread via GitHub
chris-twiner commented on PR #48477: URL: https://github.com/apache/spark/pull/48477#issuecomment-2582255465 > > extra black box indirection that will not allow optimisation (e.g. constant folding etc.) > > I am not sure I am following here. Is the black box you talking about the "co

Re: [PR] [SPARK-50783][CORE] Canonicalize JVM profiler results file name and layout on DFS [spark]

2025-01-10 Thread via GitHub
pan3793 commented on code in PR #49440: URL: https://github.com/apache/spark/pull/49440#discussion_r1910128066 ## connector/profiler/README.md: ## @@ -72,7 +72,7 @@ Then enable the profiling in the configuration. event=wall,interval=10ms,alloc=2m,lock=10ms,chunktime=300s

[PR] [MINOR][DOCS] Add new dataframe methods to API references [spark]

2025-01-10 Thread via GitHub
zhengruifeng opened a new pull request, #49439: URL: https://github.com/apache/spark/pull/49439 ### What changes were proposed in this pull request? Add new dataframe methods to API references ### Why are the changes needed? these new methods are missing in docs ##

[PR] [Only Test] conda issue [spark]

2025-01-10 Thread via GitHub
panbingkun opened a new pull request, #49441: URL: https://github.com/apache/spark/pull/49441 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How

Re: [PR] [SPARK-50624][SQL] Add TimestampNTZType to ColumnarRow/MutableColumnarRow [spark]

2025-01-10 Thread via GitHub
nastra commented on code in PR #49437: URL: https://github.com/apache/spark/pull/49437#discussion_r1910231591 ## sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java: ## @@ -188,7 +188,9 @@ public Object get(int ordinal, DataType dataType) { return

Re: [PR] [MINOR][BUILD] Pom compiler source target [spark]

2025-01-10 Thread via GitHub
github-actions[bot] commented on PR #48312: URL: https://github.com/apache/spark/pull/48312#issuecomment-2584951251 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] [SPARK-50601][SQL] Support withColumns / withColumnsRenamed in subqueries [spark]

2025-01-10 Thread via GitHub
ueshin commented on PR #49386: URL: https://github.com/apache/spark/pull/49386#issuecomment-2584956725 `SparkConnectPlanner` has an issue. Waiting for #49449. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[PR] [SPARK-50789][CONNECT] The inputs for typed aggregations should be analyzed [spark]

2025-01-10 Thread via GitHub
ueshin opened a new pull request, #49449: URL: https://github.com/apache/spark/pull/49449 ### What changes were proposed in this pull request? Fixes `SparkConnectPlanner` to analyze the inputs for typed aggregations. ### Why are the changes needed? The inputs for typed ag

Re: [PR] [SPARK-50783][CORE] Canonicalize JVM profiler results file name and layout on DFS [spark]

2025-01-10 Thread via GitHub
parthchandra commented on code in PR #49440: URL: https://github.com/apache/spark/pull/49440#discussion_r1911779031 ## connector/profiler/README.md: ## @@ -54,7 +54,7 @@ Then enable the profiling in the configuration. spark.executor.profiling.dfsDir (none) - An HD

Re: [PR] [SPARK-50652][SS] Add checks to RocksDB V2 backward compatibility [spark]

2025-01-10 Thread via GitHub
srielau commented on code in PR #49275: URL: https://github.com/apache/spark/pull/49275#discussion_r1911592696 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -2800,6 +2800,30 @@ }, "sqlState" : "42K0E" }, + "INVALID_LOG_VERSION" : { +"mes

[PR] [SPARK-50790][PYTHON] Implement parse json in pyspark [spark]

2025-01-10 Thread via GitHub
gene-db opened a new pull request, #49450: URL: https://github.com/apache/spark/pull/49450 ### What changes were proposed in this pull request? Implement the parseJson functionality in PySpark, for parsing a json string to a VariantVal. ### Why are the changes needed? Cu

Re: [PR] [SPARK-50694][SQL] Support withColumns / withColumnsRenamed in subqueries [spark]

2025-01-10 Thread via GitHub
ueshin commented on code in PR #49386: URL: https://github.com/apache/spark/pull/49386#discussion_r1911766156 ## sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala: ## @@ -1275,29 +1275,15 @@ class Dataset[T] private[sql]( require(colNames.size == cols.size, Review

Re: [PR] [MINOR][SQL] Remove `removeWhitespace helper` for DESCRIBE TABLE [spark]

2025-01-10 Thread via GitHub
cloud-fan commented on code in PR #49433: URL: https://github.com/apache/spark/pull/49433#discussion_r1910423796 ## sql/core/src/test/resources/sql-tests/results/show-tables.sql.out: ## @@ -128,6 +128,7 @@ View Schema Mode: BINDING Schema: root |-- e: integer (nullable = true

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1910491403 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,323 @@ class MySQLIntegrationSuite

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
beliefer commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1910325099 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +151,19 @@ private case class MySQLDialect() extends JdbcDialect with SQLConfHel

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1910368126 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +151,19 @@ private case class MySQLDialect() extends JdbcDialect with SQLCon

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1910365474 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala: ## @@ -374,6 +374,7 @@ abstract class JdbcDialect extends Serializable with Logging {

[PR] [WIP][SPARK-50403][SQL] Parameterized EXECUTE IMMEDIATE [spark]

2025-01-10 Thread via GitHub
MaxGekk opened a new pull request, #49442: URL: https://github.com/apache/spark/pull/49442 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1910365474 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala: ## @@ -374,6 +374,7 @@ abstract class JdbcDialect extends Serializable with Logging {

Re: [PR] [SPARK-50124][SQL][FOLLOWUP] InsertSortForLimitAndOffset should propagate missing ordering columns [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49416: URL: https://github.com/apache/spark/pull/49416#issuecomment-2582865495 +1, late LGTM, @cloud-fan . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1910376515 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,323 @@ class MySQLIntegrationSuite

Re: [PR] [SPARK-50707][SQL] Enable casting to/from char/varchar [spark]

2025-01-10 Thread via GitHub
cloud-fan commented on PR #49340: URL: https://github.com/apache/spark/pull/49340#issuecomment-2582710614 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1910374793 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,323 @@ class MySQLIntegrationSuite

Re: [PR] [SPARK-50707][SQL] Enable casting to/from char/varchar [spark]

2025-01-10 Thread via GitHub
cloud-fan closed pull request #49340: [SPARK-50707][SQL] Enable casting to/from char/varchar URL: https://github.com/apache/spark/pull/49340 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1910368940 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +151,19 @@ private case class MySQLDialect() extends JdbcDialect with SQLCon

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1910381035 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +151,19 @@ private case class MySQLDialect() extends JdbcDialect with SQLCon

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1910380266 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +151,19 @@ private case class MySQLDialect() extends JdbcDialect with SQLCon

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1910376515 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,323 @@ class MySQLIntegrationSuite

Re: [PR] [SPARK-50776][K8S][TESTS] Fix test assertions on executor service account [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun closed pull request #49428: [SPARK-50776][K8S][TESTS] Fix test assertions on executor service account URL: https://github.com/apache/spark/pull/49428 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] [SPARK-50776][K8S][TESTS] Fix test assertions on executor service account [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49428: URL: https://github.com/apache/spark/pull/49428#issuecomment-2582936276 Let me check and do the follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] [SPARK-50776][K8S][TESTS] Fix test assertions on executor service account [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49428: URL: https://github.com/apache/spark/pull/49428#issuecomment-2582935831 Oh, the CI seems to succeed even though the Scala linter fails. - https://github.com/cnauroth/spark/actions/runs/12698857446/job/35398309914 ![Screenshot 2025-01-10

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1910365474 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala: ## @@ -374,6 +374,7 @@ abstract class JdbcDialect extends Serializable with Logging {

Re: [PR] [SPARK-50776][KUBERNETES][TESTS] Invalid test assertions on executor Kubernetes service account [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49428: URL: https://github.com/apache/spark/pull/49428#issuecomment-2582912501 Thank you for making a PR, @cnauroth ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] [MINOR][SQL] Remove `removeWhitespace helper` for DESCRIBE TABLE [spark]

2025-01-10 Thread via GitHub
asl3 commented on code in PR #49433: URL: https://github.com/apache/spark/pull/49433#discussion_r1910450908 ## sql/core/src/test/resources/sql-tests/results/show-tables.sql.out: ## @@ -128,6 +128,7 @@ View Schema Mode: BINDING Schema: root |-- e: integer (nullable = true) +

Re: [PR] [SPARK-50784][TESTS] Fix `lint-scala` not to ignore `scalastyle` errors [spark]

2025-01-10 Thread via GitHub
grundprinzip commented on code in PR #49443: URL: https://github.com/apache/spark/pull/49443#discussion_r1910558682 ## dev/lint-scala: ## @@ -17,6 +17,8 @@ # limitations under the License. # +set -e + Review Comment: This is a bit of a weird patch. The mvn command should

Re: [PR] [SPARK-50784][TESTS] Fix `lint-scala` not to ignore `scalastyle` errors [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49443: URL: https://github.com/apache/spark/pull/49443#issuecomment-2583025520 I'm not sure why you say like that with `-e`. > This is a bit of a weird patch. The mvn command should actually continue checking for `$?` and not for the error output being e

Re: [PR] [MINOR][DOCS] Add new dataframe methods to API references [spark]

2025-01-10 Thread via GitHub
zhengruifeng commented on PR #49439: URL: https://github.com/apache/spark/pull/49439#issuecomment-2582512828 thanks @MaxGekk merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] [MINOR][DOCS] Add new dataframe methods to API references [spark]

2025-01-10 Thread via GitHub
zhengruifeng closed pull request #49439: [MINOR][DOCS] Add new dataframe methods to API references URL: https://github.com/apache/spark/pull/49439 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [SPARK-50704][SQL] Support more pushdown functions for MySQL connector [spark]

2025-01-10 Thread via GitHub
sunxiaoguang commented on code in PR #49335: URL: https://github.com/apache/spark/pull/49335#discussion_r1910365474 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala: ## @@ -374,6 +374,7 @@ abstract class JdbcDialect extends Serializable with Logging {

[PR] [SPARK-50784][TESTS] Fix `lint-scala` not to ignore `scalastyle` errors [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun opened a new pull request, #49443: URL: https://github.com/apache/spark/pull/49443 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### H

Re: [PR] [SPARK-40799] [BUILD] [CONNECT] Enforce scalafmt for Spark Connect module. [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on code in PR #38258: URL: https://github.com/apache/spark/pull/38258#discussion_r1910535785 ## dev/lint-scala: ## @@ -21,3 +21,12 @@ SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )" SPARK_ROOT_DIR="$(dirname $SCRIPT_DIR)" "$SCRIPT_DIR/scalastyle" "$1"

Re: [PR] [SPARK-50784][TESTS] Fix `lint-scala` not to ignore `scalastyle` errors [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49443: URL: https://github.com/apache/spark/pull/49443#issuecomment-2582969875 cc @grundprinzip, @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] [SPARK-40799] [BUILD] [CONNECT] Enforce scalafmt for Spark Connect module. [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #38258: URL: https://github.com/apache/spark/pull/38258#issuecomment-2582966632 I made a fix here. - #49443 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] [SPARK-40799] [BUILD] [CONNECT] Enforce scalafmt for Spark Connect module. [spark]

2025-01-10 Thread via GitHub
grundprinzip commented on code in PR #38258: URL: https://github.com/apache/spark/pull/38258#discussion_r1910542643 ## dev/lint-scala: ## @@ -21,3 +21,12 @@ SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )" SPARK_ROOT_DIR="$(dirname $SCRIPT_DIR)" "$SCRIPT_DIR/scalastyle" "$1"

Re: [PR] [SPARK-50776][K8S][TESTS] Fix test assertions on executor service account [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49428: URL: https://github.com/apache/spark/pull/49428#issuecomment-2582980598 It turns out that the Apache Spark's `lint-scala` script has a long standing bug. So, I made a PR to fix it. - #49443 -- This is an automated message from the Apache Git Servi

Re: [PR] [SPARK-50774][SQL] Make collation names public in CollationFactory [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49425: URL: https://github.com/apache/spark/pull/49425#issuecomment-2583195943 +1 for the above Max's suggestion too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[PR] [SPARK-50776][FOLLOWUP] Fix code style [spark]

2025-01-10 Thread via GitHub
panbingkun opened a new pull request, #49444: URL: https://github.com/apache/spark/pull/49444 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How

Re: [PR] [SPARK-50776][FOLLOWUP] Fix code style [spark]

2025-01-10 Thread via GitHub
panbingkun commented on PR #49444: URL: https://github.com/apache/spark/pull/49444#issuecomment-2583155463 cc @cnauroth @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] [SPARK-50776][FOLLOWUP] Fix code style [spark]

2025-01-10 Thread via GitHub
panbingkun closed pull request #49444: [SPARK-50776][FOLLOWUP] Fix code style URL: https://github.com/apache/spark/pull/49444 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [SPARK-50776][FOLLOWUP] Fix code style [spark]

2025-01-10 Thread via GitHub
panbingkun commented on PR #49444: URL: https://github.com/apache/spark/pull/49444#issuecomment-2583163672 closing in favor of https://github.com/apache/spark/pull/49443. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [SPARK-50780][SQL] Use `overrideStdFeatures` instead of `setFeatureMask` in `JacksonParser` [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun closed pull request #49434: [SPARK-50780][SQL] Use `overrideStdFeatures` instead of `setFeatureMask` in `JacksonParser` URL: https://github.com/apache/spark/pull/49434 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] [SPARK-50784][TESTS] Fix `lint-scala` not to ignore `scalastyle` errors [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49443: URL: https://github.com/apache/spark/pull/49443#issuecomment-2583239929 Could you review this, @panbingkun ? The CI starts to fail at compilation due to this. -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] [SPARK-50784][TESTS] Fix `lint-scala` not to ignore `scalastyle` errors [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49443: URL: https://github.com/apache/spark/pull/49443#issuecomment-2583246424 Also, cc @huaxingao and @viirya . When you have some time, could you review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] [WIP][SPARK-50739][SQL] Recursive CTE. Changes in ResolveWithCTE.scala to have [spark]

2025-01-10 Thread via GitHub
nemanjapetr-db commented on PR #49351: URL: https://github.com/apache/spark/pull/49351#issuecomment-2583331296 Please take another look. CTERelationDef does not contain anchor any longer -- when needed it is fetched from its child via pattern matching. Code is greatly simplified and a

Re: [PR] [SPARK-50784][TESTS] Fix `lint-scala` not to ignore `scalastyle` errors [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49443: URL: https://github.com/apache/spark/pull/49443#issuecomment-2583265660 Thank you, @grundprinzip . Yes, you are right. Previously, we didn't hit this corner case before. -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] [SPARK-50786][SQL] Remove `removeWhitespace helper` for DESCRIBE TABLE [spark]

2025-01-10 Thread via GitHub
asl3 commented on PR #49433: URL: https://github.com/apache/spark/pull/49433#issuecomment-2583276007 Thank you @dongjoon-hyun! Yes, the PR should be ready -- it restores the golden file to its state before this PR: https://github.com/apache/spark/pull/49139/files#diff-b6f30759017988fd0963ce

Re: [PR] [SPARK-50776][K8S][TESTS][FOLLOWUP] Fix code style [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49444: URL: https://github.com/apache/spark/pull/49444#issuecomment-2583280853 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] [SPARK-50776][K8S][TESTS][FOLLOWUP] Fix code style [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun closed pull request #49444: [SPARK-50776][K8S][TESTS][FOLLOWUP] Fix code style URL: https://github.com/apache/spark/pull/49444 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] [SPARK-50784][TESTS] Fix `lint-scala` not to ignore `scalastyle` errors [spark]

2025-01-10 Thread via GitHub
dongjoon-hyun commented on PR #49443: URL: https://github.com/apache/spark/pull/49443#issuecomment-2583283218 To recover master branch, I merged @panbingkun 's PR first because mine is still incomplete. - #49444 -- This is an automated message from the Apache Git Service. To respond to

  1   2   >