Re: [PR] [SPARK 49109][SQL] Rename leftover BinaryLcase to Lcase [spark]

2024-08-06 Thread via GitHub
cloud-fan commented on PR #47602: URL: https://github.com/apache/spark/pull/47602#issuecomment-2270544405 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK 49109][SQL] Rename leftover BinaryLcase to Lcase [spark]

2024-08-06 Thread via GitHub
cloud-fan closed pull request #47602: [SPARK 49109][SQL] Rename leftover BinaryLcase to Lcase URL: https://github.com/apache/spark/pull/47602 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] [SPARK-49048][SS] Add support for reading relevant operator metadata at given batch id [spark]

2024-08-06 Thread via GitHub
HeartSaVioR commented on PR #47528: URL: https://github.com/apache/spark/pull/47528#issuecomment-2270574979 It only failed with docker integration test. Thanks! Merging to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] [SPARK-49048][SS] Add support for reading relevant operator metadata at given batch id [spark]

2024-08-06 Thread via GitHub
HeartSaVioR closed pull request #47528: [SPARK-49048][SS] Add support for reading relevant operator metadata at given batch id URL: https://github.com/apache/spark/pull/47528 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[PR] [SPARK-49120][BUILD] Bump Gson 2.11.0 [spark]

2024-08-06 Thread via GitHub
pan3793 opened a new pull request, #47627: URL: https://github.com/apache/spark/pull/47627 ### What changes were proposed in this pull request? Currently, Spark pulls Gson 2.2.4 from `hive-exec`, which is pretty old and [vulnerability](https://cve.mitre.org/cgi-bin/cvename.cgi

[PR] [WIP][SPARK-49119][SQL] Fix the inconsistency of syntax `show columns` between v1 and v2 [spark]

2024-08-06 Thread via GitHub
panbingkun opened a new pull request, #47628: URL: https://github.com/apache/spark/pull/47628 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change?

[PR] [SPARK-49047][PYTHON][CONNECT][FOLLOWUP] Catch potential truncation failure [spark]

2024-08-06 Thread via GitHub
zhengruifeng opened a new pull request, #47629: URL: https://github.com/apache/spark/pull/47629 ### What changes were proposed in this pull request? Catch potential truncation failure ### Why are the changes needed? logging should not fail execution ### Does this PR introd

Re: [PR] [SPARK-47359][SQL][FOLLOWUP] Remove unused genCode for StringTranslate [spark]

2024-08-06 Thread via GitHub
yaooqinn closed pull request #47623: [SPARK-47359][SQL][FOLLOWUP] Remove unused genCode for StringTranslate URL: https://github.com/apache/spark/pull/47623 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [SPARK-47359][SQL][FOLLOWUP] Remove unused genCode for StringTranslate [spark]

2024-08-06 Thread via GitHub
yaooqinn commented on PR #47623: URL: https://github.com/apache/spark/pull/47623#issuecomment-2270688871 Merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-06 Thread via GitHub
yaooqinn closed pull request #47592: [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog URL: https://github.com/apache/spark/pull/47592 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-06 Thread via GitHub
yaooqinn commented on PR #47592: URL: https://github.com/apache/spark/pull/47592#issuecomment-2270742204 Merged to master, Thank you all -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] [SPARK-48978][SQL] Implement ASCII fast path in collation support for UTF8_LCASE [spark]

2024-08-06 Thread via GitHub
cloud-fan commented on PR #47326: URL: https://github.com/apache/spark/pull/47326#issuecomment-2270773188 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-48978][SQL] Implement ASCII fast path in collation support for UTF8_LCASE [spark]

2024-08-06 Thread via GitHub
cloud-fan closed pull request #47326: [SPARK-48978][SQL] Implement ASCII fast path in collation support for UTF8_LCASE URL: https://github.com/apache/spark/pull/47326 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-06 Thread via GitHub
cloud-fan commented on PR #47592: URL: https://github.com/apache/spark/pull/47592#issuecomment-2270800457 Since it's a bug fix, I backported it to 3.5 as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] [SPARK-49120][BUILD] Bump Gson 2.11.0 [spark]

2024-08-06 Thread via GitHub
pan3793 commented on PR #47627: URL: https://github.com/apache/spark/pull/47627#issuecomment-2270802518 cc @dongjoon-hyun @yaooqinn @LuciferYang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] [SPARK-49119][SQL] Fix the inconsistency of syntax `show columns` between v1 and v2 [spark]

2024-08-06 Thread via GitHub
panbingkun commented on PR #47628: URL: https://github.com/apache/spark/pull/47628#issuecomment-2270813066 cc @cloud-fan @yaooqinn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] [SPARK-49120][BUILD] Bump Gson 2.11.0 [spark]

2024-08-06 Thread via GitHub
yaooqinn commented on code in PR #47627: URL: https://github.com/apache/spark/pull/47627#discussion_r1705243660 ## dev/deps/spark-deps-hadoop-3-hive-2.3: ## @@ -62,11 +62,12 @@ derby/10.16.1.1//derby-10.16.1.1.jar derbyshared/10.16.1.1//derbyshared-10.16.1.1.jar derbytools/10.

Re: [PR] [SPARK-49120][BUILD] Bump Gson 2.11.0 [spark]

2024-08-06 Thread via GitHub
LuciferYang commented on code in PR #47627: URL: https://github.com/apache/spark/pull/47627#discussion_r1705248967 ## dev/deps/spark-deps-hadoop-3-hive-2.3: ## @@ -62,11 +62,12 @@ derby/10.16.1.1//derby-10.16.1.1.jar derbyshared/10.16.1.1//derbyshared-10.16.1.1.jar derbytools/

Re: [PR] [SPARK-49120][BUILD] Bump Gson 2.11.0 [spark]

2024-08-06 Thread via GitHub
pan3793 commented on code in PR #47627: URL: https://github.com/apache/spark/pull/47627#discussion_r1705264618 ## dev/deps/spark-deps-hadoop-3-hive-2.3: ## @@ -62,11 +62,12 @@ derby/10.16.1.1//derby-10.16.1.1.jar derbyshared/10.16.1.1//derbyshared-10.16.1.1.jar derbytools/10.1

Re: [PR] [SPARK-49120][BUILD] Bump Gson 2.11.0 [spark]

2024-08-06 Thread via GitHub
yaooqinn commented on code in PR #47627: URL: https://github.com/apache/spark/pull/47627#discussion_r1705270688 ## dev/deps/spark-deps-hadoop-3-hive-2.3: ## @@ -62,11 +62,12 @@ derby/10.16.1.1//derby-10.16.1.1.jar derbyshared/10.16.1.1//derbyshared-10.16.1.1.jar derbytools/10.

[PR] [SPARK-49099][SQL][FOLLOWUP] recover tests in DDLSuite [spark]

2024-08-06 Thread via GitHub
cloud-fan opened a new pull request, #47630: URL: https://github.com/apache/spark/pull/47630 ### What changes were proposed in this pull request? This is a followup of https://github.com/apache/spark/pull/47592 to fix test failure during backport. ### Why are the change

Re: [PR] [SPARK-49099][SQL][FOLLOWUP][3.5] recover tests in DDLSuite [spark]

2024-08-06 Thread via GitHub
yaooqinn commented on PR #47630: URL: https://github.com/apache/spark/pull/47630#issuecomment-2271012908 Merged to branch-3.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Enhance the metrics in SparkUI with logical plan stats [spark]

2024-08-06 Thread via GitHub
HyukjinKwon commented on PR #47534: URL: https://github.com/apache/spark/pull/47534#issuecomment-2271083000 Mind filing a JIRA and add it into the PR title? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [SPARK-49047][PYTHON][CONNECT][FOLLOWUP] Catch potential truncation failure [spark]

2024-08-06 Thread via GitHub
HyukjinKwon commented on PR #47629: URL: https://github.com/apache/spark/pull/47629#issuecomment-2271085888 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [SPARK-49047][PYTHON][CONNECT][FOLLOWUP] Catch potential truncation failure [spark]

2024-08-06 Thread via GitHub
HyukjinKwon closed pull request #47629: [SPARK-49047][PYTHON][CONNECT][FOLLOWUP] Catch potential truncation failure URL: https://github.com/apache/spark/pull/47629 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] [MINOR][INFRA] Mute `stale` Github Action in forks [spark]

2024-08-06 Thread via GitHub
HyukjinKwon commented on PR #47626: URL: https://github.com/apache/spark/pull/47626#issuecomment-2271087398 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [WIP][SPARK-48763][TESTS][FOLLOW-UP] Update project location in PlanGenerationTestSuite [spark]

2024-08-06 Thread via GitHub
HyukjinKwon closed pull request #47624: [WIP][SPARK-48763][TESTS][FOLLOW-UP] Update project location in PlanGenerationTestSuite URL: https://github.com/apache/spark/pull/47624 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] [MINOR][INFRA] Mute `stale` Github Action in forks [spark]

2024-08-06 Thread via GitHub
HyukjinKwon closed pull request #47626: [MINOR][INFRA] Mute `stale` Github Action in forks URL: https://github.com/apache/spark/pull/47626 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] [SPARK-49099][SQL][FOLLOWUP][3.5] recover tests in DDLSuite [spark]

2024-08-06 Thread via GitHub
yaooqinn closed pull request #47630: [SPARK-49099][SQL][FOLLOWUP][3.5] recover tests in DDLSuite URL: https://github.com/apache/spark/pull/47630 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] [SPARK-49004][CONNECT] Use separate registry for Column API internal functions [spark]

2024-08-06 Thread via GitHub
hvanhovell commented on code in PR #47572: URL: https://github.com/apache/spark/pull/47572#discussion_r1705482291 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala: ## @@ -342,7 +342,8 @@ case class UnresolvedFunction( isDistinct: Boolea

Re: [PR] [SPARK-49004][CONNECT] Use separate registry for Column API internal functions [spark]

2024-08-06 Thread via GitHub
hvanhovell commented on code in PR #47572: URL: https://github.com/apache/spark/pull/47572#discussion_r1705487892 ## sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -1614,14 +1614,23 @@ class SparkConnectPlanner( fun

[PR] [SPARK-49122][Connect][SQL] Add addArtifact API to the Spark SQL Core [spark]

2024-08-06 Thread via GitHub
xupefei opened a new pull request, #47631: URL: https://github.com/apache/spark/pull/47631 ### What changes were proposed in this pull request? This PR improves Spark SQL Core to add a bunch of `addArtifact` APIs to `SparkSession`. These APIs are first introduced to Spark Connect a wh

Re: [PR] [SPARK-49119][SQL] Fix the inconsistency of syntax `show columns` between v1 and v2 [spark]

2024-08-06 Thread via GitHub
cloud-fan commented on code in PR #47628: URL: https://github.com/apache/spark/pull/47628#discussion_r1705588735 ## sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala: ## @@ -1385,8 +1385,10 @@ abstract class DDLSuite extends QueryTest with DDLSuiteBa

Re: [PR] [SPARK-49082][SQL] Widening type promotions in `AvroDeserializer` [spark]

2024-08-06 Thread via GitHub
cloud-fan commented on code in PR #47582: URL: https://github.com/apache/spark/pull/47582#discussion_r1705597732 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -194,6 +200,9 @@ private[sql] class AvroDeserializer( case (FLOAT, Flo

Re: [PR] [SPARK-49004][CONNECT] Use separate registry for Column API internal functions [spark]

2024-08-06 Thread via GitHub
hvanhovell commented on PR #47572: URL: https://github.com/apache/spark/pull/47572#issuecomment-2271373687 Merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] [SPARK-49004][CONNECT] Use separate registry for Column API internal functions [spark]

2024-08-06 Thread via GitHub
asfgit closed pull request #47572: [SPARK-49004][CONNECT] Use separate registry for Column API internal functions URL: https://github.com/apache/spark/pull/47572 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] [SPARK-49082][SQL] Widening type promotions in `AvroDeserializer` [spark]

2024-08-06 Thread via GitHub
wayneguow commented on code in PR #47582: URL: https://github.com/apache/spark/pull/47582#discussion_r1705604844 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -194,6 +200,9 @@ private[sql] class AvroDeserializer( case (FLOAT, Flo

Re: [PR] [SPARK-49120][BUILD] Bump Gson 2.11.0 [spark]

2024-08-06 Thread via GitHub
yaooqinn closed pull request #47627: [SPARK-49120][BUILD] Bump Gson 2.11.0 URL: https://github.com/apache/spark/pull/47627 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] [SPARK-49120][BUILD] Bump Gson 2.11.0 [spark]

2024-08-06 Thread via GitHub
yaooqinn commented on PR #47627: URL: https://github.com/apache/spark/pull/47627#issuecomment-2271387176 Thank you @pan3793 @LuciferYang Merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [SPARK-49082][SQL] Widening type promotions in `AvroDeserializer` [spark]

2024-08-06 Thread via GitHub
wayneguow commented on code in PR #47582: URL: https://github.com/apache/spark/pull/47582#discussion_r1705612916 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -194,6 +200,9 @@ private[sql] class AvroDeserializer( case (FLOAT, Flo

Re: [PR] [SPARK-49082][SQL] Widening type promotions in `AvroDeserializer` [spark]

2024-08-06 Thread via GitHub
wayneguow commented on code in PR #47582: URL: https://github.com/apache/spark/pull/47582#discussion_r1705612916 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -194,6 +200,9 @@ private[sql] class AvroDeserializer( case (FLOAT, Flo

Re: [PR] [SPARK-49082][SQL] Widening type promotions in `AvroDeserializer` [spark]

2024-08-06 Thread via GitHub
cloud-fan commented on code in PR #47582: URL: https://github.com/apache/spark/pull/47582#discussion_r1705616093 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -194,6 +200,9 @@ private[sql] class AvroDeserializer( case (FLOAT, Flo

Re: [PR] [SPARK-48911][SQL][TESTS] Improve collation support testing for various expressions [spark]

2024-08-06 Thread via GitHub
cloud-fan commented on PR #47372: URL: https://github.com/apache/spark/pull/47372#issuecomment-2271405641 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-48911][SQL][TESTS] Improve collation support testing for various expressions [spark]

2024-08-06 Thread via GitHub
cloud-fan closed pull request #47372: [SPARK-48911][SQL][TESTS] Improve collation support testing for various expressions URL: https://github.com/apache/spark/pull/47372 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] [SPARK-49082][SQL] Widening type promotions in `AvroDeserializer` [spark]

2024-08-06 Thread via GitHub
wayneguow commented on code in PR #47582: URL: https://github.com/apache/spark/pull/47582#discussion_r1705625868 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -194,6 +200,9 @@ private[sql] class AvroDeserializer( case (FLOAT, Flo

Re: [PR] [SPARK-48821][SQL] Support Update in DataFrameWriterV2 [spark]

2024-08-06 Thread via GitHub
cloud-fan commented on code in PR #47233: URL: https://github.com/apache/spark/pull/47233#discussion_r1705631822 ## sql/core/src/main/scala/org/apache/spark/sql/UpdateWriter.scala: ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more +

Re: [PR] [SPARK-48821][SQL] Support Update in DataFrameWriterV2 [spark]

2024-08-06 Thread via GitHub
cloud-fan commented on code in PR #47233: URL: https://github.com/apache/spark/pull/47233#discussion_r1705634549 ## sql/core/src/main/scala/org/apache/spark/sql/UpdateWriter.scala: ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more +

Re: [PR] [SPARK-49062][SQL] Migrate XML to File Data Source V2 [spark]

2024-08-06 Thread via GitHub
wayneguow commented on PR #47539: URL: https://github.com/apache/spark/pull/47539#issuecomment-2271452001 Gentle ping @cloud-fan , if you have time, please help take a look at this code related to v2. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [SPARK-36680][SQL] Supports Dynamic Table Options for Spark SQL [spark]

2024-08-06 Thread via GitHub
advancedxy commented on PR #46707: URL: https://github.com/apache/spark/pull/46707#issuecomment-2271453058 > Other ideas welcome. Same scenario applies to UPDATE which also has single table identifier to read from and write to. I noticed SQL Server uses with to specify hints: https:/

Re: [PR] [SPARK-49095][SQL] Update `DecimalType`and `Decimal` compatible logic of `Avro` data source to avoid loss of decimal precision [spark]

2024-08-06 Thread via GitHub
wayneguow commented on PR #47584: URL: https://github.com/apache/spark/pull/47584#issuecomment-2271456407 cc @cloud-fan @LuciferYang when you have time :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[PR] [SPARK-49124][BUILD] Upgrade tink to 1.14.1 [spark]

2024-08-06 Thread via GitHub
wayneguow opened a new pull request, #47632: URL: https://github.com/apache/spark/pull/47632 ### What changes were proposed in this pull request? This PR aims to upgrade `tink` from 1.13.0 to 1.14.1. ### Why are the changes needed? There are some bug fixes and per

Re: [PR] [SPARK-49098][SQL] Add write options for INSERT [spark]

2024-08-06 Thread via GitHub
dongjoon-hyun closed pull request #47591: [SPARK-49098][SQL] Add write options for INSERT URL: https://github.com/apache/spark/pull/47591 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] [SPARK-49125][SQL] Allow duplicated column names in CSV writing [spark]

2024-08-06 Thread via GitHub
cloud-fan opened a new pull request, #47633: URL: https://github.com/apache/spark/pull/47633 ### What changes were proposed in this pull request? In file source writing, we disallow duplicated column names in the input query for all formats, because most formats don't do well

Re: [PR] [SPARK-49125][SQL] Allow duplicated column names in CSV writing [spark]

2024-08-06 Thread via GitHub
cloud-fan commented on PR #47633: URL: https://github.com/apache/spark/pull/47633#issuecomment-2271554150 cc @MaxGekk @viirya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] [SPARK-39142][SPARK-42235][PYTHON] Add overloads in pandas function stub file [spark]

2024-08-06 Thread via GitHub
pontusvision commented on PR #39974: URL: https://github.com/apache/spark/pull/39974#issuecomment-2271554177 Please reopen this; I'm also experiencing issues with pylance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

[PR] [SPARK-48843] Adding more tests that would have failed had the fix ap… [spark]

2024-08-06 Thread via GitHub
nemanjapetr-db opened a new pull request, #47634: URL: https://github.com/apache/spark/pull/47634 …ache/spark#47271 not been sumitted. Prevents regression. Hardens previous tests to catch infinite loop caused by head() and first() in Connect. Adds a test that would have caused infinite loop

Re: [PR] [SPARK-48989][SQL][FOLLOWUP] Fix SubstringIndex codegen [spark]

2024-08-06 Thread via GitHub
cloud-fan commented on PR #47610: URL: https://github.com/apache/spark/pull/47610#issuecomment-2271591986 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-48989][SQL][FOLLOWUP] Fix SubstringIndex codegen [spark]

2024-08-06 Thread via GitHub
cloud-fan closed pull request #47610: [SPARK-48989][SQL][FOLLOWUP] Fix SubstringIndex codegen URL: https://github.com/apache/spark/pull/47610 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] [SPARK-49098][SQL] Add write options for INSERT [spark]

2024-08-06 Thread via GitHub
dongjoon-hyun commented on PR #47591: URL: https://github.com/apache/spark/pull/47591#issuecomment-2271599728 BTW, could you make a new PR to revise SQL documentation with this new syntax, @szehon-ho ? -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [SPARK-49095][SQL] Update `DecimalType`and `Decimal` compatible logic of `Avro` data source to avoid loss of decimal precision [spark]

2024-08-06 Thread via GitHub
cloud-fan commented on PR #47584: URL: https://github.com/apache/spark/pull/47584#issuecomment-2271601794 Can we put more information into `Does this PR introduce any user-facing change`? Especially what kind of queries will be broken after this change. -- This is an automated message fro

Re: [PR] [SPARK-49098][SQL] Add write options for INSERT [spark]

2024-08-06 Thread via GitHub
cloud-fan commented on PR #47591: URL: https://github.com/apache/spark/pull/47591#issuecomment-2271604845 late LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] [SPARK-49095][SQL] Update `DecimalType`and `Decimal` compatible logic of `Avro` data source to avoid loss of decimal precision [spark]

2024-08-06 Thread via GitHub
wayneguow commented on PR #47584: URL: https://github.com/apache/spark/pull/47584#issuecomment-2271626302 > Can we put more information into `Does this PR introduce any user-facing change`? Especially what kind of queries will be broken after this change. Updated it. -- This is an

Re: [PR] [SPARK-49082][SQL] Widening type promotions in `AvroDeserializer` [spark]

2024-08-06 Thread via GitHub
wayneguow commented on PR #47582: URL: https://github.com/apache/spark/pull/47582#issuecomment-2271632858 Test failure is not related. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] Spark 49089 [spark]

2024-08-06 Thread via GitHub
hvanhovell opened a new pull request, #47635: URL: https://github.com/apache/spark/pull/47635 ### What changes were proposed in this pull request? This PR adds Expression only constructors to all Catalyst Expressions were still hard coded in the Connect planner and the Column API. These

[PR] [SPARK-49126][CORE] Move `spark.history.ui.maxApplications` config definition to `History.scala` [spark]

2024-08-06 Thread via GitHub
dongjoon-hyun opened a new pull request, #47636: URL: https://github.com/apache/spark/pull/47636 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change?

[PR] [SPARK-49086][CONNECT] Move ML function registration to SparkSessionExtensions [spark]

2024-08-06 Thread via GitHub
hvanhovell opened a new pull request, #47637: URL: https://github.com/apache/spark/pull/47637 ### What changes were proposed in this pull request? This PR moves ML function registration from the SparkConnectPlanner to the internal function registry. This registration is done using the Sp

Re: [PR] [SPARK-48967][SQL] Improve performance and memory footprint of "INSERT INTO ... VALUES" Statements [spark]

2024-08-06 Thread via GitHub
costas-db commented on code in PR #47428: URL: https://github.com/apache/spark/pull/47428#discussion_r1705963187 ## sql/core/src/test/scala/org/apache/spark/sql/InlineTableParsingImprovementsSuite.scala: ## @@ -0,0 +1,239 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] [SPARK-49111] DataSourceV2Strategy - Move static methods to companion object [spark]

2024-08-06 Thread via GitHub
gatorsmile commented on code in PR #47606: URL: https://github.com/apache/spark/pull/47606#discussion_r1705970068 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala: ## @@ -645,6 +633,21 @@ private[sql] object DataSourceV2Strategy

Re: [PR] [SPARK-49111] DataSourceV2Strategy - Move static methods to companion object [spark]

2024-08-06 Thread via GitHub
gatorsmile commented on code in PR #47606: URL: https://github.com/apache/spark/pull/47606#discussion_r1705971783 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala: ## @@ -645,6 +633,21 @@ private[sql] object DataSourceV2Strategy

Re: [PR] [SPARK-48967][SQL] Improve performance and memory footprint of "INSERT INTO ... VALUES" Statements [spark]

2024-08-06 Thread via GitHub
costas-db commented on code in PR #47428: URL: https://github.com/apache/spark/pull/47428#discussion_r1705979913 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/EvaluateUnresolvedInlineTable.scala: ## @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] [SPARK-48967][SQL] Improve performance and memory footprint of "INSERT INTO ... VALUES" Statements [spark]

2024-08-06 Thread via GitHub
costas-db commented on code in PR #47428: URL: https://github.com/apache/spark/pull/47428#discussion_r1705979913 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/EvaluateUnresolvedInlineTable.scala: ## @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-06 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1705992596 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulProcessorHandleImpl.scala: ## @@ -121,12 +122,15 @@ class StatefulProcessorHandleImpl(

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-06 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1705993422 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala: ## @@ -703,6 +706,24 @@ object StreamExecution { "py4j.protocol.Py4

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-06 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1705996138 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StateStoreColumnFamilySchemaUtils.scala: ## @@ -16,49 +16,115 @@ */ package org.apache.spark

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-06 Thread via GitHub
anishshri-db commented on PR #47524: URL: https://github.com/apache/spark/pull/47524#issuecomment-2271972932 @ericm-db - are the docs failures related ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] [ES-1074670][SQL] Add parquet nanosAsLong behavior change to 3.2 migration guide [spark]

2024-08-06 Thread via GitHub
asl3 opened a new pull request, #47638: URL: https://github.com/apache/spark/pull/47638 ### What changes were proposed in this pull request? Add Spark 3.2 migration guide for `CREATE TABLE AS SELECT...` behavior change. SPARK-40819 allows for nanosecond precision in Pa

[PR] [SPARK-49128][CORE] Support custom History Server UI title [spark]

2024-08-06 Thread via GitHub
dongjoon-hyun opened a new pull request, #47639: URL: https://github.com/apache/spark/pull/47639 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### H

Re: [PR] [SPARK-49045] Add docker image build for operator [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
dongjoon-hyun closed pull request #28: [SPARK-49045] Add docker image build for operator URL: https://github.com/apache/spark-kubernetes-operator/pull/28 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Operator 0.1.0 [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
dongjoon-hyun commented on PR #2: URL: https://github.com/apache/spark-kubernetes-operator/pull/2#issuecomment-2272129504 The following are merged too. - #28 - #29 Now, we can move on to the remaining modules, @jiangzho . - spark-operator-tests - spark-operator-docs -

[PR] [SPARK-49024][CONNECT] Add support for functions to column node. [spark]

2024-08-06 Thread via GitHub
hvanhovell opened a new pull request, #47640: URL: https://github.com/apache/spark/pull/47640 ### What changes were proposed in this pull request? This PR adds support for UDFs to the Column Node API. ### Why are the changes needed? We want to unify the Classic and Connect Column

Re: [PR] [SPARK-48949][SQL] SPJ: Runtime partition filtering [spark]

2024-08-06 Thread via GitHub
szehon-ho commented on code in PR #47426: URL: https://github.com/apache/spark/pull/47426#discussion_r1706121389 ## sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala: ## @@ -429,8 +429,19 @@ case class EnsureRequirements( // expres

Re: [PR] [SPARK-48700] [SQL] Mode expression for complex types (all collations) [spark]

2024-08-06 Thread via GitHub
GideonPotok commented on PR #47154: URL: https://github.com/apache/spark/pull/47154#issuecomment-2272164661 @uros-db , I just wanted to loop you in that I am just going to put finishing this PR on hold for like two weeks while I attend to some other responsibilities. Work is pretty crazy ri

[PR] [SPARK-49129] Fix `ENTRYPOINT` to point `/opt/spark-operator/operator/docker-entrypoint.sh` [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
dongjoon-hyun opened a new pull request, #31: URL: https://github.com/apache/spark-kubernetes-operator/pull/31 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing chang

Re: [PR] [SPARK-49129] Fix `ENTRYPOINT` to point `/opt/spark-operator/operator/docker-entrypoint.sh` [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
dongjoon-hyun commented on PR #31: URL: https://github.com/apache/spark-kubernetes-operator/pull/31#issuecomment-2272170330 cc @viirya and @jiangzho -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-49129] Fix `ENTRYPOINT` to point `/opt/spark-operator/operator/docker-entrypoint.sh` [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
dongjoon-hyun commented on PR #31: URL: https://github.com/apache/spark-kubernetes-operator/pull/31#issuecomment-2272171124 Thank you, @viirya ! Merged to main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] [SPARK-49129] Fix `ENTRYPOINT` to point `/opt/spark-operator/operator/docker-entrypoint.sh` [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
dongjoon-hyun closed pull request #31: [SPARK-49129] Fix `ENTRYPOINT` to point `/opt/spark-operator/operator/docker-entrypoint.sh` URL: https://github.com/apache/spark-kubernetes-operator/pull/31 -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] [SPARK-49126][CORE] Move `spark.history.ui.maxApplications` config definition to `History.scala` [spark]

2024-08-06 Thread via GitHub
dongjoon-hyun commented on PR #47636: URL: https://github.com/apache/spark/pull/47636#issuecomment-2272173543 Could you review this PR, @viirya ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] [SPARK-49128][CORE] Support custom History Server UI title [spark]

2024-08-06 Thread via GitHub
dongjoon-hyun commented on PR #47639: URL: https://github.com/apache/spark/pull/47639#issuecomment-2272173724 Could you review this PR, @viirya ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] [SPARK-49126][CORE] Move `spark.history.ui.maxApplications` config definition to `History.scala` [spark]

2024-08-06 Thread via GitHub
dongjoon-hyun commented on PR #47636: URL: https://github.com/apache/spark/pull/47636#issuecomment-2272179925 Thank you, @viirya ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] [SPARK-49128][CORE] Support custom History Server UI title [spark]

2024-08-06 Thread via GitHub
dongjoon-hyun commented on PR #47639: URL: https://github.com/apache/spark/pull/47639#issuecomment-2272180213 Thank you, @viirya ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[PR] [SPARK-45923] Verify built images in `build-image` CI job via `docker run` test [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
dongjoon-hyun opened a new pull request, #32: URL: https://github.com/apache/spark-kubernetes-operator/pull/32 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing chan

Re: [PR] [SPARK-48821][SQL] Support Update in DataFrameWriterV2 [spark]

2024-08-06 Thread via GitHub
szehon-ho commented on PR #47233: URL: https://github.com/apache/spark/pull/47233#issuecomment-2272195204 Sqashed the commits to make rebase easier, but resolved the comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [SPARK-49129] Fix `ENTRYPOINT` to point `/opt/spark-operator/operator/docker-entrypoint.sh` [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
jiangzho commented on PR #31: URL: https://github.com/apache/spark-kubernetes-operator/pull/31#issuecomment-2272194818 +1 - late LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-45923] Verify built images in `build-image` CI job via `docker run` test [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
dongjoon-hyun commented on PR #32: URL: https://github.com/apache/spark-kubernetes-operator/pull/32#issuecomment-2272199470 cc @viirya and @jiangzho -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] [SPARK-49132] Minimize docker image by removing redundant `chown` commands [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
dongjoon-hyun opened a new pull request, #33: URL: https://github.com/apache/spark-kubernetes-operator/pull/33 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing chang

Re: [PR] [SPARK-49132] Minimize docker image by removing redundant `chown` commands [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
dongjoon-hyun commented on PR #33: URL: https://github.com/apache/spark-kubernetes-operator/pull/33#issuecomment-2272226860 cc @viirya and @jiangzho -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-49129] Fix `ENTRYPOINT` to point `/opt/spark-operator/operator/docker-entrypoint.sh` [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
dongjoon-hyun commented on PR #31: URL: https://github.com/apache/spark-kubernetes-operator/pull/31#issuecomment-2272284828 Thank you, @jiangzho . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-45923] Verify built images in `build-image` CI job via `docker run` test [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
dongjoon-hyun commented on PR #32: URL: https://github.com/apache/spark-kubernetes-operator/pull/32#issuecomment-2272285043 Thank you, @jiangzho . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-49132] Minimize docker image by removing redundant `chown` commands [spark-kubernetes-operator]

2024-08-06 Thread via GitHub
dongjoon-hyun commented on PR #33: URL: https://github.com/apache/spark-kubernetes-operator/pull/33#issuecomment-2272285223 Thank you, @jiangzho . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-06 Thread via GitHub
ericm-db commented on PR #47524: URL: https://github.com/apache/spark/pull/47524#issuecomment-2272301543 @HeartSaVioR PTAL when you get a chance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

  1   2   >