[GitHub] [hudi] hudi-bot commented on pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
hudi-bot commented on pull request #4083: URL: https://github.com/apache/hudi/pull/4083#issuecomment-1008247208 ## CI report: * 00221c82e8b1693280fd72625eafcd503d54323c UNKNOWN * 46053bb143d1fd1274ac466197cc9361708e738b UNKNOWN * 020118c4b169d47e668f37240388e8d1bbdfad70 Azur

[jira] [Commented] (HUDI-3158) Reduce warn logs in Spark SQL INSERT OVERWRITE

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471296#comment-17471296 ] Raymond Xu commented on HUDI-3158: -- [~shivnarayan][~dongkelun] the needed changes look bi

[GitHub] [hudi] hudi-bot removed a comment on pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
hudi-bot removed a comment on pull request #4083: URL: https://github.com/apache/hudi/pull/4083#issuecomment-1008245348 ## CI report: * 00221c82e8b1693280fd72625eafcd503d54323c UNKNOWN * 46053bb143d1fd1274ac466197cc9361708e738b UNKNOWN * 020118c4b169d47e668f37240388e8d1bbdfa

[jira] [Updated] (HUDI-3158) Reduce warn logs in Spark SQL INSERT OVERWRITE

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3158: - Story Points: 1 (was: 0.5) > Reduce warn logs in Spark SQL INSERT OVERWRITE > ---

[hudi] branch master updated (0d8ca8d -> 3679070)

2022-01-08 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 0d8ca8d [HUDI-3104] Kafka-connect support of hadoop config environments and properties (#4451) add 3679070 [H

[jira] [Closed] (HUDI-3125) Spark SQL writing timestamp type don't need to disable `spark.sql.datetime.java8API.enabled` manually

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu closed HUDI-3125. Resolution: Fixed > Spark SQL writing timestamp type don't need to disable > `spark.sql.datetime.java8API.e

[GitHub] [hudi] xushiyan merged pull request #4471: [HUDI-3125] spark-sql write timestamp directly

2022-01-08 Thread GitBox
xushiyan merged pull request #4471: URL: https://github.com/apache/hudi/pull/4471 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr

[GitHub] [hudi] hudi-bot removed a comment on pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
hudi-bot removed a comment on pull request #4083: URL: https://github.com/apache/hudi/pull/4083#issuecomment-1003742831 ## CI report: * 00221c82e8b1693280fd72625eafcd503d54323c UNKNOWN * 46053bb143d1fd1274ac466197cc9361708e738b UNKNOWN * 020118c4b169d47e668f37240388e8d1bbdfa

[GitHub] [hudi] hudi-bot commented on pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
hudi-bot commented on pull request #4083: URL: https://github.com/apache/hudi/pull/4083#issuecomment-1008245348 ## CI report: * 00221c82e8b1693280fd72625eafcd503d54323c UNKNOWN * 46053bb143d1fd1274ac466197cc9361708e738b UNKNOWN * 020118c4b169d47e668f37240388e8d1bbdfad70 Azur

[jira] [Created] (HUDI-3198) Spark SQL create table should check partition fields

2022-01-08 Thread Raymond Xu (Jira)
Raymond Xu created HUDI-3198: Summary: Spark SQL create table should check partition fields Key: HUDI-3198 URL: https://issues.apache.org/jira/browse/HUDI-3198 Project: Apache Hudi Issue Type: Im

[jira] [Updated] (HUDI-3198) Spark SQL create table should check partition fields

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3198: - Fix Version/s: 0.11.0 > Spark SQL create table should check partition fields > ---

[jira] [Assigned] (HUDI-3198) Spark SQL create table should check partition fields

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu reassigned HUDI-3198: Assignee: Yann Byron > Spark SQL create table should check partition fields > -

[GitHub] [hudi] yihua commented on pull request #3420: [HUDI-2283] Support Clustering Command For Spark Sql

2022-01-08 Thread GitBox
yihua commented on pull request #3420: URL: https://github.com/apache/hudi/pull/3420#issuecomment-1008243291 @pengzhiwei2018 Could you rebase the PR on latest master to resolve the conflicts? -- This is an automated message from the Apache Git Service. To respond to the message, please l

[hudi] branch master updated: [HUDI-3104] Kafka-connect support of hadoop config environments and properties (#4451)

2022-01-08 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 0d8ca8d [HUDI-3104] Kafka-connect support of hadoo

[GitHub] [hudi] yihua merged pull request #4451: [HUDI-3104] Kafka-connect support hadoop config environments and properties

2022-01-08 Thread GitBox
yihua merged pull request #4451: URL: https://github.com/apache/hudi/pull/4451 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...

[GitHub] [hudi] yihua commented on a change in pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem

2022-01-08 Thread GitBox
yihua commented on a change in pull request #4458: URL: https://github.com/apache/hudi/pull/4458#discussion_r780742854 ## File path: hudi-kafka-connect/src/main/java/org/apache/hudi/connect/writers/KafkaConnectTransactionServices.java ## @@ -185,20 +187,50 @@ private void sync

[GitHub] [hudi] dongkelun commented on a change in pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
dongkelun commented on a change in pull request #4083: URL: https://github.com/apache/hudi/pull/4083#discussion_r780742793 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java ## @@ -90,6 +90,11 @@ // It is here so that bo

[GitHub] [hudi] yihua commented on a change in pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem

2022-01-08 Thread GitBox
yihua commented on a change in pull request #4458: URL: https://github.com/apache/hudi/pull/4458#discussion_r780742608 ## File path: hudi-kafka-connect/src/main/java/org/apache/hudi/connect/writers/KafkaConnectTransactionServices.java ## @@ -185,20 +187,50 @@ private void sync

[GitHub] [hudi] yihua commented on a change in pull request #4458: [HUDI-3112] Fix KafkaConnect can not sync to Hive Problem

2022-01-08 Thread GitBox
yihua commented on a change in pull request #4458: URL: https://github.com/apache/hudi/pull/4458#discussion_r780742608 ## File path: hudi-kafka-connect/src/main/java/org/apache/hudi/connect/writers/KafkaConnectTransactionServices.java ## @@ -185,20 +187,50 @@ private void sync

[GitHub] [hudi] dongkelun commented on a change in pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
dongkelun commented on a change in pull request #4083: URL: https://github.com/apache/hudi/pull/4083#discussion_r780742376 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/InputPathHandler.java ## @@ -117,9 +124,10 @@ private void parseInputPaths(Path[] inputP

[GitHub] [hudi] dongkelun commented on a change in pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
dongkelun commented on a change in pull request #4083: URL: https://github.com/apache/hudi/pull/4083#discussion_r780741795 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java ## @@ -90,6 +90,11 @@ // It is here so that bo

[GitHub] [hudi] dongkelun commented on a change in pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
dongkelun commented on a change in pull request #4083: URL: https://github.com/apache/hudi/pull/4083#discussion_r780741275 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestCreateTable.scala ## @@ -33,6 +33,10 @@ import scala.collectio

[GitHub] [hudi] dongkelun commented on a change in pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
dongkelun commented on a change in pull request #4083: URL: https://github.com/apache/hudi/pull/4083#discussion_r780741202 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java ## @@ -90,6 +90,11 @@ // It is here so that bo

[jira] [Updated] (HUDI-3197) Validate partition pruning with Hudi

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3197: - Reviewers: Raymond Xu, sivabalan narayanan (was: sivabalan narayanan) > Validate partition pruning with H

[jira] [Assigned] (HUDI-3197) Validate partition pruning with Hudi

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu reassigned HUDI-3197: Assignee: sivabalan narayanan (was: Raymond Xu) > Validate partition pruning with Hudi > -

[jira] [Closed] (HUDI-3195) optimize spark3 pom and modify build command

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu closed HUDI-3195. Resolution: Fixed > optimize spark3 pom and modify build command > -

[jira] [Updated] (HUDI-3190) Validate and certify partition pruning for hudi tables w/ spark queries

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3190: - Fix Version/s: (was: 0.10.1) > Validate and certify partition pruning for hudi tables w/ spark queries

[jira] [Closed] (HUDI-3190) Validate and certify partition pruning for hudi tables w/ spark queries

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu closed HUDI-3190. Assignee: (was: Raymond Xu) Resolution: Duplicate > Validate and certify partition pruning for hud

[jira] [Closed] (HUDI-3197) Validate partition pruning with Hudi

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu closed HUDI-3197. Reviewers: sivabalan narayanan Resolution: Done > Validate partition pruning with Hudi > --

[jira] [Comment Edited] (HUDI-3066) Very slow file listing after enabling metadata for existing tables in 0.10.0 release

2022-01-08 Thread Harsha Teja Kanna (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471280#comment-17471280 ] Harsha Teja Kanna edited comment on HUDI-3066 at 1/9/22, 6:11 AM: --

[GitHub] [hudi] YannByron commented on pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
YannByron commented on pull request #4083: URL: https://github.com/apache/hudi/pull/4083#issuecomment-1008236642 @dongkelun LGTM, just left some minor comments. @xushiyan Further review whether this strategy makes sense. -- This is an automated message from the Apache Git Service. To res

[GitHub] [hudi] YannByron commented on a change in pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
YannByron commented on a change in pull request #4083: URL: https://github.com/apache/hudi/pull/4083#discussion_r780737792 ## File path: hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/TestHoodieHFileInputFormat.java ## @@ -235,6 +235,50 @@ public void testIncrementalSimpl

[GitHub] [hudi] YannByron commented on a change in pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
YannByron commented on a change in pull request #4083: URL: https://github.com/apache/hudi/pull/4083#discussion_r780737251 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/InputPathHandler.java ## @@ -117,9 +124,10 @@ private void parseInputPaths(Path[] inputP

[GitHub] [hudi] YannByron commented on a change in pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
YannByron commented on a change in pull request #4083: URL: https://github.com/apache/hudi/pull/4083#discussion_r780736806 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieHiveUtils.java ## @@ -175,4 +177,9 @@ private static HoodieTimeline filterIf

[GitHub] [hudi] YannByron commented on a change in pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
YannByron commented on a change in pull request #4083: URL: https://github.com/apache/hudi/pull/4083#discussion_r780736465 ## File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestUtils.java ## @@ -46,6 +46,7 @@ */ public class HoodieTestUtils {

[GitHub] [hudi] nsivabalan edited a comment on issue #4170: [SUPPORT] Understanding Clustering Behavior

2022-01-08 Thread GitBox
nsivabalan edited a comment on issue #4170: URL: https://github.com/apache/hudi/issues/4170#issuecomment-1008232146 @rubenssoto : hey, any updates in this regard please. unless we get more logs, we can't do much here. -- This is an automated message from the Apache Git Service. To respo

[GitHub] [hudi] nsivabalan commented on issue #4170: [SUPPORT] Understanding Clustering Behavior

2022-01-08 Thread GitBox
nsivabalan commented on issue #4170: URL: https://github.com/apache/hudi/issues/4170#issuecomment-1008232146 @rubenssoto : hey, any updates in this regard please. unless we get more logs, we can't help much here. -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] nsivabalan commented on issue #4200: spark-sql query timestamp partition error

2022-01-08 Thread GitBox
nsivabalan commented on issue #4200: URL: https://github.com/apache/hudi/issues/4200#issuecomment-1008232053 thanks for confirming. appreciate it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hudi] nsivabalan closed issue #4200: spark-sql query timestamp partition error

2022-01-08 Thread GitBox
nsivabalan closed issue #4200: URL: https://github.com/apache/hudi/issues/4200 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...

[GitHub] [hudi] YannByron commented on a change in pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
YannByron commented on a change in pull request #4083: URL: https://github.com/apache/hudi/pull/4083#discussion_r780735886 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java ## @@ -90,6 +90,11 @@ // It is here so that bo

[GitHub] [hudi] nsivabalan commented on issue #4208: [SUPPORT] On Hudi 0.9.0 - Alter table throws java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.alterTable(java.lang.String,

2022-01-08 Thread GitBox
nsivabalan commented on issue #4208: URL: https://github.com/apache/hudi/issues/4208#issuecomment-1008232003 @YannByron @xushiyan : can you folks please follow up on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [hudi] nsivabalan commented on issue #4230: [SUPPORT] org.apache.hudi.exception.HoodieRemoteException: Failed to create marker file

2022-01-08 Thread GitBox
nsivabalan commented on issue #4230: URL: https://github.com/apache/hudi/issues/4230#issuecomment-1008231940 @yihua : gentle ping to follow up on the issue. If there is some regression, we might want to fix in 0.10.1. would appreciate if you can follow up on this. -- This is an automate

[GitHub] [hudi] nsivabalan commented on issue #4439: [BUG] ROLLBACK meet Cannot use marker based rollback strategy on completed error

2022-01-08 Thread GitBox
nsivabalan commented on issue #4439: URL: https://github.com/apache/hudi/issues/4439#issuecomment-1008231844 Hey @waywtdcc : let us know if you are looking for any more assistance. If not, feel free to close out the issue. -- This is an automated message from the Apache Git Service. To r

[GitHub] [hudi] YannByron commented on a change in pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
YannByron commented on a change in pull request #4083: URL: https://github.com/apache/hudi/pull/4083#discussion_r780735803 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala ## @@ -164,9 +169,14 @@ class

[GitHub] [hudi] nsivabalan commented on issue #4429: [SUPPORT] Spark SQL CTAS command doesn't work with 0.10.0 version and Spark 3.1.1

2022-01-08 Thread GitBox
nsivabalan commented on issue #4429: URL: https://github.com/apache/hudi/issues/4429#issuecomment-1008231688 hey folks. if the issue is resolved, can we close out the github issue. thanks to Yann for quick turn around. -- This is an automated message from the Apache Git Service. To respo

[GitHub] [hudi] nsivabalan closed issue #4419: [SUPPORT] Not An Avro File (flink)

2022-01-08 Thread GitBox
nsivabalan closed issue #4419: URL: https://github.com/apache/hudi/issues/4419 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...

[GitHub] [hudi] nsivabalan commented on issue #4419: [SUPPORT] Not An Avro File (flink)

2022-01-08 Thread GitBox
nsivabalan commented on issue #4419: URL: https://github.com/apache/hudi/issues/4419#issuecomment-1008231564 thanks for confirming. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] nsivabalan commented on issue #4457: [SUPPORT] Hudi archive stopped working

2022-01-08 Thread GitBox
nsivabalan commented on issue #4457: URL: https://github.com/apache/hudi/issues/4457#issuecomment-1008231502 @zuyanton : hey, do you have any updates for us. CC @prashantwason does something pop up for you. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] nsivabalan commented on issue #4461: [SUPPORT]Hudi(0.10.0) write to Aliyun oss using metadata table warning

2022-01-08 Thread GitBox
nsivabalan commented on issue #4461: URL: https://github.com/apache/hudi/issues/4461#issuecomment-1008231420 @nikenfls : do you have any updates for us. if the issue is resolved, can we close out the github issue. thanks. -- This is an automated message from the Apache Git Service. To re

[GitHub] [hudi] YannByron commented on a change in pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
YannByron commented on a change in pull request #4083: URL: https://github.com/apache/hudi/pull/4083#discussion_r780735473 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestCreateTable.scala ## @@ -33,6 +33,10 @@ import scala.collectio

[GitHub] [hudi] nsivabalan commented on issue #4456: [SUPPORT] MultiWriter w/ DynamoDB - Unable to acquire lock, lock object null

2022-01-08 Thread GitBox
nsivabalan commented on issue #4456: URL: https://github.com/apache/hudi/issues/4456#issuecomment-1008231311 @nochimow : a gentle reminder to respond to above question. above commentor is a Hoodie committer who added dynamoDB lock provider. So, he should be able to help in your case. --

[GitHub] [hudi] nsivabalan commented on issue #4434: [SUPPORT]why are there many files under the Hoodie file?

2022-01-08 Thread GitBox
nsivabalan commented on issue #4434: URL: https://github.com/apache/hudi/issues/4434#issuecomment-1008231118 @tieke1121 : hey are you looking for more info. let us know. if not, feel free to close out the github issue. -- This is an automated message from the Apache Git Service. To respo

[GitHub] [hudi] nsivabalan commented on issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column

2022-01-08 Thread GitBox
nsivabalan commented on issue #4477: URL: https://github.com/apache/hudi/issues/4477#issuecomment-1008231035 @YannByron : may I know whats the tracking ticket. If not, can we create one for the issue reported in this github issue. -- This is an automated message from the Apache Git Serv

[GitHub] [hudi] nsivabalan closed issue #4474: [SUPPORT] Should we shade all aws dependencies to avoid class conflicts?

2022-01-08 Thread GitBox
nsivabalan closed issue #4474: URL: https://github.com/apache/hudi/issues/4474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...

[GitHub] [hudi] nsivabalan commented on issue #4474: [SUPPORT] Should we shade all aws dependencies to avoid class conflicts?

2022-01-08 Thread GitBox
nsivabalan commented on issue #4474: URL: https://github.com/apache/hudi/issues/4474#issuecomment-1008230937 Closing the github issue as we have a tracking jira. thank you folks for chiming in. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [hudi] nsivabalan commented on issue #4541: [SUPPORT] NullPointerException while writing Bulk ingest table

2022-01-08 Thread GitBox
nsivabalan commented on issue #4541: URL: https://github.com/apache/hudi/issues/4541#issuecomment-1008230765 let's try to remove some advanced configs, and test if we can make a simple job succeed and then we can add back more configs to deduce the issue. - I see you have added lot o

[GitHub] [hudi] YannByron commented on a change in pull request #4083: [HUDI-2837] The original hoodie.table.name should be maintained in Spark SQL

2022-01-08 Thread GitBox
YannByron commented on a change in pull request #4083: URL: https://github.com/apache/hudi/pull/4083#discussion_r780735306 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java ## @@ -90,6 +90,11 @@ // It is here so that bo

[GitHub] [hudi] nsivabalan commented on issue #4539: [SUPPORT] spark 2.4.0 write data to hudi ERROR (0.10.0)

2022-01-08 Thread GitBox
nsivabalan commented on issue #4539: URL: https://github.com/apache/hudi/issues/4539#issuecomment-1008230027 2.4.0 is not supported. Can you try with 2.4.3 or higher spark versions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[jira] [Comment Edited] (HUDI-3197) Validate partition pruning with Hudi

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471283#comment-17471283 ] Raymond Xu edited comment on HUDI-3197 at 1/9/22, 4:49 AM: --- {cod

[jira] [Updated] (HUDI-44) Compaction must preserve commit timestamps of merged records #376

2022-01-08 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-44?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-44: Status: Resolved (was: Patch Available) > Compaction must preserve commit timestamps of merge

[jira] [Commented] (HUDI-3197) Validate partition pruning with Hudi

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471283#comment-17471283 ] Raymond Xu commented on HUDI-3197: -- {code:java} -- create a partitioned, preCombineField-

[jira] [Updated] (HUDI-3197) Validate partition pruning with Hudi

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3197: - Attachment: (was: Screen Shot 2022-01-08 at 8.43.50 PM.png) > Validate partition pruning with Hudi > -

[jira] [Resolved] (HUDI-2947) HoodieDeltaStreamer/DeltaSync can improperly pick up the checkpoint config from CLI in continuous mode

2022-01-08 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-2947. --- > HoodieDeltaStreamer/DeltaSync can improperly pick up the checkpoint config > from CLI i

[jira] [Updated] (HUDI-3197) Validate partition pruning with Hudi

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3197: - Attachment: Screen Shot 2022-01-08 at 8.43.50 PM.png > Validate partition pruning with Hudi >

[jira] [Commented] (HUDI-2909) Partition field parsing fails due to KeyGenerator giving inconsistent value for logical timestamp type

2022-01-08 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471282#comment-17471282 ] sivabalan narayanan commented on HUDI-2909: --- [~codope] : can you help Harsha. >

[GitHub] [hudi] hudi-bot removed a comment on pull request #4518: [HUDI-2477] Removing rollbacks instants from timeline for restore operation

2022-01-08 Thread GitBox
hudi-bot removed a comment on pull request #4518: URL: https://github.com/apache/hudi/pull/4518#issuecomment-1008222910 ## CI report: * 2b82de3c867cddb3af7f2edd7f48f662defda372 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4518: [HUDI-2477] Removing rollbacks instants from timeline for restore operation

2022-01-08 Thread GitBox
hudi-bot commented on pull request #4518: URL: https://github.com/apache/hudi/pull/4518#issuecomment-1008228966 ## CI report: * 108b27f73f4656423be54bf4b20ba9dad8a26647 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] danny0405 commented on pull request #4446: [HUDI-2917] rollback insert data appended to log file when using Hbase Index

2022-01-08 Thread GitBox
danny0405 commented on pull request #4446: URL: https://github.com/apache/hudi/pull/4446#issuecomment-1008228935 Generally i think we should figure out a way for global index how to distinguish between `INSERT` and `UPDATE` for input records instead of hacking in the partitioner for write

[GitHub] [hudi] danny0405 commented on a change in pull request #4446: [HUDI-2917] rollback insert data appended to log file when using Hbase Index

2022-01-08 Thread GitBox
danny0405 commented on a change in pull request #4446: URL: https://github.com/apache/hudi/pull/4446#discussion_r780733934 ## File path: hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/BaseJavaCommitActionExecutor.java ## @@ -90,27 +90,29 @@ publ

[GitHub] [hudi] cdmikechen commented on a change in pull request #4451: [HUDI-3104] Kafka-connect support hadoop config environments and properties

2022-01-08 Thread GitBox
cdmikechen commented on a change in pull request #4451: URL: https://github.com/apache/hudi/pull/4451#discussion_r780733280 ## File path: hudi-kafka-connect/src/main/java/org/apache/hudi/connect/utils/KafkaConnectUtils.java ## @@ -89,6 +140,23 @@ public static int getLatestNum

[GitHub] [hudi] cdmikechen commented on a change in pull request #4451: [HUDI-3104] Kafka-connect support hadoop config environments and properties

2022-01-08 Thread GitBox
cdmikechen commented on a change in pull request #4451: URL: https://github.com/apache/hudi/pull/4451#discussion_r780733010 ## File path: hudi-kafka-connect/src/main/java/org/apache/hudi/connect/utils/KafkaConnectUtils.java ## @@ -65,6 +70,52 @@ private static final Logge

[jira] [Comment Edited] (HUDI-3066) Very slow file listing after enabling metadata for existing tables in 0.10.0 release

2022-01-08 Thread Harsha Teja Kanna (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471280#comment-17471280 ] Harsha Teja Kanna edited comment on HUDI-3066 at 1/9/22, 4:15 AM: --

[jira] [Commented] (HUDI-3066) Very slow file listing after enabling metadata for existing tables in 0.10.0 release

2022-01-08 Thread Harsha Teja Kanna (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471280#comment-17471280 ] Harsha Teja Kanna commented on HUDI-3066: - Hi [~shivnarayan] Basic question I am

[GitHub] [hudi] hudi-bot commented on pull request #4542: [HUDI-3157] Remove aws jars from hudi bundles

2022-01-08 Thread GitBox
hudi-bot commented on pull request #4542: URL: https://github.com/apache/hudi/pull/4542#issuecomment-1008223765 ## CI report: * 5c4d7cfecd25cea19567f897fa0dac3f5f784baf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #4542: [HUDI-3157] Remove aws jars from hudi bundles

2022-01-08 Thread GitBox
hudi-bot removed a comment on pull request #4542: URL: https://github.com/apache/hudi/pull/4542#issuecomment-1008217433 ## CI report: * 5c4d7cfecd25cea19567f897fa0dac3f5f784baf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot removed a comment on pull request #4518: [HUDI-2477] Removing rollbacks instants from timeline for restore operation

2022-01-08 Thread GitBox
hudi-bot removed a comment on pull request #4518: URL: https://github.com/apache/hudi/pull/4518#issuecomment-1008222649 ## CI report: * 2b82de3c867cddb3af7f2edd7f48f662defda372 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4518: [HUDI-2477] Removing rollbacks instants from timeline for restore operation

2022-01-08 Thread GitBox
hudi-bot commented on pull request #4518: URL: https://github.com/apache/hudi/pull/4518#issuecomment-1008222910 ## CI report: * 2b82de3c867cddb3af7f2edd7f48f662defda372 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #4518: [HUDI-2477] Removing rollbacks instants from timeline for restore operation

2022-01-08 Thread GitBox
hudi-bot removed a comment on pull request #4518: URL: https://github.com/apache/hudi/pull/4518#issuecomment-1006204405 ## CI report: * 2b82de3c867cddb3af7f2edd7f48f662defda372 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4518: [HUDI-2477] Removing rollbacks instants from timeline for restore operation

2022-01-08 Thread GitBox
hudi-bot commented on pull request #4518: URL: https://github.com/apache/hudi/pull/4518#issuecomment-1008222649 ## CI report: * 2b82de3c867cddb3af7f2edd7f48f662defda372 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #4542: [HUDI-3157] Remove aws jars from hudi bundles

2022-01-08 Thread GitBox
hudi-bot removed a comment on pull request #4542: URL: https://github.com/apache/hudi/pull/4542#issuecomment-1008217107 ## CI report: * 5c4d7cfecd25cea19567f897fa0dac3f5f784baf UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run

[GitHub] [hudi] hudi-bot commented on pull request #4542: [HUDI-3157] Remove aws jars from hudi bundles

2022-01-08 Thread GitBox
hudi-bot commented on pull request #4542: URL: https://github.com/apache/hudi/pull/4542#issuecomment-1008217433 ## CI report: * 5c4d7cfecd25cea19567f897fa0dac3f5f784baf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot commented on pull request #4542: [HUDI-3157] Remove aws jars from hudi bundles

2022-01-08 Thread GitBox
hudi-bot commented on pull request #4542: URL: https://github.com/apache/hudi/pull/4542#issuecomment-1008217107 ## CI report: * 5c4d7cfecd25cea19567f897fa0dac3f5f784baf UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure`

[GitHub] [hudi] boneanxs commented on pull request #4542: [HUDI-3157] Remove aws jars from hudi bundles

2022-01-08 Thread GitBox
boneanxs commented on pull request #4542: URL: https://github.com/apache/hudi/pull/4542#issuecomment-1008217012 @xushiyan @nsivabalan pls take a took. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[jira] [Updated] (HUDI-3157) Remove aws jars from hudi bundles

2022-01-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3157: - Labels: pull-request-available sev:critical user-support-issues (was: sev:critical user-support-i

[GitHub] [hudi] boneanxs opened a new pull request #4542: [HUDI-3157] Remove aws jars from hudi bundles

2022-01-08 Thread GitBox
boneanxs opened a new pull request #4542: URL: https://github.com/apache/hudi/pull/4542 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpos

[jira] [Commented] (HUDI-3157) Remove aws jars from hudi bundles

2022-01-08 Thread Hui An (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471269#comment-17471269 ] Hui An commented on HUDI-3157: -- Working on this now. > Remove aws jars from hudi bundles > -

[GitHub] [hudi] hudi-bot removed a comment on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-01-08 Thread GitBox
hudi-bot removed a comment on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1008208353 ## CI report: * fa5894fba7cf168250fe52b70d8131ce0877f285 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-01-08 Thread GitBox
hudi-bot commented on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1008215773 ## CI report: * 8896e81ac168348d66de6c8cf444c4a7e2c9826e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[jira] [Comment Edited] (HUDI-2909) Partition field parsing fails due to KeyGenerator giving inconsistent value for logical timestamp type

2022-01-08 Thread Harsha Teja Kanna (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471264#comment-17471264 ] Harsha Teja Kanna edited comment on HUDI-2909 at 1/9/22, 2:05 AM: --

[jira] [Comment Edited] (HUDI-2909) Partition field parsing fails due to KeyGenerator giving inconsistent value for logical timestamp type

2022-01-08 Thread Harsha Teja Kanna (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471264#comment-17471264 ] Harsha Teja Kanna edited comment on HUDI-2909 at 1/9/22, 2:05 AM: --

[jira] [Commented] (HUDI-2909) Partition field parsing fails due to KeyGenerator giving inconsistent value for logical timestamp type

2022-01-08 Thread Harsha Teja Kanna (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471264#comment-17471264 ] Harsha Teja Kanna commented on HUDI-2909: - I am not able to determine if I fall un

[GitHub] [hudi] aznwarmonkey opened a new issue #4541: [SUPPORT] NullPointerException while writing Bulk ingest table

2022-01-08 Thread GitBox
aznwarmonkey opened a new issue #4541: URL: https://github.com/apache/hudi/issues/4541 Hello, I am currently getting an exception while writing a `hudi` talbe in `bulk_ingest` mode. Please see below for the stacktrace along with the snippet of code I a using to write the data. I am

[GitHub] [hudi] hudi-bot removed a comment on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-01-08 Thread GitBox
hudi-bot removed a comment on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1008207981 ## CI report: * fa5894fba7cf168250fe52b70d8131ce0877f285 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-01-08 Thread GitBox
hudi-bot commented on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1008208353 ## CI report: * fa5894fba7cf168250fe52b70d8131ce0877f285 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-01-08 Thread GitBox
hudi-bot removed a comment on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1007952877 ## CI report: * fa5894fba7cf168250fe52b70d8131ce0877f285 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-01-08 Thread GitBox
hudi-bot commented on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1008207981 ## CI report: * fa5894fba7cf168250fe52b70d8131ce0877f285 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #4514: [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation

2022-01-08 Thread GitBox
hudi-bot removed a comment on pull request #4514: URL: https://github.com/apache/hudi/pull/4514#issuecomment-1008177446 ## CI report: * ddc3af0c32bafef6b10c32c43132df32a5f7d83c UNKNOWN * e1ba726105dfa7ae07d802546c71a0cf1ad8b172 UNKNOWN * 306e7d462959e0249e230f60c2e9ea6602342

[GitHub] [hudi] hudi-bot commented on pull request #4514: [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation

2022-01-08 Thread GitBox
hudi-bot commented on pull request #4514: URL: https://github.com/apache/hudi/pull/4514#issuecomment-1008194396 ## CI report: * ddc3af0c32bafef6b10c32c43132df32a5f7d83c UNKNOWN * e1ba726105dfa7ae07d802546c71a0cf1ad8b172 UNKNOWN * 306e7d462959e0249e230f60c2e9ea6602342e08 UNKN

[jira] [Updated] (HUDI-3197) Validate partition pruning with Hudi

2022-01-08 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3197: -- Summary: Validate partition pruning with Hudi (was: Validate partition pruning with Spa

[jira] [Closed] (HUDI-2682) Spark schema not updated with new columns on hive sync

2022-01-08 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu closed HUDI-2682. Reviewers: Tao Meng Resolution: Fixed > Spark schema not updated with new columns on hive sync > --

  1   2   3   >