[GitHub] [hudi] fisser001 commented on issue #4887: [SUPPORT] Unexpected behaviour with partitioned hudi tables with impala as query engine

2022-04-12 Thread GitBox
fisser001 commented on issue #4887: URL: https://github.com/apache/hudi/issues/4887#issuecomment-1096204986 @garyli1019 Yes we have access to Cloudera support and we had already opened a ticket with Cloudera. But Cloudera takes the position that they do not support Hudi. I can't quite under

[GitHub] [hudi] hudi-bot commented on pull request #5296: [HUDI-3855] Fixing `FILENAME_METADATA_FIELD` not being correctly updated in `HoodieMergeHandle`

2022-04-12 Thread GitBox
hudi-bot commented on PR #5296: URL: https://github.com/apache/hudi/pull/5296#issuecomment-1096207473 ## CI report: * e5d566882c6f3ed58a65a01065f0ae99dfb420b2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=800

[GitHub] [hudi] kasured opened a new issue, #5298: [SUPPORT] File is deleted during inline compaction on MOR table causing subsequent FileNotFoundException on a reader

2022-04-12 Thread GitBox
kasured opened a new issue, #5298: URL: https://github.com/apache/hudi/issues/5298 **Describe the problem you faced** When inline compaction is turned on and when the actual compaction plan is completed, the commit file is referencing the file which has been deleted during the compac

[GitHub] [hudi] hudi-bot commented on pull request #5296: [HUDI-3855] Fixing `FILENAME_METADATA_FIELD` not being correctly updated in `HoodieMergeHandle`

2022-04-12 Thread GitBox
hudi-bot commented on PR #5296: URL: https://github.com/apache/hudi/pull/5296#issuecomment-1096256827 ## CI report: * e5d566882c6f3ed58a65a01065f0ae99dfb420b2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=800

[jira] [Created] (HUDI-3860) Write bootstrap script to prep EMR w OSS spark

2022-04-12 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-3860: - Summary: Write bootstrap script to prep EMR w OSS spark Key: HUDI-3860 URL: https://issues.apache.org/jira/browse/HUDI-3860 Project: Apache Hudi Issue Type: Test

[GitHub] [hudi] hudi-bot commented on pull request #5264: [HUDI-3818] encode bytes column value when generate HoodieKey

2022-04-12 Thread GitBox
hudi-bot commented on PR #5264: URL: https://github.com/apache/hudi/pull/5264#issuecomment-1096262951 ## CI report: * e9ea015b4417b01600dba6985602f0c66e56c666 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8001

[GitHub] [hudi] hudi-bot commented on pull request #5296: [HUDI-3855] Fixing `FILENAME_METADATA_FIELD` not being correctly updated in `HoodieMergeHandle`

2022-04-12 Thread GitBox
hudi-bot commented on PR #5296: URL: https://github.com/apache/hudi/pull/5296#issuecomment-1096263191 ## CI report: * e5d566882c6f3ed58a65a01065f0ae99dfb420b2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=800

[jira] [Commented] (HUDI-3819) upgrade spring cve-2022-22965

2022-04-12 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520967#comment-17520967 ] Sagar Sumit commented on HUDI-3819: --- These are the prerequisites for the exploit: * JDK

[GitHub] [hudi] hudi-bot commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
hudi-bot commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096306614 ## CI report: * 75051413bfb5e6e01ed8e41c98505ce592e8d81c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7992

[GitHub] [hudi] vingov commented on a diff in pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
vingov commented on code in PR #5294: URL: https://github.com/apache/hudi/pull/5294#discussion_r848111267 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java: ## @@ -477,14 +479,19 @@ private Pair>> fetchFromSourc } boolean shouldCo

[GitHub] [hudi] vingov commented on a diff in pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
vingov commented on code in PR #5294: URL: https://github.com/apache/hudi/pull/5294#discussion_r848111719 ## hudi-common/src/test/java/org/apache/hudi/avro/TestHoodieAvroUtils.java: ## @@ -227,6 +228,20 @@ public void testAddingAndRemovingMetadataFields() { assertEquals(NUM

[GitHub] [hudi] vingov commented on a diff in pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
vingov commented on code in PR #5294: URL: https://github.com/apache/hudi/pull/5294#discussion_r848112012 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java: ## @@ -133,6 +133,11 @@ public class HoodieWriteConfig extends HoodieConfig {

[GitHub] [hudi] vingov commented on a diff in pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
vingov commented on code in PR #5294: URL: https://github.com/apache/hudi/pull/5294#discussion_r848113146 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java: ## @@ -477,14 +479,19 @@ private Pair>> fetchFromSourc } boolean shouldCo

[GitHub] [hudi] hudi-bot commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
hudi-bot commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096312398 ## CI report: * 75051413bfb5e6e01ed8e41c98505ce592e8d81c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7992

[GitHub] [hudi] vingov commented on a diff in pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
vingov commented on code in PR #5294: URL: https://github.com/apache/hudi/pull/5294#discussion_r848113533 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java: ## @@ -727,6 +734,13 @@ public void setupWriteClient() throws IOException { priv

[GitHub] [hudi] codope commented on a diff in pull request #5274: [HUDI-3839] Fixing incorrect selection of MT partitions to be updated

2022-04-12 Thread GitBox
codope commented on code in PR #5274: URL: https://github.com/apache/hudi/pull/5274#discussion_r848114555 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -770,18 +774,6 @@ private MetadataRecordsGenerationParam

[GitHub] [hudi] codope merged pull request #5274: [HUDI-3839] Fixing incorrect selection of MT partitions to be updated

2022-04-12 Thread GitBox
codope merged PR #5274: URL: https://github.com/apache/hudi/pull/5274 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.or

[hudi] branch master updated (f91e9e63e1 -> 101b82a679)

2022-04-12 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from f91e9e63e1 [HUDI-3799] Fixing not deleting empty instants w/o archiving (#5261) add 101b82a679 [HUDI-3839] Fixing

[GitHub] [hudi] hudi-bot commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
hudi-bot commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096323549 ## CI report: * 75051413bfb5e6e01ed8e41c98505ce592e8d81c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7992

[GitHub] [hudi] hudi-bot commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
hudi-bot commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096333606 ## CI report: * afc2f676cd198fd6fbba9e66c27470aefad98262 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=800

[GitHub] [hudi] hudi-bot commented on pull request #5297: [HUDI-3859] Fix spark profiles and utilities-slim dep

2022-04-12 Thread GitBox
hudi-bot commented on PR #5297: URL: https://github.com/apache/hudi/pull/5297#issuecomment-1096333705 ## CI report: * cc81ebb8f2b84b9ada13927de9c30a1b69864f2f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8003

[GitHub] [hudi] codope opened a new pull request, #5299: [WIP][HUDI-3819] Upgrade spring-core in hudi-cli

2022-04-12 Thread GitBox
codope opened a new pull request, #5299: URL: https://github.com/apache/hudi/pull/5299 ## What is the purpose of the pull request To mitigate [CVE-2022-22965](https://tanzu.vmware.com/security/cve-2022-22965) The spring-shell has not been patched yet, however, I attempted to exclud

[jira] [Updated] (HUDI-3819) upgrade spring cve-2022-22965

2022-04-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3819: - Labels: pull-request-available (was: ) > upgrade spring cve-2022-22965 >

[jira] [Commented] (HUDI-3819) upgrade spring cve-2022-22965

2022-04-12 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521006#comment-17521006 ] Sagar Sumit commented on HUDI-3819: --- As suspected, there is a class not found issue afte

[GitHub] [hudi] hudi-bot commented on pull request #5299: [WIP][HUDI-3819] Upgrade spring-core in hudi-cli

2022-04-12 Thread GitBox
hudi-bot commented on PR #5299: URL: https://github.com/apache/hudi/pull/5299#issuecomment-1096344573 ## CI report: * 6bca3f4d8e426f4196f1d080adbefd0790067bb9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #5299: [WIP][HUDI-3819] Upgrade spring-core in hudi-cli

2022-04-12 Thread GitBox
hudi-bot commented on PR #5299: URL: https://github.com/apache/hudi/pull/5299#issuecomment-1096350524 ## CI report: * 6bca3f4d8e426f4196f1d080adbefd0790067bb9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8008

[GitHub] [hudi] hudi-bot commented on pull request #5279: [HUDI-3843] Make flink profiles build with scala-2.11

2022-04-12 Thread GitBox
hudi-bot commented on PR #5279: URL: https://github.com/apache/hudi/pull/5279#issuecomment-1096366712 ## CI report: * 66bc1d1b54b7d5d7fbbc4db8e29b4ced675c2c8d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8004

[GitHub] [hudi] codope commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
codope commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096376778 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

[GitHub] [hudi] hudi-bot commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
hudi-bot commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096377769 ## CI report: * afc2f676cd198fd6fbba9e66c27470aefad98262 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=800

[GitHub] [hudi] hudi-bot commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
hudi-bot commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096387568 ## CI report: * a6f16be43b9125cbf3155917096ad5b604822b42 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
hudi-bot commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096392035 ## CI report: * a6f16be43b9125cbf3155917096ad5b604822b42 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #5296: [HUDI-3855] Fixing `FILENAME_METADATA_FIELD` not being correctly updated in `HoodieMergeHandle`

2022-04-12 Thread GitBox
hudi-bot commented on PR #5296: URL: https://github.com/apache/hudi/pull/5296#issuecomment-1096396627 ## CI report: * 9458d847182b0628d228211d010310ade743d431 UNKNOWN * df54e1dd20f6e602c177a00295ac7ca616d7d029 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
hudi-bot commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096409920 ## CI report: * a6f16be43b9125cbf3155917096ad5b604822b42 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8010

[jira] [Created] (HUDI-3861) 'path' in CatalogTable#properties failed to be updated when renaming table

2022-04-12 Thread Jin Xing (Jira)
Jin Xing created HUDI-3861: -- Summary: 'path' in CatalogTable#properties failed to be updated when renaming table Key: HUDI-3861 URL: https://issues.apache.org/jira/browse/HUDI-3861 Project: Apache Hudi

[GitHub] [hudi] danny0405 commented on issue #4881: Full incremental Enable index loading to discover duplicate data(index.bootstrap.enabled)

2022-04-12 Thread GitBox
danny0405 commented on issue #4881: URL: https://github.com/apache/hudi/issues/4881#issuecomment-1096447657 > @nsivabalan Will the current issue be fixed when the next version is released Do you still use the batch bootstrap ? -- This is an automated message from the Apache Git Ser

[jira] [Updated] (HUDI-3861) 'path' in CatalogTable#properties failed to be updated when renaming table

2022-04-12 Thread Jin Xing (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jin Xing updated HUDI-3861: --- Description: Reproduce the issue as below   {code:java} 1. Create a MOR table create table mor_simple(   id

[jira] [Updated] (HUDI-3861) 'path' in CatalogTable#properties failed to be updated when renaming table

2022-04-12 Thread Jin Xing (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jin Xing updated HUDI-3861: --- Description: Reproduce the issue as below {code:java} 1. Create a MOR table create table mor_simple(   id int

[jira] [Updated] (HUDI-3861) 'path' in CatalogTable#properties failed to be updated when renaming table

2022-04-12 Thread Jin Xing (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jin Xing updated HUDI-3861: --- Description: Reproduce the issue as below   {code:java} 1. Create a MOR table create table mor_simple(   id

[jira] [Commented] (HUDI-3861) 'path' in CatalogTable#properties failed to be updated when renaming table

2022-04-12 Thread Jin Xing (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521043#comment-17521043 ] Jin Xing commented on HUDI-3861: [~xleesf]  Since you are master of HoodieCatalog, I ping

[jira] [Created] (HUDI-3862) Some default configurations in HoodieHBaseIndexConfig do not work

2022-04-12 Thread xi chaomin (Jira)
xi chaomin created HUDI-3862: Summary: Some default configurations in HoodieHBaseIndexConfig do not work Key: HUDI-3862 URL: https://issues.apache.org/jira/browse/HUDI-3862 Project: Apache Hudi

[GitHub] [hudi] hudi-bot commented on pull request #5299: [WIP][HUDI-3819] Upgrade spring-core in hudi-cli

2022-04-12 Thread GitBox
hudi-bot commented on PR #5299: URL: https://github.com/apache/hudi/pull/5299#issuecomment-1096481992 ## CI report: * 6bca3f4d8e426f4196f1d080adbefd0790067bb9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8008

[jira] [Commented] (HUDI-3861) 'path' in CatalogTable#properties failed to be updated when renaming table

2022-04-12 Thread leesf (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521054#comment-17521054 ] leesf commented on HUDI-3861: - [~jinxing6...@126.com] Thanks for reporting this, but I think r

[jira] [Updated] (HUDI-3863) Add UT for drop partition column support in deltastreamer

2022-04-12 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-3863: -- Fix Version/s: 0.12.0 > Add UT for drop partition column support in deltastreamer > ---

[jira] [Created] (HUDI-3863) Add UT for drop partition column support in deltastreamer

2022-04-12 Thread Sagar Sumit (Jira)
Sagar Sumit created HUDI-3863: - Summary: Add UT for drop partition column support in deltastreamer Key: HUDI-3863 URL: https://issues.apache.org/jira/browse/HUDI-3863 Project: Apache Hudi Issue

[GitHub] [hudi] codope commented on a diff in pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
codope commented on code in PR #5294: URL: https://github.com/apache/hudi/pull/5294#discussion_r848269526 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java: ## @@ -477,14 +479,19 @@ private Pair>> fetchFromSourc } boolean shouldCo

[jira] [Commented] (HUDI-3861) 'path' in CatalogTable#properties failed to be updated when renaming table

2022-04-12 Thread KnightChess (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521065#comment-17521065 ] KnightChess commented on HUDI-3861: --- [~xleesf] If the table is manages, the real table p

[jira] [Comment Edited] (HUDI-3861) 'path' in CatalogTable#properties failed to be updated when renaming table

2022-04-12 Thread KnightChess (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521065#comment-17521065 ] KnightChess edited comment on HUDI-3861 at 4/12/22 10:28 AM: -

[GitHub] [hudi] hudi-bot commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
hudi-bot commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096567211 ## CI report: * a6f16be43b9125cbf3155917096ad5b604822b42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8010

[GitHub] [hudi] hudi-bot commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
hudi-bot commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096576675 ## CI report: * a6f16be43b9125cbf3155917096ad5b604822b42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8010

[GitHub] [hudi] hudi-bot commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
hudi-bot commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096579737 ## CI report: * a6f16be43b9125cbf3155917096ad5b604822b42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8010

[jira] [Commented] (HUDI-3861) 'path' in CatalogTable#properties failed to be updated when renaming table

2022-04-12 Thread leesf (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521087#comment-17521087 ] leesf commented on HUDI-3861: - [~KnightChess] yeah, if the real table path is updated, the tbl

[GitHub] [hudi] cxzl25 commented on pull request #5287: [HUDI-3849] AvroDeserializer supports AVRO_REBASE_MODE_IN_READ configuration

2022-04-12 Thread GitBox
cxzl25 commented on PR #5287: URL: https://github.com/apache/hudi/pull/5287#issuecomment-1096618802 @YannByron @leesf @xushiyan please help to review this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hudi] stym06 commented on a diff in pull request #4665: [HUDI-2733] Add support for Thrift sync

2022-04-12 Thread GitBox
stym06 commented on code in PR #4665: URL: https://github.com/apache/hudi/pull/4665#discussion_r848345045 ## hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/ThriftDDLExecutor.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[jira] [Assigned] (HUDI-3861) 'path' in CatalogTable#properties failed to be updated when renaming table

2022-04-12 Thread KnightChess (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KnightChess reassigned HUDI-3861: - Assignee: KnightChess > 'path' in CatalogTable#properties failed to be updated when renaming tabl

[jira] [Commented] (HUDI-3861) 'path' in CatalogTable#properties failed to be updated when renaming table

2022-04-12 Thread KnightChess (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521104#comment-17521104 ] KnightChess commented on HUDI-3861: --- [~xleesf] I'll fix it these days. > 'path' in Cata

[GitHub] [hudi] hudi-bot commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
hudi-bot commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096655626 ## CI report: * e9164c1494191fb50af7814db9810b3afaf25ac4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8011

[GitHub] [hudi] zhilinli123 commented on issue #4881: Full incremental Enable index loading to discover duplicate data(index.bootstrap.enabled)

2022-04-12 Thread GitBox
zhilinli123 commented on issue #4881: URL: https://github.com/apache/hudi/issues/4881#issuecomment-1096660043 > > @nsivabalan Will the current issue be fixed when the next version is released > > Do you still use the batch bootstrap ? I am currently using a real-time a

[GitHub] [hudi] hudi-bot commented on pull request #5060: [HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint

2022-04-12 Thread GitBox
hudi-bot commented on PR #5060: URL: https://github.com/apache/hudi/pull/5060#issuecomment-1096665227 ## CI report: * b4f82659a528abea9ebbdc872bd8e59a80d9aa2a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7790

[GitHub] [hudi] hudi-bot commented on pull request #5279: [HUDI-3843] Make flink profiles build with scala-2.11

2022-04-12 Thread GitBox
hudi-bot commented on PR #5279: URL: https://github.com/apache/hudi/pull/5279#issuecomment-1096665626 ## CI report: * 66bc1d1b54b7d5d7fbbc4db8e29b4ced675c2c8d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #5060: [HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint

2022-04-12 Thread GitBox
hudi-bot commented on PR #5060: URL: https://github.com/apache/hudi/pull/5060#issuecomment-1096668717 ## CI report: * b4f82659a528abea9ebbdc872bd8e59a80d9aa2a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7790

[GitHub] [hudi] hudi-bot commented on pull request #5279: [HUDI-3843] Make flink profiles build with scala-2.11

2022-04-12 Thread GitBox
hudi-bot commented on PR #5279: URL: https://github.com/apache/hudi/pull/5279#issuecomment-1096669132 ## CI report: * 66bc1d1b54b7d5d7fbbc4db8e29b4ced675c2c8d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8004

[GitHub] [hudi] hudi-bot commented on pull request #5297: [HUDI-3859] Fix spark profiles and utilities-slim dep

2022-04-12 Thread GitBox
hudi-bot commented on PR #5297: URL: https://github.com/apache/hudi/pull/5297#issuecomment-1096669254 ## CI report: * cc81ebb8f2b84b9ada13927de9c30a1b69864f2f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8003

[jira] [Updated] (HUDI-3818) hudi doesn't support bytes column as primary key

2022-04-12 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3818: - Fix Version/s: 0.12.0 > hudi doesn't support bytes column as primary key > ---

[jira] [Updated] (HUDI-3818) hudi doesn't support bytes column as primary key

2022-04-12 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3818: - Priority: Major (was: Minor) > hudi doesn't support bytes column as primary key > ---

[GitHub] [hudi] hudi-bot commented on pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
hudi-bot commented on PR #5294: URL: https://github.com/apache/hudi/pull/5294#issuecomment-1096672790 ## CI report: * e9164c1494191fb50af7814db9810b3afaf25ac4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8011

[GitHub] [hudi] hudi-bot commented on pull request #5297: [HUDI-3859] Fix spark profiles and utilities-slim dep

2022-04-12 Thread GitBox
hudi-bot commented on PR #5297: URL: https://github.com/apache/hudi/pull/5297#issuecomment-1096672856 ## CI report: * cc81ebb8f2b84b9ada13927de9c30a1b69864f2f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8003

[GitHub] [hudi] codope merged pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
codope merged PR #5294: URL: https://github.com/apache/hudi/pull/5294 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.or

[hudi] branch master updated (101b82a679 -> d16740976e)

2022-04-12 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 101b82a679 [HUDI-3839] Fixing incorrect selection of MT partitions to be updated (#5274) add d16740976e [HUDI-3838

[GitHub] [hudi] hudi-bot commented on pull request #5052: [HUDI-3644] hoodie log scan bug cause data duplication bugfix

2022-04-12 Thread GitBox
hudi-bot commented on PR #5052: URL: https://github.com/apache/hudi/pull/5052#issuecomment-1096680005 ## CI report: * ed7918372fdcaca2855d285e17005ecf9cf42520 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7130

[jira] [Updated] (HUDI-3818) hudi doesn't support bytes column as primary key

2022-04-12 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3818: - Priority: Critical (was: Major) > hudi doesn't support bytes column as primary key >

[GitHub] [hudi] hudi-bot commented on pull request #5052: [HUDI-3644] hoodie log scan bug cause data duplication bugfix

2022-04-12 Thread GitBox
hudi-bot commented on PR #5052: URL: https://github.com/apache/hudi/pull/5052#issuecomment-1096683819 ## CI report: * ed7918372fdcaca2855d285e17005ecf9cf42520 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7130

[GitHub] [hudi] hudi-bot commented on pull request #5060: [HUDI-3652] Make ObjectSizeCalculator threadlocal to reduce memory footprint

2022-04-12 Thread GitBox
hudi-bot commented on PR #5060: URL: https://github.com/apache/hudi/pull/5060#issuecomment-1096747088 ## CI report: * 421abaa4fd6040ef70f4b3addda038c5b2d71eae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8012

[GitHub] [hudi] data-storyteller opened a new pull request, #5300: Intig Test Reducing partitions for log running multi partition yaml

2022-04-12 Thread GitBox
data-storyteller opened a new pull request, #5300: URL: https://github.com/apache/hudi/pull/5300 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is th

[GitHub] [hudi] nsivabalan commented on a diff in pull request #5296: [HUDI-3855] Fixing `FILENAME_METADATA_FIELD` not being correctly updated in `HoodieMergeHandle`

2022-04-12 Thread GitBox
nsivabalan commented on code in PR #5296: URL: https://github.com/apache/hudi/pull/5296#discussion_r848452061 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java: ## @@ -354,12 +349,7 @@ public void write(GenericRecord oldRecord) { if (

[GitHub] [hudi] hudi-bot commented on pull request #5300: Intig Test Reducing partitions for log running multi partition yaml

2022-04-12 Thread GitBox
hudi-bot commented on PR #5300: URL: https://github.com/apache/hudi/pull/5300#issuecomment-1096755875 ## CI report: * 67cebb44aeb65de39265991f8b7fcd06de017c20 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #5300: Intig Test Reducing partitions for log running multi partition yaml

2022-04-12 Thread GitBox
hudi-bot commented on PR #5300: URL: https://github.com/apache/hudi/pull/5300#issuecomment-1096760173 ## CI report: * 67cebb44aeb65de39265991f8b7fcd06de017c20 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8016

[GitHub] [hudi] hudi-bot commented on pull request #5297: [HUDI-3859] Fix spark profiles and utilities-slim dep

2022-04-12 Thread GitBox
hudi-bot commented on PR #5297: URL: https://github.com/apache/hudi/pull/5297#issuecomment-1096772750 ## CI report: * 6ddde8153e5f4ab5038d3e36eea65607ba77ef7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8013

[jira] [Updated] (HUDI-3749) Run latest hudi w/ EMR spark and report to aws folks

2022-04-12 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3749: -- Sprint: Hudi-Sprint-Mar-22 (was: Hudi-Sprint-Mar-22, Hudi-Sprint-Apr-12) > Run latest h

[jira] [Closed] (HUDI-3799) Understand reason behind "Not an avro data file" with hudi

2022-04-12 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-3799. - Resolution: Fixed > Understand reason behind "Not an avro data file" with hudi > -

[jira] [Commented] (HUDI-3819) upgrade spring cve-2022-22965

2022-04-12 Thread Jason-Morries Adam (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521190#comment-17521190 ] Jason-Morries Adam commented on HUDI-3819: -- Hi [~codope], the official Spring si

[GitHub] [hudi] hudi-bot commented on pull request #5052: [HUDI-3644] hoodie log scan bug cause data duplication bugfix

2022-04-12 Thread GitBox
hudi-bot commented on PR #5052: URL: https://github.com/apache/hudi/pull/5052#issuecomment-1096820582 ## CI report: * defd623805a61c4898b8f2656f356e6f5d5dd863 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8014

[jira] [Created] (HUDI-3864) Avoid fetching all files for all partitions on the read/query path for flink

2022-04-12 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-3864: - Summary: Avoid fetching all files for all partitions on the read/query path for flink Key: HUDI-3864 URL: https://issues.apache.org/jira/browse/HUDI-3864 Pr

[jira] [Updated] (HUDI-3864) Avoid fetching all files for all partitions on the read/query path for flink

2022-04-12 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3864: -- Fix Version/s: 0.12.0 > Avoid fetching all files for all partitions on the read/query pa

[jira] [Assigned] (HUDI-3864) Avoid fetching all files for all partitions on the read/query path for flink

2022-04-12 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3864: - Assignee: Danny Chen > Avoid fetching all files for all partitions on the read/qu

[jira] [Commented] (HUDI-3762) Check performance of key lookup with full scan disabled in metadata table

2022-04-12 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521206#comment-17521206 ] sivabalan narayanan commented on HUDI-3762: --- perf looks ok for Metadata based li

[jira] [Updated] (HUDI-3864) Avoid fetching all files for all partitions on the read/query path for flink

2022-04-12 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3864: -- Description: Fetching all files across all partitions should be avoided in hot path. es

[jira] [Closed] (HUDI-3762) Check performance of key lookup with full scan disabled in metadata table

2022-04-12 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-3762. - Resolution: Fixed > Check performance of key lookup with full scan disabled in metadata ta

[jira] [Closed] (HUDI-3860) Write bootstrap script to prep EMR w OSS spark

2022-04-12 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-3860. - Resolution: Done Closing. The above reference doc should work > Write bootstrap script to prep EMR w OSS

[GitHub] [hudi] nsivabalan commented on a diff in pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

2022-04-12 Thread GitBox
nsivabalan commented on code in PR #5294: URL: https://github.com/apache/hudi/pull/5294#discussion_r848568345 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java: ## @@ -478,13 +481,14 @@ private Pair>> fetchFromSourc boolean shouldCombi

[jira] [Updated] (HUDI-3577) NPE in HoodieTimelineArchiver

2022-04-12 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3577: -- Status: In Progress (was: Open) > NPE in HoodieTimelineArchiver > -

[GitHub] [hudi] yihua merged pull request #5279: [HUDI-3843] Make flink profiles build with scala-2.11

2022-04-12 Thread GitBox
yihua merged PR #5279: URL: https://github.com/apache/hudi/pull/5279 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

[hudi] branch master updated (d16740976e -> 84783b9779)

2022-04-12 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from d16740976e [HUDI-3838] Implemented drop partition column feature for delta streamer code path (#5294) add 84783b977

[GitHub] [hudi] melin opened a new issue, #5301: [SUPPORT]Support Show Data Files Command Based on Call Procedure Command for Spark SQL

2022-04-12 Thread GitBox
melin opened a new issue, #5301: URL: https://github.com/apache/hudi/issues/5301 To show all of the table's data files and each file's metadata: | content | file_path | file_format | partition | record_count | file_size_in_bytes | column_sizes| value_counts | null_value_counts | nan_

[GitHub] [hudi] yihua commented on a diff in pull request #5297: [HUDI-3859] Fix spark profiles and utilities-slim dep

2022-04-12 Thread GitBox
yihua commented on code in PR #5297: URL: https://github.com/apache/hudi/pull/5297#discussion_r848583799 ## README.md: ## @@ -72,32 +74,31 @@ mvn clean javadoc:aggregate -Pjavadocs ### Build with different Spark versions -The default Spark version supported is 2.4.4. To bui

[GitHub] [hudi] yihua commented on issue #4230: [SUPPORT] org.apache.hudi.exception.HoodieRemoteException: Failed to create marker file

2022-04-12 Thread GitBox
yihua commented on issue #4230: URL: https://github.com/apache/hudi/issues/4230#issuecomment-1096901147 @BalaMahesh Thanks for reporting your error. If you encountered the marker creation error due to timeline server connection refused for async compaction/clustering in DeltaStreamer and D

[jira] [Commented] (HUDI-3819) upgrade spring cve-2022-22965

2022-04-12 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521218#comment-17521218 ] Sagar Sumit commented on HUDI-3819: --- Ah I see, so it got renamed to spring-shell-starter

[GitHub] [hudi] yihua commented on a diff in pull request #5176: [HUDI-3700] Add hudi-utilities-slim-bundle excluding hudi-spark-datasource modules

2022-04-12 Thread GitBox
yihua commented on code in PR #5176: URL: https://github.com/apache/hudi/pull/5176#discussion_r848601404 ## README.md: ## @@ -94,6 +94,12 @@ mvn clean package -DskipTests -Dspark3.1.x Starting from versions 0.11, Hudi no longer requires `spark-avro` to be specified using `--

[GitHub] [hudi] hudi-bot commented on pull request #5300: Intig Test Reducing partitions for log running multi partition yaml

2022-04-12 Thread GitBox
hudi-bot commented on PR #5300: URL: https://github.com/apache/hudi/pull/5300#issuecomment-1096909237 ## CI report: * 67cebb44aeb65de39265991f8b7fcd06de017c20 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8016

[jira] [Commented] (HUDI-3819) upgrade spring cve-2022-22965

2022-04-12 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521222#comment-17521222 ] Sagar Sumit commented on HUDI-3819: --- So, as I mentioned before, v 2.0.1.RELEASE is still

[GitHub] [hudi] codope opened a new pull request, #5302: [HUDI-3838] Move getPartitionColumns call out of executor to the driver

2022-04-12 Thread GitBox
codope opened a new pull request, #5302: URL: https://github.com/apache/hudi/pull/5302 ## What is the purpose of the pull request Minor improvement based on [PR comment ](https://github.com/apache/hudi/pull/5294#discussion_r848568345) ## Brief change log *(for example:)*

  1   2   3   >