[jira] [Updated] (HUDI-2324) [UMBRELLA] Implement Hudi Transaction writes for Kafka Connect platform

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2324: - Fix Version/s: 0.12.0 (was: 0.11.0) > [UMBRELLA] Implement Hudi Transaction writes

[jira] [Updated] (HUDI-1859) [UMBRELLA] RFC - 14 : JDBC incremental puller

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-1859: - Component/s: reader-core > [UMBRELLA] RFC - 14 : JDBC incremental puller > ---

[GitHub] [hudi] xushiyan commented on a change in pull request #4916: Update roadmap page to reflect latest state

2022-02-28 Thread GitBox
xushiyan commented on a change in pull request #4916: URL: https://github.com/apache/hudi/pull/4916#discussion_r816519733 ## File path: website/src/pages/roadmap.md ## @@ -62,11 +62,11 @@ Next major release : [0.11.0](https://issues.apache.org/jira/projects/HUDI/versi |--

[GitHub] [hudi] danny0405 commented on issue #4558: [BUG] Bootstrap operation data loading missing

2022-02-28 Thread GitBox
danny0405 commented on issue #4558: URL: https://github.com/apache/hudi/issues/4558#issuecomment-1055120580 @waywtdcc Try 0.10.1 if possible again, the problem expects to be resolved ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] danny0405 commented on issue #3960: [SUPPORT]How to auto sync "add column" by flink ?

2022-02-28 Thread GitBox
danny0405 commented on issue #3960: URL: https://github.com/apache/hudi/issues/3960#issuecomment-1055119077 > > IMO, this requires the flink can support schema evolution, current community flink doesn't support it, you can consider use aliyun enterprise flink, we are support it. > >

[GitHub] [hudi] hudi-bot removed a comment on pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL

2022-02-28 Thread GitBox
hudi-bot removed a comment on pull request #4901: URL: https://github.com/apache/hudi/pull/4901#issuecomment-1055001727 ## CI report: * ceec40b0d2b0fb64c7288cc79a1bd127e47df822 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/

[GitHub] [hudi] hudi-bot commented on pull request #4901: [HUDI-3445] Support Clustering Command Based on Call Procedure Command for Spark SQL

2022-02-28 Thread GitBox
hudi-bot commented on pull request #4901: URL: https://github.com/apache/hudi/pull/4901#issuecomment-1055117112 ## CI report: * 7eb3a7fe67a7dde03dc1082a7e4b23c91f7d3ec0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] danny0405 commented on a change in pull request #4000: [HUDI-2631] In CompactFunction, set up the write schema each time wit…

2022-02-28 Thread GitBox
danny0405 commented on a change in pull request #4000: URL: https://github.com/apache/hudi/pull/4000#discussion_r816515465 ## File path: hudi-flink/src/main/java/org/apache/hudi/sink/compact/CompactFunction.java ## @@ -114,6 +117,17 @@ private void doCompaction(String instantT

[GitHub] [hudi] danny0405 commented on issue #4882: [SUPPORT]How to clean action files theses are not used with flink task?

2022-02-28 Thread GitBox
danny0405 commented on issue #4882: URL: https://github.com/apache/hudi/issues/4882#issuecomment-1055114614 These files should be archived then, did your job have pending compactions on the timeline ? -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] danny0405 edited a comment on issue #4881: Full incremental Enable index loading to discover duplicate data(index.bootstrap.enabled)

2022-02-28 Thread GitBox
danny0405 edited a comment on issue #4881: URL: https://github.com/apache/hudi/issues/4881#issuecomment-1055113849 Seems a known bug fixed before, does you code has this fix: https://github.com/apache/hudi/pull/3925 ? -- This is an automated message from the Apache Git Service. To respon

[GitHub] [hudi] danny0405 commented on issue #4881: Full incremental Enable index loading to discover duplicate data(index.bootstrap.enabled)

2022-02-28 Thread GitBox
danny0405 commented on issue #4881: URL: https://github.com/apache/hudi/issues/4881#issuecomment-1055113849 Seems a known bug fixed before, dod you code has this fix: https://github.com/apache/hudi/pull/3925 -- This is an automated message from the Apache Git Service. To respond to the m

[jira] [Updated] (HUDI-1042) [Umbrella] Support clustering on filegroups

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-1042: - Fix Version/s: 0.12.0 > [Umbrella] Support clustering on filegroups >

[GitHub] [hudi] danny0405 edited a comment on pull request #4880: [HUDI-2752] The MOR DELETE block breaks the event time sequence of CDC

2022-02-28 Thread GitBox
danny0405 edited a comment on pull request #4880: URL: https://github.com/apache/hudi/pull/4880#issuecomment-1053032393 > I skimmed through the patch. Looks like, we are looking to fix only a specific scenario. > > insert record with preCombine value 100. delete record with preCombin

[jira] [Updated] (HUDI-3019) Upserts with Dataype promotion only to a subset of partition fails

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3019: - Priority: Critical (was: Major) > Upserts with Dataype promotion only to a subset of partition fails > --

[jira] [Updated] (HUDI-3019) Upserts with Dataype promotion only to a subset of partition fails

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3019: - Component/s: writer-core > Upserts with Dataype promotion only to a subset of partition fails > --

[GitHub] [hudi] hudi-bot removed a comment on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-02-28 Thread GitBox
hudi-bot removed a comment on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1055106014 ## CI report: * 45bb573f0f66cbdb8dd09c5635294a5777a6ad2a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-02-28 Thread GitBox
hudi-bot commented on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1055108107 ## CI report: * 31bca2b35905b3a1207af4099d4d06b99059dc2f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?

[jira] [Updated] (HUDI-2560) Introduce id_based schema to support full schema evolution

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2560: - Priority: Blocker (was: Major) > Introduce id_based schema to support full schema evolution > ---

[jira] [Updated] (HUDI-2560) Introduce id_based schema to support full schema evolution

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2560: - Issue Type: New Feature (was: Task) > Introduce id_based schema to support full schema evolution > --

[jira] [Updated] (HUDI-2429) [UMBRELLA] Comprehensive Schema evolution in Hudi

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2429: - Priority: Blocker (was: Major) > [UMBRELLA] Comprehensive Schema evolution in Hudi >

[GitHub] [hudi] hudi-bot removed a comment on pull request #4925: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single sink table from multiple source tables

2022-02-28 Thread GitBox
hudi-bot removed a comment on pull request #4925: URL: https://github.com/apache/hudi/pull/4925#issuecomment-1054995552 ## CI report: * d0c0fc1caef055950f86b6bf09fd63fae1dbfe03 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4925: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single sink table from multiple source tables

2022-02-28 Thread GitBox
hudi-bot commented on pull request #4925: URL: https://github.com/apache/hudi/pull/4925#issuecomment-1055106425 ## CI report: * 82f08a4b57e32d64c0beb3fb952570a23b31eda1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-02-28 Thread GitBox
hudi-bot removed a comment on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1055037227 ## CI report: * 45bb573f0f66cbdb8dd09c5635294a5777a6ad2a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-02-28 Thread GitBox
hudi-bot commented on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1055106014 ## CI report: * 45bb573f0f66cbdb8dd09c5635294a5777a6ad2a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] garyli1019 commented on a change in pull request #4679: [HUDI-3315] RFC-35 Part-1 Support bucket index in Flink writer

2022-02-28 Thread GitBox
garyli1019 commented on a change in pull request #4679: URL: https://github.com/apache/hudi/pull/4679#discussion_r816499865 ## File path: hudi-flink/src/main/java/org/apache/hudi/sink/BucketStreamWriteFunction.java ## @@ -0,0 +1,158 @@ +/* + * Licensed to the Apache Software F

[jira] [Created] (HUDI-3539) Flink bucket index bucketID bootstrap optimization

2022-02-28 Thread Gary Li (Jira)
Gary Li created HUDI-3539: - Summary: Flink bucket index bucketID bootstrap optimization Key: HUDI-3539 URL: https://issues.apache.org/jira/browse/HUDI-3539 Project: Apache Hudi Issue Type: Improvemen

[GitHub] [hudi] cuibo01 commented on pull request #4909: [HUDI-3516] Implement record iterator for HoodieDataBlock

2022-02-28 Thread GitBox
cuibo01 commented on pull request #4909: URL: https://github.com/apache/hudi/pull/4909#issuecomment-1055087137 @danny0405 Thank you very much for your review :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] hudi-bot removed a comment on pull request #4866: [HUDI-3469] Refactor `HoodieTestDataGenerator` to provide for reproducible Builds

2022-02-28 Thread GitBox
hudi-bot removed a comment on pull request #4866: URL: https://github.com/apache/hudi/pull/4866#issuecomment-1054995184 ## CI report: * 180ea55d8c08a4933202dbb3cd2cc87b06e0ef3d UNKNOWN * 2c85afe1c7a84f3e7d55a8111bc6a6e9a0214c16 Azure: [FAILURE](https://dev.azure.com/apache-hud

[GitHub] [hudi] hudi-bot commented on pull request #4866: [HUDI-3469] Refactor `HoodieTestDataGenerator` to provide for reproducible Builds

2022-02-28 Thread GitBox
hudi-bot commented on pull request #4866: URL: https://github.com/apache/hudi/pull/4866#issuecomment-1055086370 ## CI report: * 180ea55d8c08a4933202dbb3cd2cc87b06e0ef3d UNKNOWN * ff1ccbd820c49ed9bfe89c762367d3a3f2b1e33d UNKNOWN * 95c0c7b9d535751ba540aa2440ad61f0f90d21c6 UNKN

[GitHub] [hudi] cuibo01 commented on pull request #4924: [WIP][CI Test Only - 2][HUDI-1180] Upgrade HBase to 2.4.9

2022-02-28 Thread GitBox
cuibo01 commented on pull request #4924: URL: https://github.com/apache/hudi/pull/4924#issuecomment-1055085042 in hbase 2.x, scanner and reader add `close` method -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [hudi] hudi-bot commented on pull request #4909: [HUDI-3516] Implement record iterator for HoodieDataBlock

2022-02-28 Thread GitBox
hudi-bot commented on pull request #4909: URL: https://github.com/apache/hudi/pull/4909#issuecomment-1055084862 ## CI report: * 4b93451c19c4f69cf04ec6139114e424a3b72f00 UNKNOWN * 9f920edbe3ed87e180b259d81a8343197080fcc1 UNKNOWN * 455953353c2d193a00578bcb11b2ae47035bf82b Azur

[GitHub] [hudi] hudi-bot removed a comment on pull request #4909: [HUDI-3516] Implement record iterator for HoodieDataBlock

2022-02-28 Thread GitBox
hudi-bot removed a comment on pull request #4909: URL: https://github.com/apache/hudi/pull/4909#issuecomment-1055073577 ## CI report: * 4b93451c19c4f69cf04ec6139114e424a3b72f00 UNKNOWN * 9f920edbe3ed87e180b259d81a8343197080fcc1 UNKNOWN * 455953353c2d193a00578bcb11b2ae47035bf

[GitHub] [hudi] hudi-bot removed a comment on pull request #4739: [HUDI-3365] Make sure Metadata Table records are updated appropriately on HDFS

2022-02-28 Thread GitBox
hudi-bot removed a comment on pull request #4739: URL: https://github.com/apache/hudi/pull/4739#issuecomment-1054991623 ## CI report: * 11f1b688459ab9017ebde2a38d1645e0f59b50c3 UNKNOWN * c243f70d774b7ecb059dad4bb03870b2c2d4436b UNKNOWN * cb354756580e1a9b6515ff1c9cc670579a3e7

[GitHub] [hudi] hudi-bot commented on pull request #4739: [HUDI-3365] Make sure Metadata Table records are updated appropriately on HDFS

2022-02-28 Thread GitBox
hudi-bot commented on pull request #4739: URL: https://github.com/apache/hudi/pull/4739#issuecomment-1055078518 ## CI report: * 11f1b688459ab9017ebde2a38d1645e0f59b50c3 UNKNOWN * c243f70d774b7ecb059dad4bb03870b2c2d4436b UNKNOWN * cb354756580e1a9b6515ff1c9cc670579a3e70f4 Azur

[jira] [Commented] (HUDI-3522) Introduce DropColumnSchemaPostProcessor to support drop columns from schema

2022-02-28 Thread Xianghu Wang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17499354#comment-17499354 ] Xianghu Wang commented on HUDI-3522: [~shivnarayan] thanks for your response. there i

[GitHub] [hudi] hudi-bot removed a comment on pull request #4909: [HUDI-3516] Implement record iterator for HoodieDataBlock

2022-02-28 Thread GitBox
hudi-bot removed a comment on pull request #4909: URL: https://github.com/apache/hudi/pull/4909#issuecomment-1054970221 ## CI report: * 4b93451c19c4f69cf04ec6139114e424a3b72f00 UNKNOWN * 9f920edbe3ed87e180b259d81a8343197080fcc1 UNKNOWN * 455953353c2d193a00578bcb11b2ae47035bf

[GitHub] [hudi] hudi-bot commented on pull request #4909: [HUDI-3516] Implement record iterator for HoodieDataBlock

2022-02-28 Thread GitBox
hudi-bot commented on pull request #4909: URL: https://github.com/apache/hudi/pull/4909#issuecomment-1055073577 ## CI report: * 4b93451c19c4f69cf04ec6139114e424a3b72f00 UNKNOWN * 9f920edbe3ed87e180b259d81a8343197080fcc1 UNKNOWN * 455953353c2d193a00578bcb11b2ae47035bf82b Azur

[jira] [Created] (HUDI-3538) Support compaction command based on Call Produce Command

2022-02-28 Thread shibei (Jira)
shibei created HUDI-3538: Summary: Support compaction command based on Call Produce Command Key: HUDI-3538 URL: https://issues.apache.org/jira/browse/HUDI-3538 Project: Apache Hudi Issue Type: New Fe

[jira] [Updated] (HUDI-3445) Support clustering command based on Call Produce Command

2022-02-28 Thread shibei (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shibei updated HUDI-3445: - Component/s: (was: compaction) Summary: Support clustering command based on Call Produce Command (was

[GitHub] [hudi] pmgod8922 opened a new issue #4929: [SUPPORT] SparkSession To Hudi Small files are not merged

2022-02-28 Thread GitBox
pmgod8922 opened a new issue #4929: URL: https://github.com/apache/hudi/issues/4929 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at dev-subscr

[GitHub] [hudi] hudi-bot removed a comment on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version

2022-02-28 Thread GitBox
hudi-bot removed a comment on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1050786941 ## CI report: * 0f9e39b14a31e8fd62f9267552de857b92cb5b34 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version

2022-02-28 Thread GitBox
hudi-bot commented on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1055051537 ## CI report: * 0f9e39b14a31e8fd62f9267552de857b92cb5b34 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] huberylee commented on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version

2022-02-28 Thread GitBox
huberylee commented on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1055050715 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] hudi-bot commented on pull request #4880: [HUDI-2752] The MOR DELETE block breaks the event time sequence of CDC

2022-02-28 Thread GitBox
hudi-bot commented on pull request #4880: URL: https://github.com/apache/hudi/pull/4880#issuecomment-1055050093 ## CI report: * e7ffd325e71f535cc6316c468a0cea353236b40b UNKNOWN * ba1ac9c3dc31cf865df4c3f254ea064c4c4f1dc1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4880: [HUDI-2752] The MOR DELETE block breaks the event time sequence of CDC

2022-02-28 Thread GitBox
hudi-bot removed a comment on pull request #4880: URL: https://github.com/apache/hudi/pull/4880#issuecomment-1054956428 ## CI report: * 1568291657e15a58d19d31dcd64c2b93fe274aa7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] melin opened a new issue #4928: [SUPPORT] JDBC Catalog

2022-02-28 Thread GitBox
melin opened a new issue #4928: URL: https://github.com/apache/hudi/issues/4928 Provides a JDBC catalog to manage Hudi tables using a relational database. Is it valuable? iceberg jdbc catalog: https://iceberg.apache.org/docs/latest/jdbc/ -- This is an automated message from the Apache

[GitHub] [hudi] hudi-bot removed a comment on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-02-28 Thread GitBox
hudi-bot removed a comment on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1055035693 ## CI report: * 45bb573f0f66cbdb8dd09c5635294a5777a6ad2a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-02-28 Thread GitBox
hudi-bot commented on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1055037227 ## CI report: * 45bb573f0f66cbdb8dd09c5635294a5777a6ad2a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-02-28 Thread GitBox
hudi-bot removed a comment on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1055029134 ## CI report: * 45bb573f0f66cbdb8dd09c5635294a5777a6ad2a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-02-28 Thread GitBox
hudi-bot commented on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1055035693 ## CI report: * 45bb573f0f66cbdb8dd09c5635294a5777a6ad2a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-02-28 Thread GitBox
hudi-bot removed a comment on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1054949433 ## CI report: * 93bec1ef3483dc43fd8ff39c4a376427c7d9f097 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4489: [HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-02-28 Thread GitBox
hudi-bot commented on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1055029134 ## CI report: * 45bb573f0f66cbdb8dd09c5635294a5777a6ad2a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[jira] [Commented] (HUDI-3334) Unable to merge HoodieMetadataPayload during partition listing

2022-02-28 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17499335#comment-17499335 ] Ethan Guo commented on HUDI-3334: - Reopening this based on another report of the same Ille

[jira] [Updated] (HUDI-3334) Unable to merge HoodieMetadataPayload during partition listing

2022-02-28 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3334: Story Points: 2 (was: 0) > Unable to merge HoodieMetadataPayload during partition listing > ---

[jira] [Updated] (HUDI-3334) Unable to merge HoodieMetadataPayload during partition listing

2022-02-28 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3334: Priority: Blocker (was: Major) > Unable to merge HoodieMetadataPayload during partition listing > -

[jira] [Reopened] (HUDI-3334) Unable to merge HoodieMetadataPayload during partition listing

2022-02-28 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reopened HUDI-3334: - > Unable to merge HoodieMetadataPayload during partition listing > ---

[GitHub] [hudi] danny0405 commented on a change in pull request #4679: [HUDI-3315] RFC-35 Part-1 Support bucket index in Flink writer

2022-02-28 Thread GitBox
danny0405 commented on a change in pull request #4679: URL: https://github.com/apache/hudi/pull/4679#discussion_r816451737 ## File path: hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java ## @@ -189,21 +191,32 @@ } public static DataStream hoodieStreamWr

[GitHub] [hudi] danny0405 commented on a change in pull request #4679: [HUDI-3315] RFC-35 Part-1 Support bucket index in Flink writer

2022-02-28 Thread GitBox
danny0405 commented on a change in pull request #4679: URL: https://github.com/apache/hudi/pull/4679#discussion_r816451410 ## File path: hudi-flink/src/main/java/org/apache/hudi/sink/BucketStreamWriteFunction.java ## @@ -0,0 +1,158 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [hudi] danny0405 commented on a change in pull request #4679: [HUDI-3315] RFC-35 Part-1 Support bucket index in Flink writer

2022-02-28 Thread GitBox
danny0405 commented on a change in pull request #4679: URL: https://github.com/apache/hudi/pull/4679#discussion_r816451150 ## File path: hudi-flink/src/main/java/org/apache/hudi/sink/BucketStreamWriteFunction.java ## @@ -0,0 +1,158 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [hudi] danny0405 commented on a change in pull request #4679: [HUDI-3315] RFC-35 Part-1 Support bucket index in Flink writer

2022-02-28 Thread GitBox
danny0405 commented on a change in pull request #4679: URL: https://github.com/apache/hudi/pull/4679#discussion_r816450304 ## File path: hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java ## @@ -310,6 +318,20 @@ private FlinkOptions() { + "Actu

[GitHub] [hudi] watermelon12138 commented on a change in pull request #4925: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single sink table from multiple source tables

2022-02-28 Thread GitBox
watermelon12138 commented on a change in pull request #4925: URL: https://github.com/apache/hudi/pull/4925#discussion_r816449044 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java ## @@ -379,23 +444,50 @@ priva

[GitHub] [hudi] danny0405 commented on pull request #3771: [HUDI-2402] Add Kerberos configuration options to Hive Sync

2022-02-28 Thread GitBox
danny0405 commented on pull request #3771: URL: https://github.com/apache/hudi/pull/3771#issuecomment-1055017841 Hi @nsivabalan , many users report this issue so i mark this PR as blocker for release-0.11 -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [hudi] danny0405 commented on pull request #4909: [HUDI-3516] Implement record iterator for HoodieDataBlock

2022-02-28 Thread GitBox
danny0405 commented on pull request #4909: URL: https://github.com/apache/hudi/pull/4909#issuecomment-1055017415 Thanks, overall looks good, i have reviewed and have another patch for this :) [HUDI-3516_2.patch.zip](https://github.com/apache/hudi/files/8158826/HUDI-3516_2.patch.zip)

[GitHub] [hudi] Gatsby-Lee commented on issue #4896: [SUPPORT] Metadata Table causes missing data.

2022-02-28 Thread GitBox
Gatsby-Lee commented on issue #4896: URL: https://github.com/apache/hudi/issues/4896#issuecomment-1055017231 @nsivabalan @yihua Thank you Please let me know if you need anything from me about this issue. -- This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [hudi] Gatsby-Lee commented on issue #4839: Hudi upsert doesnt trigger compaction for MOR

2022-02-28 Thread GitBox
Gatsby-Lee commented on issue #4839: URL: https://github.com/apache/hudi/issues/4839#issuecomment-1055016784 @glory9211 Like @nsivabalan said, if your data latency is not critical, then CoW works as well. I use CoW for some datasets that have incoming data around 600-800/sec. I design

[jira] [Updated] (HUDI-2466) Add and validate comprehensive yamls for spark dml

2022-02-28 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2466: -- Sprint: (was: Hudi-Sprint-Mar-01) > Add and validate comprehensive yamls for spark dml

[jira] [Updated] (HUDI-3530) Make Spark bundle version X.Y instead of X.Y.Z

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3530: - Sprint: Hudi-Sprint-Mar-01 > Make Spark bundle version X.Y instead of X.Y.Z >

[jira] [Assigned] (HUDI-3470) Add support to test hudi w/ failure injections

2022-02-28 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-3470: - Assignee: sivabalan narayanan > Add support to test hudi w/ failure injections >

[jira] [Updated] (HUDI-3470) Add support to test hudi w/ failure injections

2022-02-28 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3470: -- Sprint: Hudi-Sprint-Mar-01 > Add support to test hudi w/ failure injections > --

[jira] [Updated] (HUDI-1623) Support start_commit_time & end_commit_times for serializable incremental pull

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-1623: - Story Points: 5 > Support start_commit_time & end_commit_times for serializable incremental pull > ---

[jira] [Assigned] (HUDI-2860) Make timeline server work with concurrent/async table service

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu reassigned HUDI-2860: Assignee: Raymond Xu > Make timeline server work with concurrent/async table service >

[jira] [Updated] (HUDI-2860) Make timeline server work with concurrent/async table service

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2860: - Sprint: Hudi-Sprint-Mar-01 > Make timeline server work with concurrent/async table service > -

[jira] [Updated] (HUDI-3325) Query Integration: Util to get aggregate columns ranges across all files from the column index

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3325: -- Sprint: Hudi-Sprint-Mar-01 > Query Integration: Util to get aggregate columns ranges across all

[jira] [Updated] (HUDI-3468) RFC: Emit table metadata to Linkedin datahub metadata

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3468: - Sprint: (was: Hudi-Sprint-Mar-01) > RFC: Emit table metadata to Linkedin datahub metadata >

[jira] [Updated] (HUDI-3405) Query Integration: Graceful fallback when indexes are not available

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3405: -- Sprint: Hudi-Sprint-Mar-01 > Query Integration: Graceful fallback when indexes are not available

[jira] [Updated] (HUDI-3326) Query Integration: HoodieFileReader should expose API for getting range metadata

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3326: -- Sprint: Hudi-Sprint-Mar-01 > Query Integration: HoodieFileReader should expose API for getting r

[jira] [Updated] (HUDI-3324) Query Integration: Support returning file names matching the given columns and ranges

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3324: -- Sprint: Hudi-Sprint-Mar-01 > Query Integration: Support returning file names matching the given

[jira] [Updated] (HUDI-3217) RFC-46: Optimize Record Payload handling

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3217: -- Fix Version/s: (was: 0.11.0) > RFC-46: Optimize Record Payload handling > --

[jira] [Updated] (HUDI-3353) Rebase `HoodieFileWriter` to accept `HoodieRecord`

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3353: -- Priority: Major (was: Blocker) > Rebase `HoodieFileWriter` to accept `HoodieRecord` > -

[jira] [Updated] (HUDI-3350) Create Engine-specific Implementations of `HoodieRecord`

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3350: -- Priority: Major (was: Blocker) > Create Engine-specific Implementations of `HoodieRecord` > ---

[jira] [Updated] (HUDI-2566) Add multi-writer testing infra to test hudi

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2566: - Sprint: Hudi-Sprint-Mar-01 > Add multi-writer testing infra to test hudi > ---

[jira] [Updated] (HUDI-3410) Revisit Record-reading Abstractions

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3410: -- Priority: Major (was: Blocker) > Revisit Record-reading Abstractions >

[jira] [Updated] (HUDI-3379) Rebase `HoodieAppendHandle` to operate on `HoodieRecord`

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3379: -- Priority: Major (was: Blocker) > Rebase `HoodieAppendHandle` to operate on `HoodieRecord` > ---

[jira] [Updated] (HUDI-3384) Implement Spark-specific FileWriters

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3384: -- Priority: Major (was: Blocker) > Implement Spark-specific FileWriters > ---

[jira] [Updated] (HUDI-3380) Rebase `HoodieDataBlock`s to operate on `HoodieRecord`

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3380: -- Priority: Major (was: Blocker) > Rebase `HoodieDataBlock`s to operate on `HoodieRecord` > -

[jira] [Updated] (HUDI-2466) Add and validate comprehensive yamls for spark dml

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2466: - Sprint: Hudi-Sprint-Mar-01 > Add and validate comprehensive yamls for spark dml > ---

[jira] [Updated] (HUDI-3354) Rebase `HoodieRealtimeRecordReader` to return `HoodieRecord`

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3354: -- Priority: Major (was: Blocker) > Rebase `HoodieRealtimeRecordReader` to return `HoodieRecord` >

[jira] [Updated] (HUDI-3351) Rebase Record combining semantic into `HoodieRecordCombiningEngine`

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3351: -- Priority: Major (was: Blocker) > Rebase Record combining semantic into `HoodieRecordCombiningEn

[jira] [Updated] (HUDI-3349) Revisit HoodieRecord API to be able to replace HoodieRecordPayload

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3349: -- Priority: Major (was: Blocker) > Revisit HoodieRecord API to be able to replace HoodieRecordPay

[jira] [Updated] (HUDI-3378) Rebase `HoodieCreateHandle` to operate on `HoodieRecord`

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3378: -- Priority: Major (was: Blocker) > Rebase `HoodieCreateHandle` to operate on `HoodieRecord` > ---

[jira] [Updated] (HUDI-2530) Add async table services test infra to test hudi

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2530: - Sprint: Hudi-Sprint-Mar-01 > Add async table services test infra to test hudi > -

[jira] [Updated] (HUDI-3385) Implement Spark-specific `FileReader`s

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3385: -- Priority: Major (was: Blocker) > Implement Spark-specific `FileReader`s > -

[jira] [Updated] (HUDI-3177) CREATE INDEX command

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3177: - Story Points: 0 (was: 4) > CREATE INDEX command > > > Key: HUDI-3177

[jira] [Updated] (HUDI-3491) Remove PathFilter from DirectoryLister

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3491: - Story Points: 0 (was: 2) > Remove PathFilter from DirectoryLister > -

[jira] [Updated] (HUDI-3381) Rebase `HoodieMergeHandle` to operate on `HoodieRecord`

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3381: -- Fix Version/s: 0.11.0 > Rebase `HoodieMergeHandle` to operate on `HoodieRecord` > --

[jira] [Updated] (HUDI-3381) Rebase `HoodieMergeHandle` to operate on `HoodieRecord`

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3381: -- Fix Version/s: (was: 0.11.0) > Rebase `HoodieMergeHandle` to operate on `HoodieRecord` > ---

[jira] [Updated] (HUDI-3381) Rebase `HoodieMergeHandle` to operate on `HoodieRecord`

2022-02-28 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-3381: -- Priority: Major (was: Blocker) > Rebase `HoodieMergeHandle` to operate on `HoodieRecord` >

[jira] [Assigned] (HUDI-1370) Scoping work needed to support bootstrapped data table and RFC-15 together

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu reassigned HUDI-1370: Assignee: Alexey Kudinkin > Scoping work needed to support bootstrapped data table and RFC-15 toget

[jira] [Closed] (HUDI-3368) Support metadata bloom index for secondary keys

2022-02-28 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-3368. - Resolution: Won't Do > Support metadata bloom index for secondary keys > -

[jira] [Updated] (HUDI-1370) Scoping work needed to support bootstrapped data table and RFC-15 together

2022-02-28 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-1370: - Sprint: Hudi-Sprint-Jan-18, Hudi-Sprint-Jan-24, Hudi-Sprint-Jan-31, Hudi-Sprint-Feb-7, Hudi-Sprint-Feb-14,

  1   2   3   4   5   6   7   8   >