[GitHub] [hudi] nsivabalan opened a new pull request #5141: [WIP] Fixing closure of ParquetReader

2022-03-26 Thread GitBox
nsivabalan opened a new pull request #5141: URL: https://github.com/apache/hudi/pull/5141 ## What is the purpose of the pull request We were running integration tests against hudi and in recent times we are seeing "too many open files" and the spark long running COW tests fails. Look

[GitHub] [hudi] hudi-bot commented on pull request #5140: [HUDI-3722] Fix truncate hudi table's error

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5140: URL: https://github.com/apache/hudi/pull/5140#issuecomment-1079851253 ## CI report: * a43ae34ec4ca0d1c491f5d689a62fbd234131775 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #5140: [HUDI-3722] Fix truncate hudi table's error

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5140: URL: https://github.com/apache/hudi/pull/5140#issuecomment-1079850878 ## CI report: * a43ae34ec4ca0d1c491f5d689a62fbd234131775 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run

[GitHub] [hudi] hudi-bot commented on pull request #5140: [HUDI-3722] Fix truncate hudi table's error

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5140: URL: https://github.com/apache/hudi/pull/5140#issuecomment-1079850878 ## CI report: * a43ae34ec4ca0d1c491f5d689a62fbd234131775 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure`

[jira] [Updated] (HUDI-3722) Fix truncate hudi table's error

2022-03-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3722: - Labels: pull-request-available (was: ) > Fix truncate hudi table's error > --

[GitHub] [hudi] XuQianJin-Stars opened a new pull request #5140: [HUDI-3722] Fix truncate hudi table's error

2022-03-26 Thread GitBox
XuQianJin-Stars opened a new pull request #5140: URL: https://github.com/apache/hudi/pull/5140 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the

[jira] [Updated] (HUDI-3722) Fix truncate hudi table's error

2022-03-26 Thread Forward Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Forward Xu updated HUDI-3722: - Epic Link: HUDI-3431 > Fix truncate hudi table's error > --- > >

[jira] [Updated] (HUDI-3722) Fix truncate hudi table's error

2022-03-26 Thread Forward Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Forward Xu updated HUDI-3722: - Summary: Fix truncate hudi table's error (was: Fix truncate table's error) > Fix truncate hudi table's er

[jira] [Created] (HUDI-3722) Fix truncate table's error

2022-03-26 Thread Forward Xu (Jira)
Forward Xu created HUDI-3722: Summary: Fix truncate table's error Key: HUDI-3722 URL: https://issues.apache.org/jira/browse/HUDI-3722 Project: Apache Hudi Issue Type: Bug Components: sp

[jira] [Closed] (HUDI-3604) Missing to apply rollback commits to Metadata table if rollback failed mid-way

2022-03-26 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-3604. --- Resolution: Fixed > Missing to apply rollback commits to Metadata table if rollback failed mid-way > -

[hudi] branch master updated (4d940bb -> 484b340)

2022-03-26 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 4d940bb [HUDI-3716] OOM occurred when use bulk_insert cow table with flink BUCKET index (#5135) add 484b340 [HUD

[GitHub] [hudi] yihua merged pull request #5114: [HUDI-3604] Adjust the order of timeline changes in rollbacks

2022-03-26 Thread GitBox
yihua merged pull request #5114: URL: https://github.com/apache/hudi/pull/5114 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...

[GitHub] [hudi] hudi-bot commented on pull request #5114: [HUDI-3604] Adjust the order of timeline changes in rollbacks

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5114: URL: https://github.com/apache/hudi/pull/5114#issuecomment-1079843159 ## CI report: * 5bc41f79495caba40c46a34b7e71a8cdcf904878 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #5114: [HUDI-3604] Adjust the order of timeline changes in rollbacks

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5114: URL: https://github.com/apache/hudi/pull/5114#issuecomment-1079833665 ## CI report: * 77a160c6afccd444842299b2aae382b76d2dc4ef Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] YannByron commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
YannByron commented on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079839012 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079830211 ## CI report: * 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079838919 ## CI report: * dcec725d12526135dac3be0073fddadf25d654e9 UNKNOWN * eaf08d6f752baa9ca28cc3db1ff35c99e99bc207 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org

[GitHub] [hudi] xiarixiaoyao commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
xiarixiaoyao commented on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079838219 > @xiarixiaoyao thanks for fixing this! > > Can you please also add the test that you've used as a benchmark (based on JMH)? i will add benchmark to cover it. no

[GitHub] [hudi] wqwl611 commented on issue #5136: [SUPPORT] Empty parquet "is not a Parquet file (length is too low: 0)"

2022-03-26 Thread GitBox
wqwl611 commented on issue #5136: URL: https://github.com/apache/hudi/issues/5136#issuecomment-1079836598 https://user-images.githubusercontent.com/67826098/160266592-be4dd02d-dcb6-4914-837b-987d42c35a88.png";> I found maybe spark speculative execution cause this broken parquet.

[GitHub] [hudi] hudi-bot commented on pull request #5139: [WIP][HUDI-3579] Add timeline commands in hudi-cli

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5139: URL: https://github.com/apache/hudi/pull/5139#issuecomment-1079834548 ## CI report: * a732deaf15bc0b4c79fab0bcb16b6c5c5603359b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?

[GitHub] [hudi] hudi-bot removed a comment on pull request #5139: [WIP][HUDI-3579] Add timeline commands in hudi-cli

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5139: URL: https://github.com/apache/hudi/pull/5139#issuecomment-1079831937 ## CI report: * a732deaf15bc0b4c79fab0bcb16b6c5c5603359b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] alexeykudinkin commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
alexeykudinkin commented on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079834292 @xiarixiaoyao thanks for fixing this! Can you please also add the test that you've used as a benchmark (based on JMH)? -- This is an automated message from the A

[jira] [Created] (HUDI-3721) Metadata table blocks rollback and restore to savepoint before bootstrapped/init commit

2022-03-26 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-3721: --- Summary: Metadata table blocks rollback and restore to savepoint before bootstrapped/init commit Key: HUDI-3721 URL: https://issues.apache.org/jira/browse/HUDI-3721 Project: Ap

[GitHub] [hudi] codope commented on a change in pull request #4693: [HUDI-2488][HUDI-3175] Implement async metadata indexing

2022-03-26 Thread GitBox
codope commented on a change in pull request #4693: URL: https://github.com/apache/hudi/pull/4693#discussion_r835845988 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieIndexer.java ## @@ -0,0 +1,219 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] [hudi] hudi-bot removed a comment on pull request #5114: [HUDI-3604] Adjust the order of timeline changes in rollbacks

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5114: URL: https://github.com/apache/hudi/pull/5114#issuecomment-1079833277 ## CI report: * 77a160c6afccd444842299b2aae382b76d2dc4ef Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #5114: [HUDI-3604] Adjust the order of timeline changes in rollbacks

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5114: URL: https://github.com/apache/hudi/pull/5114#issuecomment-1079833665 ## CI report: * 77a160c6afccd444842299b2aae382b76d2dc4ef Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot commented on pull request #5114: [HUDI-3604] Adjust the order of timeline changes in rollbacks

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5114: URL: https://github.com/apache/hudi/pull/5114#issuecomment-1079833277 ## CI report: * 77a160c6afccd444842299b2aae382b76d2dc4ef Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #5114: [HUDI-3604] Adjust the order of timeline changes in rollbacks

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5114: URL: https://github.com/apache/hudi/pull/5114#issuecomment-1078613082 ## CI report: * 77a160c6afccd444842299b2aae382b76d2dc4ef Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot removed a comment on pull request #5139: [WIP][HUDI-3579] Add timeline commands in hudi-cli

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5139: URL: https://github.com/apache/hudi/pull/5139#issuecomment-1079831700 ## CI report: * a732deaf15bc0b4c79fab0bcb16b6c5c5603359b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run

[GitHub] [hudi] hudi-bot commented on pull request #5139: [WIP][HUDI-3579] Add timeline commands in hudi-cli

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5139: URL: https://github.com/apache/hudi/pull/5139#issuecomment-1079831937 ## CI report: * a732deaf15bc0b4c79fab0bcb16b6c5c5603359b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot commented on pull request #5139: [WIP][HUDI-3579] Add timeline commands in hudi-cli

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5139: URL: https://github.com/apache/hudi/pull/5139#issuecomment-1079831700 ## CI report: * a732deaf15bc0b4c79fab0bcb16b6c5c5603359b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure`

[jira] [Updated] (HUDI-3579) Add timeline commands in hudi-cli

2022-03-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3579: - Labels: pull-request-available (was: ) > Add timeline commands in hudi-cli >

[GitHub] [hudi] yihua opened a new pull request #5139: [WIP][HUDI-3579] Add timeline commands in hudi-cli

2022-03-26 Thread GitBox
yihua opened a new pull request #5139: URL: https://github.com/apache/hudi/pull/5139 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpose o

[jira] [Commented] (HUDI-3604) Missing to apply rollback commits to Metadata table if rollback failed mid-way

2022-03-26 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17512826#comment-17512826 ] Ethan Guo commented on HUDI-3604: - Case (a) is going to be fixed by the PR.   For case (

[jira] [Created] (HUDI-3720) Rollback reattempt fails if the commit to roll back is not present

2022-03-26 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-3720: --- Summary: Rollback reattempt fails if the commit to roll back is not present Key: HUDI-3720 URL: https://issues.apache.org/jira/browse/HUDI-3720 Project: Apache Hudi I

[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079830211 ## CI report: * 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079829847 ## CI report: * 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079829847 ## CI report: * 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079829534 ## CI report: * 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079829534 ## CI report: * 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079748188 ## CI report: * 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot removed a comment on pull request #5065: [HUDI-2566] Adding multi-writer test support to integ test

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5065: URL: https://github.com/apache/hudi/pull/5065#issuecomment-1079813281 ## CI report: * c098330db24be6455fb4951772a4a703cc9e99fa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #5065: [HUDI-2566] Adding multi-writer test support to integ test

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5065: URL: https://github.com/apache/hudi/pull/5065#issuecomment-1079829509 ## CI report: * 075f78f9caa64b6a835b4c7c66560548cbd7d802 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
xiarixiaoyao commented on a change in pull request #5137: URL: https://github.com/apache/hudi/pull/5137#discussion_r835843426 ## File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala ## @@ -62,10 +62,10 @@ object AvroConversionUtils

[jira] [Commented] (HUDI-3551) Add OCS StorageScheme to support Oracle Cloud

2022-03-26 Thread Rajesh (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17512822#comment-17512822 ] Rajesh commented on HUDI-3551: -- Looked at the PR's [~cartershanklin]  looks like the document

[GitHub] [hudi] nsivabalan commented on issue #5105: cluster with incorrect partition

2022-03-26 Thread GitBox
nsivabalan commented on issue #5105: URL: https://github.com/apache/hudi/issues/5105#issuecomment-1079817651 @suryaprasanna : Can you please assist here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hudi] nsivabalan commented on issue #5101: [SUPPORT] Deltastreamer: setting hoodie.datasource.hive_sync.partition_fields to blank results in empty partition being used in hive sync

2022-03-26 Thread GitBox
nsivabalan commented on issue #5101: URL: https://github.com/apache/hudi/issues/5101#issuecomment-1079816578 may I know why are you trying to explicitly set it to empty string. hudi will infer for you. you can skip setting it. Do you see any issues with that? -- This is an automated mes

[GitHub] [hudi] nsivabalan closed issue #5000: [SUPPORT] Update all records based on key rather than using preCombineKey field

2022-03-26 Thread GitBox
nsivabalan closed issue #5000: URL: https://github.com/apache/hudi/issues/5000 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...

[GitHub] [hudi] nsivabalan commented on issue #5036: [SUPPORT] AWS DMS and Deletes on S3 with Hudi

2022-03-26 Thread GitBox
nsivabalan commented on issue #5036: URL: https://github.com/apache/hudi/issues/5036#issuecomment-1079816097 let us know if you were able to find a solution. feel free to close out the github issue if you got it resolved. -- This is an automated message from the Apache Git Service. To r

[GitHub] [hudi] nsivabalan closed issue #5047: [SUPPORT] Small file creation while writing to a Hudi Table

2022-03-26 Thread GitBox
nsivabalan closed issue #5047: URL: https://github.com/apache/hudi/issues/5047 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...

[GitHub] [hudi] nsivabalan commented on issue #5047: [SUPPORT] Small file creation while writing to a Hudi Table

2022-03-26 Thread GitBox
nsivabalan commented on issue #5047: URL: https://github.com/apache/hudi/issues/5047#issuecomment-1079815928 recently we did fix the small file configs behavior https://github.com/apache/hudi/pull/5129 also, try setting "hoodie.parquet.block.size" in addition to "hoodie.parquet.m

[GitHub] [hudi] nsivabalan commented on issue #5058: [SUPPORT]. Hudi Deltastreamer Job getting failed with Error upserting bucketType UPDATE for partition :0

2022-03-26 Thread GitBox
nsivabalan commented on issue #5058: URL: https://github.com/apache/hudi/issues/5058#issuecomment-1079815544 closing the github issue as we don't have much AI from our side. thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [hudi] nsivabalan closed issue #5058: [SUPPORT]. Hudi Deltastreamer Job getting failed with Error upserting bucketType UPDATE for partition :0

2022-03-26 Thread GitBox
nsivabalan closed issue #5058: URL: https://github.com/apache/hudi/issues/5058 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...

[jira] [Commented] (HUDI-3409) Expose Timeline Server Metrics

2022-03-26 Thread Rajesh (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17512798#comment-17512798 ] Rajesh commented on HUDI-3409: -- [~dswift]  can you provide  little more detail on how we can

[GitHub] [hudi] hudi-bot removed a comment on pull request #5065: [HUDI-2566] Adding multi-writer test support to integ test

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5065: URL: https://github.com/apache/hudi/pull/5065#issuecomment-1079812849 ## CI report: * c098330db24be6455fb4951772a4a703cc9e99fa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #5065: [HUDI-2566] Adding multi-writer test support to integ test

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5065: URL: https://github.com/apache/hudi/pull/5065#issuecomment-1079813281 ## CI report: * c098330db24be6455fb4951772a4a703cc9e99fa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #5065: [HUDI-2566] Adding multi-writer test support to integ test

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5065: URL: https://github.com/apache/hudi/pull/5065#issuecomment-1072927851 ## CI report: * c098330db24be6455fb4951772a4a703cc9e99fa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #5065: [HUDI-2566] Adding multi-writer test support to integ test

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5065: URL: https://github.com/apache/hudi/pull/5065#issuecomment-1079812849 ## CI report: * c098330db24be6455fb4951772a4a703cc9e99fa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] nsivabalan commented on pull request #5065: [HUDI-2566] Adding multi-writer test support to integ test

2022-03-26 Thread GitBox
nsivabalan commented on pull request #5065: URL: https://github.com/apache/hudi/pull/5065#issuecomment-1079812369 @yihua : addressed all comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[jira] [Commented] (HUDI-2520) Certify sync with Hive 3

2022-03-26 Thread leesf (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17512797#comment-17512797 ] leesf commented on HUDI-2520: - [~rex_xiong] hi, are you working on a fix? > Certify sync with

[hudi] branch master updated: [HUDI-3716] OOM occurred when use bulk_insert cow table with flink BUCKET index (#5135)

2022-03-26 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 4d940bb [HUDI-3716] OOM occurred when use bulk

[GitHub] [hudi] danny0405 merged pull request #5135: [HUDI-3716] OOM occurred when use bulk_insert cow table with flink BU…

2022-03-26 Thread GitBox
danny0405 merged pull request #5135: URL: https://github.com/apache/hudi/pull/5135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubsc

[GitHub] [hudi] hudi-bot commented on pull request #5138: [HUDI-3713] Guarding archival for multi-writer

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5138: URL: https://github.com/apache/hudi/pull/5138#issuecomment-1079787512 ## CI report: * 13ad272877616750cdcaceecfc8e77b055b46983 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #5138: [HUDI-3713] Guarding archival for multi-writer

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5138: URL: https://github.com/apache/hudi/pull/5138#issuecomment-1079777075 ## CI report: * 13ad272877616750cdcaceecfc8e77b055b46983 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[hudi] branch master updated (57b4f39 -> 189d529)

2022-03-26 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 57b4f39 [HUDI-3612] Clustering strategy should create new TypedProperties when modifying it (#5027) add 189d5

[GitHub] [hudi] nsivabalan merged pull request #5129: [HUDI-3709] Fixing `ParquetWriter` impls not respecting Parquet Max File Size limit

2022-03-26 Thread GitBox
nsivabalan merged pull request #5129: URL: https://github.com/apache/hudi/pull/5129 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubs

[GitHub] [hudi] hudi-bot commented on pull request #5138: [HUDI-3713] Guarding archival with locks

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5138: URL: https://github.com/apache/hudi/pull/5138#issuecomment-1079777075 ## CI report: * 13ad272877616750cdcaceecfc8e77b055b46983 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #5138: [HUDI-3713] Guarding archival with locks

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5138: URL: https://github.com/apache/hudi/pull/5138#issuecomment-1079776700 ## CI report: * 13ad272877616750cdcaceecfc8e77b055b46983 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run

[GitHub] [hudi] hudi-bot commented on pull request #5138: [HUDI-3713] Guarding archival with locks

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5138: URL: https://github.com/apache/hudi/pull/5138#issuecomment-1079776700 ## CI report: * 13ad272877616750cdcaceecfc8e77b055b46983 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure`

[jira] [Updated] (HUDI-3713) Archival fails w/ NPE with multi-writers

2022-03-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3713: - Labels: pull-request-available (was: ) > Archival fails w/ NPE with multi-writers > -

[GitHub] [hudi] nsivabalan opened a new pull request #5138: [HUDI-3713] Guarding archival with locks

2022-03-26 Thread GitBox
nsivabalan opened a new pull request #5138: URL: https://github.com/apache/hudi/pull/5138 ## What is the purpose of the pull request When multi-writers are involved, archival fails w/ NullPointerException. Hence guarding with locks. ## Brief change log - Fix HoodieTime

[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079748188 ## CI report: * 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079727822 ## CI report: * 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4489: [WIP][HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-03-26 Thread GitBox
hudi-bot commented on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1079741048 ## CI report: * d7168e4702215d46a56254ea1ae12e90cb77e88e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #4489: [WIP][HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1079727203 ## CI report: * ad5ed9559a0811033df791682d489c51814aae6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] alexeykudinkin commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
alexeykudinkin commented on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079737555 @xiarixiaoyao @xushiyan let's also think about how we can prevent such regressions in the future. Ideally, we should check in the test that you used to validate it as a smo

[jira] [Updated] (HUDI-3495) Reading keys in parallel from HoodieMetadataMergedLogRecordReader may lead to empty results even if key exists

2022-03-26 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3495: Priority: Major (was: Blocker) > Reading keys in parallel from HoodieMetadataMergedLogRecordReader may lead

[GitHub] [hudi] alexeykudinkin commented on pull request #4709: [HUDI-3338] custom relation instead of HadoopFsRelation

2022-03-26 Thread GitBox
alexeykudinkin commented on pull request #4709: URL: https://github.com/apache/hudi/pull/4709#issuecomment-1079734704 SG, let's put it back -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [hudi] alexeykudinkin commented on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing

2022-03-26 Thread GitBox
alexeykudinkin commented on issue #5107: URL: https://github.com/apache/hudi/issues/5107#issuecomment-1079734532 Thanks for flagging this @YuweiXiao, great catch! To summarize what the issue is here: it is unfortunately a very sneaky one and it occurred accidentally during the refac

[GitHub] [hudi] nsivabalan commented on a change in pull request #4693: [HUDI-2488][HUDI-3175] Implement async metadata indexing

2022-03-26 Thread GitBox
nsivabalan commented on a change in pull request #4693: URL: https://github.com/apache/hudi/pull/4693#discussion_r835786373 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/index/RunIndexActionExecutor.java ## @@ -0,0 +1,263 @@ +/* + * Li

[GitHub] [hudi] nsivabalan commented on a change in pull request #4693: [HUDI-2488][HUDI-3175] Implement async metadata indexing

2022-03-26 Thread GitBox
nsivabalan commented on a change in pull request #4693: URL: https://github.com/apache/hudi/pull/4693#discussion_r835785960 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/index/RunIndexActionExecutor.java ## @@ -0,0 +1,189 @@ +/* + * Li

[GitHub] [hudi] nsivabalan commented on a change in pull request #4693: [HUDI-2488][HUDI-3175] Implement async metadata indexing

2022-03-26 Thread GitBox
nsivabalan commented on a change in pull request #4693: URL: https://github.com/apache/hudi/pull/4693#discussion_r835785855 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/index/RunIndexActionExecutor.java ## @@ -0,0 +1,189 @@ +/* + * Li

[GitHub] [hudi] hudi-bot commented on pull request #4945: [HUDI-3538] Support Compaction Command Based on Call Procedure Command for Spark SQL

2022-03-26 Thread GitBox
hudi-bot commented on pull request #4945: URL: https://github.com/apache/hudi/pull/4945#issuecomment-1079732557 ## CI report: * 3802bb3e07923b1904f141d62fb744667493ddfb Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #4945: [HUDI-3538] Support Compaction Command Based on Call Procedure Command for Spark SQL

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #4945: URL: https://github.com/apache/hudi/pull/4945#issuecomment-1079718404 ## CI report: * ce55ea05eb9d27917fce72a3e40678ee8a5ed55e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] alexeykudinkin commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
alexeykudinkin commented on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079732495 @xiarixiaoyao also, please update the PR description so that whoever will be reviewing this PR will not need to go to JIRA to understand what it is about -- This is an au

[GitHub] [hudi] nsivabalan commented on a change in pull request #4693: [HUDI-2488][HUDI-3175] Implement async metadata indexing

2022-03-26 Thread GitBox
nsivabalan commented on a change in pull request #4693: URL: https://github.com/apache/hudi/pull/4693#discussion_r835785470 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/index/RunIndexActionExecutor.java ## @@ -0,0 +1,189 @@ +/* + * Li

[GitHub] [hudi] alexeykudinkin commented on a change in pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
alexeykudinkin commented on a change in pull request #5137: URL: https://github.com/apache/hudi/pull/5137#discussion_r835784601 ## File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala ## @@ -62,10 +62,10 @@ object AvroConversionUtil

[GitHub] [hudi] nsivabalan commented on a change in pull request #4693: [HUDI-2488][HUDI-3175] Implement async metadata indexing

2022-03-26 Thread GitBox
nsivabalan commented on a change in pull request #4693: URL: https://github.com/apache/hudi/pull/4693#discussion_r835784016 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/index/RunIndexActionExecutor.java ## @@ -0,0 +1,189 @@ +/* + * Li

[GitHub] [hudi] nsivabalan commented on a change in pull request #4693: [HUDI-2488][HUDI-3175] Implement async metadata indexing

2022-03-26 Thread GitBox
nsivabalan commented on a change in pull request #4693: URL: https://github.com/apache/hudi/pull/4693#discussion_r835783252 ## File path: hudi-common/src/main/java/org/apache/hudi/common/bloom/BloomFilter.java ## @@ -30,6 +34,13 @@ */ void add(String key); + /** +

[GitHub] [hudi] XuQianJin-Stars removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
XuQianJin-Stars removed a comment on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079727554 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079727822 ## CI report: * 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079726338 ## CI report: * 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] XuQianJin-Stars commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
XuQianJin-Stars commented on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079727554 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[GitHub] [hudi] hudi-bot commented on pull request #4489: [WIP][HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-03-26 Thread GitBox
hudi-bot commented on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1079727203 ## CI report: * ad5ed9559a0811033df791682d489c51814aae6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #4489: [WIP][HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1079726707 ## CI report: * ad5ed9559a0811033df791682d489c51814aae6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #4489: [WIP][HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-03-26 Thread GitBox
hudi-bot commented on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1079726707 ## CI report: * ad5ed9559a0811033df791682d489c51814aae6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #4489: [WIP][HUDI-3135] Fix Delete partitions with metadata table and fix show partitions in spark sql

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #4489: URL: https://github.com/apache/hudi/pull/4489#issuecomment-1066331484 ## CI report: * ad5ed9559a0811033df791682d489c51814aae6a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] hudi-bot commented on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot commented on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079726338 ## CI report: * 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?b

[GitHub] [hudi] hudi-bot removed a comment on pull request #5137: [HUDI-3719] High performance costs of AvroSerizlizer in DataSource wr…

2022-03-26 Thread GitBox
hudi-bot removed a comment on pull request #5137: URL: https://github.com/apache/hudi/pull/5137#issuecomment-1079709771 ## CI report: * 6d18b04edbbaa9a8f8e3645e00df7fa6d76f51e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/r

[GitHub] [hudi] nsivabalan commented on a change in pull request #4693: [HUDI-2488][HUDI-3175] Implement async metadata indexing

2022-03-26 Thread GitBox
nsivabalan commented on a change in pull request #4693: URL: https://github.com/apache/hudi/pull/4693#discussion_r835782128 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieIndexer.java ## @@ -0,0 +1,276 @@ +/* + * Licensed to the Apache Software Foun

  1   2   >