Re: [PR] [HUDI-9147] Support HoodieFileGroupReader for Flink and use FileGroup reader in compaction [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13078: URL: https://github.com/apache/hudi/pull/13078#issuecomment-2781196295 ## CI report: * a7b5bf3d6d35a9dec6b5101d0b270534698f37a6 UNKNOWN * 72eb82f6c236bacde23aaeac192a807d39188cee UNKNOWN * 8ff30568268dfb24df1fc17d6a7c256d045a621f UNKNOWN *

Re: [PR] [HUDI-9147] Support HoodieFileGroupReader for Flink and use FileGroup reader in compaction [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13078: URL: https://github.com/apache/hudi/pull/13078#issuecomment-2781171663 ## CI report: * a7b5bf3d6d35a9dec6b5101d0b270534698f37a6 UNKNOWN * 72eb82f6c236bacde23aaeac192a807d39188cee UNKNOWN * 8ff30568268dfb24df1fc17d6a7c256d045a621f UNKNOWN *

Re: [PR] [DNM] test validate bundle [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13087: URL: https://github.com/apache/hudi/pull/13087#issuecomment-2781165719 ## CI report: * c480071bba7d14af4adf9ee426251533ee14d3a2 UNKNOWN * f5e10734454c53b1e9aeb85d8706ee3cda077520 UNKNOWN * 984f801303f8d8df3246942ea47ea12498f34007 UNKNOWN *

Re: [PR] [HUDI-9147] Support HoodieFileGroupReader for Flink and use FileGroup reader in compaction [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13078: URL: https://github.com/apache/hudi/pull/13078#issuecomment-2781159176 ## CI report: * a7b5bf3d6d35a9dec6b5101d0b270534698f37a6 UNKNOWN * 72eb82f6c236bacde23aaeac192a807d39188cee UNKNOWN * 8ff30568268dfb24df1fc17d6a7c256d045a621f UNKNOWN *

Re: [PR] [HUDI-9147] Support HoodieFileGroupReader for Flink and use FileGroup reader in compaction [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13078: URL: https://github.com/apache/hudi/pull/13078#issuecomment-2781158748 ## CI report: * a7b5bf3d6d35a9dec6b5101d0b270534698f37a6 UNKNOWN * 72eb82f6c236bacde23aaeac192a807d39188cee UNKNOWN * 8ff30568268dfb24df1fc17d6a7c256d045a621f UNKNOWN *

Re: [PR] [HUDI-9147] Support HoodieFileGroupReader for Flink and use FileGroup reader in compaction [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13078: URL: https://github.com/apache/hudi/pull/13078#issuecomment-2781156269 ## CI report: * a7b5bf3d6d35a9dec6b5101d0b270534698f37a6 UNKNOWN * 72eb82f6c236bacde23aaeac192a807d39188cee UNKNOWN * 8ff30568268dfb24df1fc17d6a7c256d045a621f UNKNOWN *

Re: [PR] [HUDI-9147] Support HoodieFileGroupReader for Flink and use FileGroup reader in compaction [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13078: URL: https://github.com/apache/hudi/pull/13078#issuecomment-2781155437 ## CI report: * a7b5bf3d6d35a9dec6b5101d0b270534698f37a6 UNKNOWN * 72eb82f6c236bacde23aaeac192a807d39188cee UNKNOWN * 8ff30568268dfb24df1fc17d6a7c256d045a621f UNKNOWN *

Re: [PR] [DNM] test validate bundle [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13087: URL: https://github.com/apache/hudi/pull/13087#issuecomment-2781144827 ## CI report: * c480071bba7d14af4adf9ee426251533ee14d3a2 UNKNOWN * 6c21537ed3cd2d22ab12e56598cd3a3b67bd4833 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47

Re: [PR] [DNM] test validate bundle [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13087: URL: https://github.com/apache/hudi/pull/13087#issuecomment-2781143087 ## CI report: * c480071bba7d14af4adf9ee426251533ee14d3a2 UNKNOWN * 6c21537ed3cd2d22ab12e56598cd3a3b67bd4833 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47

Re: [PR] [DNM] test validate bundle [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13087: URL: https://github.com/apache/hudi/pull/13087#issuecomment-2781134513 ## CI report: * c480071bba7d14af4adf9ee426251533ee14d3a2 UNKNOWN * 6c21537ed3cd2d22ab12e56598cd3a3b67bd4833 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47

Re: [PR] [DNM] test validate bundle [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13087: URL: https://github.com/apache/hudi/pull/13087#issuecomment-2781128488 ## CI report: * c480071bba7d14af4adf9ee426251533ee14d3a2 UNKNOWN * 6c21537ed3cd2d22ab12e56598cd3a3b67bd4833 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47

Re: [PR] [HUDI-9012] Implement and utilize native writer for HFile [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #12866: URL: https://github.com/apache/hudi/pull/12866#issuecomment-2755167805 ## CI report: * 180b65dd680bf029888057648b10bb4c9f480a97 UNKNOWN * e229e3a1b793a97339602dfc61ce0eae3668ade7 UNKNOWN * 7c9e1fbbb52ae47d26daa0e25fc3d02569c79545 UNKNOWN *

[I] [SUPPORT] HoodieCompactionException on hoodie metadata causing upserts to fail. [hudi]

2025-04-05 Thread via GitHub
gbcoder2020 opened a new issue, #13002: URL: https://github.com/apache/hudi/issues/13002 **Describe the problem you faced** In one of the job runs involving upsert data to Hudi CoW table, I observed failure corresponding to HoodieCompactionException on metadata folder. See screen

Re: [PR] [HUDI-9198] Support rate limit for append mode [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #12999: URL: https://github.com/apache/hudi/pull/12999#issuecomment-2744915203 ## CI report: * c09c0429cea5637f68849a18f2173624d5bc1a7d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4325)

Re: [PR] [HUDI-9236] Handle markers for log files in table version 6 [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13007: URL: https://github.com/apache/hudi/pull/13007#issuecomment-2763024455 ## CI report: * 15ad8f7c2f24e34c73a0d7499bc9914626c74a92 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4532)

[PR] test validating flink bundle [hudi]

2025-04-05 Thread via GitHub
cshuo opened a new pull request, #13011: URL: https://github.com/apache/hudi/pull/13011 ### Change Logs test validating flink bundle ### Impact no ### Risk level (write none, low medium or high below) low ### Documentation Update _Describe any

Re: [PR] [WIP][HUDI-8990] Partition Level Bucket Index [hudi]

2025-04-05 Thread via GitHub
zhangyue19921010 commented on code in PR #13017: URL: https://github.com/apache/hudi/pull/13017#discussion_r2014114953 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/index/bucket/TestHoodieSimpleBucketIndex.java: ## @@ -29,6 +29,7 @@ import org.apache.hudi.data.

Re: [PR] [HUDI-6895] Change default timeline timezone from local to UTC [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #9794: URL: https://github.com/apache/hudi/pull/9794#issuecomment-2753387939 ## CI report: * 1936e68fddcb9b7b3b1669bae834b6f0a05b6d95 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4419)

[jira] [Assigned] (HUDI-9230) Partition Level Bucket Index adopt Spark and Flink reader bucket id pruning

2025-04-05 Thread Yue Zhang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yue Zhang reassigned HUDI-9230: --- Assignee: Yue Zhang > Partition Level Bucket Index adopt Spark and Flink reader bucket id pruning > -

[jira] [Created] (HUDI-9201) Validate and simplify FG buffer merge logic

2025-04-05 Thread Lin Liu (Jira)
Lin Liu created HUDI-9201: - Summary: Validate and simplify FG buffer merge logic Key: HUDI-9201 URL: https://issues.apache.org/jira/browse/HUDI-9201 Project: Apache Hudi Issue Type: Bug Affects V

Re: [PR] [MINOR] Fix NumberFormatException while updating metrics for MDT in table version 6 [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13056: URL: https://github.com/apache/hudi/pull/13056#issuecomment-2763766226 ## CI report: * cfce6dd0c69d5d798ef7318d017a3a6c269f824f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-6895] Change default timeline timezone from local to UTC [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #9794: URL: https://github.com/apache/hudi/pull/9794#issuecomment-2751557279 ## CI report: * 10b69b62ee63f391d2f7e269a92d20b6fbeb117f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4394)

Re: [PR] [HUDI-9247] Re-evaluate reuse of TimeGenerator instance [hudi]

2025-04-05 Thread via GitHub
nsivabalan commented on PR #13077: URL: https://github.com/apache/hudi/pull/13077#issuecomment-2776492281 We might need something like below ``` public interface TimeGenerator { /** * Generates a globally monotonically increasing timestamp. The implementation must ensur

[PR] [HUDI-9147] Support HoodieFileGroupReader for Flink and use FileGroup reader in compaction [hudi]

2025-04-05 Thread via GitHub
cshuo opened a new pull request, #13078: URL: https://github.com/apache/hudi/pull/13078 ### Change Logs * Implement FileGroup reader for Flink * Support Flink compaction use FileGroup reader ### Impact Improve perf for Flink compaction ### Risk level (write none, lo

(hudi) branch HUDI-9207 updated: Spark Insert Overwrite Support Row Writer

2025-04-05 Thread zhangyue19921010
This is an automated email from the ASF dual-hosted git repository. zhangyue19921010 pushed a commit to branch HUDI-9207 in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/HUDI-9207 by this push: new 44bf019a0ee Spark Insert Overwrite

[jira] [Assigned] (HUDI-9236) Cherry-pick HUDI-1517 for release 1.0.2

2025-04-05 Thread Lokesh Jain (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain reassigned HUDI-9236: - Assignee: Y Ethan Guo > Cherry-pick HUDI-1517 for release 1.0.2 > ---

Re: [PR] [WIP][HUDI-8990] Partition Level Bucket Index [hudi]

2025-04-05 Thread via GitHub
zhangyue19921010 commented on code in PR #13017: URL: https://github.com/apache/hudi/pull/13017#discussion_r2011348699 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bucket/PartitionBucketIndexCalculator.java: ## @@ -0,0 +1,359 @@ +/* + * Licensed to the A

Re: [PR] [HUDI-8990] Partition bucket index supports query pruning based on bucket id [hudi]

2025-04-05 Thread via GitHub
zhangyue19921010 commented on code in PR #13060: URL: https://github.com/apache/hudi/pull/13060#discussion_r2026285922 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala: ## @@ -103,6 +104,15 @@ case class HoodieFileIndex(spark: Spark

Re: [PR] test validating flink bundle [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13011: URL: https://github.com/apache/hudi/pull/13011#issuecomment-2742828978 ## CI report: * dd0ffb4ee8bf8eb1ebab8cb50fbb6f50becdcdcf Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4306)

Re: [PR] [HUDI-1517] create marker file for every log file [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13007: URL: https://github.com/apache/hudi/pull/13007#issuecomment-2741160881 ## CI report: * e02fd7c542196d9e751b8b4427d431149dc7f88c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4288)

Re: [PR] [HUDI-9215] Set partitionColumnsWithKeyGenerator based on table version [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13025: URL: https://github.com/apache/hudi/pull/13025#issuecomment-2748968488 ## CI report: * 54bd983c9ac1b177a14cad5b3bb8bf98a961e448 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-9144] Flink writer for MOR table supports writing RowData … [hudi]

2025-04-05 Thread via GitHub
danny0405 commented on code in PR #12967: URL: https://github.com/apache/hudi/pull/12967#discussion_r2002782140 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bucket/BucketRowDataStreamWriteFunction.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apach

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2737958064 ## CI report: * 411b6b0bd0238e770d9454c88d5a1daca0af41a6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4258)

[jira] [Updated] (HUDI-9220) Cannot find write operation type if run inline log compaction

2025-04-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-9220: - Labels: pull-request-available (was: ) > Cannot find write operation type if run inline log compa

Re: [PR] [HUDI-7915] Spark 4 support [hudi]

2025-04-05 Thread via GitHub
wombatu-kun commented on PR #12772: URL: https://github.com/apache/hudi/pull/12772#issuecomment-2767730311 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] [HUDI-9144] Flink writer for MOR table supports writing RowData … [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #12967: URL: https://github.com/apache/hudi/pull/12967#issuecomment-2742473202 ## CI report: * fe88b33a3a00147d07978c47646bf7ed0de96387 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4297)

Re: [PR] [HUDI-9144] Flink writer for MOR table supports writing RowData … [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #12967: URL: https://github.com/apache/hudi/pull/12967#issuecomment-2746731729 ## CI report: * 8e29c5367e7997865cfa1c29eb3e2ca1ac4ffb28 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4310)

Re: [PR] [HUDI-8409] Fixing merge mode config during upgrade and downgrade from version 7 to 8 and back [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13046: URL: https://github.com/apache/hudi/pull/13046#issuecomment-2763081899 ## CI report: * b2934f721306a6d045fdfbfe20acde9d55c6b4be Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4534)

Re: [PR] [HUDI-9144] Flink writer for MOR table supports writing RowData … [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #12967: URL: https://github.com/apache/hudi/pull/12967#issuecomment-2741991434 ## CI report: * f3521b40d2d21e6b13f113bbe980a896e60d807a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4294)

Re: [PR] [MINOR] Fixing master for build failure [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13085: URL: https://github.com/apache/hudi/pull/13085#issuecomment-2776956305 ## CI report: * a9afa718f7bca8020a071d9c79769567a346bafa UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[jira] [Closed] (HUDI-9144) [RFC-87] Flink writer for MOR table switchs to parquet data block for log file

2025-04-05 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-9144. Reviewers: Danny Chen Resolution: Fixed Fixed via master branch: 9352576357da04080a15456a01d6993cd553de

[PR] [HUDI-8990] Partition Level Bucket Index [hudi]

2025-04-05 Thread via GitHub
zhangyue19921010 opened a new pull request, #13012: URL: https://github.com/apache/hudi/pull/13012 ### Change Logs **Completed:** 1. Common Layer Core Abstractions 2. Meta, Rule, Calculator, etc.: Development and testing completed. 3. Flink Integration => bulk_insert m

Re: [PR] [HUDI-6249] Move thread-safety to map impls for FSView [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13082: URL: https://github.com/apache/hudi/pull/13082#issuecomment-2776507495 ## CI report: * d800411050aa30d83ad546b063c03a19a2dad4eb Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4620)

Re: [PR] [WIP][HUDI-8990] Partition Level Bucket Index [hudi]

2025-04-05 Thread via GitHub
zhangyue19921010 commented on code in PR #13017: URL: https://github.com/apache/hudi/pull/13017#discussion_r2011355530 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bucket/PartitionBucketIndexCalculator.java: ## @@ -0,0 +1,359 @@ +/* + * Licensed to the A

Re: [PR] [HUDI-9249]Support displaying InsertIntoHoodieTableCommand metrics in Spark Web UI [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13068: URL: https://github.com/apache/hudi/pull/13068#issuecomment-2771441883 ## CI report: * 5afa0a422373f13699714bcbca9cceb26ad9a305 UNKNOWN * 1ac0baed776aeac81e69c197940f615edd499a01 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47

Re: [PR] [HUDI-9205] Refactor Flink tests to avoid sleeping for data results [hudi]

2025-04-05 Thread via GitHub
cshuo commented on code in PR #13027: URL: https://github.com/apache/hudi/pull/13027#discussion_r2011449334 ## hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java: ## @@ -147,20 +157,17 @@ void testStreamWriteAndReadFromSpecifiedComm

Re: [PR] [WIP][HUDI-8990] Partition Level Bucket Index [hudi]

2025-04-05 Thread via GitHub
danny0405 commented on code in PR #13017: URL: https://github.com/apache/hudi/pull/13017#discussion_r2011816952 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bucket/partition/PartitionBucketIndexUtils.java: ## @@ -0,0 +1,84 @@ +/* + * Licensed to the Apac

Re: [PR] [HUDI-9218] Implement HoodieRecordMerger for Flink HoodieRecord [hudi]

2025-04-05 Thread via GitHub
cshuo commented on code in PR #13040: URL: https://github.com/apache/hudi/pull/13040#discussion_r2016086609 ## hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/merge/TestHoodieFlinkRecordMerger.java: ## @@ -0,0 +1,180 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] [HUDI-9248] Unify code paths for all write operations about `bulk_insert` [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13066: URL: https://github.com/apache/hudi/pull/13066#issuecomment-2771187333 ## CI report: * eee072110ed173302af4757a93b77f035e56f09f UNKNOWN * fbb4d3bae14c6ed006ed2c2b57f1fa41e7b1d2f5 UNKNOWN * 76a01e4ea9f39c1b154db6ac754a9d49b2f60fcf Azure: [SUCC

Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13071: URL: https://github.com/apache/hudi/pull/13071#issuecomment-2771480829 ## CI report: * 47c18c9bb7b720dcc38465ac31d92c6cd50da0fd Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4578)

Re: [PR] [HUDI-6895] Change default timeline timezone from local to UTC [hudi]

2025-04-05 Thread via GitHub
codope commented on PR #9794: URL: https://github.com/apache/hudi/pull/9794#issuecomment-2773160677 @vinothchandar The main reason for this fix is that a user might run table service or some job from their local machine which is on a different timezone from the servers. So, we have switched

Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]

2025-04-05 Thread via GitHub
jonvex commented on code in PR #12340: URL: https://github.com/apache/hudi/pull/12340#discussion_r2027325214 ## hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java: ## @@ -0,0 +1,464 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

(hudi) 01/06: finish hashing_config initial && related UT

2025-04-05 Thread zhangyue19921010
This is an automated email from the ASF dual-hosted git repository. zhangyue19921010 pushed a commit to branch HUDI-8990 in repository https://gitbox.apache.org/repos/asf/hudi.git commit 475b4f108e01e137dcac0fde994ac63ac42e1f15 Author: YueZhang AuthorDate: Mon Mar 17 20:02:24 2025 +0800 fin

[jira] [Updated] (HUDI-9219) Follow up on all usages of Closeable iterator in spark tasks

2025-04-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-9219: -- Fix Version/s: 1.0.2 > Follow up on all usages of Closeable iterator in spark tasks > --

Re: [PR] [MINOR] stop `HoodieSparkSqlWriter` spark context with proper exit code [hudi]

2025-04-05 Thread via GitHub
danny0405 commented on PR #13006: URL: https://github.com/apache/hudi/pull/13006#issuecomment-2741932967 The is exactly the autual effect if the exit code is 0? Can you share with us a production use case. -- This is an automated message from the Apache Git Service. To respond to the mess

[jira] [Created] (HUDI-9255) Fix merge strategy inferring for custom merge mode

2025-04-05 Thread Lokesh Jain (Jira)
Lokesh Jain created HUDI-9255: - Summary: Fix merge strategy inferring for custom merge mode Key: HUDI-9255 URL: https://issues.apache.org/jira/browse/HUDI-9255 Project: Apache Hudi Issue Type: Su

Re: [PR] [HUDI-8409] Fixing merge mode config during upgrade and downgrade from version 7 to 8 and back [hudi]

2025-04-05 Thread via GitHub
yihua commented on code in PR #13046: URL: https://github.com/apache/hudi/pull/13046#discussion_r2019615082 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToSevenDowngradeHandler.java: ## @@ -161,19 +166,27 @@ static void unsetInitialVersion(H

Re: [PR] [MINOR]Fix typo and Add implementation class name in interface method of HoodieRecordMerger [hudi]

2025-04-05 Thread via GitHub
zhangyue19921010 merged PR #13059: URL: https://github.com/apache/hudi/pull/13059 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hu

Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13008: URL: https://github.com/apache/hudi/pull/13008#issuecomment-2756873488 ## CI report: * 9f31bc8eb92a06bafdd0771ae5efd342bcf637aa Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438)

Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]

2025-04-05 Thread via GitHub
linliu-code commented on code in PR #12843: URL: https://github.com/apache/hudi/pull/12843#discussion_r2004864672 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestCustomDeleteRecords.scala: ## @@ -0,0 +1,151 @@ +/* + * Licensed to the Apac

Re: [PR] [HUDI-9249]Support displaying InsertIntoHoodieTableCommand metrics in Spark Web UI [hudi]

2025-04-05 Thread via GitHub
wangyinsheng commented on PR #13068: URL: https://github.com/apache/hudi/pull/13068#issuecomment-2771412091 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [I] [SUPPORT] hive sync table cannot query throw java.lang.IllegalArgumentException [hudi]

2025-04-05 Thread via GitHub
danny0405 commented on issue #13069: URL: https://github.com/apache/hudi/issues/13069#issuecomment-2771389490 VERSION_2 is introduced in 1.x Hudi, it looks like the version is inconsistent between the Flink writer and the `hudi-hadoop-mr-bundle-0.14.1` jar, the oncoming Hudi 1.0.2 would add

[jira] [Closed] (HUDI-8933) With metadata table enabled, upgrade fails during rollback of a pending compaction commit

2025-04-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-8933. - Resolution: Fixed > With metadata table enabled, upgrade fails during rollback of a pendin

Re: [PR] [DNM] Test UT [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13081: URL: https://github.com/apache/hudi/pull/13081#issuecomment-2778053220 ## CI report: * a0c36322d53438a70f1f022da60d7f268278f751 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4649)

[jira] [Created] (HUDI-9222) Revisit usage of database name table config

2025-04-05 Thread Y Ethan Guo (Jira)
Y Ethan Guo created HUDI-9222: - Summary: Revisit usage of database name table config Key: HUDI-9222 URL: https://issues.apache.org/jira/browse/HUDI-9222 Project: Apache Hudi Issue Type: Improveme

Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #12340: URL: https://github.com/apache/hudi/pull/12340#issuecomment-2776550333 ## CI report: * f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN * 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN * 95e3598fd2de8830b945234ee25ba328376068af Azure: [FAIL

Re: [PR] [HUDI-8990] Partition Level Bucket Index [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13012: URL: https://github.com/apache/hudi/pull/13012#issuecomment-2742634917 ## CI report: * aa9ed6ce12c49d9c79034453a9bba3bfaf16a9d2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4307)

Re: [PR] [HUDI-9236] Handle markers for log files in table version 6 [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13007: URL: https://github.com/apache/hudi/pull/13007#issuecomment-2762976673 ## CI report: * 15ad8f7c2f24e34c73a0d7499bc9914626c74a92 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4532)

Re: [PR] [DNM] Test UT [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13081: URL: https://github.com/apache/hudi/pull/13081#issuecomment-2777615267 ## CI report: * 32be9857a80f85c53ad4878a9325b6cc9230fc75 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4635)

Re: [PR] [HUDI-9246] Confine protobuf shading to the hudi-io package [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13065: URL: https://github.com/apache/hudi/pull/13065#issuecomment-2767611171 ## CI report: * cf64134ef0a64d082b99e14930dbd9a73bae4962 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-9211] Fix bug with config in DataHubSyncTool [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13018: URL: https://github.com/apache/hudi/pull/13018#issuecomment-2746472290 ## CI report: * 336e6f8bff9f8f98004800c2c14ce79bbca3ebc0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4340)

Re: [PR] chore: fix codecov report generation [hudi-rs]

2025-04-05 Thread via GitHub
xushiyan merged PR #316: URL: https://github.com/apache/hudi-rs/pull/316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache

Re: [PR] [HUDI-9227] Fix bulk insert overwrite after a failed insert [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13041: URL: https://github.com/apache/hudi/pull/13041#issuecomment-2757060468 ## CI report: * f48592bbd6ed3c5d100414fe79bb1d0b36aefed6 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4479)

Re: [I] [SUPPORT] HoodieCompactionException on hoodie metadata causing upserts to fail. [hudi]

2025-04-05 Thread via GitHub
gbcoder2020 commented on issue #13002: URL: https://github.com/apache/hudi/issues/13002#issuecomment-2743922673 @nsivabalan ``` 2025-03-11T15:04:00.241Z 25/03/11 15:03:59 ERROR YarnScheduler: Lost executor 1492 on : Executor heartbeat timed out after 149584 ms 25/03/11 1

Re: [PR] [HUDI-8474] Adding upsert partitoner for metadata table [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13001: URL: https://github.com/apache/hudi/pull/13001#issuecomment-2738106106 ## CI report: * ce763ab69fadcff3acadf151eb2d35875fc80866 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4270)

[jira] [Commented] (HUDI-8796) Silent ignoring of bucket index in Flink append mode

2025-04-05 Thread Geser Dugarov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17936420#comment-17936420 ] Geser Dugarov commented on HUDI-8796: - Revisit this issue after major changes in Flink

Re: [PR] [HUDI-9259] Fixing marker reconciliation for failures during deleting additional files [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13088: URL: https://github.com/apache/hudi/pull/13088#issuecomment-2777695790 ## CI report: * 0819566d9e2a391d73e3ad92ae31425f6b713592 UNKNOWN * 683ec0fd2c136ca9c98f575229fae461c6d573bf Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47

Re: [PR] [DOCS] Data quality docs update [hudi]

2025-04-05 Thread via GitHub
danny0405 commented on PR #13061: URL: https://github.com/apache/hudi/pull/13061#issuecomment-2767798678 Oops, there are some compile errors: https://github.com/apache/hudi/actions/runs/14175094747/job/39737032336?pr=13061#step:5:36 -- This is an automated message from the Apache Git Serv

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-04-05 Thread via GitHub
yihua commented on code in PR #12991: URL: https://github.com/apache/hudi/pull/12991#discussion_r2004506918 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/TestSqlConf.scala: ## @@ -82,7 +82,7 @@ class TestSqlConf extends HoodieSparkSqlTestBas

Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]

2025-04-05 Thread via GitHub
TheR1sing3un commented on code in PR #13070: URL: https://github.com/apache/hudi/pull/13070#discussion_r2026330349 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedParquetFileFormat.scala: ## @

[jira] [Commented] (HUDI-8933) With metadata table enabled, upgrade fails during rollback of a pending compaction commit

2025-04-05 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17937232#comment-17937232 ] sivabalan narayanan commented on HUDI-8933: --- tried w/ latest master. not reprodu

Re: [PR] [MINOR] Remove warning around table version six [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13080: URL: https://github.com/apache/hudi/pull/13080#issuecomment-2775795950 ## CI report: * ac5da00e4b8e76739795681581d9121e3c1f3f0a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-9088] Fix unnecessary scanning of target table in MERGE INTO on Spark [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #12934: URL: https://github.com/apache/hudi/pull/12934#issuecomment-2756019451 ## CI report: * ac9e7a5b752f0cc80d05837f10599a4490f3b67a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4096)

[jira] [Updated] (HUDI-9211) DatahubSyncTool fails with NullPointerException

2025-04-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-9211: - Labels: pull-request-available (was: ) > DatahubSyncTool fails with NullPointerException > --

[PR] [HUDI-9249]Support displaying InsertIntoHoodieTableCommand metrics in Spark Web UI [hudi]

2025-04-05 Thread via GitHub
wangyinsheng opened a new pull request, #13068: URL: https://github.com/apache/hudi/pull/13068 ### Change Logs Support displaying InsertIntoHoodieTableCommand metrics in Spark Web UI implement steps - Change the return type of `BaseHoodieWriteClient.commit `from `boolean` to

[jira] [Updated] (HUDI-9195) Rowdata write handle builds data block using record iterator if there is no delete record

2025-04-05 Thread Shuo Cheng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuo Cheng updated HUDI-9195: - Summary: Rowdata write handle builds data block using record iterator if there is no delete record (was:

Re: [PR] [HUDI-9255] Fix inferring correct merge behavior for few scenarios [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13079: URL: https://github.com/apache/hudi/pull/13079#issuecomment-2777218884 ## CI report: * 003b62418cbfd698289a2261866567a7c21d2d33 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4630)

Re: [PR] [HUDI-9144] Flink writer for MOR table supports writing RowData … [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #12967: URL: https://github.com/apache/hudi/pull/12967#issuecomment-2742847545 ## CI report: * 27bf56019b6b5d30022131498290dd8cf1ba7b5f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4301)

Re: [PR] [HUDI-9248] Unify code paths for all write operations about `bulk_insert` [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13066: URL: https://github.com/apache/hudi/pull/13066#issuecomment-2771394395 ## CI report: * eee072110ed173302af4757a93b77f035e56f09f UNKNOWN * fbb4d3bae14c6ed006ed2c2b57f1fa41e7b1d2f5 UNKNOWN * 08836df3a4a45001d0d2c1c42338a8692f9d137f Azure: [FAIL

(hudi) branch master updated (9cd1b610ffe -> ee50661db20)

2025-04-05 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 9cd1b610ffe [HUDI-9228] Support rowdata writing for MOR table with consistent hasing (#13043) add ee50661db20 [

[jira] [Closed] (HUDI-9205) Refactor Flink tests to avoid sleeping for data results

2025-04-05 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-9205. Reviewers: Danny Chen Resolution: Fixed Fixed via master branch: b6957f020d00aca9d5c83f51b8e8f0bd102ac9

Re: [PR] [WIP][HUDI-8990] Partition Level Bucket Index [hudi]

2025-04-05 Thread via GitHub
danny0405 commented on code in PR #13017: URL: https://github.com/apache/hudi/pull/13017#discussion_r2011825123 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/index/FlinkHoodieIndexFactory.java: ## @@ -27,7 +27,7 @@ import org.apache.hudi.index.bloom.HoodieGloba

Re: [PR] [HUDI-9123][RFC-91] Add RFC for storage based lock provider using conditional writes. [hudi]

2025-04-05 Thread via GitHub
yihua commented on code in PR #12927: URL: https://github.com/apache/hudi/pull/12927#discussion_r2013079570 ## rfc/rfc-91/rfc-91.md: ## @@ -0,0 +1,145 @@ + +# RFC-91: Storage-based lock provider using conditional writes + +## Proposers + +- @alexr17 + +## Approvers + + - @yihua

Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]

2025-04-05 Thread via GitHub
lokeshj1703 commented on code in PR #13010: URL: https://github.com/apache/hudi/pull/13010#discussion_r2014807324 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/BaseHoodieLogRecordReader.java: ## @@ -136,16 +136,14 @@ public abstract class BaseHoodieLogRecordReade

Re: [PR] [MINOR] stop spark context of HoodieSparkSqlWriter with proper exit code [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13006: URL: https://github.com/apache/hudi/pull/13006#issuecomment-2740523242 ## CI report: * c57eb694f2b479f14d71c3c3924c5226399c9f15 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4286)

(hudi) branch HUDI-8990-V2 updated (aba33eaa2c6 -> 254a910b11e)

2025-04-05 Thread zhangyue19921010
This is an automated email from the ASF dual-hosted git repository. zhangyue19921010 pushed a change to branch HUDI-8990-V2 in repository https://gitbox.apache.org/repos/asf/hudi.git from aba33eaa2c6 fix ut add 254a910b11e fix ut No new revisions were added by this update. Summary of

[jira] [Updated] (HUDI-9216) 1.x spark reader enforces database from tableConfig instead of defaulting to spark.catalog.currentDatabase as fallback

2025-04-05 Thread Vinish Reddy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinish Reddy updated HUDI-9216: --- Description: Hudi as a table format doesn't enforce each table needs to be registered to one catalog.

Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]

2025-04-05 Thread via GitHub
linliu-code commented on code in PR #12843: URL: https://github.com/apache/hudi/pull/12843#discussion_r2004322820 ## hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java: ## @@ -135,13 +137,13 @@ public void processDataBlock(HoodieD

Re: [PR] [HUDI-8990] Partition bucket index supports query pruning based on bucket id [hudi]

2025-04-05 Thread via GitHub
danny0405 commented on code in PR #13060: URL: https://github.com/apache/hudi/pull/13060#discussion_r2021979593 ## hudi-common/src/main/java/org/apache/hudi/common/model/PartitionBucketIndexHashingConfig.java: ## @@ -196,24 +196,51 @@ public static Option loadHashingConfig(Hood

Re: [PR] [HUDI-9207] Spark Insert Overwrite Support Row Writer [hudi]

2025-04-05 Thread via GitHub
hudi-bot commented on PR #13014: URL: https://github.com/apache/hudi/pull/13014#issuecomment-2743867226 ## CI report: * 44bf019a0ee76212d2f8e0293995824214264c33 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4313) Azu

  1   2   3   >