Re: [PR] [HUDI-9198] Support rate limit for append mode [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12999: URL: https://github.com/apache/hudi/pull/12999#issuecomment-2735532569 ## CI report: * ac5b1ea3a5c0dff0bf02f6e67b80abfcd7bb8322 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-9198] Support rate limit for append mode [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12999: URL: https://github.com/apache/hudi/pull/12999#issuecomment-2735534572 ## CI report: * ac5b1ea3a5c0dff0bf02f6e67b80abfcd7bb8322 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4260)

Re: [PR] [HUDI-9120] Enable File Group reader by default for table version 6 [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12935: URL: https://github.com/apache/hudi/pull/12935#issuecomment-2735532327 ## CI report: * d4047fffadd48826944b6f8289546e98607aa8e7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4259)

[PR] [HUDI-9198] Support rate limit for append mode [hudi]

2025-03-18 Thread via GitHub
danny0405 opened a new pull request, #12999: URL: https://github.com/apache/hudi/pull/12999 ### Change Logs Support rate limit for append mode ### Impact none ### Risk level (write none, low medium or high below) low -- This is an automated message fr

Re: [I] [SUPPORT] why HoodieDatasetBulkInsertHelper bulkInsert method no BucketBulkInsertDataInternalWriterHelper [hudi]

2025-03-18 Thread via GitHub
leeseven1211 commented on issue #12989: URL: https://github.com/apache/hudi/issues/12989#issuecomment-2735508732 The code only matches the ConsistentBucketBulkInsertDataInternalWriterHelper, and for all cases that do not match, it uses the BulkInsertDataInternalWriterHelper. When using Buc

Re: [I] [SUPPORT] why HoodieDatasetBulkInsertHelper bulkInsert method no BucketBulkInsertDataInternalWriterHelper [hudi]

2025-03-18 Thread via GitHub
leeseven1211 commented on issue #12989: URL: https://github.com/apache/hudi/issues/12989#issuecomment-2735495056 this is class name : HoodieDatasetBulkInsertHelper , method is : bulkInsert ,this method why no match : case HoodieIndex.IndexType.BUCKET if writeConfig.getBucketIndexEng

[jira] [Updated] (HUDI-9195) [RFC-87] Rowdata write handle builds data block using record iterator if there is no delete record

2025-03-18 Thread Shuo Cheng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuo Cheng updated HUDI-9195: - Summary: [RFC-87] Rowdata write handle builds data block using record iterator if there is no delete recor

[jira] [Updated] (HUDI-9192) [RFC-87] RowData log handle supports writing record positions to log block

2025-03-18 Thread Shuo Cheng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuo Cheng updated HUDI-9192: - Summary: [RFC-87] RowData log handle supports writing record positions to log block (was: RowData log han

[jira] [Updated] (HUDI-9189) [RFC-87] Flink writer support binary buffer based on Flink manged memory

2025-03-18 Thread Shuo Cheng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuo Cheng updated HUDI-9189: - Summary: [RFC-87] Flink writer support binary buffer based on Flink manged memory (was: Flink writer supp

[jira] [Updated] (HUDI-9191) [RFC-87] Support collecting column range metadata for AVRO log block efficiently

2025-03-18 Thread Shuo Cheng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuo Cheng updated HUDI-9191: - Summary: [RFC-87] Support collecting column range metadata for AVRO log block efficiently (was: Support c

[jira] [Updated] (HUDI-9151) [RFC-87] Flink writer/reader for CDC mode switchs to row data

2025-03-18 Thread Shuo Cheng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuo Cheng updated HUDI-9151: - Summary: [RFC-87] Flink writer/reader for CDC mode switchs to row data (was: Flink writer/reader for CDC

[jira] [Updated] (HUDI-9190) [RFC-87] RowData log write handle supports writing Avro data block

2025-03-18 Thread Shuo Cheng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuo Cheng updated HUDI-9190: - Summary: [RFC-87] RowData log write handle supports writing Avro data block (was: RowData log handle supp

[jira] [Updated] (HUDI-9149) [RFC-87] Flink writer for cow supports row data writing/merging

2025-03-18 Thread Shuo Cheng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuo Cheng updated HUDI-9149: - Summary: [RFC-87] Flink writer for cow supports row data writing/merging (was: Flink writer for cow suppo

[jira] [Updated] (HUDI-9144) [RFC-87] Flink writer for MOR table switchs to parquet data block for log file

2025-03-18 Thread Shuo Cheng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuo Cheng updated HUDI-9144: - Summary: [RFC-87] Flink writer for MOR table switchs to parquet data block for log file (was: Flink write

Re: [PR] [RFC-92] Pluggable Table Format Support [hudi]

2025-03-18 Thread via GitHub
danny0405 commented on code in PR #12998: URL: https://github.com/apache/hudi/pull/12998#discussion_r2002498822 ## rfc/rfc-92/rfc-92.md: ## @@ -0,0 +1,102 @@ + +# RFC-92: Pluggable Table Formats in Hudi + +## Proposers + +* Balaji Varadarajan + +## Approvers + +* Vinoth Chan

[jira] [Created] (HUDI-9198) Support rate limit for append mode

2025-03-18 Thread Danny Chen (Jira)
Danny Chen created HUDI-9198: Summary: Support rate limit for append mode Key: HUDI-9198 URL: https://issues.apache.org/jira/browse/HUDI-9198 Project: Apache Hudi Issue Type: New Feature

Re: [PR] [HUDI-9120] Fix delete ordering comparison issue [hudi]

2025-03-18 Thread via GitHub
linliu-code commented on PR #12979: URL: https://github.com/apache/hudi/pull/12979#issuecomment-2732052332 Close this one since it is not a correct fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
linliu-code opened a new pull request, #12990: URL: https://github.com/apache/hudi/pull/12990 ### Change Logs Previously when the configuration "hoodie.record.merge.mode" is null (by default), we will not add precombine field to the required schema since we assume it is commit time o

Re: [PR] [HUDI-9120] Enable File Group reader by default for table version 6 [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12935: URL: https://github.com/apache/hudi/pull/12935#issuecomment-2735398415 ## CI report: * 3498922c1b7993f4919f9bb4400fc8a8565ccdac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4247)

Re: [PR] [HUDI-9120] Enable File Group reader by default for table version 6 [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12935: URL: https://github.com/apache/hudi/pull/12935#issuecomment-2735396350 ## CI report: * 3498922c1b7993f4919f9bb4400fc8a8565ccdac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4247)

hudi.git: Error while running github feature from master:.asf.yaml

2025-03-18 Thread Apache Infrastructure
An error occurred while processing the github feature in .asf.yaml: GitHub discussions can only be enabled if a mailing list target exists for it. --- With regards, ASF Infra.

Re: [PR] [HUDI-7037] Fix colstats reading for Decimal field [hudi]

2025-03-18 Thread via GitHub
codope commented on code in PR #12993: URL: https://github.com/apache/hudi/pull/12993#discussion_r2002300483 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/ColumnStatsIndexSupport.scala: ## @@ -455,10 +456,10 @@ object ColumnStatsIndexSupport {

[PR] [RFC-92] Pluggable Table Format Support [hudi]

2025-03-18 Thread via GitHub
bvaradar opened a new pull request, #12998: URL: https://github.com/apache/hudi/pull/12998 ### Change Logs Pluggable Table Format Support in Hudi ### Impact Pluggable Table Format Support in Hudi ### Risk level (write none, low medium or high below) none

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2735342081 ## CI report: * 411b6b0bd0238e770d9454c88d5a1daca0af41a6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4258)

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
linliu-code closed pull request #12990: [HUDI-9120] Add precombine field if possible URL: https://github.com/apache/hudi/pull/12990 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] [HUDI-7037] Fix colstats reading for Decimal field [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12993: URL: https://github.com/apache/hudi/pull/12993#issuecomment-2735142827 ## CI report: * c81a532851ea54fcb58262145d016323a1e42ac7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4248)

Re: [PR] [HUDI-9196] Disable flaky tests on Flink [hudi]

2025-03-18 Thread via GitHub
danny0405 commented on PR #12994: URL: https://github.com/apache/hudi/pull/12994#issuecomment-2735118369 Let's see if the fix https://github.com/apache/hudi/pull/12995 resolve all these flakiness. -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] [HUDI-7037] Fix colstats reading for Decimal field [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12993: URL: https://github.com/apache/hudi/pull/12993#issuecomment-2735251387 ## CI report: * ca52bb6677971593da5f246468ce260096c88d8a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4256)

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2735247377 ## CI report: * b5e64be8b802c3d2cb048b11c3c83d1296dc2d41 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4257)

Re: [PR] [HUDI-7037] Fix colstats reading for Decimal field [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12993: URL: https://github.com/apache/hudi/pull/12993#issuecomment-2734289239 ## CI report: * c81a532851ea54fcb58262145d016323a1e42ac7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4248)

Re: [PR] [HUDI-9120] Enable File Group reader by default for table version 6 [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12935: URL: https://github.com/apache/hudi/pull/12935#issuecomment-2734375355 ## CI report: * 3498922c1b7993f4919f9bb4400fc8a8565ccdac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4247)

Re: [PR] [HUDI-9144] Flink writer for MOR table supports writing RowData … [hudi]

2025-03-18 Thread via GitHub
cshuo commented on code in PR #12967: URL: https://github.com/apache/hudi/pull/12967#discussion_r167891 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkWriteHandleFactory.java: ## @@ -283,4 +283,30 @@ public static DeltaCommitWriteHandleFactory getIn

Re: [I] [DISCUSSION] Should we treat `COMMIT_TIME_ORDERING` as a special case of `EVENT_TIME_ORDERING` ? [hudi]

2025-03-18 Thread via GitHub
TheR1sing3un commented on issue #12997: URL: https://github.com/apache/hudi/issues/12997#issuecomment-2735238725 > user want to force the merging to be based on input sequence. If according to what I said above, when we process duplicate record, we update it in turn according to the

Re: [I] [DISCUSSION] Should we treat `COMMIT_TIME_ORDERING` as a special case of `EVENT_TIME_ORDERING` ? [hudi]

2025-03-18 Thread via GitHub
TheR1sing3un commented on issue #12997: URL: https://github.com/apache/hudi/issues/12997#issuecomment-2735234924 in addition, I think we can treat all log records with the same `record_key` as state transition logs similar to state machines (this record's value), and we can handle them sequ

Re: [I] [SUPPORT] Slow commit times with Spark Structured Streaming from Kinesis to MOR Hudi table [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12412: URL: https://github.com/apache/hudi/issues/12412#issuecomment-2735222177 hey @ad1happy2go : whats the latest on this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] [SUPPORT] Queries are very memory intensive due to low read parallelism in HoodieMergeOnReadRDD [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12434: URL: https://github.com/apache/hudi/issues/12434#issuecomment-2735221825 any update on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] [SUPPORT] Hoodie Custom Merge Paylod results in UnsupportedOperationException [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12571: URL: https://github.com/apache/hudi/issues/12571#issuecomment-2735220428 hey folks, whats the status here. did we find the root cause, or not reproducible. we are trying to collect issues to be targetted for 1.0.2. So, trying to gauge the status of

Re: [I] [SUPPORT]Caused by: org.apache.hudi.exception.HoodieIOException: Exception create input stream from file: HoodieLogFile{pathStr='hdfs://nameservice1/xxx/.00000056-15ec-459f-bb67-5f8c2b319203_2

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12554: URL: https://github.com/apache/hudi/issues/12554#issuecomment-2735221004 hey @ad1happy2go @rangareddy : who is following up here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [I] Log files in Hudi MOR table are not getting deleted [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12702: URL: https://github.com/apache/hudi/issues/12702#issuecomment-2735217750 hey @ad1happy2go : whats the status on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] [SUPPORT] docker demo not working: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/parquet/format/TypeDefinedOrder [hudi]

2025-03-18 Thread via GitHub
Souldiv commented on issue #12946: URL: https://github.com/apache/hudi/issues/12946#issuecomment-2735217288 hey @rangareddy I have followed the steps outlined [here](https://hudi.apache.org/docs/docker_demo/) I get that error when I try to run the sync tool for hive. I believe it might be

Re: [I] [SUPPORT] Does Hudi re-create record level index during an upsert operation? [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12783: URL: https://github.com/apache/hudi/issues/12783#issuecomment-2735216669 you should see this only in first batch after you enable RLI. once its fully initialized, subsequent batchs should use RLI instead of global simple. but the instantiation of

Re: [I] [SUPPORT] - Records deleted with via "hard delete" appear after next commit [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12833: URL: https://github.com/apache/hudi/issues/12833#issuecomment-2735213212 hey @RuyRoaV : gentle ping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] [SUPPORT] How to Suppress the HoodieWriterCommitMessage on each Parquet file it Writes HoodieWriterCommitMessage [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12854: URL: https://github.com/apache/hudi/issues/12854#issuecomment-2735212873 hey @rangareddy : can you try it out to suppress the logging for the given class of interest and report back. -- This is an automated message from the Apache Git Service. To

Re: [I] [SUPPORT] docker demo not working: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/parquet/format/TypeDefinedOrder [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12946: URL: https://github.com/apache/hudi/issues/12946#issuecomment-2735210997 hey @rangareddy : did you try docker demo with 0.15.0 branch. can you report it back once you could get it working successfully on your end. -- This is an automated message

Re: [I] [DISCUSSION] Should we treat `COMMIT_TIME_ORDERING` as a special case of `EVENT_TIME_ORDERING` ? [hudi]

2025-03-18 Thread via GitHub
TheR1sing3un commented on issue #12997: URL: https://github.com/apache/hudi/issues/12997#issuecomment-2735198627 @yihua @danny0405 I'd love to hear what you think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[I] [DISCUSSION] Should we treat `COMMIT_TIME_ORDERING` as a special case of `EVENT_TIME_ORDERING` ? [hudi]

2025-03-18 Thread via GitHub
TheR1sing3un opened a new issue, #12997: URL: https://github.com/apache/hudi/issues/12997 From the current code structure, we would treat the merge policy of `COMMIT_TIME_ORDERING` as a separate logic, but from a business perspective, should we treat it as a special case of `EVENT_TIME_ORDE

Re: [I] [SUPPORT] When writing to a Hudi MOR table using Flink, data merging did not occur based on the expected value of "precombine.field". [hudi]

2025-03-18 Thread via GitHub
Toroidals closed issue #12996: [SUPPORT] When writing to a Hudi MOR table using Flink, data merging did not occur based on the expected value of "precombine.field". URL: https://github.com/apache/hudi/issues/12996 -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [HUDI-9197] Fix flaky test for flink: testDynamicPartitionPrune [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12995: URL: https://github.com/apache/hudi/pull/12995#issuecomment-2735191670 ## CI report: * a27ffd4b4687f9fb983d5914f2d060d0ce4f6956 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4254)

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2735188604 ## CI report: * b5e64be8b802c3d2cb048b11c3c83d1296dc2d41 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4257)

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2735187205 ## CI report: * 6330602b196e68b6fe9f2e1612dec8590dce073c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4251)

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2735185144 ## CI report: * 6330602b196e68b6fe9f2e1612dec8590dce073c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4251)

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-03-18 Thread via GitHub
linliu-code commented on code in PR #12991: URL: https://github.com/apache/hudi/pull/12991#discussion_r2002315944 ## hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderSchemaHandler.java: ## @@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Softw

[I] [SUPPORT] When writing to a Hudi MOR table using Flink, data merging did not occur based on the expected value of "precombine.field". [hudi]

2025-03-18 Thread via GitHub
Toroidals opened a new issue, #12996: URL: https://github.com/apache/hudi/issues/12996 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? y - Join the mailing list to engage in conversations and get faster support at dev-su

Re: [PR] [HUDI-7037] Fix colstats reading for Decimal field [hudi]

2025-03-18 Thread via GitHub
codope commented on code in PR #12993: URL: https://github.com/apache/hudi/pull/12993#discussion_r2002297504 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/ColumnStatsIndexSupport.scala: ## @@ -490,13 +491,19 @@ object ColumnStatsIndexSupport {

(hudi) branch master updated: [HUDI-9196] Disable flaky tests on Flink (#12994)

2025-03-18 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 8b7bd1391bd [HUDI-9196] Disable flaky tests on Flin

Re: [PR] [HUDI-7037] Fix colstats reading for Decimal field [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12993: URL: https://github.com/apache/hudi/pull/12993#issuecomment-2735144248 ## CI report: * c81a532851ea54fcb58262145d016323a1e42ac7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4248)

Re: [PR] [HUDI-9196] Disable flaky tests on Flink [hudi]

2025-03-18 Thread via GitHub
yihua merged PR #12994: URL: https://github.com/apache/hudi/pull/12994 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] [HUDI-9197] Fix flaky test for flink: testDynamicPartitionPrune [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12995: URL: https://github.com/apache/hudi/pull/12995#issuecomment-2735100011 ## CI report: * a27ffd4b4687f9fb983d5914f2d060d0ce4f6956 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4254)

Re: [PR] [HUDI-9187] Use thread safe data structures in ExternalSpillableMap [hudi]

2025-03-18 Thread via GitHub
the-other-tim-brown commented on code in PR #12982: URL: https://github.com/apache/hudi/pull/12982#discussion_r2001062318 ## hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java: ## @@ -92,7 +92,7 @@ public class ExternalSpillableMap implem

Re: [PR] [HUDI-9197] Fix flaky test for flink: testDynamicPartitionPrune [hudi]

2025-03-18 Thread via GitHub
cshuo commented on PR #12995: URL: https://github.com/apache/hudi/pull/12995#issuecomment-2735114203 cc @danny0405 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] [HUDI-9083] Fixing flakiness with multi writer test [hudi]

2025-03-18 Thread via GitHub
yihua merged PR #12987: URL: https://github.com/apache/hudi/pull/12987 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] [HUDI-9013] Add backwards compatible MDT writer support and reader support with tbl v6 [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on code in PR #12948: URL: https://github.com/apache/hudi/pull/12948#discussion_r2001850161 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -193,6 +194,10 @@ protected HoodieBackedTableMet

[jira] [Updated] (HUDI-9197) ITTestHoodieDataSource.testDynamicPartitionPrune is flaky

2025-03-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-9197: - Labels: pull-request-available (was: ) > ITTestHoodieDataSource.testDynamicPartitionPrune is flak

Re: [PR] [HUDI-9197] Fix flaky test for flink: testDynamicPartitionPrune [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12995: URL: https://github.com/apache/hudi/pull/12995#issuecomment-2735097969 ## CI report: * a27ffd4b4687f9fb983d5914f2d060d0ce4f6956 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[PR] [HUDI-9197] Fix flaky test for flink: testDynamicPartitionPrune [hudi]

2025-03-18 Thread via GitHub
cshuo opened a new pull request, #12995: URL: https://github.com/apache/hudi/pull/12995 ### Change Logs When `read.streaming.enabled` is enabled, flink incremental reading will continue running indefinitely until explicitly cancelled, and for IT test, we use sleep() to make sure expe

[jira] [Created] (HUDI-9197) ITTestHoodieDataSource.testDynamicPartitionPrune is flaky

2025-03-18 Thread Shuo Cheng (Jira)
Shuo Cheng created HUDI-9197: Summary: ITTestHoodieDataSource.testDynamicPartitionPrune is flaky Key: HUDI-9197 URL: https://issues.apache.org/jira/browse/HUDI-9197 Project: Apache Hudi Issue Typ

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
yihua commented on code in PR #12991: URL: https://github.com/apache/hudi/pull/12991#discussion_r2001831896 ## hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReaderSchemaHandler.java: ## @@ -199,7 +199,8 @@ private static String[] getMandatoryFieldsF

Re: [PR] [HUDI-9196] Disable flaky tests on Flink [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12994: URL: https://github.com/apache/hudi/pull/12994#issuecomment-2735004365 ## CI report: * 5165b88a1f4b1a31785997453238e62744d41993 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-9188] Fixing RLI record generation to account for deletes with lower ordering values in MOR log files [hudi]

2025-03-18 Thread via GitHub
danny0405 commented on code in PR #12984: URL: https://github.com/apache/hudi/pull/12984#discussion_r2002213885 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -901,6 +919,53 @@ public static HoodieData convertMetadataToRecordIndexRecor

Re: [PR] [HUDI-9188] Fixing RLI record generation to account for deletes with lower ordering values in MOR log files [hudi]

2025-03-18 Thread via GitHub
danny0405 commented on code in PR #12984: URL: https://github.com/apache/hudi/pull/12984#discussion_r2002213885 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -901,6 +919,53 @@ public static HoodieData convertMetadataToRecordIndexRecor

Re: [I] [SUPPORT] why HoodieDatasetBulkInsertHelper bulkInsert method no BucketBulkInsertDataInternalWriterHelper [hudi]

2025-03-18 Thread via GitHub
danny0405 commented on issue #12989: URL: https://github.com/apache/hudi/issues/12989#issuecomment-2735037324 Can you at least tell us the class name that the code snippet belongs to, so that we can see if the fix is necessary. BTW, the bucket index bulk_insert is definitely supported, so I

Re: [PR] [HUDI-9188] Fixing RLI record generation to account for deletes with lower ordering values in MOR log files [hudi]

2025-03-18 Thread via GitHub
yihua commented on code in PR #12984: URL: https://github.com/apache/hudi/pull/12984#discussion_r2002208429 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -901,6 +919,53 @@ public static HoodieData convertMetadataToRecordIndexRecords(H

[jira] [Commented] (HUDI-8375) Implement CompletionTimeQueryViewV1.getStartTimes method needed by IncrementalQeuryAnalyzer

2025-03-18 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17936651#comment-17936651 ] Y Ethan Guo commented on HUDI-8375: --- IncrementalQeuryAnalyzer is only used by V2 classes

[jira] [Closed] (HUDI-8375) Implement CompletionTimeQueryViewV1.getStartTimes method needed by IncrementalQeuryAnalyzer

2025-03-18 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo closed HUDI-8375. - Resolution: Invalid > Implement CompletionTimeQueryViewV1.getStartTimes method needed by > IncrementalQe

Re: [I] [SUPPORT] Hudi DELETE operation in Flink scans all partitions despite partition predicate [hudi]

2025-03-18 Thread via GitHub
danny0405 commented on issue #12988: URL: https://github.com/apache/hudi/issues/12988#issuecomment-2735034204 I see the index_bootstrap op is busy in your case, we can eliminate it for the `DELETE` use case. -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-03-18 Thread via GitHub
yihua commented on code in PR #12991: URL: https://github.com/apache/hudi/pull/12991#discussion_r2002199160 ## hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderSchemaHandler.java: ## @@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] [HUDI-9083] Fixing flakiness with multi writer test [hudi]

2025-03-18 Thread via GitHub
yihua commented on PR #12987: URL: https://github.com/apache/hudi/pull/12987#issuecomment-2735009507 Flink CI failure is irrelevant. Merging this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [HUDI-9196] Disable flaky tests on Flink [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12994: URL: https://github.com/apache/hudi/pull/12994#issuecomment-2735005948 ## CI report: * 5165b88a1f4b1a31785997453238e62744d41993 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4252)

[jira] [Updated] (HUDI-9196) Investigate Flink flaky tests

2025-03-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-9196: - Labels: pull-request-available (was: ) > Investigate Flink flaky tests >

[PR] [HUDI-9196] Disable flaky tests on Flink [hudi]

2025-03-18 Thread via GitHub
yihua opened a new pull request, #12994: URL: https://github.com/apache/hudi/pull/12994 ### Change Logs This PR disables flaky tests on Flink to unblock CI. HUDI-9196 for further investigation. ### Impact Unblocks CI. ### Risk level none ### Documen

[jira] [Updated] (HUDI-9196) Investigate Flink flaky tests

2025-03-18 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo updated HUDI-9196: -- Description: Two Flink tests are flaky: [https://github.com/apache/hudi/actions/runs/13914701358/job/38

[jira] [Updated] (HUDI-9196) Investigate Flink flaky tests

2025-03-18 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo updated HUDI-9196: -- Description: Two Flink tests are flaky: [https://github.com/apache/hudi/actions/runs/13914701358/job/38

[jira] [Created] (HUDI-9196) Investigate Flink flaky tests

2025-03-18 Thread Y Ethan Guo (Jira)
Y Ethan Guo created HUDI-9196: - Summary: Investigate Flink flaky tests Key: HUDI-9196 URL: https://issues.apache.org/jira/browse/HUDI-9196 Project: Apache Hudi Issue Type: Bug Reporte

[jira] [Updated] (HUDI-9196) Investigate Flink flaky tests

2025-03-18 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo updated HUDI-9196: -- Fix Version/s: 1.0.2 > Investigate Flink flaky tests > - > >

[jira] [Updated] (HUDI-9196) Investigate Flink flaky tests

2025-03-18 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo updated HUDI-9196: -- Priority: Blocker (was: Major) > Investigate Flink flaky tests > - > >

Re: [PR] [HUDI-9083] Fixing flakiness with multi writer test [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12987: URL: https://github.com/apache/hudi/pull/12987#issuecomment-2734977491 ## CI report: * 8f98d0ff87fd8d21365696b22af77caac421cdd5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4250)

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2734932914 ## CI report: * 6330602b196e68b6fe9f2e1612dec8590dce073c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4251)

Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]

2025-03-18 Thread via GitHub
linliu-code commented on code in PR #12843: URL: https://github.com/apache/hudi/pull/12843#discussion_r2002139125 ## hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java: ## @@ -143,6 +148,29 @@ public HoodieBaseFileGroupRecordBuffer(HoodieRead

Re: [PR] [HUDI-9022] Handle records with custom delete markers for MOR [hudi]

2025-03-18 Thread via GitHub
linliu-code commented on code in PR #12843: URL: https://github.com/apache/hudi/pull/12843#discussion_r2002132006 ## hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java: ## @@ -143,6 +148,29 @@ public HoodieBaseFileGroupRecordBuffer(HoodieRead

[jira] [Closed] (HUDI-8655) Create Tests for Filegroup reader for Schema Cache and for Spillable Map

2025-03-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-8655. - Resolution: Fixed > Create Tests for Filegroup reader for Schema Cache and for Spillable M

Re: [PR] [HUDI-9083] Fixing flakiness with multi writer test [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12987: URL: https://github.com/apache/hudi/pull/12987#issuecomment-2734877234 ## CI report: * 8f98d0ff87fd8d21365696b22af77caac421cdd5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4250)

Re: [PR] [HUDI-9083] Fixing flakiness with multi writer test [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12987: URL: https://github.com/apache/hudi/pull/12987#issuecomment-2734875401 ## CI report: * 8f98d0ff87fd8d21365696b22af77caac421cdd5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[jira] [Updated] (HUDI-9119) Hudi 1.0.1 cannot write MOR tables

2025-03-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-9119: -- Parent: HUDI-8724 Issue Type: Sub-task (was: Bug) > Hudi 1.0.1 cannot write MOR

[jira] [Closed] (HUDI-9127) Fix completion time generation to honor the time zone set in table config

2025-03-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-9127. - Resolution: Fixed [https://github.com/apache/hudi/commit/9baaed9409a1a5b654e88d50ac6826a96

[jira] [Closed] (HUDI-9086) Master is broken Feb 27, 2025

2025-03-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-9086. - Resolution: Fixed > Master is broken Feb 27, 2025 > - > >

[jira] [Updated] (HUDI-8655) Create Tests for Filegroup reader for Schema Cache and for Spillable Map

2025-03-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-8655: -- Fix Version/s: 1.0.1 (was: 1.0.2) > Create Tests for Filegroup re

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2734822641 ## CI report: * 5eb43137bc60a32dee13000bff27b3b08e3694d3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4249)

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2734820536 ## CI report: * 5eb43137bc60a32dee13000bff27b3b08e3694d3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4249)

[PR] [HUDI-9140] Fix log block io type and other rollback strategy fixes for table version 6 [hudi]

2025-03-18 Thread via GitHub
lokeshj1703 opened a new pull request, #12992: URL: https://github.com/apache/hudi/pull/12992 ### Change Logs The PR fixes the iotype as APPEND for log blocks in table version 6. It also reverts some changes made to MarkerBasedRollbackStrategy for table version 6 in HUDI-9030.

Re: [PR] [HUDI-8178] Fix CI failures for partition-stats enablement [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on PR #12081: URL: https://github.com/apache/hudi/pull/12081#issuecomment-2734569017 We fixed both Date and LocalDate with col stats and partition stats. https://github.com/apache/hudi/blob/1f43b231763a978bef8d340a654e9f6287241ec9/hudi-common/src/main/java/org/ap

  1   2   >