Re: [PR] [HUDI-9120] Enable File Group reader by default for table version 6 [hudi]

2025-03-18 Thread via GitHub
lokeshj1703 commented on PR #12935: URL: https://github.com/apache/hudi/pull/12935#issuecomment-2733818315 @linliu-code @yihua @nsivabalan The PR now cherry-picks Lin's fix and removes all the older fixes which were added earlier. It also reverts the changes made in HUDI-9030 for removing F

Re: [PR] [HUDI-9120] Enable File Group reader by default for table version 6 [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12935: URL: https://github.com/apache/hudi/pull/12935#issuecomment-2733830603 ## CI report: * d5f94e7afb5865449c7796b03cc4b9c786061ec2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4152)

Re: [PR] [HUDI-9120] Enable File Group reader by default for table version 6 [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12935: URL: https://github.com/apache/hudi/pull/12935#issuecomment-2733835632 ## CI report: * 5154a0d5f9b8adecd1c675f05405e732b2f1e9fe Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4246)

Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on code in PR #12340: URL: https://github.com/apache/hudi/pull/12340#discussion_r2001378215 ## hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala: ## @@ -165,7 +165,7 @@ class SparkFileFormatInternalRo

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2732226321 ## CI report: * d85952ac252b3a9cc7677188dac340ece0efcc1d Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4242)

Re: [PR] [HUDI-9013] Add backwards compatible MDT writer support and reader support with tbl v6 [hudi]

2025-03-18 Thread via GitHub
lokeshj1703 commented on PR #12948: URL: https://github.com/apache/hudi/pull/12948#issuecomment-2732856286 Azure CI passed: https://github.com/user-attachments/assets/05ba161a-8999-42a8-b125-f9e2a5a9cef6"; /> -- This is an automated message from the Apache Git Service. To respond

[jira] [Closed] (HUDI-8969) Analyze how to write `RowData` directly

2025-03-18 Thread Geser Dugarov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geser Dugarov closed HUDI-8969. --- Resolution: Fixed This task will be done under RFC-87. > Analyze how to write `RowData` directly > --

Re: [PR] [HUDI-9120] Enable File Group reader by default for table version 6 [hudi]

2025-03-18 Thread via GitHub
lokeshj1703 commented on code in PR #12935: URL: https://github.com/apache/hudi/pull/12935#discussion_r2001484794 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/SparkBroadcastManager.java: ## @@ -71,6 +73,7 @@ public class SparkBroadcastManager extends Eng

Re: [PR] [HUDI-9144] Flink writer for MOR table supports writing RowData … [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12967: URL: https://github.com/apache/hudi/pull/12967#issuecomment-2733608704 ## CI report: * 21ba47d8c7b86b1c92a311e216c4b85dc17ed046 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4245)

Re: [PR] [HUDI-9120] Enable File Group reader by default for table version 6 [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12935: URL: https://github.com/apache/hudi/pull/12935#issuecomment-2733817403 ## CI report: * d5f94e7afb5865449c7796b03cc4b9c786061ec2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4152)

Re: [PR] [HUDI-9120] Enable File Group reader by default for table version 6 [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12935: URL: https://github.com/apache/hudi/pull/12935#issuecomment-2733813222 ## CI report: * d5f94e7afb5865449c7796b03cc4b9c786061ec2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4152)

[jira] [Updated] (HUDI-9140) Follow up from 9030

2025-03-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-9140: - Labels: pull-request-available (was: ) > Follow up from 9030 > --- > >

Re: [PR] [HUDI-9140] Fix log block io type and other rollback strategy fixes for table version 6 [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12992: URL: https://github.com/apache/hudi/pull/12992#issuecomment-2733101341 ## CI report: * 99916b679c4915e811845ae6261e9b3263c0feea UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-9140] Fix log block io type and other rollback strategy fixes for table version 6 [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12992: URL: https://github.com/apache/hudi/pull/12992#issuecomment-2733104919 ## CI report: * 99916b679c4915e811845ae6261e9b3263c0feea Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4244)

[I] [SUPPORT] Hudi DELETE operation in Flink scans all partitions despite partition predicate [hudi]

2025-03-18 Thread via GitHub
maheshguptags opened a new issue, #12988: URL: https://github.com/apache/hudi/issues/12988 I am experiencing an issue when trying to delete records from a Hudi table where data is ingested using Flink streaming, and deletion is attempted using a Hudi batch processing job. Despite specif

Re: [PR] [HUDI-9120] Enable File Group reader by default for table version 6 [hudi]

2025-03-18 Thread via GitHub
lokeshj1703 commented on code in PR #12935: URL: https://github.com/apache/hudi/pull/12935#discussion_r2001501987 ## hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java: ## @@ -112,6 +112,7 @@ public HoodieFileGroupReader(HoodieReaderContext r

[jira] [Updated] (HUDI-7037) Column Stats for Decimal Field From Metadata table is read as Bytes

2025-03-18 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7037: -- Priority: Blocker (was: Critical) > Column Stats for Decimal Field From Metadata table is read as Bytes

[jira] [Updated] (HUDI-7037) Column Stats for Decimal Field From Metadata table is read as Bytes

2025-03-18 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7037: -- Status: In Progress (was: Open) > Column Stats for Decimal Field From Metadata table is read as Bytes >

Re: [PR] [HUDI-7037] Fix colstats reading for Decimal field [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12993: URL: https://github.com/apache/hudi/pull/12993#issuecomment-2734076458 ## CI report: * c81a532851ea54fcb58262145d016323a1e42ac7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4248)

Re: [PR] [HUDI-9144] Flink writer for MOR table supports writing RowData … [hudi]

2025-03-18 Thread via GitHub
cshuo commented on code in PR #12967: URL: https://github.com/apache/hudi/pull/12967#discussion_r2000890050 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/v2/HandleRecords.java: ## @@ -0,0 +1,79 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] [HUDI-7930] Flink Support for Array of Row and Map of Row value [hudi]

2025-03-18 Thread via GitHub
David-N-Perkins commented on PR #11727: URL: https://github.com/apache/hudi/pull/11727#issuecomment-2732958024 @empcl If I remember correctly, it was needed to get consistent names and structure in the Parquet files. I was seeing differences depending on whether the operation was "insert",

Re: [PR] [HUDI-9120] Enable File Group reader by default for table version 6 [hudi]

2025-03-18 Thread via GitHub
lokeshj1703 commented on code in PR #12935: URL: https://github.com/apache/hudi/pull/12935#discussion_r2001485609 ## hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala: ## @@ -59,17 +60,16 @@ import scala.collection.mutable

[jira] [Commented] (HUDI-9043) Analyze possibility to optimize `FlinkWriteHelper::deduplicateRecords`

2025-03-18 Thread Geser Dugarov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17936425#comment-17936425 ] Geser Dugarov commented on HUDI-9043: - {color:#00}`RowDataStreamWriteFunction{colo

[jira] [Updated] (HUDI-9043) Analyze possibility to optimize `FlinkWriteHelper::deduplicateRecords`

2025-03-18 Thread Geser Dugarov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geser Dugarov updated HUDI-9043: Status: Open (was: In Progress) > Analyze possibility to optimize `FlinkWriteHelper::deduplicateRec

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2732680960 ## CI report: * 87512b51170f102d612c11b44bea7534a684c51d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4243)

[jira] [Updated] (HUDI-8796) Silent ignoring of bucket index in Flink append mode

2025-03-18 Thread Geser Dugarov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geser Dugarov updated HUDI-8796: Status: Open (was: In Progress) > Silent ignoring of bucket index in Flink append mode > --

Re: [PR] [HUDI-9144] Flink writer for MOR table supports writing RowData … [hudi]

2025-03-18 Thread via GitHub
Alowator commented on code in PR #12967: URL: https://github.com/apache/hudi/pull/12967#discussion_r2000181537 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/RowDataStreamWriteFunction.java: ## @@ -0,0 +1,563 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] [HUDI-8796] Restrict insert operation with bucket index for Flink [hudi]

2025-03-18 Thread via GitHub
geserdugarov commented on PR #12545: URL: https://github.com/apache/hudi/pull/12545#issuecomment-2732158325 I will revisit this issue after major changes in Flink write into Hudi by RFC-87. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[jira] [Updated] (HUDI-8969) Analyze how to write `RowData` directly

2025-03-18 Thread Geser Dugarov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geser Dugarov updated HUDI-8969: Status: Open (was: In Progress) > Analyze how to write `RowData` directly > ---

Re: [PR] [HUDI-9144] Flink writer for MOR table supports writing RowData … [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12967: URL: https://github.com/apache/hudi/pull/12967#issuecomment-2733240438 ## CI report: * b591ad3b0092eec900475590089dd05f58570d5d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4222)

Re: [PR] [HUDI-9140] Fix log block io type and other rollback strategy fixes for table version 6 [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12992: URL: https://github.com/apache/hudi/pull/12992#issuecomment-2733436576 ## CI report: * 99916b679c4915e811845ae6261e9b3263c0feea Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4244)

Re: [PR] [HUDI-9120] Fix delete ordering comparison issue [hudi]

2025-03-18 Thread via GitHub
linliu-code commented on PR #12979: URL: https://github.com/apache/hudi/pull/12979#issuecomment-2733434279 This is not needed since we have found a better fix: https://github.com/apache/hudi/pull/12991 -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [HUDI-9120] Fix delete ordering comparison issue [hudi]

2025-03-18 Thread via GitHub
linliu-code closed pull request #12979: [HUDI-9120] Fix delete ordering comparison issue URL: https://github.com/apache/hudi/pull/12979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2732038143 ## CI report: * d85952ac252b3a9cc7677188dac340ece0efcc1d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-9013] Add backwards compatible MDT writer support and reader support with tbl v6 [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12948: URL: https://github.com/apache/hudi/pull/12948#issuecomment-2731725478 ## CI report: * 047885b4286dae609122e6573117cfd5dcdca572 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4221)

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2732053104 ## CI report: * d85952ac252b3a9cc7677188dac340ece0efcc1d Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4242)

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2732041000 ## CI report: * d85952ac252b3a9cc7677188dac340ece0efcc1d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4242)

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2732049001 ## CI report: * d85952ac252b3a9cc7677188dac340ece0efcc1d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4242)

Re: [I] [SUPPORT] why HoodieDatasetBulkInsertHelper bulkInsert method no BucketBulkInsertDataInternalWriterHelper [hudi]

2025-03-18 Thread via GitHub
danny0405 commented on issue #12989: URL: https://github.com/apache/hudi/issues/12989#issuecomment-2731982548 Do you want to do some code refactoring or encounter some issues for your use case? -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] [SUPPORT] why HoodieDatasetBulkInsertHelper bulkInsert method no BucketBulkInsertDataInternalWriterHelper [hudi]

2025-03-18 Thread via GitHub
leeseven1211 commented on issue #12989: URL: https://github.com/apache/hudi/issues/12989#issuecomment-2731993287 While using bulk insert to batch write data into Hudi, I noticed that the written files were not bucketed according to the bucket index. After adding this piece of code,

Re: [PR] [HUDI-9083] Fixing flakiness with multi writer test [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12987: URL: https://github.com/apache/hudi/pull/12987#issuecomment-2731998178 ## CI report: * 1969b9f2ad75790d1058e6b66ae0995793c3082d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4240)

Re: [I] [SUPPORT] Hudi DELETE operation in Flink scans all partitions despite partition predicate [hudi]

2025-03-18 Thread via GitHub
danny0405 commented on issue #12988: URL: https://github.com/apache/hudi/issues/12988#issuecomment-2732019149 Are there any other failures in the JM log? Can you also show me the Flink UI operator DAG? -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12990: URL: https://github.com/apache/hudi/pull/12990#issuecomment-2732029371 ## CI report: * daa0efee55176b3f6441a960a796322b6adec941 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
linliu-code opened a new pull request, #12991: URL: https://github.com/apache/hudi/pull/12991 ### Change Logs Previously when the configuration "hoodie.record.merge.mode" is null (by default), we will not add precombine field to the required schema since we assume it is commit time o

Re: [I] [SUPPORT] why HoodieDatasetBulkInsertHelper bulkInsert method no BucketBulkInsertDataInternalWriterHelper [hudi]

2025-03-18 Thread via GitHub
danny0405 commented on issue #12989: URL: https://github.com/apache/hudi/issues/12989#issuecomment-2732033960 Can you share you configurations and the code link that you want to put a patch to? -- This is an automated message from the Apache Git Service. To respond to the message, please

[jira] [Created] (HUDI-9195) rowdata write handle builds data block using record iterator if there is no delete record

2025-03-18 Thread Shuo Cheng (Jira)
Shuo Cheng created HUDI-9195: Summary: rowdata write handle builds data block using record iterator if there is no delete record Key: HUDI-9195 URL: https://issues.apache.org/jira/browse/HUDI-9195 Project

[I] [SUPPORT] why HoodieDatasetBulkInsertHelper bulkInsert method no BucketBulkInsertDataInternalWriterHelper [hudi]

2025-03-18 Thread via GitHub
leeseven1211 opened a new issue, #12989: URL: https://github.com/apache/hudi/issues/12989 use bulk insert , only ConsistentBucketBulkInsertDataInternalWriterHelper and BulkInsertDataInternalWriterHelper ,why is not support BucketBulkInsertDataInternalWriterHelper Below is the code s

Re: [PR] [HUDI-9170] Fixing schema projection with file group reader [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12970: URL: https://github.com/apache/hudi/pull/12970#issuecomment-2731183347 ## CI report: * 09b4ba83b5d61cd777c577e483bfe21098725ecc UNKNOWN * b31778b0dd6ccaf619321c6f9b397f7a388c8717 UNKNOWN * d8264536a187f6e213ed1eb08d941c0fc86a1e55 UNKNOWN *

Re: [I] org.apache.hudi.exception.HoodieException: org.apache.avro.AvroTypeException: Cannot encode decimal with precision 14 as max precision 13 [hudi]

2025-03-18 Thread via GitHub
imonteroq commented on issue #11335: URL: https://github.com/apache/hudi/issues/11335#issuecomment-2734199198 I am also getting the same issue. Running Hudi 0.15 on EMR serverless using Spark/Scala. I have saved the incoming data to a new table and it has absolutely NO decimal fields with p

Re: [PR] [HUDI-9013] Add backwards compatible MDT writer support and reader support with tbl v6 [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on PR #12948: URL: https://github.com/apache/hudi/pull/12948#issuecomment-2734247170 CI is failing due to known flaky test ITTestHoodieDataSource. testIncrementalReadArchivedCommits -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] [HUDI-9083] Fixing flakiness with multi writer test [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12987: URL: https://github.com/apache/hudi/pull/12987#issuecomment-2734539112 ## CI report: * 1969b9f2ad75790d1058e6b66ae0995793c3082d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4240)

[jira] [Updated] (HUDI-7037) Column Stats for Decimal Field From Metadata table is read as Bytes

2025-03-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7037: - Labels: pull-request-available (was: ) > Column Stats for Decimal Field From Metadata table is re

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2734200114 ## CI report: * 87512b51170f102d612c11b44bea7534a684c51d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4243)

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
yihua commented on code in PR #12991: URL: https://github.com/apache/hudi/pull/12991#discussion_r2001740210 ## hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReaderSchemaHandler.java: ## @@ -199,7 +199,8 @@ private static String[] getMandatoryFieldsF

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
yihua commented on code in PR #12991: URL: https://github.com/apache/hudi/pull/12991#discussion_r2001740210 ## hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReaderSchemaHandler.java: ## @@ -199,7 +199,8 @@ private static String[] getMandatoryFieldsF

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2734455487 ## CI report: * 5eb43137bc60a32dee13000bff27b3b08e3694d3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4249)

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2735187205 ## CI report: * 6330602b196e68b6fe9f2e1612dec8590dce073c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4251)

[I] [DISCUSSION] Should we treat `COMMIT_TIME_ORDERING` as a special case of `EVENT_TIME_ORDERING` ? [hudi]

2025-03-18 Thread via GitHub
TheR1sing3un opened a new issue, #12997: URL: https://github.com/apache/hudi/issues/12997 From the current code structure, we would treat the merge policy of `COMMIT_TIME_ORDERING` as a separate logic, but from a business perspective, should we treat it as a special case of `EVENT_TIME_ORDE

Re: [I] [DISCUSSION] Should we treat `COMMIT_TIME_ORDERING` as a special case of `EVENT_TIME_ORDERING` ? [hudi]

2025-03-18 Thread via GitHub
TheR1sing3un commented on issue #12997: URL: https://github.com/apache/hudi/issues/12997#issuecomment-2735198627 @yihua @danny0405 I'd love to hear what you think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2735188604 ## CI report: * b5e64be8b802c3d2cb048b11c3c83d1296dc2d41 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4257)

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2735185144 ## CI report: * 6330602b196e68b6fe9f2e1612dec8590dce073c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4251)

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-03-18 Thread via GitHub
linliu-code commented on code in PR #12991: URL: https://github.com/apache/hudi/pull/12991#discussion_r2002315944 ## hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderSchemaHandler.java: ## @@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Softw

Re: [I] [SUPPORT] When writing to a Hudi MOR table using Flink, data merging did not occur based on the expected value of "precombine.field". [hudi]

2025-03-18 Thread via GitHub
Toroidals closed issue #12996: [SUPPORT] When writing to a Hudi MOR table using Flink, data merging did not occur based on the expected value of "precombine.field". URL: https://github.com/apache/hudi/issues/12996 -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [HUDI-9197] Fix flaky test for flink: testDynamicPartitionPrune [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12995: URL: https://github.com/apache/hudi/pull/12995#issuecomment-2735191670 ## CI report: * a27ffd4b4687f9fb983d5914f2d060d0ce4f6956 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4254)

Re: [I] [SUPPORT] docker demo not working: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/parquet/format/TypeDefinedOrder [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12946: URL: https://github.com/apache/hudi/issues/12946#issuecomment-2735210997 hey @rangareddy : did you try docker demo with 0.15.0 branch. can you report it back once you could get it working successfully on your end. -- This is an automated message

Re: [I] [SUPPORT] How to Suppress the HoodieWriterCommitMessage on each Parquet file it Writes HoodieWriterCommitMessage [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12854: URL: https://github.com/apache/hudi/issues/12854#issuecomment-2735212873 hey @rangareddy : can you try it out to suppress the logging for the given class of interest and report back. -- This is an automated message from the Apache Git Service. To

Re: [I] [SUPPORT]Caused by: org.apache.hudi.exception.HoodieIOException: Exception create input stream from file: HoodieLogFile{pathStr='hdfs://nameservice1/xxx/.00000056-15ec-459f-bb67-5f8c2b319203_2

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12554: URL: https://github.com/apache/hudi/issues/12554#issuecomment-2735221004 hey @ad1happy2go @rangareddy : who is following up here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [I] [SUPPORT] - Records deleted with via "hard delete" appear after next commit [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12833: URL: https://github.com/apache/hudi/issues/12833#issuecomment-2735213212 hey @RuyRoaV : gentle ping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] [SUPPORT] Does Hudi re-create record level index during an upsert operation? [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12783: URL: https://github.com/apache/hudi/issues/12783#issuecomment-2735216669 you should see this only in first batch after you enable RLI. once its fully initialized, subsequent batchs should use RLI instead of global simple. but the instantiation of

Re: [I] [SUPPORT] docker demo not working: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/parquet/format/TypeDefinedOrder [hudi]

2025-03-18 Thread via GitHub
Souldiv commented on issue #12946: URL: https://github.com/apache/hudi/issues/12946#issuecomment-2735217288 hey @rangareddy I have followed the steps outlined [here](https://hudi.apache.org/docs/docker_demo/) I get that error when I try to run the sync tool for hive. I believe it might be

Re: [I] Log files in Hudi MOR table are not getting deleted [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12702: URL: https://github.com/apache/hudi/issues/12702#issuecomment-2735217750 hey @ad1happy2go : whats the status on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] [SUPPORT] Hoodie Custom Merge Paylod results in UnsupportedOperationException [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12571: URL: https://github.com/apache/hudi/issues/12571#issuecomment-2735220428 hey folks, whats the status here. did we find the root cause, or not reproducible. we are trying to collect issues to be targetted for 1.0.2. So, trying to gauge the status of

Re: [I] [SUPPORT] Queries are very memory intensive due to low read parallelism in HoodieMergeOnReadRDD [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12434: URL: https://github.com/apache/hudi/issues/12434#issuecomment-2735221825 any update on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] [SUPPORT] Slow commit times with Spark Structured Streaming from Kinesis to MOR Hudi table [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on issue #12412: URL: https://github.com/apache/hudi/issues/12412#issuecomment-2735222177 hey @ad1happy2go : whats the latest on this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [HUDI-9120] Enable File Group reader by default for table version 6 [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12935: URL: https://github.com/apache/hudi/pull/12935#issuecomment-2734044775 ## CI report: * 5154a0d5f9b8adecd1c675f05405e732b2f1e9fe Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4246)

[jira] [Updated] (HUDI-7037) Column Stats for Decimal Field From Metadata table is read as Bytes

2025-03-18 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7037: -- Status: Patch Available (was: In Progress) > Column Stats for Decimal Field From Metadata table is read

Re: [PR] [HUDI-7037] Fix colstats reading for Decimal field [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12993: URL: https://github.com/apache/hudi/pull/12993#issuecomment-2734070157 ## CI report: * c81a532851ea54fcb58262145d016323a1e42ac7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[PR] [HUDI-7037] Fix colstats reading for Decimal field [hudi]

2025-03-18 Thread via GitHub
codope opened a new pull request, #12993: URL: https://github.com/apache/hudi/pull/12993 ### Change Logs When reading column statistics for Decimal fields from the metadata table, the unwrapped decimal values are incorrectly handled. Specifically: - **Type Loss in Unwrapping:**

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2734820536 ## CI report: * 5eb43137bc60a32dee13000bff27b3b08e3694d3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4249)

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2734822641 ## CI report: * 5eb43137bc60a32dee13000bff27b3b08e3694d3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4249)

[jira] [Updated] (HUDI-8655) Create Tests for Filegroup reader for Schema Cache and for Spillable Map

2025-03-18 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-8655: -- Fix Version/s: 1.0.1 (was: 1.0.2) > Create Tests for Filegroup re

[PR] [HUDI-9197] Fix flaky test for flink: testDynamicPartitionPrune [hudi]

2025-03-18 Thread via GitHub
cshuo opened a new pull request, #12995: URL: https://github.com/apache/hudi/pull/12995 ### Change Logs When `read.streaming.enabled` is enabled, flink incremental reading will continue running indefinitely until explicitly cancelled, and for IT test, we use sleep() to make sure expe

Re: [PR] [HUDI-9197] Fix flaky test for flink: testDynamicPartitionPrune [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12995: URL: https://github.com/apache/hudi/pull/12995#issuecomment-2735097969 ## CI report: * a27ffd4b4687f9fb983d5914f2d060d0ce4f6956 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-9083] Fixing flakiness with multi writer test [hudi]

2025-03-18 Thread via GitHub
yihua merged PR #12987: URL: https://github.com/apache/hudi/pull/12987 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] [HUDI-9013] Add backwards compatible MDT writer support and reader support with tbl v6 [hudi]

2025-03-18 Thread via GitHub
nsivabalan commented on code in PR #12948: URL: https://github.com/apache/hudi/pull/12948#discussion_r2001850161 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -193,6 +194,10 @@ protected HoodieBackedTableMet

[jira] [Updated] (HUDI-9197) ITTestHoodieDataSource.testDynamicPartitionPrune is flaky

2025-03-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-9197: - Labels: pull-request-available (was: ) > ITTestHoodieDataSource.testDynamicPartitionPrune is flak

Re: [PR] [HUDI-9197] Fix flaky test for flink: testDynamicPartitionPrune [hudi]

2025-03-18 Thread via GitHub
cshuo commented on PR #12995: URL: https://github.com/apache/hudi/pull/12995#issuecomment-2735114203 cc @danny0405 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] [HUDI-9187] Use thread safe data structures in ExternalSpillableMap [hudi]

2025-03-18 Thread via GitHub
the-other-tim-brown commented on code in PR #12982: URL: https://github.com/apache/hudi/pull/12982#discussion_r2001062318 ## hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java: ## @@ -92,7 +92,7 @@ public class ExternalSpillableMap implem

Re: [PR] [HUDI-9197] Fix flaky test for flink: testDynamicPartitionPrune [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12995: URL: https://github.com/apache/hudi/pull/12995#issuecomment-2735100011 ## CI report: * a27ffd4b4687f9fb983d5914f2d060d0ce4f6956 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4254)

Re: [PR] [HUDI-9196] Disable flaky tests on Flink [hudi]

2025-03-18 Thread via GitHub
yihua merged PR #12994: URL: https://github.com/apache/hudi/pull/12994 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] [HUDI-9120] Add precombine field if possible [hudi]

2025-03-18 Thread via GitHub
hudi-bot commented on PR #12991: URL: https://github.com/apache/hudi/pull/12991#issuecomment-2734206025 ## CI report: * 87512b51170f102d612c11b44bea7534a684c51d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4243)

Re: [PR] [HUDI-8178] Fix CI failures for partition-stats enablement [hudi]

2025-03-18 Thread via GitHub
nsivabalan closed pull request #12081: [HUDI-8178] Fix CI failures for partition-stats enablement URL: https://github.com/apache/hudi/pull/12081 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] [HUDI-9083] Fixing flakiness with multi writer test [hudi]

2025-03-18 Thread via GitHub
yihua commented on PR #12987: URL: https://github.com/apache/hudi/pull/12987#issuecomment-2735009507 Flink CI failure is irrelevant. Merging this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [HUDI-9120] Fix merge mode inference for table version 6 in file group reader [hudi]

2025-03-18 Thread via GitHub
yihua commented on code in PR #12991: URL: https://github.com/apache/hudi/pull/12991#discussion_r2002199160 ## hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderSchemaHandler.java: ## @@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Software Fo

Re: [I] [SUPPORT] Hudi DELETE operation in Flink scans all partitions despite partition predicate [hudi]

2025-03-18 Thread via GitHub
danny0405 commented on issue #12988: URL: https://github.com/apache/hudi/issues/12988#issuecomment-2735034204 I see the index_bootstrap op is busy in your case, we can eliminate it for the `DELETE` use case. -- This is an automated message from the Apache Git Service. To respond to the me

[jira] [Closed] (HUDI-8375) Implement CompletionTimeQueryViewV1.getStartTimes method needed by IncrementalQeuryAnalyzer

2025-03-18 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo closed HUDI-8375. - Resolution: Invalid > Implement CompletionTimeQueryViewV1.getStartTimes method needed by > IncrementalQe

[jira] [Commented] (HUDI-8375) Implement CompletionTimeQueryViewV1.getStartTimes method needed by IncrementalQeuryAnalyzer

2025-03-18 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17936651#comment-17936651 ] Y Ethan Guo commented on HUDI-8375: --- IncrementalQeuryAnalyzer is only used by V2 classes

Re: [PR] [HUDI-9188] Fixing RLI record generation to account for deletes with lower ordering values in MOR log files [hudi]

2025-03-18 Thread via GitHub
yihua commented on code in PR #12984: URL: https://github.com/apache/hudi/pull/12984#discussion_r2002208429 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -901,6 +919,53 @@ public static HoodieData convertMetadataToRecordIndexRecords(H

Re: [I] [SUPPORT] why HoodieDatasetBulkInsertHelper bulkInsert method no BucketBulkInsertDataInternalWriterHelper [hudi]

2025-03-18 Thread via GitHub
danny0405 commented on issue #12989: URL: https://github.com/apache/hudi/issues/12989#issuecomment-2735037324 Can you at least tell us the class name that the code snippet belongs to, so that we can see if the fix is necessary. BTW, the bucket index bulk_insert is definitely supported, so I

Re: [PR] [HUDI-9188] Fixing RLI record generation to account for deletes with lower ordering values in MOR log files [hudi]

2025-03-18 Thread via GitHub
danny0405 commented on code in PR #12984: URL: https://github.com/apache/hudi/pull/12984#discussion_r2002213885 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -901,6 +919,53 @@ public static HoodieData convertMetadataToRecordIndexRecor

  1   2   >