Re: [PR] [HUDI-8766] Enabling cols stats by default with writer [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12596: URL: https://github.com/apache/hudi/pull/12596#issuecomment-2581981304 ## CI report: * ae2ca606c6cd125f31b7ed029968d0993b1bb0bd UNKNOWN * 71b6a13890909b81c74ce7b138237ab695a08782 UNKNOWN * 2710a96832046a764b7125c1152c788d96c6e1f9 Azure: [SUCC

[jira] [Closed] (HUDI-8775) Expression index on a column should get tracked at partition level if partition stats index is turned on

2025-01-09 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-8775. - Resolution: Fixed > Expression index on a column should get tracked at partition level if > partition sta

(hudi) branch master updated (5f591eec223 -> dc001ea4828)

2025-01-09 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 5f591eec223 [HUDI-8762] Fix a typo in Fix a typo in TestIncrementalQueryWithArchivedInstants (#12611) add dc001ea4

Re: [PR] [HUDI-8775] Expression index on a column should get tracked at partition level if partition stats index is turned on [hudi]

2025-01-09 Thread via GitHub
codope merged PR #12558: URL: https://github.com/apache/hudi/pull/12558 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.

Re: [PR] [HUDI-8766] Enabling cols stats by default with writer [hudi]

2025-01-09 Thread via GitHub
codope commented on code in PR #12596: URL: https://github.com/apache/hudi/pull/12596#discussion_r1909901588 ## azure-pipelines-20230430.yml: ## @@ -214,7 +214,7 @@ stages: displayName: Top 100 long-running testcases - job: UT_FT_3 displayName: UT sp

Re: [PR] [HUDI-8602] Fix a bug for incremental query [hudi]

2025-01-09 Thread via GitHub
linliu-code commented on code in PR #12385: URL: https://github.com/apache/hudi/pull/12385#discussion_r1909901739 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala: ## @@ -209,7 +209,14 @@ trait HoodieIncrementalRelati

Re: [PR] [HUDI-8766] Enabling cols stats by default with writer [hudi]

2025-01-09 Thread via GitHub
codope commented on code in PR #12596: URL: https://github.com/apache/hudi/pull/12596#discussion_r1909901588 ## azure-pipelines-20230430.yml: ## @@ -214,7 +214,7 @@ stages: displayName: Top 100 long-running testcases - job: UT_FT_3 displayName: UT sp

Re: [PR] [HUDI-8775] Expression index on a column should get tracked at partition level if partition stats index is turned on [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12558: URL: https://github.com/apache/hudi/pull/12558#issuecomment-2581900400 ## CI report: * e1e0ea9214a05cd585989e228639f213cc8f033f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2792)

Re: [PR] [HUDI-8775] Expression index on a column should get tracked at partition level if partition stats index is turned on [hudi]

2025-01-09 Thread via GitHub
codope commented on code in PR #12558: URL: https://github.com/apache/hudi/pull/12558#discussion_r1909890911 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -395,7 +392,9 @@ public static Map> convertMetadataToRecords(Hoo if (enabl

Re: [PR] [HUDI-8766] Enabling cols stats by default with writer [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12596: URL: https://github.com/apache/hudi/pull/12596#issuecomment-2581888527 ## CI report: * ae2ca606c6cd125f31b7ed029968d0993b1bb0bd UNKNOWN * 71b6a13890909b81c74ce7b138237ab695a08782 UNKNOWN * 15866ae0099c3b58d22329be0e5008b3149cb95f Azure: [FAIL

Re: [PR] [HUDI-8766] Enabling cols stats by default with writer [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12596: URL: https://github.com/apache/hudi/pull/12596#issuecomment-2581867489 ## CI report: * ae2ca606c6cd125f31b7ed029968d0993b1bb0bd UNKNOWN * 71b6a13890909b81c74ce7b138237ab695a08782 UNKNOWN * 15866ae0099c3b58d22329be0e5008b3149cb95f Azure: [FAIL

[jira] [Updated] (HUDI-8796) Silent ignoring of bucket index in Flink append mode

2025-01-09 Thread Geser Dugarov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geser Dugarov updated HUDI-8796: Description: Currently, there is no exception when we try to write data in Flink append mode using b

[jira] [Updated] (HUDI-8796) Silent ignoring of bucket index in Flink append mode

2025-01-09 Thread Geser Dugarov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geser Dugarov updated HUDI-8796: Summary: Silent ignoring of bucket index in Flink append mode (was: Silent ignoring of simple bucke

Re: [PR] [HUDI-8766] Enabling cols stats by default with writer [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12596: URL: https://github.com/apache/hudi/pull/12596#issuecomment-2581822443 ## CI report: * ae2ca606c6cd125f31b7ed029968d0993b1bb0bd UNKNOWN * 71b6a13890909b81c74ce7b138237ab695a08782 UNKNOWN * 15866ae0099c3b58d22329be0e5008b3149cb95f Azure: [FAIL

Re: [PR] [HUDI-8775] Expression index on a column should get tracked at partition level if partition stats index is turned on [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12558: URL: https://github.com/apache/hudi/pull/12558#issuecomment-2581813248 ## CI report: * c5912b6788b23621a4dcc609a4d5b4e6ae0af6da Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2767)

Re: [PR] [HUDI-8775] Expression index on a column should get tracked at partition level if partition stats index is turned on [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12558: URL: https://github.com/apache/hudi/pull/12558#issuecomment-2581811657 ## CI report: * c5912b6788b23621a4dcc609a4d5b4e6ae0af6da Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2767)

Re: [PR] [HUDI-8800] Introduce SingleSparkConsistentBucketClusteringExecutionStrategy to improve performance [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12537: URL: https://github.com/apache/hudi/pull/12537#issuecomment-2581802070 ## CI report: * 64ad84f40ff6a47df76979a382525fee0cc67d2e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2790)

Re: [PR] [HUDI-8796] Silent ignoring of simple bucket index in Flink append mode [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12545: URL: https://github.com/apache/hudi/pull/12545#issuecomment-2581794386 ## CI report: * 3efc78274b41c22ac6d2695e715fd157a9b9a9b8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2789)

[jira] [Updated] (HUDI-8172) Make primaryKey and other column configs case insensitive

2025-01-09 Thread Vova Kolmakov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vova Kolmakov updated HUDI-8172: Status: In Progress (was: Open) > Make primaryKey and other column configs case insensitive > -

[jira] [Assigned] (HUDI-8172) Make primaryKey and other column configs case insensitive

2025-01-09 Thread Vova Kolmakov (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vova Kolmakov reassigned HUDI-8172: --- Assignee: Vova Kolmakov > Make primaryKey and other column configs case insensitive > ---

Re: [PR] [HUDI-8766] Enabling cols stats by default with writer [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12596: URL: https://github.com/apache/hudi/pull/12596#issuecomment-2581720604 ## CI report: * ae2ca606c6cd125f31b7ed029968d0993b1bb0bd UNKNOWN * 71b6a13890909b81c74ce7b138237ab695a08782 UNKNOWN * a0efb5a7f12042228a5444aeab00f98827dfad3a Azure: [FAIL

Re: [PR] [HUDI-8624] Avoid check metadata for archived commits in incremental queries [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12613: URL: https://github.com/apache/hudi/pull/12613#issuecomment-2581719260 ## CI report: * 8fe93c788b78c9239f8feb90d3d78a90b8153914 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2788)

Re: [PR] [HUDI-8800] Introduce SingleSparkConsistentBucketClusteringExecutionStrategy to improve performance [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12537: URL: https://github.com/apache/hudi/pull/12537#issuecomment-2581699957 ## CI report: * 6198247de5d01f8edaf4976efffdffa6e6674b64 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2763)

Re: [PR] [HUDI-8766] Enabling cols stats by default with writer [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12596: URL: https://github.com/apache/hudi/pull/12596#issuecomment-2581698677 ## CI report: * ae2ca606c6cd125f31b7ed029968d0993b1bb0bd UNKNOWN * 71b6a13890909b81c74ce7b138237ab695a08782 UNKNOWN * a0efb5a7f12042228a5444aeab00f98827dfad3a Azure: [FAIL

Re: [PR] [HUDI-8800] Introduce SingleSparkConsistentBucketClusteringExecutionStrategy to improve performance [hudi]

2025-01-09 Thread via GitHub
TheR1sing3un commented on code in PR #12537: URL: https://github.com/apache/hudi/pull/12537#discussion_r1909786316 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/strategy/ClusteringExecutionStrategy.java: ## @@ -67,4 +85,69 @@ protected Hood

Re: [PR] [HUDI-8800] Introduce SingleSparkConsistentBucketClusteringExecutionStrategy to improve performance [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12537: URL: https://github.com/apache/hudi/pull/12537#issuecomment-2581698449 ## CI report: * 6198247de5d01f8edaf4976efffdffa6e6674b64 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2763)

[jira] [Updated] (HUDI-8854) Support LocalDate with ordering value in DeleteRecord

2025-01-09 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-8854: -- Fix Version/s: 1.0.2 > Support LocalDate with ordering value in DeleteRecord > -

[jira] [Created] (HUDI-8854) Support LocalDate with ordering value in DeleteRecord

2025-01-09 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-8854: - Summary: Support LocalDate with ordering value in DeleteRecord Key: HUDI-8854 URL: https://issues.apache.org/jira/browse/HUDI-8854 Project: Apache Hudi

[jira] [Assigned] (HUDI-8854) Support LocalDate with ordering value in DeleteRecord

2025-01-09 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-8854: - Assignee: sivabalan narayanan > Support LocalDate with ordering value in DeleteRe

Re: [PR] [HUDI-8796] Silent ignoring of simple bucket index in Flink append mode [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12545: URL: https://github.com/apache/hudi/pull/12545#issuecomment-2581691059 ## CI report: * 20a6a8c042d092026fbed250e5b313e366d2cf61 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2786)

Re: [PR] [HUDI-8796] Silent ignoring of simple bucket index in Flink append mode [hudi]

2025-01-09 Thread via GitHub
geserdugarov commented on PR #12545: URL: https://github.com/apache/hudi/pull/12545#issuecomment-2581656971 @zhangyue19921010 , @danny0405 , I've switched fix for bucket index support for append mode to its restriction, due to major problem with only one expected base file for bucket ind

Re: [PR] [HUDI-8624] Avoid check metadata for archived commits in incremental queries [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12613: URL: https://github.com/apache/hudi/pull/12613#issuecomment-2581651747 ## CI report: * 39ca7fae423367a6f48c5139b257176d22beac02 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2785)

Re: [PR] [HUDI-8796] Silent ignoring of simple bucket index in Flink append mode [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12545: URL: https://github.com/apache/hudi/pull/12545#issuecomment-2581639192 ## CI report: * 1a9b2ad8ba31a4bfb0c41f65af7d76841a946720 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2576)

Re: [PR] [HUDI-8796] Silent ignoring of simple bucket index in Flink append mode [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12545: URL: https://github.com/apache/hudi/pull/12545#issuecomment-2581649185 ## CI report: * 20a6a8c042d092026fbed250e5b313e366d2cf61 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2786)

Re: [PR] [HUDI-8796] Silent ignoring of simple bucket index in Flink append mode [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12545: URL: https://github.com/apache/hudi/pull/12545#issuecomment-2581645223 ## CI report: * 1a9b2ad8ba31a4bfb0c41f65af7d76841a946720 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2576)

Re: [PR] [HUDI-8553] Support writing record positions to log blocks from Spark SQL UPDATE and DELETE statements [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12612: URL: https://github.com/apache/hudi/pull/12612#issuecomment-2581639502 ## CI report: * 099eea2fba303c305950fad54010c503aff5c41e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2784)

Re: [PR] [HUDI-8796] Silent ignoring of simple bucket index in Flink append mode [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12545: URL: https://github.com/apache/hudi/pull/12545#issuecomment-2581637712 ## CI report: * 1a9b2ad8ba31a4bfb0c41f65af7d76841a946720 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2576)

[jira] [Commented] (HUDI-8762) Fix issues around incremental query

2025-01-09 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911745#comment-17911745 ] Danny Chen commented on HUDI-8762: -- A minor fix is merged via master: 5f591eec22337263f70

(hudi) branch master updated: [HUDI-8762] Fix a typo in Fix a typo in TestIncrementalQueryWithArchivedInstants (#12611)

2025-01-09 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 5f591eec223 [HUDI-8762] Fix a typo in Fix a typ

Re: [PR] [HUDI-8762] Fix a typo in TestIncrementalQueryWithArchivedInstants [hudi]

2025-01-09 Thread via GitHub
danny0405 merged PR #12611: URL: https://github.com/apache/hudi/pull/12611 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apac

Re: [PR] [Early feedback] [HUDI-8163] Refactor UnMergedLogHandler with iterators [hudi]

2025-01-09 Thread via GitHub
danny0405 commented on code in PR #12608: URL: https://github.com/apache/hudi/pull/12608#discussion_r1909708993 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -1656,9 +1664,17 @@ public static List> getLogFileColumnRangeM .

Re: [PR] [Early feedback] [HUDI-8163] Refactor UnMergedLogHandler with iterators [hudi]

2025-01-09 Thread via GitHub
danny0405 commented on code in PR #12608: URL: https://github.com/apache/hudi/pull/12608#discussion_r1909706326 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieUnMergedLogRecordScanner.java: ## @@ -185,16 +213,6 @@ public Builder withInstantRange(Option inst

Re: [PR] [Early feedback] [HUDI-8163] Refactor UnMergedLogHandler with iterators [hudi]

2025-01-09 Thread via GitHub
danny0405 commented on code in PR #12608: URL: https://github.com/apache/hudi/pull/12608#discussion_r1909703029 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java: ## @@ -617,6 +617,8 @@ && compareTimestamps(logBlock.getLogBlockHea

Re: [PR] [HUDI-8762] Fix a typo in a test [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12611: URL: https://github.com/apache/hudi/pull/12611#issuecomment-2581618440 ## CI report: * 441dfd77c5036cfac3ce7a84cd7984408f5b6b64 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2783)

Re: [PR] [Early feedback] [HUDI-8163] Refactor UnMergedLogHandler with iterators [hudi]

2025-01-09 Thread via GitHub
danny0405 commented on code in PR #12608: URL: https://github.com/apache/hudi/pull/12608#discussion_r1909701282 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java: ## @@ -190,9 +190,16 @@ private static List getUnmergedLogF

[jira] [Closed] (HUDI-8173) Defualt value of hoodie.avro.schema.validate silently getting updated if not passed explicitly

2025-01-09 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-8173. Resolution: Fixed Fixed via master branch: 6b3e076205884e69e4ae8c8980026abaae74d03c > Defualt value of hood

[jira] [Updated] (HUDI-8173) Defualt value of hoodie.avro.schema.validate silently getting updated if not passed explicitly

2025-01-09 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-8173: - Status: Open (was: In Progress) > Defualt value of hoodie.avro.schema.validate silently getting updated i

(hudi) branch master updated: [HUDI-8173] Default value of hoodie.avro.schema.validate silently getting updated if not passed explicitly (#12606)

2025-01-09 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 6b3e0762058 [HUDI-8173] Default value of hoodie

Re: [PR] [HUDI-8173] Default value of hoodie.avro.schema.validate silently getting updated if not passed explicitly [hudi]

2025-01-09 Thread via GitHub
danny0405 merged PR #12606: URL: https://github.com/apache/hudi/pull/12606 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apac

Re: [PR] [HUDI-8800] Introduce SingleSparkConsistentBucketClusteringExecutionStrategy to improve performance [hudi]

2025-01-09 Thread via GitHub
danny0405 commented on code in PR #12537: URL: https://github.com/apache/hudi/pull/12537#discussion_r1909691146 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/strategy/ClusteringExecutionStrategy.java: ## @@ -67,4 +85,69 @@ protected HoodieE

Re: [PR] [HUDI-8624] Avoid check metadata for archived commits in incremental queries [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12613: URL: https://github.com/apache/hudi/pull/12613#issuecomment-2581605163 ## CI report: * 39ca7fae423367a6f48c5139b257176d22beac02 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2785)

Re: [PR] [HUDI-8624] Avoid check metadata for archived commits in incremental queries [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12613: URL: https://github.com/apache/hudi/pull/12613#issuecomment-2581596398 ## CI report: * 39ca7fae423367a6f48c5139b257176d22beac02 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2785)

Re: [PR] [HUDI-8780][RFC-83][WIP] Incremental Table Service [hudi]

2025-01-09 Thread via GitHub
zhangyue19921010 commented on PR #12601: URL: https://github.com/apache/hudi/pull/12601#issuecomment-2581604354 Hi @TheR1sing3un , Thanks for your attention. > @zhangyue19921010 Hi, judging from the rfc content, the goal this time is to use incremental `compaction` and `clustering`. Shou

Re: [PR] [HUDI-8624] Avoid check metadata for archived commits in incremental queries [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12613: URL: https://github.com/apache/hudi/pull/12613#issuecomment-2581603749 ## CI report: * 39ca7fae423367a6f48c5139b257176d22beac02 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2785)

[jira] [Updated] (HUDI-8762) Fix issues around incremental query

2025-01-09 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Liu updated HUDI-8762: -- Status: Patch Available (was: In Progress) > Fix issues around incremental query >

[jira] [Updated] (HUDI-8635) Revisit stats generated in HoodieSparkFileGroupReaderBasedMergeHandle

2025-01-09 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo updated HUDI-8635: -- Status: In Progress (was: Open) > Revisit stats generated in HoodieSparkFileGroupReaderBasedMergeHandle

Re: [PR] [HUDI-8624] Avoid check metadata for archived commits in incremental queries [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12613: URL: https://github.com/apache/hudi/pull/12613#issuecomment-2581592281 ## CI report: * 39ca7fae423367a6f48c5139b257176d22beac02 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[jira] [Updated] (HUDI-8624) Revisit commitsMetadata fetching from timeline history in MergeOnReadIncrementalRelation

2025-01-09 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Liu updated HUDI-8624: -- Status: Patch Available (was: In Progress) > Revisit commitsMetadata fetching from timeline history in > Merge

[jira] [Updated] (HUDI-8624) Revisit commitsMetadata fetching from timeline history in MergeOnReadIncrementalRelation

2025-01-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-8624: - Labels: pull-request-available (was: ) > Revisit commitsMetadata fetching from timeline history i

[PR] [HUDI-8624] Avoid check metadata for archived commits in incremental queries [hudi]

2025-01-09 Thread via GitHub
linliu-code opened a new pull request, #12613: URL: https://github.com/apache/hudi/pull/12613 ### Change Logs When start commit is archived, we fall back to full scan. ### Impact Avoid expensive metadata fetching for archived instants. ### Risk level (write none, l

[jira] [Commented] (HUDI-8553) Spark SQL UPDATE and DELETE should write record positions

2025-01-09 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911738#comment-17911738 ] Y Ethan Guo commented on HUDI-8553: --- I have a draft PR up which makes the prepped upsert

Re: [PR] [HUDI-8553] Support writing record positions to log blocks from Spark SQL UPDATE and DELETE statements [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12612: URL: https://github.com/apache/hudi/pull/12612#issuecomment-2581582716 ## CI report: * 099eea2fba303c305950fad54010c503aff5c41e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2784)

Re: [PR] [HUDI-8553] Support writing record positions to log blocks from Spark SQL UPDATE and DELETE statements [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12612: URL: https://github.com/apache/hudi/pull/12612#issuecomment-2581581182 ## CI report: * 099eea2fba303c305950fad54010c503aff5c41e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[jira] [Updated] (HUDI-8553) Spark SQL UPDATE and DELETE should write record positions

2025-01-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-8553: - Labels: pull-request-available (was: ) > Spark SQL UPDATE and DELETE should write record position

[PR] [HUDI-8553] Support writing record positions to log blocks from Spark SQL UPDATE and DELETE statements [hudi]

2025-01-09 Thread via GitHub
yihua opened a new pull request, #12612: URL: https://github.com/apache/hudi/pull/12612 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performance

Re: [PR] [HUDI-8839] CdcFileGroupIterator use spillable hashmap [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12592: URL: https://github.com/apache/hudi/pull/12592#issuecomment-2581548015 ## CI report: * 423421ec00e72021f081c901ac74891a266b8aa5 UNKNOWN * a5544e3e3d5aa734348b7bfd63820d5b8d98cc33 UNKNOWN * 7ca86e570e17a7db2c7394d62f9d95bda8f439db Azure: [SUCC

Re: [PR] [HUDI-8824] MIT should error out for some assignment clause patterns [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12584: URL: https://github.com/apache/hudi/pull/12584#issuecomment-2581546292 ## CI report: * ffad81180c72f871a9677549e38f1915e5668adb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2781)

Re: [PR] [HUDI-8762] Fix a typo in a test [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12611: URL: https://github.com/apache/hudi/pull/12611#issuecomment-2581544722 ## CI report: * 441dfd77c5036cfac3ce7a84cd7984408f5b6b64 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2783)

Re: [PR] [HUDI-8762] Fix a typo in a test [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12611: URL: https://github.com/apache/hudi/pull/12611#issuecomment-2581543130 ## CI report: * 441dfd77c5036cfac3ce7a84cd7984408f5b6b64 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

[jira] [Updated] (HUDI-8762) Fix issues around incremental query

2025-01-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-8762: - Labels: pull-request-available (was: ) > Fix issues around incremental query > --

[PR] [HUDI-8762] Fix a typo in a test [hudi]

2025-01-09 Thread via GitHub
linliu-code opened a new pull request, #12611: URL: https://github.com/apache/hudi/pull/12611 ### Change Logs The config was not set correctly. ### Impact Fixed a typo. ### Risk level (write none, low medium or high below) None. ### Documentation Upda

Re: [PR] [HUDI-8828] Test coverage of MIT partial update [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12583: URL: https://github.com/apache/hudi/pull/12583#issuecomment-2581510528 ## CI report: * 757290d3cf1ab9027f2f14f3cd22097f50939a56 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2780)

[jira] [Commented] (HUDI-8837) Fix reading partition path field on metadata bootstrap table

2025-01-09 Thread Davis Zhang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911714#comment-17911714 ] Davis Zhang commented on HUDI-8837: --- so we can remove the .drop(partitionColName) in the

[jira] [Assigned] (HUDI-8837) Fix reading partition path field on metadata bootstrap table

2025-01-09 Thread Davis Zhang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davis Zhang reassigned HUDI-8837: - Assignee: Y Ethan Guo (was: Davis Zhang) > Fix reading partition path field on metadata bootstra

[jira] [Commented] (HUDI-8553) Spark SQL UPDATE and DELETE should write record positions

2025-01-09 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911713#comment-17911713 ] Y Ethan Guo commented on HUDI-8553: --- In the UPDATE and DELETE command, we'll try creatin

[jira] [Commented] (HUDI-8837) Fix reading partition path field on metadata bootstrap table

2025-01-09 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911710#comment-17911710 ] Y Ethan Guo commented on HUDI-8837: --- The test is added in https://github.com/apache/hudi

Re: [PR] [HUDI-8832] Add merge mode test coverage for DML [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12610: URL: https://github.com/apache/hudi/pull/12610#issuecomment-2581484606 ## CI report: * 5fbd4a15950f9d2b214ce3617164f68ac96fdc4b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2778)

[jira] [Updated] (HUDI-8837) Fix reading partition path field on metadata bootstrap table

2025-01-09 Thread Davis Zhang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davis Zhang updated HUDI-8837: -- Status: In Progress (was: Open) > Fix reading partition path field on metadata bootstrap table > --

[jira] [Assigned] (HUDI-8837) Fix reading partition path field on metadata bootstrap table

2025-01-09 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo reassigned HUDI-8837: - Assignee: Davis Zhang > Fix reading partition path field on metadata bootstrap table > --

Re: [PR] [HUDI-8839] CdcFileGroupIterator use spillable hashmap [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12592: URL: https://github.com/apache/hudi/pull/12592#issuecomment-2581481123 ## CI report: * 28247026a78dda613a41ed2f039cbf11bb7d5d95 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2779)

Re: [PR] [HUDI-8824] MIT should error out for some assignment clause patterns [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12584: URL: https://github.com/apache/hudi/pull/12584#issuecomment-2581481057 ## CI report: * 631494d4f6e8389bf8c7a7d90a360fc1ea2d159d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2770)

Re: [PR] [HUDI-8824] MIT should error out for some assignment clause patterns [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12584: URL: https://github.com/apache/hudi/pull/12584#issuecomment-2581479281 ## CI report: * 631494d4f6e8389bf8c7a7d90a360fc1ea2d159d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2770)

[jira] [Assigned] (HUDI-8624) Revisit commitsMetadata fetching from timeline history in MergeOnReadIncrementalRelation

2025-01-09 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Liu reassigned HUDI-8624: - Assignee: Lin Liu > Revisit commitsMetadata fetching from timeline history in > MergeOnReadIncrementalRe

[jira] [Updated] (HUDI-8839) [Ethan pls check worklog] CDC query: The beforeImageRecords and afterImageRecords are both in-memory hash map, they should be changes to spillable map.

2025-01-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-8839: - Labels: pull-request-available (was: ) > [Ethan pls check worklog] CDC query: The beforeImageReco

Re: [PR] [HUDI-8839] CdcFileGroupIterator use spillable hashmap [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12592: URL: https://github.com/apache/hudi/pull/12592#issuecomment-2581471756 ## CI report: * 28247026a78dda613a41ed2f039cbf11bb7d5d95 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2779)

[jira] [Updated] (HUDI-8839) [Ethan pls check worklog] CDC query: The beforeImageRecords and afterImageRecords are both in-memory hash map, they should be changes to spillable map.

2025-01-09 Thread Davis Zhang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davis Zhang updated HUDI-8839: -- Summary: [Ethan pls check worklog] CDC query: The beforeImageRecords and  afterImageRecords are both in-m

[jira] [Comment Edited] (HUDI-8853) Spark sql ALTER TABLE queries are failing on EMR

2025-01-09 Thread Mansi Patel (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911702#comment-17911702 ] Mansi Patel edited comment on HUDI-8853 at 1/9/25 11:27 PM: AL

[jira] [Commented] (HUDI-8853) Spark sql ALTER TABLE queries are failing on EMR

2025-01-09 Thread Mansi Patel (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911702#comment-17911702 ] Mansi Patel commented on HUDI-8853: --- ALTER COLUMN is also causing issue. {code:java} spa

Re: [PR] [HUDI-7081] Enable fg reader for tests with enum typed data [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12609: URL: https://github.com/apache/hudi/pull/12609#issuecomment-2581434137 ## CI report: * 05110e0db27b9669466f1955fc925bdb5d64b3e8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2775)

Re: [PR] [HUDI-8828] Test coverage of MIT partial update [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12583: URL: https://github.com/apache/hudi/pull/12583#issuecomment-2581432166 ## CI report: * 5912957233547cef72a3427e482c176537a164b2 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2776)

[jira] [Updated] (HUDI-8839) CDC query: The beforeImageRecords and afterImageRecords are both in-memory hash map, they should be changes to spillable map.

2025-01-09 Thread Davis Zhang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davis Zhang updated HUDI-8839: -- Status: Patch Available (was: In Progress) > CDC query: The beforeImageRecords and afterImageRecords ar

Re: [I] Unable to alter column name for a Hudi table in AWS [hudi]

2025-01-09 Thread via GitHub
mansipp commented on issue #9780: URL: https://github.com/apache/hudi/issues/9780#issuecomment-2581429815 Facing similar issue with hudi 0.15.0 on EMR EC2. https://issues.apache.org/jira/browse/HUDI-8853 -- This is an automated message from the Apache Git Service. To respond to the messag

[jira] [Updated] (HUDI-8853) Spark sql ALTER TABLE queries are failing on EMR

2025-01-09 Thread Y Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y Ethan Guo updated HUDI-8853: -- Fix Version/s: 1.0.1 > Spark sql ALTER TABLE queries are failing on EMR > --

Re: [PR] [Hudi-8839] CdcFileGroupIterator use spillable hashmap [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12592: URL: https://github.com/apache/hudi/pull/12592#issuecomment-2581425493 ## CI report: * 28247026a78dda613a41ed2f039cbf11bb7d5d95 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2779)

Re: [PR] [Hudi-8839] CdcFileGroupIterator use spillable hashmap [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12592: URL: https://github.com/apache/hudi/pull/12592#issuecomment-2581423654 ## CI report: * e720dcfa5656730d01e5f22e5f9a890c08c60e0d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2738)

Re: [PR] [Hudi-8839] CdcFileGroupIterator use spillable hashmap [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12592: URL: https://github.com/apache/hudi/pull/12592#issuecomment-2581416796 ## CI report: * e720dcfa5656730d01e5f22e5f9a890c08c60e0d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2738)

[jira] [Updated] (HUDI-8853) Spark sql ALTER TABLE queries are failing on EMR

2025-01-09 Thread Mansi Patel (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mansi Patel updated HUDI-8853: -- Component/s: spark-sql > Spark sql ALTER TABLE queries are failing on EMR >

[jira] [Updated] (HUDI-8853) Spark sql ALTER TABLE queries are failing on EMR

2025-01-09 Thread Mansi Patel (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mansi Patel updated HUDI-8853: -- Affects Version/s: 0.15.0 > Spark sql ALTER TABLE queries are failing on EMR > -

[jira] [Comment Edited] (HUDI-8853) Spark sql ALTER TABLE queries are failing on EMR

2025-01-09 Thread Mansi Patel (Jira)
[ https://issues.apache.org/jira/browse/HUDI-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911694#comment-17911694 ] Mansi Patel edited comment on HUDI-8853 at 1/9/25 11:04 PM: Re

Re: [PR] [Hudi-8839] CdcFileGroupIterator use spillable hashmap [hudi]

2025-01-09 Thread via GitHub
hudi-bot commented on PR #12592: URL: https://github.com/apache/hudi/pull/12592#issuecomment-2581414514 ## CI report: * e720dcfa5656730d01e5f22e5f9a890c08c60e0d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2738)

  1   2   3   >