Re: [PR] [HUDI-6139] Add support for Transformer schema validation in DeltaStreamer [hudi]

2024-09-05 Thread via GitHub
lokeshj1703 closed pull request #8574: [HUDI-6139] Add support for Transformer schema validation in DeltaStreamer URL: https://github.com/apache/hudi/pull/8574 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [DOCS] [WIP] Add doc update for HUDI-5295 [hudi]

2024-09-05 Thread via GitHub
lokeshj1703 closed pull request #7932: [DOCS] [WIP] Add doc update for HUDI-5295 URL: https://github.com/apache/hudi/pull/7932 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] [HUDI-5374] Use KeyGeneratorFactory class for instantiating a KeyGenerator [hudi]

2024-09-05 Thread via GitHub
lokeshj1703 closed pull request #7432: [HUDI-5374] Use KeyGeneratorFactory class for instantiating a KeyGenerator URL: https://github.com/apache/hudi/pull/7432 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] [SUPPORT] Hudi table created with dataframe API becomes unwritable to INSERT queries due to config conflict [hudi]

2024-09-05 Thread via GitHub
CTTY commented on issue #11772: URL: https://github.com/apache/hudi/issues/11772#issuecomment-2333271572 @mzheng-plaid have you tried `spark.sql("set hoodie.datasource.write.precombine.field=")` in your session? -- This is an automated message from the Apache Git Service. To respond to t

Re: [I] [SUPPORT] Hudi table created with dataframe API becomes unwritable to INSERT queries due to config conflict [hudi]

2024-09-05 Thread via GitHub
mzheng-plaid commented on issue #11772: URL: https://github.com/apache/hudi/issues/11772#issuecomment-2333273303 @CTTY thanks for the suggestion, I'll try that tomorrow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

2024-09-05 Thread via GitHub
prabodh1194 closed pull request #11628: [MINOR] fix the target location for auxlib download in hudi CLI URL: https://github.com/apache/hudi/pull/11628 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

2024-09-05 Thread via GitHub
prabodh1194 commented on PR #11628: URL: https://github.com/apache/hudi/pull/11628#issuecomment-204944 this is not much needed. will close this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]

2024-09-06 Thread via GitHub
lokeshj1703 commented on code in PR #11816: URL: https://github.com/apache/hudi/pull/11816#discussion_r1746745167 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala: ## @@ -442,13 +442,93 @@ class TestSparkSqlWi

Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]

2024-09-06 Thread via GitHub
lokeshj1703 commented on code in PR #11816: URL: https://github.com/apache/hudi/pull/11816#discussion_r1746745451 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala: ## @@ -442,13 +442,93 @@ class TestSparkSqlWi

Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11816: URL: https://github.com/apache/hudi/pull/11816#issuecomment-2333576413 ## CI report: * 5e15f2e5209875fcfb6796dc94da7a095ddb99e9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296)

[I] [SUPPORT] [hudi]

2024-09-06 Thread via GitHub
Maher3000 opened a new issue, #11899: URL: https://github.com/apache/hudi/issues/11899 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at dev-subs

Re: [PR] [MINOR] Fix LongWritable cannot be cast to TimestampWritable for MOR table with timestamp column and schema evolution enabled. [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11898: URL: https://github.com/apache/hudi/pull/11898#issuecomment-2333949416 ## CI report: * 8ea5de98bf028ef2d4491d03609a032425be3ee0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=295)

Re: [PR] [MINOR] Fix LongWritable cannot be cast to TimestampWritable for MOR table with timestamp column and schema evolution enabled. [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11898: URL: https://github.com/apache/hudi/pull/11898#issuecomment-2333951546 ## CI report: * 8ea5de98bf028ef2d4491d03609a032425be3ee0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=295)

Re: [PR] [HUDI-8071] Handle skew for user defined sort columns in BULK_INSERT [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11758: URL: https://github.com/apache/hudi/pull/11758#issuecomment-2333957340 ## CI report: * 8aebb9546b92beb496dc409407d4cd9a516b1adc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=23) *

Re: [PR] [HUDI-8071] Handle skew for user defined sort columns in BULK_INSERT [hudi]

2024-09-06 Thread via GitHub
vinishjail97 commented on code in PR #11758: URL: https://github.com/apache/hudi/pull/11758#discussion_r1747053625 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/BulkInsertPartitioner.java: ## @@ -83,20 +84,28 @@ default Option getWriteHandleFactory(int p

Re: [PR] [HUDI-8071] Handle skew for user defined sort columns in BULK_INSERT [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11758: URL: https://github.com/apache/hudi/pull/11758#issuecomment-2333961504 ## CI report: * 0b034367ad627ce492f3ba0e397a2f5e30e6cdd8 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=299)

Re: [PR] [HUDI-7900] Delete metadata partitions when downgrade [hudi]

2024-09-06 Thread via GitHub
linliu-code commented on code in PR #11889: URL: https://github.com/apache/hudi/pull/11889#discussion_r1747166375 ## hudi-client/hudi-client-common/pom.xml: ## @@ -179,7 +179,11 @@ - + + org.mockito + mockito-inline Review Comment: Remov

[PR] [HUDI-7902] Cherry-pick partition field format changes from master to 0.x [hudi]

2024-09-06 Thread via GitHub
lokeshj1703 opened a new pull request, #11901: URL: https://github.com/apache/hudi/pull/11901 ### Change Logs We need to cherry-pick commits made in jira HUDI-7902 to 0.x branch. HUDI-7902 changes the partition field format so that partition field type is stored in table config as we

Re: [PR] [HUDI-7902] Cherry-pick partition field format changes from master to 0.x [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11901: URL: https://github.com/apache/hudi/pull/11901#issuecomment-2334145541 ## CI report: * 5308f67b9b8a110fc8ee2be748831e2f2bf99e06 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-7902] Cherry-pick partition field format changes from master to 0.x [hudi]

2024-09-06 Thread via GitHub
lokeshj1703 commented on PR #11901: URL: https://github.com/apache/hudi/pull/11901#issuecomment-2334149167 The changes in https://issues.apache.org/jira/browse/HUDI-7902, assume that the partition field format would be changed as part of hudi upgrade. We need a different way of making su

Re: [PR] [HUDI-7902] Cherry-pick partition field format changes from master to 0.x [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11901: URL: https://github.com/apache/hudi/pull/11901#issuecomment-2334265583 ## CI report: * 5308f67b9b8a110fc8ee2be748831e2f2bf99e06 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=301)

Re: [PR] [HUDI-8175] Fix LongWritable cannot be cast to TimestampWritable for MOR table with timestamp column and schema evolution enabled [hudi]

2024-09-06 Thread via GitHub
yihua merged PR #11898: URL: https://github.com/apache/hudi/pull/11898 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] [HUDI-8092] Replace FileSystem and related classes to dehadoop hudi-client-common [hudi]

2024-09-06 Thread via GitHub
yihua commented on code in PR #11805: URL: https://github.com/apache/hudi/pull/11805#discussion_r1747307358 ## hudi-io/src/main/java/org/apache/hudi/storage/HoodieStorage.java: ## @@ -426,6 +426,26 @@ public List listDirectEntries(List pathList) throw return result; }

Re: [PR] test delete cache [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11902: URL: https://github.com/apache/hudi/pull/11902#issuecomment-2334321413 ## CI report: * 0b77fe26b7823b7d5c1ac316143228212b92017a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=302)

Re: [PR] test delete cache [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11902: URL: https://github.com/apache/hudi/pull/11902#issuecomment-2334341813 ## CI report: * 0b77fe26b7823b7d5c1ac316143228212b92017a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=302)

Re: [PR] [HUDI-8067] Use exec to run the IT [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11903: URL: https://github.com/apache/hudi/pull/11903#issuecomment-2334377604 ## CI report: * d4b87249cdadb3bc0f7d1f2908a2107867664e09 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-8067] Use exec to run the IT [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11903: URL: https://github.com/apache/hudi/pull/11903#issuecomment-2334380471 ## CI report: * d4b87249cdadb3bc0f7d1f2908a2107867664e09 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=303)

Re: [PR] [HUDI-8180] Delete the caches with workflow [hudi]

2024-09-06 Thread via GitHub
yihua merged PR #11904: URL: https://github.com/apache/hudi/pull/11904 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] test delete cache [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11902: URL: https://github.com/apache/hudi/pull/11902#issuecomment-2334426995 ## CI report: * 0b77fe26b7823b7d5c1ac316143228212b92017a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=302)

[PR] [HUDI-8180] use individual cache key when deleting caches [hudi]

2024-09-06 Thread via GitHub
jonvex opened a new pull request, #11905: URL: https://github.com/apache/hudi/pull/11905 ### Change Logs --all flag didn't work. Try to go back to deleting 1 by 1 ### Impact fix ci ### Risk level (write none, low medium or high below) low ### Document

Re: [PR] [HUDI-8180] use individual cache key when deleting caches [hudi]

2024-09-06 Thread via GitHub
yihua merged PR #11905: URL: https://github.com/apache/hudi/pull/11905 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

[PR] [HUDI-8180] give all permissions to scheduled workflow [hudi]

2024-09-06 Thread via GitHub
jonvex opened a new pull request, #11906: URL: https://github.com/apache/hudi/pull/11906 ### Change Logs delete cache doesn't have required permissions ### Impact fix ci ### Risk level (write none, low medium or high below) low ### Documentation Updat

Re: [PR] [HUDI-8180] give all permissions to scheduled workflow [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11906: URL: https://github.com/apache/hudi/pull/11906#issuecomment-2334457865 ## CI report: * f7959fdbf47b92f238788eed8e144a3794946425 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=307)

Re: [PR] [HUDI-8180] Give all permissions to scheduled workflow [hudi]

2024-09-06 Thread via GitHub
yihua merged PR #11906: URL: https://github.com/apache/hudi/pull/11906 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] [HUDI-7900] Delete metadata partitions when downgrade [hudi]

2024-09-06 Thread via GitHub
linliu-code commented on code in PR #11889: URL: https://github.com/apache/hudi/pull/11889#discussion_r1747466775 ## hudi-client/hudi-client-common/src/test/java/org/apache/hudi/table/upgrade/TestEightToSevenDowngradeHandler.java: ## @@ -0,0 +1,86 @@ +/* + * Licensed to the Apac

Re: [PR] [HUDI-7900] Delete metadata partitions when downgrade [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11889: URL: https://github.com/apache/hudi/pull/11889#issuecomment-2334495153 ## CI report: * 8b9c81b708c64ae021fea0da15185d11be6d5d07 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=272)

[PR] [HUDI-8180] make permissions more strict for scheduled workflow [hudi]

2024-09-06 Thread via GitHub
jonvex opened a new pull request, #11907: URL: https://github.com/apache/hudi/pull/11907 ### Change Logs make permissions more strict now that we know the delete cache workflow works. According to https://github.com/actions/gh-actions-cache/issues/85 it is the write action permiss

Re: [PR] [HUDI-8180] make permissions more strict for scheduled workflow [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11907: URL: https://github.com/apache/hudi/pull/11907#issuecomment-2334498182 ## CI report: * fef95ace533d58e6d7d9fab5e87ab5b35766b593 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-8180] make permissions more strict for scheduled workflow [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11907: URL: https://github.com/apache/hudi/pull/11907#issuecomment-2334501323 ## CI report: * fef95ace533d58e6d7d9fab5e87ab5b35766b593 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=310)

Re: [I] [SUPPORT] Hudi CLI. java.lang.NoClassDefFoundError: org/apache/hudi/avro/model/HoodieWriteStat [hudi]

2024-09-06 Thread via GitHub
alberttwong closed issue #11908: [SUPPORT] Hudi CLI. java.lang.NoClassDefFoundError: org/apache/hudi/avro/model/HoodieWriteStat URL: https://github.com/apache/hudi/issues/11908 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[I] [SUPPORT] Hudi CLI conf is hard coded to /opt/hudi/packaging/hudi-cli-bundle/conf/hudi-defaults.conf [hudi]

2024-09-06 Thread via GitHub
alberttwong opened a new issue, #11909: URL: https://github.com/apache/hudi/issues/11909 Hudi CLI conf is hard coded to /opt/hudi/packaging/hudi-cli-bundle/conf/hudi-defaults.conf A clear and concise description of the problem. **To Reproduce** Steps to reproduce the beh

Re: [PR] [HUDI-8180] make permissions more strict for scheduled workflow [hudi]

2024-09-06 Thread via GitHub
yihua merged PR #11907: URL: https://github.com/apache/hudi/pull/11907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] [HUDI-7900] Delete metadata partitions when downgrade [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11889: URL: https://github.com/apache/hudi/pull/11889#issuecomment-2334519267 ## CI report: * 8b9c81b708c64ae021fea0da15185d11be6d5d07 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=272)

Re: [PR] [HUDI-7900] Delete metadata partitions when downgrade [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11889: URL: https://github.com/apache/hudi/pull/11889#issuecomment-2334521084 ## CI report: * 1519be6b694a65a489a94d9103e1ee13de9c68e0 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=309)

Re: [PR] [HUDI-7900] Delete metadata partitions when downgrade [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11889: URL: https://github.com/apache/hudi/pull/11889#issuecomment-2334540719 ## CI report: * 1519be6b694a65a489a94d9103e1ee13de9c68e0 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=309)

Re: [PR] test delete cache [hudi]

2024-09-06 Thread via GitHub
jonvex closed pull request #11902: test delete cache URL: https://github.com/apache/hudi/pull/11902 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: comm

Re: [I] [SUPPORT] Hudi table created with dataframe API becomes unwritable to INSERT queries due to config conflict [hudi]

2024-09-06 Thread via GitHub
CTTY closed issue #11772: [SUPPORT] Hudi table created with dataframe API becomes unwritable to INSERT queries due to config conflict URL: https://github.com/apache/hudi/issues/11772 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] [HUDI-7900] Delete metadata partitions when downgrade [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11889: URL: https://github.com/apache/hudi/pull/11889#issuecomment-2334589104 ## CI report: * 1519be6b694a65a489a94d9103e1ee13de9c68e0 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=309)

Re: [PR] [HUDI-8067] Use exec to run the IT [hudi]

2024-09-06 Thread via GitHub
yihua merged PR #11903: URL: https://github.com/apache/hudi/pull/11903 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] delete cache on failure [hudi]

2024-09-06 Thread via GitHub
jonvex closed pull request #11894: delete cache on failure URL: https://github.com/apache/hudi/pull/11894 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [I] [SUPPORT] PartialUpdateAvroPayload still overwriting undefined columns as NULL [hudi]

2024-09-06 Thread via GitHub
joelwalden commented on issue #11726: URL: https://github.com/apache/hudi/issues/11726#issuecomment-2334645720 @danny0405 Is there a way to debug this in the Hudi logs or metadata? I'm logging out my Hudi options just before write and the correct class is being passed (as shown above) --

Re: [PR] [HUDI-7900] Delete metadata partitions when downgrade [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11889: URL: https://github.com/apache/hudi/pull/11889#issuecomment-2334680218 ## CI report: * 84648b03e2c78b64d6ce8fdb758c09e1fe28c451 UNKNOWN * b097072200e6a14f876e68d8201897e0812c1c65 UNKNOWN * 1b6072ac5fba314e63753b246106fb620a5ad7f4 Azure: [FAIL

Re: [PR] [HUDI-1441] Fixing HoodieAvroUtils.rewriteRecord for nested record schema evolution [hudi]

2024-09-06 Thread via GitHub
nsivabalan closed pull request #2982: [HUDI-1441] Fixing HoodieAvroUtils.rewriteRecord for nested record schema evolution URL: https://github.com/apache/hudi/pull/2982 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] [HUDI-7712] Fixing RLI initialization to account for file slices instead of just base files while initializing [hudi]

2024-09-06 Thread via GitHub
vinothchandar commented on code in PR #11153: URL: https://github.com/apache/hudi/pull/11153#discussion_r1747660447 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -181,6 +183,10 @@ protected HoodieBackedTable

Re: [PR] [HUDI-8103] Introduce Table Write Version and version table configs [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11866: URL: https://github.com/apache/hudi/pull/11866#issuecomment-2334834725 ## CI report: * 1f3cea1f76c6d7d613de85e425888941d00a3e0c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=256)

Re: [PR] [HUDI-8103] Introduce Table Write Version and version table configs [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11866: URL: https://github.com/apache/hudi/pull/11866#issuecomment-2334835967 ## CI report: * 1f3cea1f76c6d7d613de85e425888941d00a3e0c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=256)

Re: [I] [SUPPORT] Hudi CLI conf is hard coded to /opt/hudi/packaging/hudi-cli-bundle/conf/hudi-defaults.conf [hudi]

2024-09-06 Thread via GitHub
alberttwong commented on issue #11909: URL: https://github.com/apache/hudi/issues/11909#issuecomment-2334850059 it runs if you don't set `export HUDI_CONF_DIR=/opt/hudicli/conf/` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[PR] [WEBSITE] Ad scarf for website analytics [hudi]

2024-09-06 Thread via GitHub
bhasudha opened a new pull request, #11911: URL: https://github.com/apache/hudi/pull/11911 ### Change Logs Add scarf pixel in the top nav bar so all website pages can be tracked. ### Impact Use scarf.sh for website analytics. Approved by asf: https://privacy.apache.org/f

Re: [PR] [WEBSITE] Ad scarf for website analytics [hudi]

2024-09-06 Thread via GitHub
bhasudha merged PR #11911: URL: https://github.com/apache/hudi/pull/11911 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apach

Re: [I] [SUPPORT] Docker demo on website does not work as expected [hudi]

2024-09-06 Thread via GitHub
alberttwong commented on issue #11797: URL: https://github.com/apache/hudi/issues/11797#issuecomment-2335048175 We are in testing with new instructions. https://github.com/alberttwong/onehouse-demos/tree/main/trino-prestodb-spark-minio -- This is an automated message from the Apach

Re: [PR] [MINOR] Replace hard-coded path by temp dir in tests [hudi]

2024-09-06 Thread via GitHub
hudi-bot commented on PR #11912: URL: https://github.com/apache/hudi/pull/11912#issuecomment-2335049272 ## CI report: * 204718c728928b20de0509d3cf8513e26a005bec UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-7040] Handle dropping of partition columns with populate meta fields disabled [hudi]

2024-09-06 Thread via GitHub
codope commented on code in PR #10272: URL: https://github.com/apache/hudi/pull/10272#discussion_r1747946464 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala: ## @@ -365,6 +365,52 @@ class TestHoodieSparkSqlWriter { testBulkI

Re: [PR] [HUDI-8071] Handle skew for user defined sort columns in BULK_INSERT [hudi]

2024-09-06 Thread via GitHub
yihua commented on code in PR #11758: URL: https://github.com/apache/hudi/pull/11758#discussion_r1747950600 ## hudi-client/hudi-java-client/src/test/java/org/apache/hudi/execution/bulkinsert/TestJavaBulkInsertInternalPartitioner.java: ## @@ -60,7 +60,7 @@ public static Map gene

Re: [PR] [HUDI-2955] Support Hadoop 3.x Hive 3.x and Spark 3.2.x default (rebase) [hudi]

2024-09-06 Thread via GitHub
yihua closed pull request #5786: [HUDI-2955] Support Hadoop 3.x Hive 3.x and Spark 3.2.x default (rebase) URL: https://github.com/apache/hudi/pull/5786 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Hdp3 hive3 spark3 branch rebase hudi shaded [hudi]

2024-09-06 Thread via GitHub
yihua closed pull request #6092: Hdp3 hive3 spark3 branch rebase hudi shaded URL: https://github.com/apache/hudi/pull/6092 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] [HUDI-4794] add an option of the log file block size [hudi]

2024-09-06 Thread via GitHub
yihua commented on PR #6366: URL: https://github.com/apache/hudi/pull/6366#issuecomment-2335078184 @danny0405 do you think we still need this change on the log block size in Hudi 1.0? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [HUDI-4964] inline all the getter methods that have no logic … [hudi]

2024-09-06 Thread via GitHub
yihua closed pull request #6850: [HUDI-4964] inline all the getter methods that have no logic … URL: https://github.com/apache/hudi/pull/6850 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[I] [SUPPORT] Hudi Streamer EMR 7.0.0 with Hudi Extension (DELTA|ICEBERG) [hudi]

2024-09-07 Thread via GitHub
soumilshah1995 opened a new issue, #11913: URL: https://github.com/apache/hudi/issues/11913 ### Steps to reproduce Step1 : Download dataset upload it to S3 ![image](https://github.com/user-attachments/assets/149120f3-0648-4a49-850a-fdc864146ebf) https://

Re: [I] [SUPPORT] Hudi Streamer EMR Serverless ( 7.0.0) with Hudi Extension ( DELTA| ICEBERG ) [hudi]

2024-09-07 Thread via GitHub
soumilshah1995 commented on issue #11913: URL: https://github.com/apache/hudi/issues/11913#issuecomment-2336397762 Also weird issue is if it works at times if you delete the folder on S3 and glue tables and re run the same job it fails -- This is an automated message from the Apache

Re: [PR] [HUDI-5583] Add UT on multiple Java client writers while cleaning to validate no deadlock [hudi]

2024-09-07 Thread via GitHub
yihua commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-2336474230 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] [HUDI-5583] Add UT on multiple Java client writers while cleaning to validate no deadlock [hudi]

2024-09-07 Thread via GitHub
hudi-bot commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-2336474274 ## CI report: * e1e8daa6007e3750729ddd950edabb7015ee1465 UNKNOWN * 2f6d7d9847c1943378266e8e52d24e11d7bf027c UNKNOWN Bot commands @hudi-bot supports the following

Re: [PR] [HUDI-5583] Add UT on multiple Java client writers while cleaning to validate no deadlock [hudi]

2024-09-07 Thread via GitHub
hudi-bot commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-2336474499 ## CI report: * 2461980697426b4d6c92ab96a8728566ffd54e8e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [HUDI-5583] Add UT on multiple Java client writers while cleaning to validate no deadlock [hudi]

2024-09-07 Thread via GitHub
hudi-bot commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-2336474797 ## CI report: * 2461980697426b4d6c92ab96a8728566ffd54e8e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [HUDI-5583] Add UT on multiple Java client writers while cleaning to validate no deadlock [hudi]

2024-09-07 Thread via GitHub
hudi-bot commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-2336475953 ## CI report: * 2461980697426b4d6c92ab96a8728566ffd54e8e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [MINOR] Add remote file system props to meta sync properties [hudi]

2024-09-07 Thread via GitHub
yihua merged PR #11846: URL: https://github.com/apache/hudi/pull/11846 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.o

Re: [PR] [HUDI-8135] Limit number of meta clients created in StreamSync [hudi]

2024-09-07 Thread via GitHub
hudi-bot commented on PR #11853: URL: https://github.com/apache/hudi/pull/11853#issuecomment-2336483475 ## CI report: * 07b78b44283ed5625df5fec3a5f18622c9859cda Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=193)

Re: [PR] [HUDI-8135] Limit number of meta clients created in StreamSync [hudi]

2024-09-07 Thread via GitHub
hudi-bot commented on PR #11853: URL: https://github.com/apache/hudi/pull/11853#issuecomment-2336483648 ## CI report: * 07b78b44283ed5625df5fec3a5f18622c9859cda Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=193)

Re: [PR] [HUDI-8135] Limit number of meta clients created in StreamSync [hudi]

2024-09-07 Thread via GitHub
hudi-bot commented on PR #11853: URL: https://github.com/apache/hudi/pull/11853#issuecomment-2336502591 ## CI report: * bbcf7ca5135a7a03db515e9a613f0b31bfdc4c52 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=328)

Re: [PR] [HUDI-8084] Support Sort Merge Join Compaction [hudi]

2024-09-07 Thread via GitHub
hudi-bot commented on PR #11872: URL: https://github.com/apache/hudi/pull/11872#issuecomment-2336539716 ## CI report: * cbc1eeec261dcc48ba3dc02bd0ae1569dd06acac Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=240)

Re: [PR] [HUDI-5583] Add UT on multiple Java client writers while cleaning to validate no deadlock [hudi]

2024-09-07 Thread via GitHub
yihua merged PR #7704: URL: https://github.com/apache/hudi/pull/7704 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

Re: [PR] [HUDI-8182]Cache internalSchema for hive read, avoid each split reloa… [hudi]

2024-09-08 Thread via GitHub
hudi-bot commented on PR #11914: URL: https://github.com/apache/hudi/pull/11914#issuecomment-2336633304 ## CI report: * 018164ec3ed4da728e21e4c9a9d6a8a65019042e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run th

Re: [PR] [HUDI-8182]Cache internalSchema for hive read, avoid each split reloa… [hudi]

2024-09-08 Thread via GitHub
hudi-bot commented on PR #11914: URL: https://github.com/apache/hudi/pull/11914#issuecomment-2336633664 ## CI report: * 018164ec3ed4da728e21e4c9a9d6a8a65019042e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=331)

Re: [PR] [HUDI-8182]Cache internalSchema for hive read, avoid each split reloa… [hudi]

2024-09-08 Thread via GitHub
hudi-bot commented on PR #11914: URL: https://github.com/apache/hudi/pull/11914#issuecomment-2336636244 ## CI report: * 018164ec3ed4da728e21e4c9a9d6a8a65019042e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=331)

Re: [PR] [HUDI-8182]Cache internalSchema for hive read, avoid each split reloa… [hudi]

2024-09-08 Thread via GitHub
hudi-bot commented on PR #11914: URL: https://github.com/apache/hudi/pull/11914#issuecomment-2336636646 ## CI report: * 018164ec3ed4da728e21e4c9a9d6a8a65019042e Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=331)

Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

2024-09-08 Thread via GitHub
hudi-bot commented on PR #11580: URL: https://github.com/apache/hudi/pull/11580#issuecomment-2336750272 ## CI report: * 3a1c57e3dc77d325881e8093a72bff4927cad160 UNKNOWN * 42332f58f2204b2d9c368752de5ce2fcfaed8624 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4

Re: [PR] [HUDI-4240] Fix test Avro schema to be compatible across Avro versions [hudi]

2024-09-08 Thread via GitHub
hudi-bot commented on PR #5868: URL: https://github.com/apache/hudi/pull/5868#issuecomment-2336787035 ## CI report: * 98c3ff7a9397ebed46f0130acec2af6b5fdafd9a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [HUDI-4240] Fix test Avro schema to be compatible across Avro versions [hudi]

2024-09-08 Thread via GitHub
hudi-bot commented on PR #5868: URL: https://github.com/apache/hudi/pull/5868#issuecomment-2336788233 ## CI report: * 98c3ff7a9397ebed46f0130acec2af6b5fdafd9a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=334)

Re: [PR] [HUDI-4240] Fix test Avro schema to be compatible across Avro versions [hudi]

2024-09-08 Thread via GitHub
hudi-bot commented on PR #5868: URL: https://github.com/apache/hudi/pull/5868#issuecomment-2336811647 ## CI report: * 98c3ff7a9397ebed46f0130acec2af6b5fdafd9a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=334)

[PR] [ENG-13924] Fix empty meta sync class issue [hudi]

2024-09-08 Thread via GitHub
linliu-code opened a new pull request, #11915: URL: https://github.com/apache/hudi/pull/11915 ### Change Logs When meta sync is enabled, if the meta sync class name is empty string, java reflection would fail due to the empty class name. ### Impact Reduce the error messa

Re: [PR] [ENG-13924] Fix empty meta sync class issue [hudi]

2024-09-08 Thread via GitHub
hudi-bot commented on PR #11915: URL: https://github.com/apache/hudi/pull/11915#issuecomment-2336816953 ## CI report: * c2df0fbb63344d9129cb3b74007d6869e8350c59 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=339)

Re: [PR] [RFC-79] Improving reliability of concurrent table service executions and rollbacks [hudi]

2024-09-08 Thread via GitHub
nsivabalan commented on code in PR #11555: URL: https://github.com/apache/hudi/pull/11555#discussion_r1749333846 ## rfc/rfc-79/rfc-79.md: ## @@ -0,0 +1,116 @@ + +# RFC-79: Improving reliability of concurrent table service executions and rollbacks + +## Proposers + +- @kbuci +-

Re: [PR] [HUDI-8182]Cache internalSchema for hive read, avoid each split reloa… [hudi]

2024-09-08 Thread via GitHub
danny0405 commented on code in PR #11914: URL: https://github.com/apache/hudi/pull/11914#discussion_r1749394265 ## hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/SchemaEvolutionContext.java: ## @@ -98,21 +101,52 @@ public SchemaEvolutionContext(InputSplit split, JobConf jo

Re: [PR] [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue [hudi]

2024-09-08 Thread via GitHub
danny0405 commented on PR #7105: URL: https://github.com/apache/hudi/pull/7105#issuecomment-2336864244 > @danny0405 is this still a bug or we can close the PR? I kind of think it makes sense, we better to unify the instantiation of the fs objects. -- This is an automated message fr

Re: [I] org.apache.hudi.exception.HoodieException: org.apache.avro.AvroTypeException: Cannot encode decimal with precision 14 as max precision 13 [hudi]

2024-09-08 Thread via GitHub
danny0405 commented on issue #11335: URL: https://github.com/apache/hudi/issues/11335#issuecomment-233683 > Issue is caused by replace commit's new parquet file written. Noticed that new parquet file had different datatype for these columns. It's weird the decimal switched to long

Re: [PR] [ENG-13924] Fix empty meta sync class issue [hudi]

2024-09-08 Thread via GitHub
yihua commented on code in PR #11915: URL: https://github.com/apache/hudi/pull/11915#discussion_r1749484706 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java: ## @@ -1037,6 +1037,11 @@ public void runMetaSync() { Map failedMetaSyncs = ne

Re: [PR] [HUDI-8170] Create Reader State Class to remove state from fg reader [hudi]

2024-09-08 Thread via GitHub
yihua commented on code in PR #11827: URL: https://github.com/apache/hudi/pull/11827#discussion_r1749508430 ## hudi-common/src/main/java/org/apache/hudi/common/engine/FileGroupReaderState.java: ## @@ -0,0 +1,187 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] [HUDI-8160] Verify the consistency of the user-defined schema and the existing hoodie scheme when creating the hoodie table [hudi]

2024-09-08 Thread via GitHub
yihua commented on code in PR #11869: URL: https://github.com/apache/hudi/pull/11869#discussion_r1749515878 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableCommand.scala: ## @@ -88,6 +92,20 @@ case class CreateHoodieTa

Re: [PR] [HUDI-8160] Verify the consistency of the user-defined schema and the existing hoodie scheme when creating the hoodie table [hudi]

2024-09-08 Thread via GitHub
yihua commented on code in PR #11869: URL: https://github.com/apache/hudi/pull/11869#discussion_r1749516209 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableCommand.scala: ## @@ -88,6 +92,20 @@ case class CreateHoodieTa

Re: [PR] [HUDI-7975] Test PR to identify the functional test execution [hudi]

2024-09-08 Thread via GitHub
yihua commented on PR #11719: URL: https://github.com/apache/hudi/pull/11719#issuecomment-2337054923 Closing this for now. @suryaprasanna Have you figure out the fix to run CLI tests in CI? This is the tracking ticket: HUDI-7614. -- This is an automated message from the Apache Git Servi

Re: [PR] [minor]The Kryo of Hudi conflicts with the Kryo of Flink [hudi]

2024-09-08 Thread via GitHub
yihua closed pull request #6522: [minor]The Kryo of Hudi conflicts with the Kryo of Flink URL: https://github.com/apache/hudi/pull/6522 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

  1   2   3   4   5   6   7   8   9   10   >