[GitHub] [hudi] danny0405 commented on issue #4188: [SUPPORT] NullPointerException in HoodieROTablePathFilter Hudi 0.10.0

2021-12-03 Thread GitBox
danny0405 commented on issue #4188: URL: https://github.com/apache/hudi/issues/4188#issuecomment-985294375 > Thanks, I will have to wait for my team. will test it tomorrow and update the result here. You should be fast, the 0.10.0 RC#3 is cut tomorrow, if you want the fix in the rel

[GitHub] [hudi] danny0405 edited a comment on issue #4188: [SUPPORT] NullPointerException in HoodieROTablePathFilter Hudi 0.10.0

2021-12-03 Thread GitBox
danny0405 edited a comment on issue #4188: URL: https://github.com/apache/hudi/issues/4188#issuecomment-985294375 > Thanks, I will have to wait for my team. will test it tomorrow and update the result here. You should be as quickly as possible, the 0.10.0 RC#3 is cut tomorrow, if yo

[GitHub] [hudi] punish-yh commented on issue #4204: [SUPPORT] FlinkSQL write to Hudi, throw InvalidAvroMagicException: Not an Avro data file

2021-12-03 Thread GitBox
punish-yh commented on issue #4204: URL: https://github.com/apache/hudi/issues/4204#issuecomment-985299547 @danny0405 well,in job root exception, i only find these, the jobmanager exception same as this ![7A6E58CCDCA2670B42B6386B965CE0AA](https://user-images.githubusercontent.com/59658

[GitHub] [hudi] punish-yh edited a comment on issue #4204: [SUPPORT] FlinkSQL write to Hudi, throw InvalidAvroMagicException: Not an Avro data file

2021-12-03 Thread GitBox
punish-yh edited a comment on issue #4204: URL: https://github.com/apache/hudi/issues/4204#issuecomment-985299547 @danny0405 well,in job root exception, i only find these, the jobmanager exception same as this ![image](https://user-images.githubusercontent.com/59658062/144567211-a0

[GitHub] [hudi] hudi-bot commented on pull request #4199: [HUDI-2924] Refresh the fs view on successful checkpoints for write p…

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4199: URL: https://github.com/apache/hudi/pull/4199#issuecomment-985300044 ## CI report: * 44e5f689561137dc89b369862e82d8f6594a1240 UNKNOWN * 84e8af3268fb2ebebb8673b211db8397240f5245 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/

[GitHub] [hudi] hudi-bot removed a comment on pull request #4199: [HUDI-2924] Refresh the fs view on successful checkpoints for write p…

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4199: URL: https://github.com/apache/hudi/pull/4199#issuecomment-985272979 ## CI report: * 44e5f689561137dc89b369862e82d8f6594a1240 UNKNOWN * fd95743abab9a751bc2162a967884062717873de Azure: [CANCELED](https://dev.azure.com/apache-hud

[GitHub] [hudi] danny0405 merged pull request #4199: [HUDI-2924] Refresh the fs view on successful checkpoints for write p…

2021-12-03 Thread GitBox
danny0405 merged pull request #4199: URL: https://github.com/apache/hudi/pull/4199 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubsc

[hudi] branch master updated (f74b3d1 -> 0699521)

2021-12-03 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from f74b3d1 [minor] Refactor write profile to always generate fs view (#4198) add 0699521 [HUDI-2924] Refresh th

[jira] [Resolved] (HUDI-2924) Refresh the fs view on successful checkpoints for write profile

2021-12-03 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-2924. -- > Refresh the fs view on successful checkpoints for write profile >

[jira] [Commented] (HUDI-2924) Refresh the fs view on successful checkpoints for write profile

2021-12-03 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452814#comment-17452814 ] Danny Chen commented on HUDI-2924: -- Fixed via master branch: 0699521f83eab843279ff7d78ac2

[GitHub] [hudi] danny0405 commented on issue #4204: [SUPPORT] FlinkSQL write to Hudi, throw InvalidAvroMagicException: Not an Avro data file

2021-12-03 Thread GitBox
danny0405 commented on issue #4204: URL: https://github.com/apache/hudi/issues/4204#issuecomment-985302580 Seems there is a crush metadata file, can you delete it manually and try again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[jira] [Created] (HUDI-2926) Inline compaction of MOR table does not merge last log blocks when metadata table is enabled

2021-12-03 Thread Prashant Wason (Jira)
Prashant Wason created HUDI-2926: Summary: Inline compaction of MOR table does not merge last log blocks when metadata table is enabled Key: HUDI-2926 URL: https://issues.apache.org/jira/browse/HUDI-2926

[jira] [Commented] (HUDI-2926) Inline compaction of MOR table does not merge last log blocks when metadata table is enabled

2021-12-03 Thread Prashant Wason (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452815#comment-17452815 ] Prashant Wason commented on HUDI-2926: -- This has been fixed in 0.10 [by this PR|[htt

[jira] [Comment Edited] (HUDI-2926) Inline compaction of MOR table does not merge last log blocks when metadata table is enabled

2021-12-03 Thread Prashant Wason (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452815#comment-17452815 ] Prashant Wason edited comment on HUDI-2926 at 12/3/21, 8:15 AM:

[GitHub] [hudi] xushiyan commented on pull request #4201: Revert "[HUDI-2495] Resolve inconsistent key generation for timestamp…

2021-12-03 Thread GitBox
xushiyan commented on pull request #4201: URL: https://github.com/apache/hudi/pull/4201#issuecomment-985305969 > @YannByron Why revert it? I've added a fix for [HUDI-2909](https://issues.apache.org/jira/browse/HUDI-2909) : #4203 Can you please review? the key consideration is: keyge

[GitHub] [hudi] forest455 commented on issue #4200: spark-sql query timestamp partition error

2021-12-03 Thread GitBox
forest455 commented on issue #4200: URL: https://github.com/apache/hudi/issues/4200#issuecomment-985310884 i think there have a bug -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[GitHub] [hudi] fireking77 commented on issue #3945: [SUPPORT] hoodie.datasource.write.drop.partition.columns not working as expected (deltastreamer)

2021-12-03 Thread GitBox
fireking77 commented on issue #3945: URL: https://github.com/apache/hudi/issues/3945#issuecomment-985311538 Hi Guys! +1 here I have the same issue with quite a same setup (EMR 6.4) but with Hudi version 0.8 wich is built into that EMR version. Thanks, Darvi -- Th

[jira] [Commented] (HUDI-2926) Inline compaction of MOR table does not merge last log blocks when metadata table is enabled

2021-12-03 Thread Prashant Wason (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452829#comment-17452829 ] Prashant Wason commented on HUDI-2926: -- Patch to fix this issue in 0.9   diff --git

[GitHub] [hudi] xushiyan merged pull request #4182: [MINOR] use catalog schema if can not find table schema

2021-12-03 Thread GitBox
xushiyan merged pull request #4182: URL: https://github.com/apache/hudi/pull/4182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr

[hudi] branch master updated (0699521 -> ca42724)

2021-12-03 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 0699521 [HUDI-2924] Refresh the fs view on successful checkpoints for write profile (#4199) add ca42724 [MINO

[GitHub] [hudi] xushiyan edited a comment on pull request #4201: Revert "[HUDI-2495] Resolve inconsistent key generation for timestamp…

2021-12-03 Thread GitBox
xushiyan edited a comment on pull request #4201: URL: https://github.com/apache/hudi/pull/4201#issuecomment-985305969 > @YannByron Why revert it? I've added a fix for [HUDI-2909](https://issues.apache.org/jira/browse/HUDI-2909) : #4203 Can you please review? the key consideration is

[GitHub] [hudi] hudi-bot commented on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-985323557 ## CI report: * e12df0fe85613a48adfec2f57b0a6a27bd3919f4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?bu

[GitHub] [hudi] hudi-bot removed a comment on pull request #4203: [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4203: URL: https://github.com/apache/hudi/pull/4203#issuecomment-985283216 ## CI report: * e12df0fe85613a48adfec2f57b0a6a27bd3919f4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] lsyldliu commented on pull request #4153: [HUDI-2877] Support flink catalog to help user use flink table conveniently

2021-12-03 Thread GitBox
lsyldliu commented on pull request #4153: URL: https://github.com/apache/hudi/pull/4153#issuecomment-985345913 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[GitHub] [hudi] hudi-bot commented on pull request #4153: [HUDI-2877] Support flink catalog to help user use flink table conveniently

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4153: URL: https://github.com/apache/hudi/pull/4153#issuecomment-985346167 ## CI report: * 4d0c55169a2ff9652bb63e92544d5a89dc9da71f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?bu

[GitHub] [hudi] hudi-bot removed a comment on pull request #4153: [HUDI-2877] Support flink catalog to help user use flink table conveniently

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4153: URL: https://github.com/apache/hudi/pull/4153#issuecomment-981826925 ## CI report: * 4d0c55169a2ff9652bb63e92544d5a89dc9da71f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] hudi-bot removed a comment on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985279261 ## CI report: * 9c0ecdf8e29889a221b6d2e90cc105987657290b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] hudi-bot commented on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985346397 ## CI report: * 9c0ecdf8e29889a221b6d2e90cc105987657290b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?bu

[GitHub] [hudi] hudi-bot removed a comment on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985346397 ## CI report: * 9c0ecdf8e29889a221b6d2e90cc105987657290b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] hudi-bot commented on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985351437 ## CI report: * 9c0ecdf8e29889a221b6d2e90cc105987657290b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?bu

[GitHub] [hudi] hudi-bot removed a comment on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985351437 ## CI report: * 9c0ecdf8e29889a221b6d2e90cc105987657290b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] hudi-bot commented on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985354926 ## CI report: * 9c0ecdf8e29889a221b6d2e90cc105987657290b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?bu

[GitHub] [hudi] manojpec commented on a change in pull request #4194: [HUDI-2902] Fixing populate meta fields with Hfile writers

2021-12-03 Thread GitBox
manojpec commented on a change in pull request #4194: URL: https://github.com/apache/hudi/pull/4194#discussion_r761768414 ## File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java ## @@ -346,6 +363,56 @@ public vo

[GitHub] [hudi] danny0405 commented on pull request #4153: [HUDI-2877] Support flink catalog to help user use flink table conveniently

2021-12-03 Thread GitBox
danny0405 commented on pull request #4153: URL: https://github.com/apache/hudi/pull/4153#issuecomment-985370391 Thanks for the contribution @lsyldliu , reviewing now ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [hudi] hudi-bot removed a comment on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985354926 ## CI report: * 9c0ecdf8e29889a221b6d2e90cc105987657290b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] hudi-bot commented on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985373529 ## CI report: * 9c0ecdf8e29889a221b6d2e90cc105987657290b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?bu

[GitHub] [hudi] hudi-bot commented on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985375475 ## CI report: * 78c68edc8debb60e13b15883597dedcfa37f67ba UNKNOWN * b94ff683ec547f1418a2433bfd267da26a3603d4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985373529 ## CI report: * 9c0ecdf8e29889a221b6d2e90cc105987657290b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] fireking77 opened a new issue #4205: [SUPPORT] Target Parquet schema / metadata info / LogicalType

2021-12-03 Thread GitBox
fireking77 opened a new issue #4205: URL: https://github.com/apache/hudi/issues/4205 **Describe the problem you faced** Hi Guys, My config: Amazon EMR 6.4 with Hudi 0.8 I am very newbie with HUDI, and when I checked the materialized parquet schema, I found some intresti

[GitHub] [hudi] codope commented on pull request #4201: Revert "[HUDI-2495] Resolve inconsistent key generation for timestamp…

2021-12-03 Thread GitBox
codope commented on pull request #4201: URL: https://github.com/apache/hudi/pull/4201#issuecomment-985380602 > > @YannByron Why revert it? I've added a fix for [HUDI-2909](https://issues.apache.org/jira/browse/HUDI-2909) : #4203 Can you please review? > > the key consideration is: k

[GitHub] [hudi] hudi-bot removed a comment on pull request #4153: [HUDI-2877] Support flink catalog to help user use flink table conveniently

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4153: URL: https://github.com/apache/hudi/pull/4153#issuecomment-985346167 ## CI report: * 4d0c55169a2ff9652bb63e92544d5a89dc9da71f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] hudi-bot commented on pull request #4153: [HUDI-2877] Support flink catalog to help user use flink table conveniently

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4153: URL: https://github.com/apache/hudi/pull/4153#issuecomment-985396342 ## CI report: * 4d0c55169a2ff9652bb63e92544d5a89dc9da71f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?bu

[GitHub] [hudi] hudi-bot commented on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985396444 ## CI report: * 78c68edc8debb60e13b15883597dedcfa37f67ba UNKNOWN * b94ff683ec547f1418a2433bfd267da26a3603d4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985375475 ## CI report: * 78c68edc8debb60e13b15883597dedcfa37f67ba UNKNOWN * b94ff683ec547f1418a2433bfd267da26a3603d4 Azure: [CANCELED](https://dev.azure.com/apache-hud

[GitHub] [hudi] danny0405 commented on a change in pull request #4153: [HUDI-2877] Support flink catalog to help user use flink table conveniently

2021-12-03 Thread GitBox
danny0405 commented on a change in pull request #4153: URL: https://github.com/apache/hudi/pull/4153#discussion_r761809811 ## File path: hudi-flink/src/test/java/org/apache/hudi/table/catalog/TestHoodieCatalog.java ## @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software F

[GitHub] [hudi] hudi-bot removed a comment on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985396444 ## CI report: * 78c68edc8debb60e13b15883597dedcfa37f67ba UNKNOWN * b94ff683ec547f1418a2433bfd267da26a3603d4 Azure: [CANCELED](https://dev.azure.com/apache-hud

[GitHub] [hudi] hudi-bot commented on pull request #4193: [HUDI-2915] bugfix field not found error for sparksql

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985434927 ## CI report: * 78c68edc8debb60e13b15883597dedcfa37f67ba UNKNOWN * 9712293bd4c13b5ef0d89ec1b5cdcb30ef447fb4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/

[GitHub] [hudi] XuQianJin-Stars commented on pull request #4193: [HUDI-2915] Fix field not found error for sparksql

2021-12-03 Thread GitBox
XuQianJin-Stars commented on pull request #4193: URL: https://github.com/apache/hudi/pull/4193#issuecomment-985448474 hi @leesf Do you have time to review this pr? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[GitHub] [hudi] nsivabalan commented on a change in pull request #4194: [HUDI-2902] Fixing populate meta fields with Hfile writers

2021-12-03 Thread GitBox
nsivabalan commented on a change in pull request #4194: URL: https://github.com/apache/hudi/pull/4194#discussion_r761888049 ## File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieMetadataBase.java ## @@ -292,7 +292,7 @@ protected

[GitHub] [hudi] nsivabalan merged pull request #4194: [HUDI-2902] Fixing populate meta fields with Hfile writers

2021-12-03 Thread GitBox
nsivabalan merged pull request #4194: URL: https://github.com/apache/hudi/pull/4194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubs

[hudi] branch master updated: [HUDI-2902] Fixing populate meta fields with Hfile writers and Disabling virtual keys by default for metadata table (#4194)

2021-12-03 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new e483f7c [HUDI-2902] Fixing populate meta field

[jira] [Resolved] (HUDI-2902) Metadata table virtual keys are not effective in base files

2021-12-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-2902. --- > Metadata table virtual keys are not effective in base files > --

[jira] [Updated] (HUDI-2918) Metadata table virtual keys need to be disabled by default

2021-12-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2918: -- Status: Resolved (was: Patch Available) > Metadata table virtual keys need to be disabl

[jira] [Commented] (HUDI-2918) Metadata table virtual keys need to be disabled by default

2021-12-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452967#comment-17452967 ] sivabalan narayanan commented on HUDI-2918: --- Addressed as part of 2902. > Metad

[GitHub] [hudi] nsivabalan merged pull request #4195: [HUDI-2911] Removing default value for `PARTITIONPATH_FIELD_NAME` resulting in incorrect `KeyGenerator` configuration

2021-12-03 Thread GitBox
nsivabalan merged pull request #4195: URL: https://github.com/apache/hudi/pull/4195 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubs

[hudi] branch master updated: [HUDI-2911] Removing default value for `PARTITIONPATH_FIELD_NAME` resulting in incorrect `KeyGenerator` configuration (#4195)

2021-12-03 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new bed7f98 [HUDI-2911] Removing default value for

[jira] [Resolved] (HUDI-2911) Writing non-partitioned table produces incorrect "hoodie.properties" file

2021-12-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-2911. --- > Writing non-partitioned table produces incorrect "hoodie.properties" file >

[jira] [Assigned] (HUDI-2911) Writing non-partitioned table produces incorrect "hoodie.properties" file

2021-12-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-2911: - Assignee: Alexey Kudinkin > Writing non-partitioned table produces incorrect "hoo

[jira] [Updated] (HUDI-2911) Writing non-partitioned table produces incorrect "hoodie.properties" file

2021-12-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2911: -- Fix Version/s: 0.10.0 (was: 0.11.0) > Writing non-partitioned tab

[GitHub] [hudi] nsivabalan commented on pull request #4201: Revert "[HUDI-2495] Resolve inconsistent key generation for timestamp…

2021-12-03 Thread GitBox
nsivabalan commented on pull request #4201: URL: https://github.com/apache/hudi/pull/4201#issuecomment-985508006 I get the intention behind the original fix. But I am most concerned that, for the same record (non-row writer path) in 0.9.0, and 0.10.0, we might return diff key gen values an

[jira] [Commented] (HUDI-2925) Cleaner may attempt to delete the same file twice when metadata table is enabled

2021-12-03 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17453050#comment-17453050 ] sivabalan narayanan commented on HUDI-2925: --- I have some clarifications wrt the

[GitHub] [hudi] nsivabalan opened a new pull request #4206: [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

2021-12-03 Thread GitBox
nsivabalan opened a new pull request #4206: URL: https://github.com/apache/hudi/pull/4206 ## What is the purpose of the pull request - When metadata table has an inflight compaction, snapshot view fails since latest file slice view does not return any base files nor log files.

[jira] [Updated] (HUDI-2923) Unable to read from metadata table when a compaction is in progress or has failed

2021-12-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2923: - Labels: pull-request-available (was: ) > Unable to read from metadata table when a compaction is

[GitHub] [hudi] hudi-bot commented on pull request #4206: [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4206: URL: https://github.com/apache/hudi/pull/4206#issuecomment-985572894 ## CI report: * bff2f468fef2b4fa86d4f6aab083d9216b555869 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` r

[GitHub] [hudi] hudi-bot commented on pull request #4206: [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4206: URL: https://github.com/apache/hudi/pull/4206#issuecomment-985574901 ## CI report: * bff2f468fef2b4fa86d4f6aab083d9216b555869 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?bu

[GitHub] [hudi] hudi-bot removed a comment on pull request #4206: [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4206: URL: https://github.com/apache/hudi/pull/4206#issuecomment-985572894 ## CI report: * bff2f468fef2b4fa86d4f6aab083d9216b555869 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run

[jira] [Resolved] (HUDI-2813) Claim RFC number for RFC for spark datasource V2 Integration

2021-12-03 Thread leesf (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] leesf resolved HUDI-2813. - > Claim RFC number for RFC for spark datasource V2 Integration >

[jira] [Updated] (HUDI-2813) Claim RFC number for RFC for spark datasource V2 Integration

2021-12-03 Thread leesf (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] leesf updated HUDI-2813: Fix Version/s: 0.11.0 > Claim RFC number for RFC for spark datasource V2 Integration >

[jira] [Closed] (HUDI-2813) Claim RFC number for RFC for spark datasource V2 Integration

2021-12-03 Thread leesf (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] leesf closed HUDI-2813. --- > Claim RFC number for RFC for spark datasource V2 Integration >

[GitHub] [hudi] hudi-bot removed a comment on pull request #4206: [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4206: URL: https://github.com/apache/hudi/pull/4206#issuecomment-985574901 ## CI report: * bff2f468fef2b4fa86d4f6aab083d9216b555869 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] hudi-bot commented on pull request #4206: [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4206: URL: https://github.com/apache/hudi/pull/4206#issuecomment-985618361 ## CI report: * bff2f468fef2b4fa86d4f6aab083d9216b555869 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?bu

[GitHub] [hudi] vinothchandar commented on pull request #4201: Revert "[HUDI-2495] Resolve inconsistent key generation for timestamp…

2021-12-03 Thread GitBox
vinothchandar commented on pull request #4201: URL: https://github.com/apache/hudi/pull/4201#issuecomment-985628475 I think its safer to revert the original PR (i.e this PR) and follow up the right way after the release. Keep existing behavior for 0.10? Wdyt -- This is an automated messa

[GitHub] [hudi] nsivabalan merged pull request #4201: Revert "[HUDI-2495] Resolve inconsistent key generation for timestamp…

2021-12-03 Thread GitBox
nsivabalan merged pull request #4201: URL: https://github.com/apache/hudi/pull/4201 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubs

[hudi] branch master updated (bed7f98 -> 2f96f43)

2021-12-03 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from bed7f98 [HUDI-2911] Removing default value for `PARTITIONPATH_FIELD_NAME` resulting in incorrect `KeyGenerator` co

[GitHub] [hudi] fireking77 commented on issue #4205: [SUPPORT] Target Parquet schema / metadata info / LogicalType

2021-12-03 Thread GitBox
fireking77 commented on issue #4205: URL: https://github.com/apache/hudi/issues/4205#issuecomment-985649438 Guys, I am sorry very much, this was a bug at me! It contains the right meta. Column(report_time) name: report_time path: report_time ma

[GitHub] [hudi] fireking77 closed issue #4205: [SUPPORT] Target Parquet schema / metadata info / LogicalType

2021-12-03 Thread GitBox
fireking77 closed issue #4205: URL: https://github.com/apache/hudi/issues/4205 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...

[jira] [Updated] (HUDI-2919) Migrate Hudi to proper Incremental Build System (Bazel)

2021-12-03 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-2919: -- Summary: Migrate Hudi to proper Incremental Build System (Bazel) (was: Migrate Hudi to Bazel)

[GitHub] [hudi] vinothchandar commented on a change in pull request #4206: [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

2021-12-03 Thread GitBox
vinothchandar commented on a change in pull request #4206: URL: https://github.com/apache/hudi/pull/4206#discussion_r762091656 ## File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java ## @@ -413,6 +413,58 @@ publ

[GitHub] [hudi] hudi-bot removed a comment on pull request #4175: [HUDI-2883] WIP Refactor hive sync tool / config to use reflection and standardize configs

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4175: URL: https://github.com/apache/hudi/pull/4175#issuecomment-985155867 ## CI report: * 1dd9878f06deeddf5287212cb15759b978c07815 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] hudi-bot commented on pull request #4175: [HUDI-2883] WIP Refactor hive sync tool / config to use reflection and standardize configs

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4175: URL: https://github.com/apache/hudi/pull/4175#issuecomment-985676561 ## CI report: * 1dd9878f06deeddf5287212cb15759b978c07815 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?bu

[GitHub] [hudi] hudi-bot commented on pull request #4175: [HUDI-2883] WIP Refactor hive sync tool / config to use reflection and standardize configs

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4175: URL: https://github.com/apache/hudi/pull/4175#issuecomment-985678498 ## CI report: * 1dd9878f06deeddf5287212cb15759b978c07815 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?bu

[GitHub] [hudi] hudi-bot removed a comment on pull request #4175: [HUDI-2883] WIP Refactor hive sync tool / config to use reflection and standardize configs

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4175: URL: https://github.com/apache/hudi/pull/4175#issuecomment-985676561 ## CI report: * 1dd9878f06deeddf5287212cb15759b978c07815 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] h7kanna commented on issue #4188: [SUPPORT] NullPointerException in HoodieROTablePathFilter Hudi 0.10.0

2021-12-03 Thread GitBox
h7kanna commented on issue #4188: URL: https://github.com/apache/hudi/issues/4188#issuecomment-985685786 Hi, Fails with same error even with concurrentHashMap 43131 [task-result-getter-2] WARN org.apache.spark.scheduler.TaskSetManager - Lost task 19.0 in stage 0.0 (TID 19) (hos

[GitHub] [hudi] BenjMaq opened a new issue #4207: [SUPPORT] Hudi 0.10.0 throws java.lang.NoSuchMethodError: com.amazonaws.http.AmazonHttpClient.disableStrictHostnameVerification

2021-12-03 Thread GitBox
BenjMaq opened a new issue #4207: URL: https://github.com/apache/hudi/issues/4207 **Describe the problem you faced** I'm currently using Hudi 0.9.0 and it runs fine. I can insert/delete data without problems (using Hudi Spark-SQL). Switching to Hudi 0.10.0 throws `java.lang.N

[GitHub] [hudi] BenjMaq opened a new issue #4208: [SUPPORT] On Hudi 0.9.0 - Alter table throws java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.alterTable(java.lang.String, org

2021-12-03 Thread GitBox
BenjMaq opened a new issue #4208: URL: https://github.com/apache/hudi/issues/4208 **Describe the problem you faced** I'm trying to run a Spark SQL statement to alter a Hudi table. At some point my Spark job fails and the errror `java.lang.NoSuchMethodException: org.apache.hadoop.

[GitHub] [hudi] hudi-bot commented on pull request #4175: [HUDI-2883] WIP Refactor hive sync tool / config to use reflection and standardize configs

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4175: URL: https://github.com/apache/hudi/pull/4175#issuecomment-985717523 ## CI report: * 2f6c1eacb6a03f23e975446c2d8b86ede81b063d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?bu

[GitHub] [hudi] hudi-bot removed a comment on pull request #4175: [HUDI-2883] WIP Refactor hive sync tool / config to use reflection and standardize configs

2021-12-03 Thread GitBox
hudi-bot removed a comment on pull request #4175: URL: https://github.com/apache/hudi/pull/4175#issuecomment-985678498 ## CI report: * 1dd9878f06deeddf5287212cb15759b978c07815 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/re

[GitHub] [hudi] nsivabalan merged pull request #4185: [HUDI-2894][HUDI-2905] Metadata table - avoiding key lookup failures on base files over S3

2021-12-03 Thread GitBox
nsivabalan merged pull request #4185: URL: https://github.com/apache/hudi/pull/4185 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubs

[hudi] branch master updated (2f96f43 -> 383d5ed)

2021-12-03 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 2f96f43 Revert "[HUDI-2495] Resolve inconsistent key generation for timestamp types by GenericRecord and Row (#39

[GitHub] [hudi] nsivabalan commented on a change in pull request #4206: [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

2021-12-03 Thread GitBox
nsivabalan commented on a change in pull request #4206: URL: https://github.com/apache/hudi/pull/4206#discussion_r762186377 ## File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java ## @@ -352,7 +352,7 @@ public static int mapRecordKeyToFile

[GitHub] [hudi] nsivabalan commented on a change in pull request #4206: [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight

2021-12-03 Thread GitBox
nsivabalan commented on a change in pull request #4206: URL: https://github.com/apache/hudi/pull/4206#discussion_r762186699 ## File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java ## @@ -413,6 +413,58 @@ public

[jira] [Created] (HUDI-2927) Experiment running Hudi services using jemalloc

2021-12-03 Thread Alexey Kudinkin (Jira)
Alexey Kudinkin created HUDI-2927: - Summary: Experiment running Hudi services using jemalloc Key: HUDI-2927 URL: https://issues.apache.org/jira/browse/HUDI-2927 Project: Apache Hudi Issue Typ

[jira] [Assigned] (HUDI-2927) Experiment running Hudi services using jemalloc

2021-12-03 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin reassigned HUDI-2927: - Assignee: Alexey Kudinkin > Experiment running Hudi services using jemalloc > ---

[jira] [Updated] (HUDI-2927) Experiment running Hudi services using jemalloc

2021-12-03 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin updated HUDI-2927: -- Priority: Minor (was: Major) > Experiment running Hudi services using jemalloc > --

[jira] [Commented] (HUDI-2927) Experiment running Hudi services using jemalloc

2021-12-03 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17453191#comment-17453191 ] Alexey Kudinkin commented on HUDI-2927: --- Some examples of using jemalloc in producti

[jira] [Commented] (HUDI-2151) Make performant out-of-box configs

2021-12-03 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17453199#comment-17453199 ] Raymond Xu commented on HUDI-2151: -- Codec change: change default codec to SNAPPY instead

[GitHub] [hudi] zhedoubushishi opened a new pull request #4209: [MINOR] Use maven-shade-plugin version for hudi-timeline-server-bundle from main pom.xml

2021-12-03 Thread GitBox
zhedoubushishi opened a new pull request #4209: URL: https://github.com/apache/hudi/pull/4209 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the

[jira] [Created] (HUDI-2928) Evaluate rebasing Hudi's default compression from Gzip to Snappy

2021-12-03 Thread Alexey Kudinkin (Jira)
Alexey Kudinkin created HUDI-2928: - Summary: Evaluate rebasing Hudi's default compression from Gzip to Snappy Key: HUDI-2928 URL: https://issues.apache.org/jira/browse/HUDI-2928 Project: Apache Hudi

[jira] [Assigned] (HUDI-2928) Evaluate rebasing Hudi's default compression from Gzip to Snappy

2021-12-03 Thread Alexey Kudinkin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Kudinkin reassigned HUDI-2928: - Assignee: Alexey Kudinkin > Evaluate rebasing Hudi's default compression from Gzip to Sna

[GitHub] [hudi] hudi-bot commented on pull request #4209: [MINOR] Use maven-shade-plugin version for hudi-timeline-server-bundle from main pom.xml

2021-12-03 Thread GitBox
hudi-bot commented on pull request #4209: URL: https://github.com/apache/hudi/pull/4209#issuecomment-985820260 ## CI report: * f7c04a2ba6bac9b0eb171f991fa7c92751bfdb00 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` r

  1   2   3   >