[jira] [Assigned] (HUDI-6401) should not throw exception when create marker file for log file

2023-06-16 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan reassigned HUDI-6401: --- Assignee: ZiyueGuan > should not throw exception when create marker file for log file > -

[jira] [Updated] (HUDI-6401) should not throw exception when create marker file for log file

2023-06-16 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan updated HUDI-6401: Description: when spark task failed or speculation enabled, different task will create marker file for the s

[jira] [Created] (HUDI-6401) should not throw exception when create marker file for log file

2023-06-16 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-6401: --- Summary: should not throw exception when create marker file for log file Key: HUDI-6401 URL: https://issues.apache.org/jira/browse/HUDI-6401 Project: Apache Hudi Issu

[jira] [Commented] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index

2023-05-28 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17726947#comment-17726947 ] ZiyueGuan commented on HUDI-3026: - This bug is fixed by HUDI-1517. In HUDI-1517, we allow

[jira] [Created] (HUDI-4965) automatically adapt COMMITS_ARCHIVAL_BATCH_SIZE

2022-10-01 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-4965: --- Summary: automatically adapt COMMITS_ARCHIVAL_BATCH_SIZE Key: HUDI-4965 URL: https://issues.apache.org/jira/browse/HUDI-4965 Project: Apache Hudi Issue Type: Improveme

[jira] [Created] (HUDI-4912) Make write status idempotent

2022-09-24 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-4912: --- Summary: Make write status idempotent Key: HUDI-4912 URL: https://issues.apache.org/jira/browse/HUDI-4912 Project: Apache Hudi Issue Type: Bug Components: in

[jira] [Commented] (HUDI-4055) use loop replace recursive call in ratelimiter

2022-05-08 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533485#comment-17533485 ] ZiyueGuan commented on HUDI-4055: - https://github.com/apache/hudi/pull/5530 > use loop re

[jira] [Created] (HUDI-4055) use loop replace recursive call in ratelimiter

2022-05-06 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-4055: --- Summary: use loop replace recursive call in ratelimiter Key: HUDI-4055 URL: https://issues.apache.org/jira/browse/HUDI-4055 Project: Apache Hudi Issue Type: Bug

[jira] [Updated] (HUDI-3694) Not use magic number of next block to determine current log block

2022-03-23 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan updated HUDI-3694: Description: HoodieLogFileReader use magic number of next log block to determine if current log block is cor

[jira] [Created] (HUDI-3694) Not use magic number of next block to determine current log block

2022-03-23 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-3694: --- Summary: Not use magic number of next block to determine current log block Key: HUDI-3694 URL: https://issues.apache.org/jira/browse/HUDI-3694 Project: Apache Hudi Is

[jira] [Commented] (HUDI-1517) Create marker file for every log file

2022-02-20 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17495169#comment-17495169 ] ZiyueGuan commented on HUDI-1517: - Hi [~shivnarayan],  Do you have a plan to pick this up

[jira] [Commented] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index

2022-01-05 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469656#comment-17469656 ] ZiyueGuan commented on HUDI-3026: - Thanks for your kind explanation. I have few experience

[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index

2022-01-05 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan updated HUDI-3026: Description: Problem: a same key may occur in two file group when Hbase index is used. These two file group

[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index

2021-12-26 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan updated HUDI-3026: Description: Problem: a same key may occur in two file group when Hbase index is used. These two file group

[jira] [Assigned] (HUDI-2917) Rollback may be incorrect for canIndexLogFile index

2021-12-25 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan reassigned HUDI-2917: --- Assignee: ZiyueGuan > Rollback may be incorrect for canIndexLogFile index > -

[jira] [Assigned] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index

2021-12-19 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan reassigned HUDI-3026: --- Assignee: ZiyueGuan > HoodieAppendhandle may result in duplicate key for hbase index > --

[jira] [Created] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index

2021-12-15 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-3026: --- Summary: HoodieAppendhandle may result in duplicate key for hbase index Key: HUDI-3026 URL: https://issues.apache.org/jira/browse/HUDI-3026 Project: Apache Hudi Issue

[jira] [Closed] (HUDI-2400) Allow timeline server correctly sync when concurrent write to timeline

2021-12-11 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan closed HUDI-2400. --- Resolution: Duplicate > Allow timeline server correctly sync when concurrent write to timeline > -

[jira] [Commented] (HUDI-2400) Allow timeline server correctly sync when concurrent write to timeline

2021-12-11 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457811#comment-17457811 ] ZiyueGuan commented on HUDI-2400: - Duplicate with https://issues.apache.org/jira/browse/HU

[jira] [Commented] (HUDI-2761) IllegalArgException from timeline server when serving getLastestBaseFiles with multi-writer

2021-12-11 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457810#comment-17457810 ] ZiyueGuan commented on HUDI-2761: - Close 2400 as it seems to be the same problem with this

[jira] [Updated] (HUDI-2875) Concurrent call to HoodieMergeHandler cause parquet corruption

2021-12-10 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan updated HUDI-2875: Description: Problem: Some corrupted parquet files are generated and exceptions will be thrown when read.

[jira] [Updated] (HUDI-2875) Concurrent call to HoodieMergeHandler cause parquet corruption

2021-12-10 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan updated HUDI-2875: Description: Problem: Some corrupted parquet files are generated and exceptions will be thrown when read.

[jira] [Created] (HUDI-2917) Rollback may be incorrect for canIndexLogFile index

2021-12-02 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-2917: --- Summary: Rollback may be incorrect for canIndexLogFile index Key: HUDI-2917 URL: https://issues.apache.org/jira/browse/HUDI-2917 Project: Apache Hudi Issue Type: Bug

[jira] [Created] (HUDI-2875) Concurrent call to HoodieMergeHandler cause parquet corruption

2021-11-27 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-2875: --- Summary: Concurrent call to HoodieMergeHandler cause parquet corruption Key: HUDI-2875 URL: https://issues.apache.org/jira/browse/HUDI-2875 Project: Apache Hudi Issue

[jira] [Assigned] (HUDI-2875) Concurrent call to HoodieMergeHandler cause parquet corruption

2021-11-27 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan reassigned HUDI-2875: --- Assignee: ZiyueGuan > Concurrent call to HoodieMergeHandler cause parquet corruption > --

[jira] [Closed] (HUDI-2771) Handle FileNotExist exception in parquet Utils

2021-11-18 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan closed HUDI-2771. --- Resolution: Invalid > Handle FileNotExist exception in parquet Utils > ---

[jira] [Assigned] (HUDI-2771) Handle FileNotExist exception in parquet Utils

2021-11-16 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan reassigned HUDI-2771: --- Assignee: ZiyueGuan > Handle FileNotExist exception in parquet Utils > --

[jira] [Created] (HUDI-2771) Handle FileNotExist exception in parquet Utils

2021-11-16 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-2771: --- Summary: Handle FileNotExist exception in parquet Utils Key: HUDI-2771 URL: https://issues.apache.org/jira/browse/HUDI-2771 Project: Apache Hudi Issue Type: Bug

[jira] [Commented] (HUDI-2031) JVM occasionally crashes during compaction when spark speculative execution is enabled

2021-11-03 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17437937#comment-17437937 ] ZiyueGuan commented on HUDI-2031: - Any guys know the root cause of this problem? Curious a

[jira] [Updated] (HUDI-2665) Overflow of DataOutputStream may lead to corrupted log block

2021-11-02 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan updated HUDI-2665: Priority: Minor (was: Major) > Overflow of DataOutputStream may lead to corrupted log block > -

[jira] [Assigned] (HUDI-2665) Overflow of DataOutputStream may lead to corrupted log block

2021-11-02 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan reassigned HUDI-2665: --- Assignee: ZiyueGuan > Overflow of DataOutputStream may lead to corrupted log block >

[jira] [Created] (HUDI-2665) Overflow of DataOutputStream may lead to corrupted log block

2021-11-02 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-2665: --- Summary: Overflow of DataOutputStream may lead to corrupted log block Key: HUDI-2665 URL: https://issues.apache.org/jira/browse/HUDI-2665 Project: Apache Hudi Issue T

[jira] [Created] (HUDI-2400) Allow timeline server correctly sync when concurrent write to timeline

2021-09-05 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-2400: --- Summary: Allow timeline server correctly sync when concurrent write to timeline Key: HUDI-2400 URL: https://issues.apache.org/jira/browse/HUDI-2400 Project: Apache Hudi

[jira] [Created] (HUDI-1875) Improve perf of MOR table upsert based on HDFS

2021-05-04 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-1875: --- Summary: Improve perf of MOR table upsert based on HDFS Key: HUDI-1875 URL: https://issues.apache.org/jira/browse/HUDI-1875 Project: Apache Hudi Issue Type: Improvemen

[jira] [Updated] (HUDI-1796) allow ExternalSpillMap use accurate payload size rather than estimated

2021-04-15 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan updated HUDI-1796: Description: Situation: In ExternalSpillMap, we need to control the amount of data in memory map to avoid O

[jira] [Closed] (HUDI-1795) allow ExternalSpillMap use accurate payload size rather than estimated

2021-04-14 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan closed HUDI-1795. --- Resolution: Duplicate > allow ExternalSpillMap use accurate payload size rather than estimated > -

[jira] [Created] (HUDI-1796) allow ExternalSpillMap use accurate payload size rather than estimated

2021-04-14 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-1796: --- Summary: allow ExternalSpillMap use accurate payload size rather than estimated Key: HUDI-1796 URL: https://issues.apache.org/jira/browse/HUDI-1796 Project: Apache Hudi

[jira] [Created] (HUDI-1795) allow ExternalSpillMap use accurate payload size rather than estimated

2021-04-14 Thread ZiyueGuan (Jira)
ZiyueGuan created HUDI-1795: --- Summary: allow ExternalSpillMap use accurate payload size rather than estimated Key: HUDI-1795 URL: https://issues.apache.org/jira/browse/HUDI-1795 Project: Apache Hudi