[
https://issues.apache.org/jira/browse/FLINK-27557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17539352#comment-17539352
]
Jane Chan edited comment on FLINK-27557 at 5/20/22 3:18 AM:
------------------------------------------------------------
{quote}Can we just use overwrite?
* overwrite specific manifest entries.
* overwrite whole table or some whole partition. (eg: when rescale in
compaction){quote}
Overwrite means the writer cannot accept the specified manifest entries as
restored files, so how to pass them to FileStoreCommit as compact before(mark
as delete) is a new question. Beyond that, overwrite corresponds to the commit
kind "OVERWRITE", but it should be more suitable to use "COMPACT" in this
situation.
I rethink it, and maybe we don't need to generate the new files for the
non-rescale compaction.
* The simplest way is to build the levels from the restored files, don't sink
records to MemTable, and submit compaction when precommit is invoked.
* In the future, maybe we can introduce a new dedicated compaction source to
restore LSM, perform compaction and commit, which avoids the shuffle cost.
was (Author: qingyue):
{quote}Can we just use overwrite?
* overwrite specific manifest entries.
* overwrite whole table or some whole partition. (eg: when rescale in
compaction){quote}
Overwrite means the writer cannot accept the specified manifest entries as
restored files, so how to pass them to FileStoreCommit as compact before(mark
as delete) is a new question. Beyond that, overwrite corresponds to the commit
kind "OVERWRITE", but it should be more suitable to use "COMPACT" in this
situation.
I rethink it and maybe we don't need to generate the new files for the normal
compaction.
* The simplest way is to build the levels from the restored files, and don't
sink record to MemTable, and submit compaction when precommit is invoked.
* In the future, maybe we can introduce a new dedicated compaction source to
restore LSM, perform compaction and commit, which avoids the shuffle cost.
> Let FileStoreWrite accept pre-planned manifest entries
> ------------------------------------------------------
>
> Key: FLINK-27557
> URL: https://issues.apache.org/jira/browse/FLINK-27557
> Project: Flink
> Issue Type: Sub-task
> Components: Table Store
> Affects Versions: table-store-0.2.0
> Reporter: Jane Chan
> Priority: Major
> Fix For: table-store-0.2.0
>
> Attachments: normal-compaction.png, optimized-normal-compaction.png,
> rescale-bucket-compaction.png
>
>
> Currently, FileStoreWrite will scan and plan files when creating a non-empty
> writer. We should also create a non-empty writer for non-rescale compaction
> cases but use the pre-planned manifest entries instead.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
