[
https://issues.apache.org/jira/browse/FLINK-28035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jane Chan updated FLINK-28035:
------------------------------
Description:
For an ordinary read-write job, the scan will check the numBuckets read from
manifests against the current numBuckets, to avoid data corruption. See
FLINK-27316.
However, this can be improved as follows.
* If no new writes happen after changing the bucket number, the reads should
not be blocked.
* For rescale overwrite, we should support scan as the old bucket num, rescale
and commit as the new bucket num.
* The streaming job can be suspended and recovered from the rescaled data
layout.
was:
For an ordinary read-write job, the scan will check the numBuckets read from
manifests against the current numBuckets, to avoid data corruption. See
FLINK-27316.
However, this can be improved as follows.
* If no new writes happen after changing the bucket number, the reads should
not be blocked.
* For rescale overwrite, we should support scan as the old bucket num, rescale
and commit as the new bucket num.
* The streaming job can be suspended, and recovered from the rescaled data
layout.
> Support rescale overwrite
> -------------------------
>
> Key: FLINK-28035
> URL: https://issues.apache.org/jira/browse/FLINK-28035
> Project: Flink
> Issue Type: Sub-task
> Components: Table Store
> Affects Versions: table-store-0.2.0
> Reporter: Jane Chan
> Priority: Major
> Labels: pull-request-available
> Fix For: table-store-0.2.0
>
>
> For an ordinary read-write job, the scan will check the numBuckets read from
> manifests against the current numBuckets, to avoid data corruption. See
> FLINK-27316.
>
> However, this can be improved as follows.
> * If no new writes happen after changing the bucket number, the reads should
> not be blocked.
> * For rescale overwrite, we should support scan as the old bucket num,
> rescale and commit as the new bucket num.
> * The streaming job can be suspended and recovered from the rescaled data
> layout.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
