[
https://issues.apache.org/jira/browse/FLINK-36039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
RocMarshal updated FLINK-36039:
-------------------------------
Description:
Currently, the autoscaler generates a large amount of historical data for event
handlers. As the system runs for a long time, the volume of historical data
will continue to grow. It is necessary to support automatic cleanup of data
within a fixed period.
Based on the creation time timestamp, the following approach for cleaning up
historical data might be a way:
* Introduce the parameter {{autoscaler.standalone.jdbc-event-handler.ttl}}
*
** Type: Duration
** Default value: 90 days
** Setting it to 0 means disabling the cleanup functionality.
* In the {{JdbcAutoScalerEventHandler}} constructor, introduce a scheduled
job. Also, add an internal interface method {{close}} for
{{AutoScalerEventHandler & JobAutoScaler}} to stop and clean up related logic.
* Cleanup logic:
#
## Query the messages with {{create_time}} less than {{(currentTime - ttl)}}
and find the maximum {{maxId}} in this collection.
## Delete 4096 messages at a time from the collection with IDs less than
{{{}maxId{}}}.
## Wait 10 ms between each deletion until the cleanup is complete.
## Scan and delete expired data daily
was:
Currently, the autoscaler generates a large amount of historical data for event
handlers. As the system runs for a long time, the volume of historical data
will continue to grow. It is necessary to support automatic cleanup of data
within a fixed period.
Based on the creation time timestamp, the following approach for cleaning up
historical data might be a way:
* Introduce the parameter {{autoscaler.standalone.jdbc-event-handler.ttl}}
*
** Type: Duration
** Default value: 90 days
** Setting it to 0 means disabling the cleanup functionality.
* In the {{JdbcAutoScalerEventHandler}} constructor, introduce a scheduled
job. Also, add an internal interface method {{close}} for
{{AutoScalerEventHandler}} to stop and clean up related logic.
* Cleanup logic:
#
## Query the messages with {{create_time}} less than {{(currentTime - ttl)}}
and find the maximum {{maxId}} in this collection.
## Delete 4096 messages at a time from the collection with IDs less than
{{{}maxId{}}}.
## Wait 10 ms between each deletion until the cleanup is complete.
## Scan and delete expired data daily
> Support clean historical event handler records in JDBC event handler
> --------------------------------------------------------------------
>
> Key: FLINK-36039
> URL: https://issues.apache.org/jira/browse/FLINK-36039
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler
> Reporter: RocMarshal
> Assignee: RocMarshal
> Priority: Minor
>
> Currently, the autoscaler generates a large amount of historical data for
> event handlers. As the system runs for a long time, the volume of historical
> data will continue to grow. It is necessary to support automatic cleanup of
> data within a fixed period.
> Based on the creation time timestamp, the following approach for cleaning up
> historical data might be a way:
> * Introduce the parameter {{autoscaler.standalone.jdbc-event-handler.ttl}}
> *
> ** Type: Duration
> ** Default value: 90 days
> ** Setting it to 0 means disabling the cleanup functionality.
> * In the {{JdbcAutoScalerEventHandler}} constructor, introduce a scheduled
> job. Also, add an internal interface method {{close}} for
> {{AutoScalerEventHandler & JobAutoScaler}} to stop and clean up related
> logic.
> * Cleanup logic:
> #
> ## Query the messages with {{create_time}} less than {{(currentTime - ttl)}}
> and find the maximum {{maxId}} in this collection.
> ## Delete 4096 messages at a time from the collection with IDs less than
> {{{}maxId{}}}.
> ## Wait 10 ms between each deletion until the cleanup is complete.
> ## Scan and delete expired data daily
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
