[
https://issues.apache.org/jira/browse/SPARK-47013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-47013:
----------------------------------
Issue Type: Improvement (was: Documentation)
> Promote `spark.sql.streaming.minBatchesToRetain` to a public config
> -------------------------------------------------------------------
>
> Key: SPARK-47013
> URL: https://issues.apache.org/jira/browse/SPARK-47013
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 4.3.0
> Reporter: Lingeshwaran Radhakrishnan
> Assignee: Brij Raj Kishore
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> Add the config spark.sql.streaming.minBatchesToRetain to the [streaming
> docs|https://spark.apache.org/docs/latest/configuration.html#spark-streaming]
> page which basically controls the minimum number of batches that must be
> retained and made recoverable.
> This would also help control the lifecycle of the state files held in the
> checkpoint folder i.e, State files are cleaned up based on the config
> spark.sql.streaming.minBatchesToRetain
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
