JingGe commented on code in PR #24394: URL: https://github.com/apache/flink/pull/24394#discussion_r1514281015
########## docs/content.zh/release-notes/flink-1.19.md: ########## @@ -0,0 +1,354 @@ +--- +title: "Release Notes - Flink 1.19" +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 1.19 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 1.18 and Flink 1.19. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.19. + +## Dependency upgrades + +#### Drop support for python 3.7 + +##### [FLINK-33029](https://issues.apache.org/jira/browse/FLINK-33029) + +#### Add support for python 3.11 + +##### [FLINK-33030](https://issues.apache.org/jira/browse/FLINK-33030) + +## Checkpoints + +#### Deprecate RestoreMode#LEGACY + +##### [FLINK-34190](https://issues.apache.org/jira/browse/FLINK-34190) + +`RestoreMode#LEGACY` is deprecated. Please use `RestoreMode#CLAIM` or `RestoreMode#NO_CLAIM` mode +instead to get a clear state file ownership when restoring. + +#### CheckpointsCleaner clean individual checkpoint states in parallel + +##### [FLINK-33090](https://issues.apache.org/jira/browse/FLINK-33090) + +Now when disposing of no longer needed checkpoints, every state handle/state file will be disposed +in parallel by the ioExecutor, vastly improving the disposing speed of a single checkpoint (for +large checkpoints the disposal time can be improved from 10 minutes to < 1 minute) . The old +behavior can be restored by setting `state.checkpoint.cleaner.parallel-mode` to false. + +#### Support using larger checkpointing interval when source is processing backlog + +##### [FLINK-32514](https://issues.apache.org/jira/browse/FLINK-32514) + +`ProcessingBacklog` is introduced to demonstrate whether a record should be processed with low latency +or high throughput. `ProcessingBacklog` can be set by source operators, and can be used to change the +checkpoint internal of a job during runtime. Review Comment: ```suggestion checkpoint interval of a job during runtime. ``` ########## docs/content.zh/release-notes/flink-1.19.md: ########## @@ -0,0 +1,354 @@ +--- +title: "Release Notes - Flink 1.19" +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 1.19 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 1.18 and Flink 1.19. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.19. + +## Dependency upgrades + +#### Drop support for python 3.7 + +##### [FLINK-33029](https://issues.apache.org/jira/browse/FLINK-33029) + +#### Add support for python 3.11 + +##### [FLINK-33030](https://issues.apache.org/jira/browse/FLINK-33030) + +## Checkpoints + +#### Deprecate RestoreMode#LEGACY + +##### [FLINK-34190](https://issues.apache.org/jira/browse/FLINK-34190) + +`RestoreMode#LEGACY` is deprecated. Please use `RestoreMode#CLAIM` or `RestoreMode#NO_CLAIM` mode +instead to get a clear state file ownership when restoring. + +#### CheckpointsCleaner clean individual checkpoint states in parallel + +##### [FLINK-33090](https://issues.apache.org/jira/browse/FLINK-33090) + +Now when disposing of no longer needed checkpoints, every state handle/state file will be disposed +in parallel by the ioExecutor, vastly improving the disposing speed of a single checkpoint (for +large checkpoints the disposal time can be improved from 10 minutes to < 1 minute) . The old Review Comment: ```suggestion large checkpoints, the disposal time can be improved from 10 minutes to < 1 minute) . The old ``` ########## docs/content.zh/release-notes/flink-1.19.md: ########## @@ -0,0 +1,354 @@ +--- +title: "Release Notes - Flink 1.19" +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 1.19 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 1.18 and Flink 1.19. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.19. + +## Dependency upgrades + +#### Drop support for python 3.7 + +##### [FLINK-33029](https://issues.apache.org/jira/browse/FLINK-33029) + +#### Add support for python 3.11 + +##### [FLINK-33030](https://issues.apache.org/jira/browse/FLINK-33030) + +## Checkpoints + +#### Deprecate RestoreMode#LEGACY + +##### [FLINK-34190](https://issues.apache.org/jira/browse/FLINK-34190) + +`RestoreMode#LEGACY` is deprecated. Please use `RestoreMode#CLAIM` or `RestoreMode#NO_CLAIM` mode +instead to get a clear state file ownership when restoring. + +#### CheckpointsCleaner clean individual checkpoint states in parallel + +##### [FLINK-33090](https://issues.apache.org/jira/browse/FLINK-33090) + +Now when disposing of no longer needed checkpoints, every state handle/state file will be disposed +in parallel by the ioExecutor, vastly improving the disposing speed of a single checkpoint (for +large checkpoints the disposal time can be improved from 10 minutes to < 1 minute) . The old +behavior can be restored by setting `state.checkpoint.cleaner.parallel-mode` to false. + +#### Support using larger checkpointing interval when source is processing backlog + +##### [FLINK-32514](https://issues.apache.org/jira/browse/FLINK-32514) + +`ProcessingBacklog` is introduced to demonstrate whether a record should be processed with low latency +or high throughput. `ProcessingBacklog` can be set by source operators, and can be used to change the Review Comment: ```suggestion or high throughput. `ProcessingBacklog` can be set by source operators and can be used to change the ``` ########## docs/content.zh/release-notes/flink-1.19.md: ########## @@ -0,0 +1,354 @@ +--- +title: "Release Notes - Flink 1.19" +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 1.19 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 1.18 and Flink 1.19. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.19. + +## Dependency upgrades + +#### Drop support for python 3.7 + +##### [FLINK-33029](https://issues.apache.org/jira/browse/FLINK-33029) + +#### Add support for python 3.11 + +##### [FLINK-33030](https://issues.apache.org/jira/browse/FLINK-33030) + +## Checkpoints + +#### Deprecate RestoreMode#LEGACY + +##### [FLINK-34190](https://issues.apache.org/jira/browse/FLINK-34190) + +`RestoreMode#LEGACY` is deprecated. Please use `RestoreMode#CLAIM` or `RestoreMode#NO_CLAIM` mode +instead to get a clear state file ownership when restoring. + +#### CheckpointsCleaner clean individual checkpoint states in parallel + +##### [FLINK-33090](https://issues.apache.org/jira/browse/FLINK-33090) + +Now when disposing of no longer needed checkpoints, every state handle/state file will be disposed +in parallel by the ioExecutor, vastly improving the disposing speed of a single checkpoint (for +large checkpoints the disposal time can be improved from 10 minutes to < 1 minute) . The old +behavior can be restored by setting `state.checkpoint.cleaner.parallel-mode` to false. + +#### Support using larger checkpointing interval when source is processing backlog + +##### [FLINK-32514](https://issues.apache.org/jira/browse/FLINK-32514) + +`ProcessingBacklog` is introduced to demonstrate whether a record should be processed with low latency +or high throughput. `ProcessingBacklog` can be set by source operators, and can be used to change the +checkpoint internal of a job during runtime. + +#### Allow triggering Checkpoints through command line client + +##### [FLINK-6755](https://issues.apache.org/jira/browse/FLINK-6755) + +The command line interface supports triggering a checkpoint manually. Usage: +``` +./bin/flink checkpoint $JOB_ID [-full] +``` +By specifying the '-full' option, a full checkpoint is triggered. Otherwise an incremental +checkpoint is triggered if the job is configured to take incremental ones periodically. + + +## Runtime & Coordination + +#### Migrate TypeserializeSnapshotr#resolveSchemaCompatibility + +##### [FLINK-30613](https://issues.apache.org/jira/browse/FLINK-30613) + +In Flink 1.19, the old method of resolving schema compatibility has been deprecated and the new one +is introduced. See [FLIP-263](https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility?src=contextnavpagetreemode) for more details. +Please migrate to the new method following [link](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/custom_serialization/#migrating-from-deprecated-typeserializersnapshotresolveschemacompatibilityt). + +#### Deprecate old serialization config methods and options + +##### [FLINK-34122](https://issues.apache.org/jira/browse/FLINK-34122) + +Configuring serialization behavior through hard codes is deprecated, because you need to modify the +codes when upgrading the job version. You should configure this via options +`pipeline.serialization-config`, `pipeline.force-avro`, `pipeline.force-kryo`, and `pipeline.generic-types`. +Registration of instance-level serializers is deprecated, using class-level serializers instead. + +#### Migrate string configuration key to ConfigOption + +##### [FLINK-34079](https://issues.apache.org/jira/browse/FLINK-34079) + +We have deprecated all setXxx and getXxx methods except `getString(String key, String defaultValue)` +and `setString(String key, String value)`, such as: `setInteger`, `setLong`, `getInteger` and `getLong` etc. +We strongly recommend users and developers use get and set methods directly. +In addition, we recommend users to use ConfigOption instead of string as key. + +#### Support System out and err to be redirected to LOG or discarded + +##### [FLINK-33625](https://issues.apache.org/jira/browse/FLINK-33625) + +`System.out` and `System.err` output the content to the `taskmanager.out` and `taskmanager.err` files. +In a production environment, if flink users use them to print a lot of content, the limits of yarn +or kubernetes may be exceeded, eventually causing the TaskManager to be killed. Flink supports +redirecting the `System.out` and `System.err` to the log file, and the log file can be rolled to +avoid unlimited disk usage. + +#### Support standard YAML for FLINK configuration + +##### [FLINK-33297](https://issues.apache.org/jira/browse/FLINK-33297) + +Starting with Flink 1.19, Flink has officially introduced full support for the standard YAML 1.2 +syntax ([FLIP-366](https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration?src=contextnavpagetreemode)). The default configuration file has been changed to `config.yaml` and placed in the +`conf/` directory. Users should directly modify this file to configure Flink. +If users want to use the legacy configuration file `flink-conf.yaml`, users just need to copy this +file into the `conf/` directory. Once the legacy configuration file `flink-conf.yaml` is detected, +Flink will prioritize using it as the configuration file.And in the upcoming Flink 2.0, the +`flink-conf.yaml` configuration file will no longer work. +More details could be found at [flink-configuration-file](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#flink-configuration-file). + +#### Add config options for administrator JVM options + +##### [FLINK-33221](https://issues.apache.org/jira/browse/FLINK-33221) + +A set of administrator JVM options are available, which prepend the user-set extra JVM options for +platform-wide JVM tuning. + +#### Flink Job stuck in suspend state after losing leadership in HA Mode + +##### [FLINK-34007](https://issues.apache.org/jira/browse/FLINK-34007) + +Fixes a bug where the leader election wasn't able to pick up leadership again after renewing the +lease token caused a leadership loss. This required `fabric8io:kubernetes-client` to be upgraded +from v6.6.2 to v6.9.0. + +#### Support dynamic source parallelism inference for batch jobs + +##### [FLINK-33768](https://issues.apache.org/jira/browse/FLINK-33768) + +In Flink 1.19, we have supported dynamic source parallelism inference for batch jobs, which allows +source connectors to dynamically infer the parallelism based on the actual amount of data to consume. +This feature is a significant improvement over previous versions, which only assigned a fixed default +parallelism to source vertices. +Source connectors need to implement the inference interface to enable dynamic parallelism inference. +Currently, the FileSource connector has already been developed with this functionality in place. +Additionally, the configuration `execution.batch.adaptive.auto-parallelism.default-source-parallelism` +will be used as the upper bound of source parallelism inference. And now it will not default to 1. Review Comment: ```suggestion will be used as the upper bound of source parallelism inference. And now it will be set to 1 by default. ``` ########## docs/content.zh/release-notes/flink-1.19.md: ########## @@ -0,0 +1,354 @@ +--- +title: "Release Notes - Flink 1.19" +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 1.19 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 1.18 and Flink 1.19. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.19. + +## Dependency upgrades + +#### Drop support for python 3.7 + +##### [FLINK-33029](https://issues.apache.org/jira/browse/FLINK-33029) + +#### Add support for python 3.11 + +##### [FLINK-33030](https://issues.apache.org/jira/browse/FLINK-33030) + +## Checkpoints + +#### Deprecate RestoreMode#LEGACY + +##### [FLINK-34190](https://issues.apache.org/jira/browse/FLINK-34190) + +`RestoreMode#LEGACY` is deprecated. Please use `RestoreMode#CLAIM` or `RestoreMode#NO_CLAIM` mode +instead to get a clear state file ownership when restoring. + +#### CheckpointsCleaner clean individual checkpoint states in parallel + +##### [FLINK-33090](https://issues.apache.org/jira/browse/FLINK-33090) + +Now when disposing of no longer needed checkpoints, every state handle/state file will be disposed +in parallel by the ioExecutor, vastly improving the disposing speed of a single checkpoint (for +large checkpoints the disposal time can be improved from 10 minutes to < 1 minute) . The old +behavior can be restored by setting `state.checkpoint.cleaner.parallel-mode` to false. + +#### Support using larger checkpointing interval when source is processing backlog + +##### [FLINK-32514](https://issues.apache.org/jira/browse/FLINK-32514) + +`ProcessingBacklog` is introduced to demonstrate whether a record should be processed with low latency +or high throughput. `ProcessingBacklog` can be set by source operators, and can be used to change the +checkpoint internal of a job during runtime. + +#### Allow triggering Checkpoints through command line client + +##### [FLINK-6755](https://issues.apache.org/jira/browse/FLINK-6755) + +The command line interface supports triggering a checkpoint manually. Usage: +``` +./bin/flink checkpoint $JOB_ID [-full] +``` +By specifying the '-full' option, a full checkpoint is triggered. Otherwise an incremental +checkpoint is triggered if the job is configured to take incremental ones periodically. + + +## Runtime & Coordination + +#### Migrate TypeserializeSnapshotr#resolveSchemaCompatibility + +##### [FLINK-30613](https://issues.apache.org/jira/browse/FLINK-30613) + +In Flink 1.19, the old method of resolving schema compatibility has been deprecated and the new one +is introduced. See [FLIP-263](https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility?src=contextnavpagetreemode) for more details. +Please migrate to the new method following [link](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/custom_serialization/#migrating-from-deprecated-typeserializersnapshotresolveschemacompatibilityt). + +#### Deprecate old serialization config methods and options + +##### [FLINK-34122](https://issues.apache.org/jira/browse/FLINK-34122) + +Configuring serialization behavior through hard codes is deprecated, because you need to modify the +codes when upgrading the job version. You should configure this via options +`pipeline.serialization-config`, `pipeline.force-avro`, `pipeline.force-kryo`, and `pipeline.generic-types`. +Registration of instance-level serializers is deprecated, using class-level serializers instead. + +#### Migrate string configuration key to ConfigOption + +##### [FLINK-34079](https://issues.apache.org/jira/browse/FLINK-34079) + +We have deprecated all setXxx and getXxx methods except `getString(String key, String defaultValue)` +and `setString(String key, String value)`, such as: `setInteger`, `setLong`, `getInteger` and `getLong` etc. +We strongly recommend users and developers use get and set methods directly. +In addition, we recommend users to use ConfigOption instead of string as key. + +#### Support System out and err to be redirected to LOG or discarded + +##### [FLINK-33625](https://issues.apache.org/jira/browse/FLINK-33625) + +`System.out` and `System.err` output the content to the `taskmanager.out` and `taskmanager.err` files. +In a production environment, if flink users use them to print a lot of content, the limits of yarn +or kubernetes may be exceeded, eventually causing the TaskManager to be killed. Flink supports +redirecting the `System.out` and `System.err` to the log file, and the log file can be rolled to +avoid unlimited disk usage. + +#### Support standard YAML for FLINK configuration + +##### [FLINK-33297](https://issues.apache.org/jira/browse/FLINK-33297) + +Starting with Flink 1.19, Flink has officially introduced full support for the standard YAML 1.2 +syntax ([FLIP-366](https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration?src=contextnavpagetreemode)). The default configuration file has been changed to `config.yaml` and placed in the +`conf/` directory. Users should directly modify this file to configure Flink. +If users want to use the legacy configuration file `flink-conf.yaml`, users just need to copy this +file into the `conf/` directory. Once the legacy configuration file `flink-conf.yaml` is detected, +Flink will prioritize using it as the configuration file.And in the upcoming Flink 2.0, the +`flink-conf.yaml` configuration file will no longer work. +More details could be found at [flink-configuration-file](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#flink-configuration-file). + +#### Add config options for administrator JVM options + +##### [FLINK-33221](https://issues.apache.org/jira/browse/FLINK-33221) + +A set of administrator JVM options are available, which prepend the user-set extra JVM options for +platform-wide JVM tuning. + +#### Flink Job stuck in suspend state after losing leadership in HA Mode + +##### [FLINK-34007](https://issues.apache.org/jira/browse/FLINK-34007) + +Fixes a bug where the leader election wasn't able to pick up leadership again after renewing the +lease token caused a leadership loss. This required `fabric8io:kubernetes-client` to be upgraded +from v6.6.2 to v6.9.0. + +#### Support dynamic source parallelism inference for batch jobs + +##### [FLINK-33768](https://issues.apache.org/jira/browse/FLINK-33768) + +In Flink 1.19, we have supported dynamic source parallelism inference for batch jobs, which allows +source connectors to dynamically infer the parallelism based on the actual amount of data to consume. +This feature is a significant improvement over previous versions, which only assigned a fixed default +parallelism to source vertices. +Source connectors need to implement the inference interface to enable dynamic parallelism inference. +Currently, the FileSource connector has already been developed with this functionality in place. +Additionally, the configuration `execution.batch.adaptive.auto-parallelism.default-source-parallelism` +will be used as the upper bound of source parallelism inference. And now it will not default to 1. +Instead, if it is not set, the upper bound of allowed parallelism set via +`execution.batch.adaptive.auto-parallelism.max-parallelism` will be used. If that configuration is Review Comment: ```suggestion `execution.batch.adaptive.auto-parallelism.max-parallelism` will be used instead. If that configuration is ``` ########## docs/content.zh/release-notes/flink-1.19.md: ########## @@ -0,0 +1,354 @@ +--- +title: "Release Notes - Flink 1.19" +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 1.19 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 1.18 and Flink 1.19. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.19. + +## Dependency upgrades + +#### Drop support for python 3.7 + +##### [FLINK-33029](https://issues.apache.org/jira/browse/FLINK-33029) + +#### Add support for python 3.11 + +##### [FLINK-33030](https://issues.apache.org/jira/browse/FLINK-33030) + +## Checkpoints + +#### Deprecate RestoreMode#LEGACY + +##### [FLINK-34190](https://issues.apache.org/jira/browse/FLINK-34190) + +`RestoreMode#LEGACY` is deprecated. Please use `RestoreMode#CLAIM` or `RestoreMode#NO_CLAIM` mode +instead to get a clear state file ownership when restoring. + +#### CheckpointsCleaner clean individual checkpoint states in parallel + +##### [FLINK-33090](https://issues.apache.org/jira/browse/FLINK-33090) + +Now when disposing of no longer needed checkpoints, every state handle/state file will be disposed +in parallel by the ioExecutor, vastly improving the disposing speed of a single checkpoint (for +large checkpoints the disposal time can be improved from 10 minutes to < 1 minute) . The old +behavior can be restored by setting `state.checkpoint.cleaner.parallel-mode` to false. + +#### Support using larger checkpointing interval when source is processing backlog + +##### [FLINK-32514](https://issues.apache.org/jira/browse/FLINK-32514) + +`ProcessingBacklog` is introduced to demonstrate whether a record should be processed with low latency +or high throughput. `ProcessingBacklog` can be set by source operators, and can be used to change the +checkpoint internal of a job during runtime. + +#### Allow triggering Checkpoints through command line client + +##### [FLINK-6755](https://issues.apache.org/jira/browse/FLINK-6755) + +The command line interface supports triggering a checkpoint manually. Usage: +``` +./bin/flink checkpoint $JOB_ID [-full] +``` +By specifying the '-full' option, a full checkpoint is triggered. Otherwise an incremental +checkpoint is triggered if the job is configured to take incremental ones periodically. + + +## Runtime & Coordination + +#### Migrate TypeserializeSnapshotr#resolveSchemaCompatibility + +##### [FLINK-30613](https://issues.apache.org/jira/browse/FLINK-30613) + +In Flink 1.19, the old method of resolving schema compatibility has been deprecated and the new one +is introduced. See [FLIP-263](https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility?src=contextnavpagetreemode) for more details. +Please migrate to the new method following [link](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/custom_serialization/#migrating-from-deprecated-typeserializersnapshotresolveschemacompatibilityt). + +#### Deprecate old serialization config methods and options + +##### [FLINK-34122](https://issues.apache.org/jira/browse/FLINK-34122) + +Configuring serialization behavior through hard codes is deprecated, because you need to modify the +codes when upgrading the job version. You should configure this via options +`pipeline.serialization-config`, `pipeline.force-avro`, `pipeline.force-kryo`, and `pipeline.generic-types`. +Registration of instance-level serializers is deprecated, using class-level serializers instead. + +#### Migrate string configuration key to ConfigOption + +##### [FLINK-34079](https://issues.apache.org/jira/browse/FLINK-34079) + +We have deprecated all setXxx and getXxx methods except `getString(String key, String defaultValue)` +and `setString(String key, String value)`, such as: `setInteger`, `setLong`, `getInteger` and `getLong` etc. +We strongly recommend users and developers use get and set methods directly. Review Comment: What is exactly recommended? This description is confusing. ########## docs/content.zh/release-notes/flink-1.19.md: ########## @@ -0,0 +1,354 @@ +--- +title: "Release Notes - Flink 1.19" +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 1.19 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 1.18 and Flink 1.19. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.19. + +## Dependency upgrades + +#### Drop support for python 3.7 + +##### [FLINK-33029](https://issues.apache.org/jira/browse/FLINK-33029) + +#### Add support for python 3.11 + +##### [FLINK-33030](https://issues.apache.org/jira/browse/FLINK-33030) + +## Checkpoints + +#### Deprecate RestoreMode#LEGACY + +##### [FLINK-34190](https://issues.apache.org/jira/browse/FLINK-34190) + +`RestoreMode#LEGACY` is deprecated. Please use `RestoreMode#CLAIM` or `RestoreMode#NO_CLAIM` mode +instead to get a clear state file ownership when restoring. + +#### CheckpointsCleaner clean individual checkpoint states in parallel + +##### [FLINK-33090](https://issues.apache.org/jira/browse/FLINK-33090) + +Now when disposing of no longer needed checkpoints, every state handle/state file will be disposed +in parallel by the ioExecutor, vastly improving the disposing speed of a single checkpoint (for +large checkpoints the disposal time can be improved from 10 minutes to < 1 minute) . The old +behavior can be restored by setting `state.checkpoint.cleaner.parallel-mode` to false. + +#### Support using larger checkpointing interval when source is processing backlog + +##### [FLINK-32514](https://issues.apache.org/jira/browse/FLINK-32514) + +`ProcessingBacklog` is introduced to demonstrate whether a record should be processed with low latency +or high throughput. `ProcessingBacklog` can be set by source operators, and can be used to change the +checkpoint internal of a job during runtime. + +#### Allow triggering Checkpoints through command line client + +##### [FLINK-6755](https://issues.apache.org/jira/browse/FLINK-6755) + +The command line interface supports triggering a checkpoint manually. Usage: +``` +./bin/flink checkpoint $JOB_ID [-full] +``` +By specifying the '-full' option, a full checkpoint is triggered. Otherwise an incremental +checkpoint is triggered if the job is configured to take incremental ones periodically. + + +## Runtime & Coordination + +#### Migrate TypeserializeSnapshotr#resolveSchemaCompatibility + +##### [FLINK-30613](https://issues.apache.org/jira/browse/FLINK-30613) + +In Flink 1.19, the old method of resolving schema compatibility has been deprecated and the new one +is introduced. See [FLIP-263](https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility?src=contextnavpagetreemode) for more details. +Please migrate to the new method following [link](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/custom_serialization/#migrating-from-deprecated-typeserializersnapshotresolveschemacompatibilityt). + +#### Deprecate old serialization config methods and options + +##### [FLINK-34122](https://issues.apache.org/jira/browse/FLINK-34122) + +Configuring serialization behavior through hard codes is deprecated, because you need to modify the +codes when upgrading the job version. You should configure this via options +`pipeline.serialization-config`, `pipeline.force-avro`, `pipeline.force-kryo`, and `pipeline.generic-types`. +Registration of instance-level serializers is deprecated, using class-level serializers instead. + +#### Migrate string configuration key to ConfigOption + +##### [FLINK-34079](https://issues.apache.org/jira/browse/FLINK-34079) + +We have deprecated all setXxx and getXxx methods except `getString(String key, String defaultValue)` +and `setString(String key, String value)`, such as: `setInteger`, `setLong`, `getInteger` and `getLong` etc. +We strongly recommend users and developers use get and set methods directly. +In addition, we recommend users to use ConfigOption instead of string as key. + +#### Support System out and err to be redirected to LOG or discarded + +##### [FLINK-33625](https://issues.apache.org/jira/browse/FLINK-33625) + +`System.out` and `System.err` output the content to the `taskmanager.out` and `taskmanager.err` files. +In a production environment, if flink users use them to print a lot of content, the limits of yarn +or kubernetes may be exceeded, eventually causing the TaskManager to be killed. Flink supports +redirecting the `System.out` and `System.err` to the log file, and the log file can be rolled to +avoid unlimited disk usage. + +#### Support standard YAML for FLINK configuration + +##### [FLINK-33297](https://issues.apache.org/jira/browse/FLINK-33297) + +Starting with Flink 1.19, Flink has officially introduced full support for the standard YAML 1.2 +syntax ([FLIP-366](https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration?src=contextnavpagetreemode)). The default configuration file has been changed to `config.yaml` and placed in the +`conf/` directory. Users should directly modify this file to configure Flink. +If users want to use the legacy configuration file `flink-conf.yaml`, users just need to copy this Review Comment: ```suggestion If users want to use the legacy configuration file `flink-conf.yaml`, they need to copy this ``` ########## docs/content.zh/release-notes/flink-1.19.md: ########## @@ -0,0 +1,354 @@ +--- +title: "Release Notes - Flink 1.19" +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 1.19 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 1.18 and Flink 1.19. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.19. + +## Dependency upgrades + +#### Drop support for python 3.7 + +##### [FLINK-33029](https://issues.apache.org/jira/browse/FLINK-33029) + +#### Add support for python 3.11 + +##### [FLINK-33030](https://issues.apache.org/jira/browse/FLINK-33030) + +## Checkpoints + +#### Deprecate RestoreMode#LEGACY + +##### [FLINK-34190](https://issues.apache.org/jira/browse/FLINK-34190) + +`RestoreMode#LEGACY` is deprecated. Please use `RestoreMode#CLAIM` or `RestoreMode#NO_CLAIM` mode +instead to get a clear state file ownership when restoring. + +#### CheckpointsCleaner clean individual checkpoint states in parallel + +##### [FLINK-33090](https://issues.apache.org/jira/browse/FLINK-33090) + +Now when disposing of no longer needed checkpoints, every state handle/state file will be disposed +in parallel by the ioExecutor, vastly improving the disposing speed of a single checkpoint (for +large checkpoints the disposal time can be improved from 10 minutes to < 1 minute) . The old +behavior can be restored by setting `state.checkpoint.cleaner.parallel-mode` to false. + +#### Support using larger checkpointing interval when source is processing backlog + +##### [FLINK-32514](https://issues.apache.org/jira/browse/FLINK-32514) + +`ProcessingBacklog` is introduced to demonstrate whether a record should be processed with low latency +or high throughput. `ProcessingBacklog` can be set by source operators, and can be used to change the +checkpoint internal of a job during runtime. + +#### Allow triggering Checkpoints through command line client + +##### [FLINK-6755](https://issues.apache.org/jira/browse/FLINK-6755) + +The command line interface supports triggering a checkpoint manually. Usage: +``` +./bin/flink checkpoint $JOB_ID [-full] +``` +By specifying the '-full' option, a full checkpoint is triggered. Otherwise an incremental +checkpoint is triggered if the job is configured to take incremental ones periodically. + + +## Runtime & Coordination + +#### Migrate TypeserializeSnapshotr#resolveSchemaCompatibility + +##### [FLINK-30613](https://issues.apache.org/jira/browse/FLINK-30613) + +In Flink 1.19, the old method of resolving schema compatibility has been deprecated and the new one +is introduced. See [FLIP-263](https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility?src=contextnavpagetreemode) for more details. +Please migrate to the new method following [link](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/custom_serialization/#migrating-from-deprecated-typeserializersnapshotresolveschemacompatibilityt). + +#### Deprecate old serialization config methods and options + +##### [FLINK-34122](https://issues.apache.org/jira/browse/FLINK-34122) + +Configuring serialization behavior through hard codes is deprecated, because you need to modify the +codes when upgrading the job version. You should configure this via options +`pipeline.serialization-config`, `pipeline.force-avro`, `pipeline.force-kryo`, and `pipeline.generic-types`. +Registration of instance-level serializers is deprecated, using class-level serializers instead. Review Comment: ```suggestion Registration of instance-level serializers is deprecated, using class-level serializers instead. For more information and code examples, please refer to [link](https://cwiki.apache.org/confluence/display/FLINK/FLIP-398:+Improve+Serialization+Configuration+And+Usage+In+Flink). ``` ########## docs/content.zh/release-notes/flink-1.19.md: ########## @@ -0,0 +1,354 @@ +--- +title: "Release Notes - Flink 1.19" +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 1.19 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 1.18 and Flink 1.19. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.19. + +## Dependency upgrades + +#### Drop support for python 3.7 + +##### [FLINK-33029](https://issues.apache.org/jira/browse/FLINK-33029) + +#### Add support for python 3.11 + +##### [FLINK-33030](https://issues.apache.org/jira/browse/FLINK-33030) + +## Checkpoints + +#### Deprecate RestoreMode#LEGACY + +##### [FLINK-34190](https://issues.apache.org/jira/browse/FLINK-34190) + +`RestoreMode#LEGACY` is deprecated. Please use `RestoreMode#CLAIM` or `RestoreMode#NO_CLAIM` mode +instead to get a clear state file ownership when restoring. + +#### CheckpointsCleaner clean individual checkpoint states in parallel + +##### [FLINK-33090](https://issues.apache.org/jira/browse/FLINK-33090) + +Now when disposing of no longer needed checkpoints, every state handle/state file will be disposed +in parallel by the ioExecutor, vastly improving the disposing speed of a single checkpoint (for +large checkpoints the disposal time can be improved from 10 minutes to < 1 minute) . The old +behavior can be restored by setting `state.checkpoint.cleaner.parallel-mode` to false. + +#### Support using larger checkpointing interval when source is processing backlog + +##### [FLINK-32514](https://issues.apache.org/jira/browse/FLINK-32514) + +`ProcessingBacklog` is introduced to demonstrate whether a record should be processed with low latency +or high throughput. `ProcessingBacklog` can be set by source operators, and can be used to change the +checkpoint internal of a job during runtime. + +#### Allow triggering Checkpoints through command line client + +##### [FLINK-6755](https://issues.apache.org/jira/browse/FLINK-6755) + +The command line interface supports triggering a checkpoint manually. Usage: +``` +./bin/flink checkpoint $JOB_ID [-full] +``` +By specifying the '-full' option, a full checkpoint is triggered. Otherwise an incremental +checkpoint is triggered if the job is configured to take incremental ones periodically. + + +## Runtime & Coordination + +#### Migrate TypeserializeSnapshotr#resolveSchemaCompatibility + +##### [FLINK-30613](https://issues.apache.org/jira/browse/FLINK-30613) + +In Flink 1.19, the old method of resolving schema compatibility has been deprecated and the new one +is introduced. See [FLIP-263](https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility?src=contextnavpagetreemode) for more details. +Please migrate to the new method following [link](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/custom_serialization/#migrating-from-deprecated-typeserializersnapshotresolveschemacompatibilityt). + +#### Deprecate old serialization config methods and options + +##### [FLINK-34122](https://issues.apache.org/jira/browse/FLINK-34122) + +Configuring serialization behavior through hard codes is deprecated, because you need to modify the +codes when upgrading the job version. You should configure this via options +`pipeline.serialization-config`, `pipeline.force-avro`, `pipeline.force-kryo`, and `pipeline.generic-types`. +Registration of instance-level serializers is deprecated, using class-level serializers instead. + +#### Migrate string configuration key to ConfigOption + +##### [FLINK-34079](https://issues.apache.org/jira/browse/FLINK-34079) + +We have deprecated all setXxx and getXxx methods except `getString(String key, String defaultValue)` +and `setString(String key, String value)`, such as: `setInteger`, `setLong`, `getInteger` and `getLong` etc. +We strongly recommend users and developers use get and set methods directly. +In addition, we recommend users to use ConfigOption instead of string as key. + +#### Support System out and err to be redirected to LOG or discarded + +##### [FLINK-33625](https://issues.apache.org/jira/browse/FLINK-33625) + +`System.out` and `System.err` output the content to the `taskmanager.out` and `taskmanager.err` files. +In a production environment, if flink users use them to print a lot of content, the limits of yarn +or kubernetes may be exceeded, eventually causing the TaskManager to be killed. Flink supports +redirecting the `System.out` and `System.err` to the log file, and the log file can be rolled to Review Comment: log files? one file for each of them maybe? ########## docs/content.zh/release-notes/flink-1.19.md: ########## @@ -0,0 +1,354 @@ +--- +title: "Release Notes - Flink 1.19" +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 1.19 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 1.18 and Flink 1.19. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.19. + +## Dependency upgrades + +#### Drop support for python 3.7 + +##### [FLINK-33029](https://issues.apache.org/jira/browse/FLINK-33029) + +#### Add support for python 3.11 + +##### [FLINK-33030](https://issues.apache.org/jira/browse/FLINK-33030) + +## Checkpoints + +#### Deprecate RestoreMode#LEGACY + +##### [FLINK-34190](https://issues.apache.org/jira/browse/FLINK-34190) + +`RestoreMode#LEGACY` is deprecated. Please use `RestoreMode#CLAIM` or `RestoreMode#NO_CLAIM` mode +instead to get a clear state file ownership when restoring. + +#### CheckpointsCleaner clean individual checkpoint states in parallel + +##### [FLINK-33090](https://issues.apache.org/jira/browse/FLINK-33090) + +Now when disposing of no longer needed checkpoints, every state handle/state file will be disposed +in parallel by the ioExecutor, vastly improving the disposing speed of a single checkpoint (for +large checkpoints the disposal time can be improved from 10 minutes to < 1 minute) . The old +behavior can be restored by setting `state.checkpoint.cleaner.parallel-mode` to false. + +#### Support using larger checkpointing interval when source is processing backlog + +##### [FLINK-32514](https://issues.apache.org/jira/browse/FLINK-32514) + +`ProcessingBacklog` is introduced to demonstrate whether a record should be processed with low latency +or high throughput. `ProcessingBacklog` can be set by source operators, and can be used to change the +checkpoint internal of a job during runtime. + +#### Allow triggering Checkpoints through command line client + +##### [FLINK-6755](https://issues.apache.org/jira/browse/FLINK-6755) + +The command line interface supports triggering a checkpoint manually. Usage: +``` +./bin/flink checkpoint $JOB_ID [-full] +``` +By specifying the '-full' option, a full checkpoint is triggered. Otherwise an incremental +checkpoint is triggered if the job is configured to take incremental ones periodically. + + +## Runtime & Coordination + +#### Migrate TypeserializeSnapshotr#resolveSchemaCompatibility + +##### [FLINK-30613](https://issues.apache.org/jira/browse/FLINK-30613) + +In Flink 1.19, the old method of resolving schema compatibility has been deprecated and the new one +is introduced. See [FLIP-263](https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility?src=contextnavpagetreemode) for more details. +Please migrate to the new method following [link](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/custom_serialization/#migrating-from-deprecated-typeserializersnapshotresolveschemacompatibilityt). + +#### Deprecate old serialization config methods and options + +##### [FLINK-34122](https://issues.apache.org/jira/browse/FLINK-34122) + +Configuring serialization behavior through hard codes is deprecated, because you need to modify the +codes when upgrading the job version. You should configure this via options +`pipeline.serialization-config`, `pipeline.force-avro`, `pipeline.force-kryo`, and `pipeline.generic-types`. +Registration of instance-level serializers is deprecated, using class-level serializers instead. + +#### Migrate string configuration key to ConfigOption + +##### [FLINK-34079](https://issues.apache.org/jira/browse/FLINK-34079) + +We have deprecated all setXxx and getXxx methods except `getString(String key, String defaultValue)` +and `setString(String key, String value)`, such as: `setInteger`, `setLong`, `getInteger` and `getLong` etc. +We strongly recommend users and developers use get and set methods directly. +In addition, we recommend users to use ConfigOption instead of string as key. + +#### Support System out and err to be redirected to LOG or discarded + +##### [FLINK-33625](https://issues.apache.org/jira/browse/FLINK-33625) + +`System.out` and `System.err` output the content to the `taskmanager.out` and `taskmanager.err` files. +In a production environment, if flink users use them to print a lot of content, the limits of yarn +or kubernetes may be exceeded, eventually causing the TaskManager to be killed. Flink supports +redirecting the `System.out` and `System.err` to the log file, and the log file can be rolled to +avoid unlimited disk usage. + +#### Support standard YAML for FLINK configuration + +##### [FLINK-33297](https://issues.apache.org/jira/browse/FLINK-33297) + +Starting with Flink 1.19, Flink has officially introduced full support for the standard YAML 1.2 +syntax ([FLIP-366](https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration?src=contextnavpagetreemode)). The default configuration file has been changed to `config.yaml` and placed in the +`conf/` directory. Users should directly modify this file to configure Flink. +If users want to use the legacy configuration file `flink-conf.yaml`, users just need to copy this +file into the `conf/` directory. Once the legacy configuration file `flink-conf.yaml` is detected, +Flink will prioritize using it as the configuration file.And in the upcoming Flink 2.0, the +`flink-conf.yaml` configuration file will no longer work. +More details could be found at [flink-configuration-file](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#flink-configuration-file). + +#### Add config options for administrator JVM options + +##### [FLINK-33221](https://issues.apache.org/jira/browse/FLINK-33221) + +A set of administrator JVM options are available, which prepend the user-set extra JVM options for +platform-wide JVM tuning. + +#### Flink Job stuck in suspend state after losing leadership in HA Mode + +##### [FLINK-34007](https://issues.apache.org/jira/browse/FLINK-34007) + +Fixes a bug where the leader election wasn't able to pick up leadership again after renewing the +lease token caused a leadership loss. This required `fabric8io:kubernetes-client` to be upgraded +from v6.6.2 to v6.9.0. + +#### Support dynamic source parallelism inference for batch jobs + +##### [FLINK-33768](https://issues.apache.org/jira/browse/FLINK-33768) + +In Flink 1.19, we have supported dynamic source parallelism inference for batch jobs, which allows +source connectors to dynamically infer the parallelism based on the actual amount of data to consume. +This feature is a significant improvement over previous versions, which only assigned a fixed default +parallelism to source vertices. +Source connectors need to implement the inference interface to enable dynamic parallelism inference. +Currently, the FileSource connector has already been developed with this functionality in place. +Additionally, the configuration `execution.batch.adaptive.auto-parallelism.default-source-parallelism` +will be used as the upper bound of source parallelism inference. And now it will not default to 1. +Instead, if it is not set, the upper bound of allowed parallelism set via Review Comment: ```suggestion If it is not set, the upper bound of allowed parallelism set via ``` ########## docs/content.zh/release-notes/flink-1.19.md: ########## @@ -0,0 +1,354 @@ +--- +title: "Release Notes - Flink 1.19" +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 1.19 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 1.18 and Flink 1.19. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.19. + +## Dependency upgrades + +#### Drop support for python 3.7 + +##### [FLINK-33029](https://issues.apache.org/jira/browse/FLINK-33029) + +#### Add support for python 3.11 + +##### [FLINK-33030](https://issues.apache.org/jira/browse/FLINK-33030) + +## Checkpoints + +#### Deprecate RestoreMode#LEGACY + +##### [FLINK-34190](https://issues.apache.org/jira/browse/FLINK-34190) + +`RestoreMode#LEGACY` is deprecated. Please use `RestoreMode#CLAIM` or `RestoreMode#NO_CLAIM` mode +instead to get a clear state file ownership when restoring. + +#### CheckpointsCleaner clean individual checkpoint states in parallel + +##### [FLINK-33090](https://issues.apache.org/jira/browse/FLINK-33090) + +Now when disposing of no longer needed checkpoints, every state handle/state file will be disposed +in parallel by the ioExecutor, vastly improving the disposing speed of a single checkpoint (for +large checkpoints the disposal time can be improved from 10 minutes to < 1 minute) . The old +behavior can be restored by setting `state.checkpoint.cleaner.parallel-mode` to false. + +#### Support using larger checkpointing interval when source is processing backlog + +##### [FLINK-32514](https://issues.apache.org/jira/browse/FLINK-32514) + +`ProcessingBacklog` is introduced to demonstrate whether a record should be processed with low latency +or high throughput. `ProcessingBacklog` can be set by source operators, and can be used to change the +checkpoint internal of a job during runtime. + +#### Allow triggering Checkpoints through command line client + +##### [FLINK-6755](https://issues.apache.org/jira/browse/FLINK-6755) + +The command line interface supports triggering a checkpoint manually. Usage: +``` +./bin/flink checkpoint $JOB_ID [-full] +``` +By specifying the '-full' option, a full checkpoint is triggered. Otherwise an incremental +checkpoint is triggered if the job is configured to take incremental ones periodically. + + +## Runtime & Coordination + +#### Migrate TypeserializeSnapshotr#resolveSchemaCompatibility + +##### [FLINK-30613](https://issues.apache.org/jira/browse/FLINK-30613) + +In Flink 1.19, the old method of resolving schema compatibility has been deprecated and the new one +is introduced. See [FLIP-263](https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility?src=contextnavpagetreemode) for more details. +Please migrate to the new method following [link](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/custom_serialization/#migrating-from-deprecated-typeserializersnapshotresolveschemacompatibilityt). + +#### Deprecate old serialization config methods and options + +##### [FLINK-34122](https://issues.apache.org/jira/browse/FLINK-34122) + +Configuring serialization behavior through hard codes is deprecated, because you need to modify the +codes when upgrading the job version. You should configure this via options +`pipeline.serialization-config`, `pipeline.force-avro`, `pipeline.force-kryo`, and `pipeline.generic-types`. +Registration of instance-level serializers is deprecated, using class-level serializers instead. + +#### Migrate string configuration key to ConfigOption + +##### [FLINK-34079](https://issues.apache.org/jira/browse/FLINK-34079) + +We have deprecated all setXxx and getXxx methods except `getString(String key, String defaultValue)` +and `setString(String key, String value)`, such as: `setInteger`, `setLong`, `getInteger` and `getLong` etc. +We strongly recommend users and developers use get and set methods directly. +In addition, we recommend users to use ConfigOption instead of string as key. + +#### Support System out and err to be redirected to LOG or discarded + +##### [FLINK-33625](https://issues.apache.org/jira/browse/FLINK-33625) + +`System.out` and `System.err` output the content to the `taskmanager.out` and `taskmanager.err` files. +In a production environment, if flink users use them to print a lot of content, the limits of yarn +or kubernetes may be exceeded, eventually causing the TaskManager to be killed. Flink supports +redirecting the `System.out` and `System.err` to the log file, and the log file can be rolled to +avoid unlimited disk usage. + +#### Support standard YAML for FLINK configuration + +##### [FLINK-33297](https://issues.apache.org/jira/browse/FLINK-33297) + +Starting with Flink 1.19, Flink has officially introduced full support for the standard YAML 1.2 +syntax ([FLIP-366](https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration?src=contextnavpagetreemode)). The default configuration file has been changed to `config.yaml` and placed in the +`conf/` directory. Users should directly modify this file to configure Flink. +If users want to use the legacy configuration file `flink-conf.yaml`, users just need to copy this +file into the `conf/` directory. Once the legacy configuration file `flink-conf.yaml` is detected, +Flink will prioritize using it as the configuration file.And in the upcoming Flink 2.0, the Review Comment: ```suggestion Flink will prioritize using it as the configuration file. In the upcoming Flink 2.0, the ``` ########## docs/content.zh/release-notes/flink-1.19.md: ########## @@ -0,0 +1,354 @@ +--- +title: "Release Notes - Flink 1.19" +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Release notes - Flink 1.19 + +These release notes discuss important aspects, such as configuration, behavior or dependencies, +that changed between Flink 1.18 and Flink 1.19. Please read these notes carefully if you are +planning to upgrade your Flink version to 1.19. + +## Dependency upgrades + +#### Drop support for python 3.7 + +##### [FLINK-33029](https://issues.apache.org/jira/browse/FLINK-33029) + +#### Add support for python 3.11 + +##### [FLINK-33030](https://issues.apache.org/jira/browse/FLINK-33030) + +## Checkpoints + +#### Deprecate RestoreMode#LEGACY + +##### [FLINK-34190](https://issues.apache.org/jira/browse/FLINK-34190) + +`RestoreMode#LEGACY` is deprecated. Please use `RestoreMode#CLAIM` or `RestoreMode#NO_CLAIM` mode +instead to get a clear state file ownership when restoring. + +#### CheckpointsCleaner clean individual checkpoint states in parallel + +##### [FLINK-33090](https://issues.apache.org/jira/browse/FLINK-33090) + +Now when disposing of no longer needed checkpoints, every state handle/state file will be disposed +in parallel by the ioExecutor, vastly improving the disposing speed of a single checkpoint (for +large checkpoints the disposal time can be improved from 10 minutes to < 1 minute) . The old +behavior can be restored by setting `state.checkpoint.cleaner.parallel-mode` to false. + +#### Support using larger checkpointing interval when source is processing backlog + +##### [FLINK-32514](https://issues.apache.org/jira/browse/FLINK-32514) + +`ProcessingBacklog` is introduced to demonstrate whether a record should be processed with low latency +or high throughput. `ProcessingBacklog` can be set by source operators, and can be used to change the +checkpoint internal of a job during runtime. + +#### Allow triggering Checkpoints through command line client + +##### [FLINK-6755](https://issues.apache.org/jira/browse/FLINK-6755) + +The command line interface supports triggering a checkpoint manually. Usage: +``` +./bin/flink checkpoint $JOB_ID [-full] +``` +By specifying the '-full' option, a full checkpoint is triggered. Otherwise an incremental +checkpoint is triggered if the job is configured to take incremental ones periodically. + + +## Runtime & Coordination + +#### Migrate TypeserializeSnapshotr#resolveSchemaCompatibility + +##### [FLINK-30613](https://issues.apache.org/jira/browse/FLINK-30613) + +In Flink 1.19, the old method of resolving schema compatibility has been deprecated and the new one +is introduced. See [FLIP-263](https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility?src=contextnavpagetreemode) for more details. +Please migrate to the new method following [link](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/custom_serialization/#migrating-from-deprecated-typeserializersnapshotresolveschemacompatibilityt). + +#### Deprecate old serialization config methods and options + +##### [FLINK-34122](https://issues.apache.org/jira/browse/FLINK-34122) + +Configuring serialization behavior through hard codes is deprecated, because you need to modify the +codes when upgrading the job version. You should configure this via options +`pipeline.serialization-config`, `pipeline.force-avro`, `pipeline.force-kryo`, and `pipeline.generic-types`. +Registration of instance-level serializers is deprecated, using class-level serializers instead. + +#### Migrate string configuration key to ConfigOption + +##### [FLINK-34079](https://issues.apache.org/jira/browse/FLINK-34079) + +We have deprecated all setXxx and getXxx methods except `getString(String key, String defaultValue)` +and `setString(String key, String value)`, such as: `setInteger`, `setLong`, `getInteger` and `getLong` etc. +We strongly recommend users and developers use get and set methods directly. +In addition, we recommend users to use ConfigOption instead of string as key. + +#### Support System out and err to be redirected to LOG or discarded + +##### [FLINK-33625](https://issues.apache.org/jira/browse/FLINK-33625) + +`System.out` and `System.err` output the content to the `taskmanager.out` and `taskmanager.err` files. +In a production environment, if flink users use them to print a lot of content, the limits of yarn +or kubernetes may be exceeded, eventually causing the TaskManager to be killed. Flink supports +redirecting the `System.out` and `System.err` to the log file, and the log file can be rolled to +avoid unlimited disk usage. + +#### Support standard YAML for FLINK configuration + +##### [FLINK-33297](https://issues.apache.org/jira/browse/FLINK-33297) + +Starting with Flink 1.19, Flink has officially introduced full support for the standard YAML 1.2 +syntax ([FLIP-366](https://cwiki.apache.org/confluence/display/FLINK/FLIP-366%3A+Support+standard+YAML+for+FLINK+configuration?src=contextnavpagetreemode)). The default configuration file has been changed to `config.yaml` and placed in the +`conf/` directory. Users should directly modify this file to configure Flink. +If users want to use the legacy configuration file `flink-conf.yaml`, users just need to copy this +file into the `conf/` directory. Once the legacy configuration file `flink-conf.yaml` is detected, +Flink will prioritize using it as the configuration file.And in the upcoming Flink 2.0, the +`flink-conf.yaml` configuration file will no longer work. +More details could be found at [flink-configuration-file](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#flink-configuration-file). + +#### Add config options for administrator JVM options + +##### [FLINK-33221](https://issues.apache.org/jira/browse/FLINK-33221) + +A set of administrator JVM options are available, which prepend the user-set extra JVM options for +platform-wide JVM tuning. + +#### Flink Job stuck in suspend state after losing leadership in HA Mode + +##### [FLINK-34007](https://issues.apache.org/jira/browse/FLINK-34007) + +Fixes a bug where the leader election wasn't able to pick up leadership again after renewing the Review Comment: ```suggestion Fixed a bug where the leader election wasn't able to pick up leadership again after renewing the ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
