Juliusz Nadberezny created FLINK-40531:
------------------------------------------
Summary: K8s Operator Autoscaler Parallelism overrides serialized
in standard-YAML format break the deployed job
Key: FLINK-40531
URL: https://issues.apache.org/jira/browse/FLINK-40531
Project: Flink
Issue Type: Bug
Components: Autoscaler, Kubernetes Operator
Reporter: Juliusz Nadberezny
{{Affects unreleased 1.17.0 version}}
{{}}
Flink can write map-valued config settings in two styles: the old {{a:1,b:2}}
and the newer {{{}{a: '1', b: '2'}{}}}. Every Flink version can read the old
style; only newer parsers can read the new one.
{{}}
The autoscaler lets Flink choose the style automatically, and that choice
depends on which config file the operator itself was started with. So if the
operator runs with {{{}config.yaml{}}}, it writes parallelism overrides in the
new style into every FlinkDeployment it scales. Any JobManager still reading a
legacy {{flink-conf.yaml}} then fails to start, because it cannot parse the
value. The operator itself reports no problem — only the job breaks.
{{KubernetesScalingRealizer}} and the autoscaler state stores serialize the
parallelism overrides map with {{{}ConfigurationUtils.convertValue(overrides,
String.class){}}}. The two-arg overload picks its format from
{{{}GlobalConfiguration.isStandardYaml(){}}}, which is derived from the config
file the _operator's own JVM_ loaded.
If the operator runs with {{config.yaml}} (standard YAML) instead of
{{{}flink-conf.yaml{}}}, it writes {{pipeline.jobvertex-parallelism-overrides:
\{a: '1', b: '2'}}} into every FlinkDeployment it scales. A JobManager loading
a legacy {{flink-conf.yaml}} cannot parse that and fails on startup with
{{{}Could not split string. Illegal quoting at position: 5{}}}. The operator
logs nothing — only the deployed job breaks.
The legacy {{a:1,b:2}} form is parseable by all Flink versions, including 2.x
(which reads both forms even though it serializes the new one). The flow form
is not. So serialization should be pinned to the portable format rather than
inherited from ambient operator state.
Made worse by a self-inflicted trap: {{getOverrideString}} reuses the previous
string verbatim when the override map is unchanged, to avoid a no-op redeploy.
That pins a broken string in place permanently, so an affected job cannot
recover on its own.
*Steps to reproduce*
# Deploy the operator with {{defaultConfiguration.config.yaml}} instead of
{{{}flink-conf.yaml{}}}.
# Let the autoscaler scale a {{v1_20}} FlinkDeployment.
# Inspect {{spec.flinkConfiguration}} — the override string is {{{...}}} and
the JobManager crash-loops.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
