[
https://issues.apache.org/jira/browse/FLINK-40587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis-Mircea Ciupitu updated FLINK-40587:
------------------------------------------
Description:
Currently, deleting and creating a FlinkDeloyment that has a previous
autoscaler state (aka autoscaler ConfigMap owned by the resource is in place)
expose the following behavior:
- Throws the following exception:
{code:bash}
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure
executing: PUT at:
https://10.96.0.1:443/api/v1/namespaces/<namespace>/configmaps/autoscaler-<flinkdeployment-name>.
Message: configmaps "autoscaler-<flinkdeployment-name>" not found. Received
status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[],
group=null, kind=configmaps, name=autoscaler-<flinkdeployment-name>,
retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
message=configmaps "autoscaler-<flinkdeployment-name>" not found,
metadata=ListMeta(_continue=null, remainingItemCount=null,
resourceVersion=null, selfLink=null, shardInfo=null, additionalProperties={}),
reason=NotFound, status=Failure, additionalProperties={}).
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:642)
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:622)
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:582)
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda<span>handleResponse</span>0(OperationSupport.java:549)
at
java.base/java.util.concurrent.CompletableFuture<span>UniApply.tryFire(Unknown
Source) at
java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source) at
io.fabric8.kubernetes.client.http.StandardHttpClient.lambda</span>completeOrCancel<span>10(StandardHttpClient.java:149)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
Source) at
java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(Unknown
Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
at
io.fabric8.kubernetes.client.utils.AsyncUtils.lambda<span>retryWithExponentialBackoff</span>3(AsyncUtils.java:91)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
Source)
at
java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(Unknown
Source) at
java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postFire(Unknown Source) at
java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(Unknown
Source)
at
java.base/java.util.concurrent.CompletableFuture<span>Completion.exec(Unknown
Source) at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at
java.base/java.util.concurrent.ForkJoinPool</span>WorkQueue.topLevelExec(Unknown
Source)
at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
{code}
- Emit the the following event/log:
{code:java}
INFO org.apache.flink.kubernetes.operator.listener.AuditUtils - >>> Event[Job]
| Warning | AUTOSCALERERROR | Failure executing: ...
{code}
The root cause comes from the fact that the operator autoscaler mechanism keeps
the autoscaler ConfigMap cached, and doesn't react when the Kubernetes
garbage-collects the FlinkDeployment-owned resources. The cache is keyed by
namespace and name, not by owner UUID, and it is only cleared on the error
path, so after a recreate the store keeps writing through a view of a ConfigMap
that no longer exists, receiving a 404 REST response.
On top of that, a 404 REST response on flush is treated like any other failure.
As it is expected here, it does not make sense to treat it the same as the
other exceptions and mark it as an AUTOSCALERERROR event.
was:
Currently, deleting and creating a FlinkDeloyment that has a previous
autoscaler state (aka autoscaler ConfigMap owned by the resource is in place)
expose the following behavior:
- Throws the following exception:
{code:bash}
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure
executing: PUT at:
https://10.96.0.1:443/api/v1/namespaces/str-local/configmaps/autoscaler-<flinkdeployment-name>.
Message: configmaps "autoscaler-<flinkdeployment-name>" not found. Received
status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[],
group=null, kind=configmaps, name=autoscaler-<flinkdeployment-name>,
retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
message=configmaps "autoscaler-<flinkdeployment-name>" not found,
metadata=ListMeta(_continue=null, remainingItemCount=null,
resourceVersion=null, selfLink=null, shardInfo=null, additionalProperties={}),
reason=NotFound, status=Failure, additionalProperties={}).
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:642)
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:622)
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:582)
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda<span>handleResponse</span>0(OperationSupport.java:549)
at
java.base/java.util.concurrent.CompletableFuture<span>UniApply.tryFire(Unknown
Source) at
java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source) at
io.fabric8.kubernetes.client.http.StandardHttpClient.lambda</span>completeOrCancel<span>10(StandardHttpClient.java:149)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
Source) at
java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(Unknown
Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
at
io.fabric8.kubernetes.client.utils.AsyncUtils.lambda<span>retryWithExponentialBackoff</span>3(AsyncUtils.java:91)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
Source)
at
java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(Unknown
Source) at
java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postFire(Unknown Source) at
java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(Unknown
Source)
at
java.base/java.util.concurrent.CompletableFuture<span>Completion.exec(Unknown
Source) at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at
java.base/java.util.concurrent.ForkJoinPool</span>WorkQueue.topLevelExec(Unknown
Source)
at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
{code}
- Emit the the following event/log:
{code}
INFO org.apache.flink.kubernetes.operator.listener.AuditUtils - >>> Event[Job]
| Warning | AUTOSCALERERROR | Failure executing: ...
{code}
The root cause comes from the fact that the operator autoscaler mechanism keeps
the autoscaler ConfigMap cached, and doesn't react when the Kubernetes
garbage-collects the FlinkDeployment-owned resources. The cache is keyed by
namespace and name, not by owner UUID, and it is only cleared on the error
path, so after a recreate the store keeps writing through a view of a ConfigMap
that no longer exists, receiving a 404 REST response.
On top of that, a 404 REST response on flush is treated like any other failure.
As it is expected here, it does not make sense to treat it the same as the
other exceptions and mark it as an AUTOSCALERERROR event.
> Recreating a FlinkDeployment throws an autoscaler exception
> -----------------------------------------------------------
>
> Key: FLINK-40587
> URL: https://issues.apache.org/jira/browse/FLINK-40587
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler, Kubernetes Operator
> Reporter: Dennis-Mircea Ciupitu
> Priority: Major
>
> Currently, deleting and creating a FlinkDeloyment that has a previous
> autoscaler state (aka autoscaler ConfigMap owned by the resource is in place)
> expose the following behavior:
> - Throws the following exception:
> {code:bash}
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure
> executing: PUT at:
> https://10.96.0.1:443/api/v1/namespaces/<namespace>/configmaps/autoscaler-<flinkdeployment-name>.
> Message: configmaps "autoscaler-<flinkdeployment-name>" not found. Received
> status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[],
> group=null, kind=configmaps, name=autoscaler-<flinkdeployment-name>,
> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
> message=configmaps "autoscaler-<flinkdeployment-name>" not found,
> metadata=ListMeta(_continue=null, remainingItemCount=null,
> resourceVersion=null, selfLink=null, shardInfo=null,
> additionalProperties={}), reason=NotFound, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:642)
> at
> io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:622)
> at
> io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:582)
> at
> io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda<span>handleResponse</span>0(OperationSupport.java:549)
> at
> java.base/java.util.concurrent.CompletableFuture<span>UniApply.tryFire(Unknown
> Source) at
> java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
> at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
> at
> io.fabric8.kubernetes.client.http.StandardHttpClient.lambda</span>completeOrCancel<span>10(StandardHttpClient.java:149)
> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
> Source) at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(Unknown
> Source)
> at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown
> Source)
> at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
> at
> io.fabric8.kubernetes.client.utils.AsyncUtils.lambda<span>retryWithExponentialBackoff</span>3(AsyncUtils.java:91)
> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
> Source)
> at
> java.base/java.util.concurrent.CompletableFuture<span>UniWhenComplete.tryFire(Unknown
> Source) at
> java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
> at java.base/java.util.concurrent.CompletableFuture.postFire(Unknown Source)
> at
> java.base/java.util.concurrent.CompletableFuture</span>UniWhenComplete.tryFire(Unknown
> Source)
> at
> java.base/java.util.concurrent.CompletableFuture<span>Completion.exec(Unknown
> Source) at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
> at
> java.base/java.util.concurrent.ForkJoinPool</span>WorkQueue.topLevelExec(Unknown
> Source)
> at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
> at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
> at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
> {code}
> - Emit the the following event/log:
> {code:java}
> INFO org.apache.flink.kubernetes.operator.listener.AuditUtils - >>>
> Event[Job] | Warning | AUTOSCALERERROR | Failure executing: ...
> {code}
> The root cause comes from the fact that the operator autoscaler mechanism
> keeps the autoscaler ConfigMap cached, and doesn't react when the Kubernetes
> garbage-collects the FlinkDeployment-owned resources. The cache is keyed by
> namespace and name, not by owner UUID, and it is only cleared on the error
> path, so after a recreate the store keeps writing through a view of a
> ConfigMap that no longer exists, receiving a 404 REST response.
> On top of that, a 404 REST response on flush is treated like any other
> failure. As it is expected here, it does not make sense to treat it the same
> as the other exceptions and mark it as an AUTOSCALERERROR event.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
