[
https://issues.apache.org/jira/browse/FLINK-36020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
siyuan zheng updated FLINK-36020:
---------------------------------
Description:
Running a Flink job on YARN, setting
'taskmanager.network.memory.buffer-debloat.period' to true may cause this error
to occur intermittently. However, increasing the value of
'taskmanager.network.memory.buffer-debloat.period' reduces the frequency of
these errors.
{code:java}
java.lang.IllegalArgumentException: Time should be non negative
at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138)
at
org.apache.flink.runtime.throughput.ThroughputCalculator.calculateThroughput(ThroughputCalculator.java:80)
at
org.apache.flink.runtime.throughput.ThroughputCalculator.calculateThroughput(ThroughputCalculator.java:71)
at
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.triggerDebloating(SingleInputGate.java:464)
at
org.apache.flink.runtime.taskmanager.InputGateWithMetrics.triggerDebloating(InputGateWithMetrics.java:90)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.debloat(StreamTask.java:894)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$6(StreamTask.java:885)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858)
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.lang.Thread.run(Thread.java:745)
{code}
was:
Running a Flink job on YARN, setting
'taskmanager.network.memory.buffer-debloat.period' to true may cause this error
to occur intermittently. However, increasing the value of
'taskmanager.network.memory.buffer-debloat.period' reduces the frequency of
these errors.
It appears that this issue may be caused by host clock synchronization.
However, I believe that a rollback of the host time should not result in an
error. Fault tolerance should be enhanced to handle this situation.
{code:java}
java.lang.IllegalArgumentException: Time should be non negative
at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138)
at
org.apache.flink.runtime.throughput.ThroughputCalculator.calculateThroughput(ThroughputCalculator.java:80)
at
org.apache.flink.runtime.throughput.ThroughputCalculator.calculateThroughput(ThroughputCalculator.java:71)
at
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.triggerDebloating(SingleInputGate.java:464)
at
org.apache.flink.runtime.taskmanager.InputGateWithMetrics.triggerDebloating(InputGateWithMetrics.java:90)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.debloat(StreamTask.java:894)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$6(StreamTask.java:885)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858)
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.lang.Thread.run(Thread.java:745)
{code}
> An exception occurred when enabling the buffer debloating mechanism: Time
> should be non negative
> -------------------------------------------------------------------------------------------------
>
> Key: FLINK-36020
> URL: https://issues.apache.org/jira/browse/FLINK-36020
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
> Affects Versions: 1.19.1
> Reporter: siyuan zheng
> Priority: Major
>
> Running a Flink job on YARN, setting
> 'taskmanager.network.memory.buffer-debloat.period' to true may cause this
> error to occur intermittently. However, increasing the value of
> 'taskmanager.network.memory.buffer-debloat.period' reduces the frequency of
> these errors.
>
> {code:java}
> java.lang.IllegalArgumentException: Time should be non negative
> at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138)
> at
> org.apache.flink.runtime.throughput.ThroughputCalculator.calculateThroughput(ThroughputCalculator.java:80)
>
> at
> org.apache.flink.runtime.throughput.ThroughputCalculator.calculateThroughput(ThroughputCalculator.java:71)
>
> at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.triggerDebloating(SingleInputGate.java:464)
>
> at
> org.apache.flink.runtime.taskmanager.InputGateWithMetrics.triggerDebloating(InputGateWithMetrics.java:90)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.debloat(StreamTask.java:894)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$6(StreamTask.java:885)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
>
> at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367)
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352)
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858)
>
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
>
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
> at java.lang.Thread.run(Thread.java:745)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
