Hello All,
Any idea how to debug this and find out which stage, which DoFn or which
side input is causing the problem?
Do I need to override OnTimer with every DoFn to avoid this problem?
I thought that some uncaught exceptions were causing this and added various
checks and exception handling in all DoFn and still seeing this issue.
It has been driving me nuts. And now forget DRAIN, it happens during normal
functioning as well. Any help would be appreciated.
java.lang.UnsupportedOperationException: Attempt to deliver a timer to a
DoFn, but timers are not supported in Dataflow.
1.
1. at org.apache.beam.runners.dataflow.worker.
StreamingSideInputDoFnRunner.onTimer (
StreamingSideInputDoFnRunner.java:86
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
)
2. at org.apache.beam.runners.dataflow.worker.
SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
)
3. at org.apache.beam.runners.dataflow.worker.SimpleParDoFn.access$600
(SimpleParDoFn.java:73
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
)
4. at org.apache.beam.runners.dataflow.worker.
SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
)
5. at org.apache.beam.runners.dataflow.worker.
SimpleParDoFn.processTimers (SimpleParDoFn.java:473
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
)
6. at org.apache.beam.runners.dataflow.worker.
SimpleParDoFn.processTimers (SimpleParDoFn.java:353
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
)
7. at org.apache.beam.runners.dataflow.worker.util.common.worker.
ParDoOperation.finish (ParDoOperation.java:52
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
)
8. at org.apache.beam.runners.dataflow.worker.util.common.worker.
MapTaskExecutor.execute (MapTaskExecutor.java:85
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
)
9. at org.apache.beam.runners.dataflow.worker.
StreamingDataflowWorker.process (StreamingDataflowWorker.java:1350
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
)
10. at org.apache.beam.runners.dataflow.worker.
StreamingDataflowWorker.access$1100 (StreamingDataflowWorker.java:152
<https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-26_20_35_51-7756840794032391970&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
)
2.
1. at org.apache.beam.runners.dataflow.worker.
StreamingDataflowWorker$7.run (StreamingDataflowWorker.java:1073)
2. at java.util.concurrent.ThreadPoolExecutor.runWorker (
ThreadPoolExecutor.java:1149)
3. at java.util.concurrent.ThreadPoolExecutor$Worker.run (
ThreadPoolExecutor.java:624)
4. at java.lang.Thread.run (Thread.java:748)
Thanks
Mohil
On Sun, Jul 26, 2020 at 1:50 PM Mohil Khare <mo...@prosimo.io> wrote:
> and it seems be due to TimerType User
>
> Thanks
> Mohil
>
> On Sun, Jul 26, 2020 at 1:42 PM Mohil Khare <mo...@prosimo.io> wrote:
>
>> Hello,
>>
>> I was looking at source code of
>> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunner.java
>> ,
>>
>> It seems the default implementation of OnTimer is to throw (Attempt to
>> deliver a timer to a DoFn, but timers are not supported in Dataflow).
>>
>> Do you know under what circumstances, My code might be throwing this ?
>> Not sure if its some issue in 2.19 which might have been fixed now with
>> 2.22
>>
>> Thanks and Regards
>> Mohil
>>
>>
>> On Fri, Jul 24, 2020 at 5:21 PM Mohil Khare <mo...@prosimo.io> wrote:
>>
>>> Actually NOT JUST DRAIN, seeing it during regular RUN as well. It's
>>> getting flooded with this exception
>>>
>>> Attempt to deliver a timer to a DoFn, but timers are not supported in
>>> Dataflow.
>>> The changes that I did are the following:
>>> 1. Read one set of logs from one of the kafka topics and create KV(ID,
>>> log1)
>>> 2. Read 2nd set of logs from another kafka topic and create KV(ID, log2)
>>> 3. Put above in session window
>>> 4. CoGroupByKey both logs
>>>
>>> Apart from aforementioned exceptions, I am also seeing the following in
>>> worker logs. What I have noticed is that it happens when the pipeline is
>>> silent for a while. i.e. no new logs to be read from Kafka (Not sure if it
>>> is the actual reason).
>>>
>>> 2020-07-24 17:06:43.532 PDT
>>> Execution of work for P188 for key
>>> cloud@prosimo.ioHe8fc8079-c844-11ea-a6d5-dabe1eb9c630 failed. Will
>>> retry locally.
>>>
>>> <https://console.cloud.google.com/logs/viewer?advancedFilter=insertId%3D%224729149996009968456:27267:0:42419165%22%20resource.type%3D%22dataflow_step%22%20resource.labels.job_id%3D%222020-07-24_16_23_20-4320159532106629306%22%20(logName%3D%22projects%2Fprosimo-test%2Flogs%2Fdataflow.googleapis.com%252Fworker%22)%20severity%3E%3DERROR%0A(timestamp%3E%222020-07-25T00:06:33.224Z%22%20OR%20(insertId%3E%224729149996009968456:27267:0:42264272%22%20AND%20timestamp%3D%222020-07-25T00:06:33.224Z%22))&expandAll=true&interval=JUMP_TO_TIME&dateRangeStart=2020-07-25T00:06:43.532Z&dateRangeEnd=2020-07-25T00:06:43.532Z&project=prosimo-test>
>>> 2020-07-24 17:06:53.863 PDT
>>> Uncaught exception:
>>>
>>>
>>>
>>> On Fri, Jul 24, 2020 at 2:59 PM Mohil Khare <mo...@prosimo.io> wrote:
>>>
>>>> Hello,
>>>>
>>>> I am on java sdk 2.19 and using dataflow for beam job. I use Timers for
>>>> my stateful transformations, but recently I started seeing the following
>>>> exception on DRAINING a job. It used to work fine and not sure what
>>>> changed.
>>>>
>>>> *java.lang.UnsupportedOperationException:*
>>>>
>>> *Attempt to deliver a timer to a DoFn, but timers are not supported in
>>> Dataflow. *
>>>
>>>>
>>>> 1.
>>>> 1. atorg.apache.beam.runners.dataflow.worker.
>>>> StreamingSideInputDoFnRunner.onTimer (
>>>> StreamingSideInputDoFnRunner.java:86
>>>>
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingSideInputDoFnRunner.java&line=86&project=prosimo-test>
>>>> )
>>>> 2. atorg.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn.processUserTimer (SimpleParDoFn.java:360
>>>>
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=360&project=prosimo-test>
>>>> )
>>>> 3. atorg.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn.access$600 (SimpleParDoFn.java:73
>>>>
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=73&project=prosimo-test>
>>>> )
>>>> 4. atorg.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn$TimerType$1.processTimer (SimpleParDoFn.java:444
>>>>
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=444&project=prosimo-test>
>>>> )
>>>> 5. atorg.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:473
>>>>
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=473&project=prosimo-test>
>>>> )
>>>> 6. atorg.apache.beam.runners.dataflow.worker.
>>>> SimpleParDoFn.processTimers (SimpleParDoFn.java:353
>>>>
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FSimpleParDoFn.java&line=353&project=prosimo-test>
>>>> )
>>>> 7. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
>>>> ParDoOperation.finish (ParDoOperation.java:52
>>>>
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FParDoOperation.java&line=52&project=prosimo-test>
>>>> )
>>>> 8. atorg.apache.beam.runners.dataflow.worker.util.common.worker.
>>>> MapTaskExecutor.execute (MapTaskExecutor.java:85
>>>>
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java&line=85&project=prosimo-test>
>>>> )
>>>> 9. atorg.apache.beam.runners.dataflow.worker.
>>>> StreamingDataflowWorker.process (
>>>> StreamingDataflowWorker.java:1350
>>>>
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=1350&project=prosimo-test>
>>>> )
>>>> 10. atorg.apache.beam.runners.dataflow.worker.
>>>> StreamingDataflowWorker.access$1100 (
>>>> StreamingDataflowWorker.java:152
>>>>
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs&appVersion=2020-07-24_14_14_42-5619042994908733980&file=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java&line=152&project=prosimo-test>
>>>> )
>>>> 2. Sometime back I opened jira for an issue which is related to
>>>> this while doing DRAIN:
>>>> https://issues.apache.org/jira/browse/BEAM-10053 (Looks like no one
>>>> has taken a stab on this Jira)
>>>> 3. Not sure if the reason is same and due to multiple side inputs
>>>> that i use in my PTransforms.
>>>>
>>>>
>>>> Any help would be appreciated.
>>>>
>>>> Thanks and Regards
>>>> Mohil
>>>>
>>>>
