No, we are not seeing anything specific about request timeouts in the logs.
Typically, the only thing we do see in the logs is the following:
21/04/21 21:44:52 ERROR ScalaDriverLocal: User Code Stack Trace:
> java.lang.RuntimeException: org.apache.spark.SparkException: Job aborted
> due to stage
> On 27 Apr 2021, at 08:39, Thomas Fredriksen(External)
> wrote:
>
> Thank you, this is very informative.
>
> We tried reducing the JdbcIO batch size from 1 to 1000, then to 100. In
> our runs, we no longer see the explicit OOM-error, but we are seeing executor
> heartbeat timeouts. Fro
Thank you, this is very informative.
We tried reducing the JdbcIO batch size from 1 to 1000, then to 100. In
our runs, we no longer see the explicit OOM-error, but we are seeing
executor heartbeat timeouts. From what we understand, this is typically
caused by OOM-errors also. However, the stag
> On 26 Apr 2021, at 13:34, Thomas Fredriksen(External)
> wrote:
>
> The stack-trace for the OOM:
>
> 21/04/21 21:40:43 WARN TaskSetManager: Lost task 1.2 in stage 2.0 (TID 57,
> 10.139.64.6, executor 3): org.apache.beam.sdk.util.UserCodeException:
> java.lang.OutOfMemoryError: GC overhead
The stack-trace for the OOM:
21/04/21 21:40:43 WARN TaskSetManager: Lost task 1.2 in stage 2.0 (TID 57,
> 10.139.64.6, executor 3): org.apache.beam.sdk.util.UserCodeException:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeEx
Hi Thomas,
Could you share the stack trace of your OOM and, if possible, the code snippet
of your pipeline?
Afaik, usually only “large" GroupByKey transforms, caused by “hot keys”, may
lead to OOM with SparkRunner.
—
Alexey
> On 26 Apr 2021, at 08:23, Thomas Fredriksen(External)
> wrote:
>
