[
https://issues.apache.org/jira/browse/BEAM-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521686#comment-17521686
]
Bjorn De Bakker commented on BEAM-14184:
----------------------------------------
Hey Luke
I reported the original error, but for some reason I can't comment or reply to
that email thread. Thanks again for your response, but I have a question on
some of the stuff you mentioned. (Caveat, I'm not a data scientist, nor do I
have a lot of experience with Apache Beam, so apologies for the entry level
questions).
_You could swap and use the experiment "beam_fn_api_buffered_stream" which will
give you a different implementation which shouldn't experience this issue. You
can also control the buffer size with the experiment
"beam_fn_api_buffered_stream_buffer_size=<BUFFER_SIZE>" with the default
BUFFER_SIZE=64 if unspecified._
Can you provide a bit more guidance on where we can configure this parameter?
Is that something we need to set in our pipeline code, or can we configure that
elsewhere, for example on the TaskManager in Flink?
Thanks.
Best regards,
Bjorn
> DirectStreamObserver does not respect channel isReady
> -----------------------------------------------------
>
> Key: BEAM-14184
> URL: https://issues.apache.org/jira/browse/BEAM-14184
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-harness
> Reporter: Luke Cwik
> Assignee: Luke Cwik
> Priority: P2
>
> Leads to OOMs like:
> {noformat}
> Output channel stalled for 1023s, outbound thread CHAIN MapPartition
> (MapPartition at [1]PerformInference) -> FlatMap (FlatMap at
> ExtractOutput[0]) -> Map (Key Extractor) -> GroupCombine (GroupCombine at
> GroupCombine:
> PerformInferenceAndCombineResults_dep_049/GroupPredictionsByImage) -> Map
> (Key Extractor) (1/1). See: https://issues.apache.org/jira/browse/BEAM-4280
> for the history for this issue.
> Feb 18, 2022 11:51:05 AM
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.netty.NettyServerTransport
> notifyTerminated
> INFO: Transport failed
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.internal.OutOfDirectMemoryError:
> failed to allocate 2097152 byte(s) of direct memory (used: 1205862679, max:
> 1207959552)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:754)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:709)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:645)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:621)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:204)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:188)
> {noformat}
> See more context in
> https://lists.apache.org/thread/llmxodbmczhn10c98prs8wmd5hy4nvff
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
