[
https://issues.apache.org/jira/browse/BEAM-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521695#comment-17521695
]
Luke Cwik commented on BEAM-14184:
----------------------------------
https://github.com/apache/beam/pull/17358 is out for review with a fix that
should make it into the next release.
You need to set the experiments pipeline option. You can do this during
pipeline construction or on the command line when launching your pipeline via
*--experiments=...*. See section 2.1 of the [programming
guide|https://beam.apache.org/documentation/programming-guide/] for more
details on setting pipeline options.
> DirectStreamObserver does not respect channel isReady
> -----------------------------------------------------
>
> Key: BEAM-14184
> URL: https://issues.apache.org/jira/browse/BEAM-14184
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-harness
> Reporter: Luke Cwik
> Assignee: Luke Cwik
> Priority: P2
>
> Leads to OOMs like:
> {noformat}
> Output channel stalled for 1023s, outbound thread CHAIN MapPartition
> (MapPartition at [1]PerformInference) -> FlatMap (FlatMap at
> ExtractOutput[0]) -> Map (Key Extractor) -> GroupCombine (GroupCombine at
> GroupCombine:
> PerformInferenceAndCombineResults_dep_049/GroupPredictionsByImage) -> Map
> (Key Extractor) (1/1). See: https://issues.apache.org/jira/browse/BEAM-4280
> for the history for this issue.
> Feb 18, 2022 11:51:05 AM
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.netty.NettyServerTransport
> notifyTerminated
> INFO: Transport failed
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.internal.OutOfDirectMemoryError:
> failed to allocate 2097152 byte(s) of direct memory (used: 1205862679, max:
> 1207959552)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:754)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:709)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:645)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:621)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:204)
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:188)
> {noformat}
> See more context in
> https://lists.apache.org/thread/llmxodbmczhn10c98prs8wmd5hy4nvff
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
