Hi Alex,
thanks for the update. I'm happy to hear you were able to resolve the issue.
How are the other ad-hoc streaming pipelines setup?
Maybe these pipelines use a different threading model than Flink. In Flink,
we often have many instances of the same serializer running in the same
JVM. Maybe th
Hello,
Just to follow up on this issue: after collecting some data and setting up
additional tests we have managed to pinpoint the issue to the the
ScalaBuff-generated code that decodes enumerations. After switching to use
ScalaPB generator instead, the problem was gone.
One thing peculiar about
Hi Alex,
I suspect its a GC issue with the code generated by ScalaBuff. Can you
maybe try to do something like a standalone test where use use a
while(true) loop to see how fast you can deserialize elements from your Foo
type?
Maybe you'll find that the JVM is growing all the time. Then there's
pro
Hey Alex,
(1) Which Flink version are you using for this?
(2) Can you also get a heap dump after the job slows down? Slow downs
like this are often caused by some component leaking memory, maybe in
Flink, maybe the Scalabuff deserializer. Can you also share the Foo
code?
– Ufuk
On Mon, Apr 18,
Hello,
Has anyone tried using ScalaBuff (https://github.com/SandroGrzicic/ScalaBuff)
with Flink? We’re trying to consume Protobuf messages from Kafka 0.8 and
have hit a performance issue. We run this code:
https://gist.github.com/clayrat/05ac17523fcaa52fcc5165d9edb406b8 (where Foo
is pre-gene
