Checkpoint failures due to other subtasks sharing the ChannelState file (Caused the Job to Stall)

Hi Everyone, We are observing an interesting issue with continuous checkpoint failures in our job causing the event to not be forwarded through the pipeline. We saw a spam of the below log in all our task manager instances. Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Th

?????? checkpoint upload thread

Hi,Yanfei:   What do you mean by using the word possible in statment it is possible to use the same connection for an operator chain? Meaning able to be done but not applied in fact? Or actually applied but with applied probability?   Thanks. --  -- ??

Re: KafkaSink self-defined RoundRobinPartitioner not able to discover new partitions

Seems i needn't to define a FlinkRoundRoubinPartitioner and just use the RoundRobinPartitioner suppllied in kafka: props.setProperty(ProducerConfig.PARTITIONER_CLASS_CONFIG, RoundRobinPartitioner.class.getName()); In this way the new partitions will be found dynamically. Howerever, there's a bug i

Re: [Request Help] Flink StreamRecord granularity latency metrics

Hi Yubin, We implement it in this manner. For every record, we define several time fields. When the record first enters the system, set one field to current time. After several complex calculation operator, set another field to currentTime. Just calculate the difference between the two values. Hop

StreamTaskException: Could not serialize object for key serializedUDF

Hi, I'm using Flink 1.17.1 streaming API, on YARN. My app first stuck at process func serialization. I know Avro Schema is not serializable so I removed all references from my process functions. Now it passes first round, but stuck again at the following error: org.apache.flink.client.program.Pr

Re: Elasticsearch 8.x Connector in Maven

Hi Ahmed,Thanks for the response, I’ll reach out to the devs list and go from there.Thanks again,Rion On Jul 31, 2024, at 9:18 AM, Ahmed Hamdy wrote:Hi RionIt seems that ES 8 was supported ahead of 3.1 release[1], which seems to not be released yet hence not published to maven.Given the importanc

Re: Elasticsearch 8.x Connector in Maven

Hi Rion It seems that ES 8 was supported ahead of 3.1 release[1], which seems to not be released yet hence not published to maven. Given the importance of ES 8 and the fact that elastic search still depends on Flink 1.18 while we are releasing 1.20, I would suggest nudging the dev list[2] for a rel

Re: Using state processor for a custom windowed aggregate function

Hi again, I realized it's easy to create a reproducible example, see this specific commit: https://github.com/asardaes/test-flink-state-processor/commit/95e65f88fd1e38bcba63ebca68e3128789c0d2f2 When I run that application, I see the following output: Savepoint created KEY=GenericServiceComposit

Re: Using state processor for a custom windowed aggregate function

Hi Matthias, This indeed compiles, I am able to actually generate a savepoint, it's just that all the window states in that savepoint appear to be null. The second argument of withOperator(...) is specified via OperatorTransformation...aggregate(), so the final transformation is built by WindowedS

Re: Elasticsearch 8.x Connector in Maven

Hi again all, Just following up on this as I’ve scoured around trying to find any documentation for using the ES 8.x connector, however everything only appears to reference 6/7. The ES 8.x seems to have been released for quite a bit of time, so I’m curious how others are using it. I’d really l

RE: Using state processor for a custom windowed aggregate function

Hi Alexis, Just a couple of points to double-check: * Does your code compile? (the second argument of withOperator(..) should derive StateBootstrapTransformation instead of SingleOutputStreamOperator) * From the documentation of savepoint API you’ll find examples for each type of state