Re: Strange issue with exactly once checkpoints and the kafka sink

2022-11-16 Thread Salva Alcántara
As noted in the SO, it's a bit confusing to me how the `checkpointing.mode` delivery guarantees with the ones for the different sinks, and in particular with the kafka one. Based on the error I had, I understand that if I use `EXACTLY_ONCE` for the checkpoints and I indicate nothing in the kafka s

Re: Dependency resolution issue with apache-flink 1.16.0 python package.

2022-11-16 Thread Xingbo Huang
Hi Yogi, I think the problem comes from poetry depending on the metadata in PyPI. This problem has been reported in https://issues.apache.org/jira/browse/FLINK-29817 and I will fix it in 1.16.1. Best, Xingbo Yogi Devendra 于2022年11月17日周四 06:21写道: > Dear community/maintainers, > > Thanks for the

Dependency resolution issue with apache-flink 1.16.0 python package.

2022-11-16 Thread Yogi Devendra
Dear community/maintainers, Thanks for the lovely platform of Apache Flink. I got following error when add apache-flink 1.16.0 dependency in my python project. Given below is the stack trace for further investigation. When I tried using lower version (1.15.2) for the same; I was able to move forw

Re: Kafka transactioins & flink checkpoints

2022-11-16 Thread Yaroslav Tkachenko
I gave a talk about that setup: https://www.youtube.com/watch?v=tiGxEGPyqCg&ab_channel=FlinkForward The documentation suggests using unaligned checkpoints in case of backpressure ( https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/state/checkpointing_under_backpressure/#unaligned

Kubernetes operator and jobs with last-state upgrades

2022-11-16 Thread Alexis Sarda-Espinosa
Hello, I am doing some tests with the operator and, if I'm not mistaken, using last-state upgrade means that, when something is changed in the CR, no savepoint is taken and the pods are simply terminated. Is that a requirement from Flink HA? I would have thought last-state would still use savepoin

Re: Owner reference with the Kubernetes operator

2022-11-16 Thread Alexis Sarda-Espinosa
Ah I see, cool, thanks. Regards, Alexis. Am Mi., 16. Nov. 2022 um 15:50 Uhr schrieb Gyula Fóra : > This has been changed in the current snapshot release: > https://issues.apache.org/jira/browse/FLINK-28979 > > It will be part of the 1.3.0 version. > > On Wed, Nov 16, 2022 at 3:32 PM Alexis Sarda

Re: Owner reference with the Kubernetes operator

2022-11-16 Thread Gyula Fóra
This has been changed in the current snapshot release: https://issues.apache.org/jira/browse/FLINK-28979 It will be part of the 1.3.0 version. On Wed, Nov 16, 2022 at 3:32 PM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hello, > > Is there a particular reason the operator doesn't s

Savepoint restore mode for the Kubernetes operator

2022-11-16 Thread Alexis Sarda-Espinosa
Hello, Is there a recommended configuration for the restore mode of jobs managed by the operator? Since the documentation states that the operator keeps a savepoint history to perform cleanup, I imagine restore mode should always be NO_CLAIM, but I just want to confirm. Regards, Alexis.

Owner reference with the Kubernetes operator

2022-11-16 Thread Alexis Sarda-Espinosa
Hello, Is there a particular reason the operator doesn't set owner references for the Deployments it creates as a result of a FlinkDeployment CR? This makes tracking in the Argo CD UI impossible. (To be clear, I mean a reference from the Deployment to the FlinkDeployment). Regards, Alexis.

Re: Kafka transactioins & flink checkpoints

2022-11-16 Thread Vishal Surana
Yes. I do use RocksDB for (incremental) checkpointing. During each checkpoint 15-20GB of state gets created (new state added, some expired). I make use of FIFO compaction. I’m a bit surprised you were able to run with 10+TB state without unaligned checkpoints because the performance in my appli

Re: Kafka transactions drastically limit usability of Flink savepoints

2022-11-16 Thread Yordan Pavlov
Hi Piotr, the option you mention is applicable only for the deprecated KafkaProducer, is there an equivalent to the modern KafkaSink? I found this article comparing the behavior of the two: https://ververica.zendesk.com/hc/en-us/articles/360013269680-Best-Practices-for-Using-Kafka-Sources-Sinks-in

Re: Reading Parquet file with array of structs cause error

2022-11-16 Thread Jing Ge
Hi Michael, yeah, it will be addressed in Flink-28867. Best regards, Jing On Wed, Nov 16, 2022 at 2:58 AM liu ron wrote: > It will be addressed in FLINK-28867. > > Best, > Ron > > Benenson, Michael via user 于2022年11月16日周三 08:47写道: > >> Thanks, Jing >> >> >> >> Do you know, if this problem wi