[ANNOUNCE] Apache Flink 1.16.0 released

2022-10-27 Thread Xingbo Huang
The Apache Flink community is very happy to announce the release of Apache Flink 1.16.0, which is the first release for the Apache Flink 1.16 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applicat

Re: Broadcast state and job restarts

2022-10-27 Thread Zakelly Lan
Hi Alexis, Broadcast state is one type of the Operator State, which is included in savepoints and checkpoints and won't be lost. Please refer to https://stackoverflow.com/questions/62509773/flink-broadcast-state-rocksdb-state-backend/62510423#62510423 Best, Zakelly On Fri, Oct 28, 2022 at 4:41 A

Flink Kubernetes rocksdb backend volume type

2022-10-27 Thread Vignesh Kumar Kathiresan via user
Hi all, What is the recommended practice of volume mounting for the rocksdb backend in a flink kubernetes standalone setup. We currently have a "local" type. We have local persistent volumes and persistent volume claims for each node's local ssd. And use one PVC in the task manager deployment. A

Broadcast state and job restarts

2022-10-27 Thread Alexis Sarda-Espinosa
Hello, The documentation for broadcast state specifies that it is always kept in memory. My assumptions based on this statement are: 1. If a job restarts in the same Flink cluster (i.e. using a restart strategy), the tasks' attempt number increases and the broadcast state is restored since it's n

Re: [DISCUSS ] add --jars to support users dependencies jars.

2022-10-27 Thread Yang Wang
Thanks Jacky Lau for starting this discussion. I understand that you are trying to find a convenient way to specify dependency jars along with user jar. However, let's try to narrow down by differentiating deployment modes. # Standalone mode No matter you are using the standalone mode on virtual

Re: Concatenating a bounded and unbounded stream

2022-10-27 Thread Noel OConnor
yea I tried this with two pulsar streams but I'm hitting a runtime exception shown below. I think it's related to https://issues.apache.org/jira/browse/FLINK-25444 --- The program finished with the following exception: The implementation o

Re: State Processor API - VoidNamespaceSerializer must be compatible with the old namespace serializer LongSerializer

2022-10-27 Thread Filip Karnicki
Hi Gordon I would like to carry over *all* of the internal states for the FunctionGroupOperator, only changing my own state. I was under the impression that the only way to do that is to call mySvepoint.removeOperator("operator-uid").withOperator("operator-uid", transformation) (where transforma

RE: Concatenating a bounded and unbounded stream

2022-10-27 Thread Schwalbe Matthias
Sorry, I’ve got thing really mixed up, I meant to reply to this other thread … ☹ Thias From: Schwalbe Matthias Sent: Thursday, October 27, 2022 9:14 AM To: 'Tzu-Li (Gordon) Tai' ; Filip Karnicki Cc: user Subject: RE: State Processor API - VoidNamespaceSerializer must be compatible with the ol

RE: State Processor API - VoidNamespaceSerializer must be compatible with the old namespace serializer LongSerializer

2022-10-27 Thread Schwalbe Matthias
Hi Filip, Hi Tsu-Li, @Tsu-Li: long time not seen 😊 (it is time for an on-site FlinkForward in Berlin again next year 😊 ) Considering Tsu-Li’s proposal, there is a restriction, at the time being you can only create a HybridSource from other sources that have exactly the same type. This is not a