Re: How to clean up RocksDB local directory in K8s statefulset

Hi Allen, what volumes do you use for your TM pod? If you want your data to be deleted when the pod restarts, you can use an ephemeral volume like EmptyDir. And Flink should remove temporary files automatically when they are not needed anymore(see this discussion

Re: How to convert Table containing TIMESTAMP_LTZ into DataStream in PyFlink 1.15.0?

Hi Dian, Thanks, much appreciated. Kind regards, John Sent from my iPhone On 27 Jun 2022, at 03:43, Dian Fu wrote:  Hi John, This seems like a bug and I have created a ticket https://issues.apache.org/jira/browse/FLINK-28253 to track it. For now, you could try replacing to_data_stream wi

How to clean up RocksDB local directory in K8s statefulset

Hi Folks, We created a stateful job using SessionWindow and RocksDB state backend and deployed it on Kubernetes Statefulset with persisted volumes. The Flink version we used is 1.14. After the job runs for some time, we observed that the size of the local RocksDB directory started to grow and the

Re: Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

This sounds very useful! Another potential use case: - Consuming from multiple kafka clusters in different datacenters/regions. I'm not sure if we would ultimately want to do this, but having it as an option when you need events from multiple kafka clusters to get the full history of changes (in

RE: Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

Hi Mason, Thanks for starting this discussion! The proposed Source sounds awesome and we would be interested in taking a look at the source code and evaluating our use cases. We can provide information and review on a potential FLIP based on other use cases. Do you have a fork/branch that you are

[ANNOUNCE] Apache Flink Kubernetes Operator 1.0.1 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.0.1. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl.

Re: Flink k8s Operator on AWS?

The problem was a misconfiguration of the initContainer which would copy my artifacts from s3 to an ephemeral volume. This caused the task manager to get started for a bit and then to be shut down. It was hard to get logging about this since the pods were gone before I could get logging from it.

RE: Synchronizing streams in coprocessfunction

Hi Gopi, Your use case is a little under-specified to give a specific answer, especially to the nature of the two input streams and the way events of both streams are correlated (joined): * Is your fast-stream keyed? * If yes: keyed state and timers can be used, otherwise only operat

Re: Synchronizing streams in coprocessfunction

Thanks Quingsheng, that would definitely work. But I'm unable to figure out how I can apply this with CoProcessFunction. One stream is windowed and trigger implementation uses the 2nd stream. On Mon, Jun 27, 2022 at 3:29 PM Qingsheng Ren wrote: > Hi Gopi, > > What about using a window with a cus

Re: Synchronizing streams in coprocessfunction

Hi Gopi, What about using a window with a custom trigger? The window is doing nothing but aggregating your input to a collection. The trigger accepts metadata from the low input stream so it can fire and purge the window (emit all elements in the window to downstream) on arrival of metadata.

[FINAL CALL] - Travel Assistance to ApacheCon New Orleans 2022

To all committers and non-committers. This is a final call to apply for travel/hotel assistance to get to and stay in New Orleans for ApacheCon 2022. Applications have been extended by one week and so the application deadline is now the 8th July 2022. The rest of this email is a copy of what ha