Flink version upgrade issues (schema register forbidden error + incorrect kafka offset issue)

Hi team, we encountered the schema forbidden issue during deployment with the changes of flink version upgrade (1.16.1 -> 1.17.1), and also encountered incorrect reset kafka offset issue during restoring our flink job from this change. Hope to get some help here. *1 - schema forbidden issue* In th

Re: changing the 'flink-main-container' name

Hi, By naming the container flink-main-container, Flink will know which container spec it should use for the Flink containers. If you change the name Flink won't know which container spec to use for the Flink container, and will probably think it's just a sidecar container, and there will still be

Re: Bloom Filter for Rocksdb

Hi, There have been no reports about setting this configuration causing any issues. I would guess it's off by default because it can increase the memory usage by an unpredictable amount. I would say feel free to enable it, from what you've said I also think that this would improve the performance

Re: Dealing with stale Watermark

Hi Irakli, If you see the watermarks tab on the operator do you see being propagated? If for example your source has multiple splits (like Kafka partitions) if one is idle or stays behind the watermark won’t be propagated as it is the minimum from all inputs (partitions/splits) Best On Fri, 20 Oc

Re: Bloom Filter for Rocksdb

I don’t know much about the performance improvements that may come from using bloom filters, but I believe you can also improve RocksDB performance by increasing managed memory https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#taskmanager-memory-managed-fraction which

Dealing with stale Watermark

Hello, I have a Flink application that is consuming events from the Kafka topic and builds sessions from them. I'm using the Keyed stream. The application runs fine initially, but after some time it is getting "stuck". I can see that the "processElement" function is processing the incoming even

Bloom Filter for Rocksdb

Can someone tell the exact performance effect of enabling bloom filter? May enabling it cause some unpredictable performance problems? I read what it is and how it works and it makes sense but I also asked myself why the default value of state.backend.rocksdb.use-bloom-filter is false. We have a

Instructions / steps regarding contributing : flink-training

Hi I want to add / modify a few unit tests for flink-training. I have came across following instructions : Contributing I am not sure about the formal process of submitting the change or any JIRA ticket etc requirements.

Re: Flink SQL exception on using cte

Thanks Robin and Aniket for the suggestions you have given. Will try and update on the same. Thanks, Elakiya On Fri, Oct 20, 2023 at 2:34 AM Robin Moffatt wrote: > CTEs are supported, you can see an example in the docs [1] [2]. In the > latter doc, it also says > > > CTEs are supported in View

changing the 'flink-main-container' name

Hello, We just adopted the flink operator. According to this link https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/ it prescribes a pod template containing among other things the following: containers:# Do not change the main container

Re: Flink SQL: non-deterministic join behavior, is it expected?

Hi Xuyang, A shuffle by join key is what I'd expect, but I don't see it. The issue only happens with parallelism > 1. > do you mean the one +I record and two +U records arrive the sink with random order? Yes. On Fri, Oct 20, 2023 at 4:48 AM Xuyang wrote: > Hi. Actually the results that arrive