Sending Avro Messages via Kafka From Flink

2022-12-16 Thread Paul O'Neill via user
Hello I have a generated Avro class (i.e. extends SpecificRecordBase). I am serializing instances of it in my flink application using: org.apache.avro avro ${avro.version} I send the resulting byte array out through my Flink Kafka Producer to a Kafka consumer (separate JVM, non-Flink) where I

Re: Slow restart from savepoint with large broadcast state when increasing parallelism

2022-12-16 Thread Jun Qin
Hi Ken, > Broadcast state is weird in that it’s duplicated, apparently avoid “hot > spots” when restoring from state. So I’m wondering how Flink handles the case > of restoring broadcast state when the parallelism increases. The Flink doc is here: https://nightlies.apache.org/flink/flink-docs-

Re: Slow restart from savepoint with large broadcast state when increasing parallelism

2022-12-16 Thread Ken Krugler
Hi Jun, Thanks for following up. The state storage is internal at a client, and isn’t throttled. Also restoring from the savepoint when we didn’t change the parallelism was fine. I didn’t see any errors in the TM logs, but I didn’t carefully inspect them - I’ll do that when we give this anoth

Re: Backpressure due to busy sub-tasks

2022-12-16 Thread Alexis Sarda-Espinosa
Hi Martijn, yes, that's what I meant, the throughput in the process function(s) didn't change, so even if they were busy 100% of the time with parallelism=2, they were processing data quickly enough. Regards, Alexis. Am Fr., 16. Dez. 2022 um 14:20 Uhr schrieb Martijn Visser < martijnvis...@apach

Re: Certificate rotation when using SSL

2022-12-16 Thread Steve Niemitz
I ended up submitting a PR that allows operators to implement rotation and more complicated TLS setups: https://github.com/apache/flink/pull/21457 On Fri, Dec 16, 2022 at 8:37 AM Őrhidi Mátyás wrote: > We've added something similar to the FKO: > https://github.com/apache/flink-kubernetes-operato

Re: Certificate rotation when using SSL

2022-12-16 Thread Őrhidi Mátyás
We've added something similar to the FKO: https://github.com/apache/flink-kubernetes-operator/pull/364 Best, Matyas On Fri, Dec 16, 2022 at 5:11 AM Martijn Visser wrote: > Hi Steve, > > I don't think that Flink has added support for certificate rotation. It > would be quite a nice feature if so

Re: Can't use nested attributes as watermarks in Table

2022-12-16 Thread Martijn Visser
Hi Theo, The most logical reason is that nested attributes were added later than watermarks were :) I agree that it's something that would be worthwhile to improve. If you can and want to make a contribution on this, that would be great. Best regards, Martijn On Wed, Dec 14, 2022 at 9:24 AM The

Re: Backpressure due to busy sub-tasks

2022-12-16 Thread Martijn Visser
Hi, Backpressure implies that it's actually a later operator that is busy. So in this case, that would be your process function that can't handle the incoming load from your Kafka source. Best regards, Martijn On Tue, Dec 13, 2022 at 7:46 PM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wro

Re: unsubscribe

2022-12-16 Thread Martijn Visser
Hi, You'll need to send an email to user-unsubscr...@flink.apache.org to unsubscribe yourself. Best Regards, Martijn On Sun, Dec 11, 2022 at 5:50 AM Saver Chia wrote: > unsubscribe >

Re: unsubscribe

2022-12-16 Thread Martijn Visser
Hi, You'll need to send an email to user-unsubscr...@flink.apache.org to unsubscribe yourself. Best Regards, Martijn On Sun, Dec 11, 2022 at 4:22 AM ganlute wrote: > unsubscribe >

Re: Certificate rotation when using SSL

2022-12-16 Thread Martijn Visser
Hi Steve, I don't think that Flink has added support for certificate rotation. It would be quite a nice feature if someone could contribute to it. Best regards, Martijn On Fri, Dec 2, 2022 at 3:51 PM Steve Niemitz wrote: > We're investigating using internal SSL for our flink deployments, but

Re: Reduce checkpoint-induced latency by tuning the amount of resource available for checkpoints

2022-12-16 Thread Hangxiang Yu
Hi, Robin. >From your code path (*FullSnapshotAsyncWriter.writeKVStateData*), I guess your version of Flink was below 1.16 and you adapted the default config of 'state.backend.incremental'. In the version below 1.16, RocksDBStateBackend will use savepoint format as its full snapshot[1]. So it will

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.3.0 released

2022-12-16 Thread Biao Geng
Congratulations! Thanks a lot for the awesome work! Best regards, Biao Geng Őrhidi Mátyás 于2022年12月15日周四 01:30写道: > The Apache Flink community is very happy to announce the release of Apache > Flink Kubernetes Operator 1.3.0. > > Release highlights: > >- Upgrade to Fabric8 6.x.x and JOSDK

Re: Savepoint a failing job

2022-12-16 Thread Martijn Visser
Hi Tim, If I understand correctly, you need to deploy a new SQL statement in order to fix your issue? If so, the problem is that a new SQL statement might lead to a different execution plan which can't be restored. See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/ov