Re: Using Prometheus Client Metrics in Flink

Rion, Regarding the second question , you can aggregate by using sum function sum(metric_name{jobb_name="JOBNAME"}) . This works is you are using the metric counter. Prasanna. On Sat, Feb 27, 2021 at 9:01 PM Rion Williams wrote: > Hi folks, > > I’ve just recently started working with Flink an

Re: Producer Configuration

They are picked up, otherwise you would not able to write any messages at all. I believe the page you referring is not for displaying Kafka properties (in my case it is empty as well, but Kafka works). Check logs. From: Claude M Sent: Saturday, February 27, 2021

Re: Producer Configuration

Yes, the flink job also works in producing messages. It's just that after a short period of time, it fails w/ a timeout. That is why I'm trying to set a longer timeout period but it doesn't seem like the properties are being picked up. On Sat, Feb 27, 2021 at 1:17 PM Alexey Trenikhun wrote: >

Re: Does the Kafka source perform retractions on Key?

Hi Arvid, >If you are not rereading the topics, why do you compact them? We are rereading the topics, at any time we might want a completely different materialized view for a different web service for some new application feature. Other jobs / new jobs need to read all the up-to-date rows from the

Re: Does the Kafka source perform retractions on Key?

Hi Rex, Your initial question was about the impact of compaction on your CDC application logic. I have been (unsuccessfully) trying to tell you that you do not need compaction and it's counterproductive. If you are not rereading the topics, why do you compact them? It's lost compute time and I/O

Re: Does the Kafka source perform retractions on Key?

Hi Arvid, I really appreciate the thorough response but I don't think this contradicts our use case. In servicing web applications we're doing nothing more than taking data from giant databases we use, and performing joins and denormalizing aggs strictly for performance reasons (joining across a l

Re: Producer Configuration

Can you produce messages using Kafka console producer connect using same properties ? From: Claude M Sent: Saturday, February 27, 2021 8:05 AM To: Alexey Trenikhun Cc: user Subject: Re: Producer Configuration Thanks for your reply, yes it was specified. Sorry

Job downgrade

Hello, Let's have version 1 of my job uses keyed state with name "a" and type A, which some Avro generated class. Then I upgrade to version 2, which in addition uses keyed state "b" and type B (another concrete Avro generated class), I take savepoint with version 2 and decided to downgrade to ve

Stateful functions 2.2 and stop with savepoint

I have an embedded function with a SinkFunction as an egress, implemented as this pseudo-code: val serializationSchema = KafkaSchemaSerializationSchema(... props required to use a Confluent Schema Registry with Avro, auth etc ...) return SinkFunctionSpec(EGRESS_ID, FlinkKafkaProducer(serializati

Re: Handling Data Separation / Watermarking from Kafka in Flink

Thanks David, I figured that the correct approach would obviously be to adopt a keying strategy upstream to ensure the same data that I used as a key downstream fell on the same partition (ensuring the ordering guarantees I’m looking for). I’m guessing implementation-wise, when I would normally

Re: Handling Data Separation / Watermarking from Kafka in Flink

Rion, If you can arrange for each tenant's events to be in only one kafka partition, that should be the best way to simplify the processing you need to do. Otherwise, a simple change that may help would be to increase the bounded delay you use in calculating your own per-tenant watermarks, thereby

Re: Producer Configuration

Thanks for your reply, yes it was specified. Sorry I forgot to include it: properties.setProperty("bootstrap.servers", "localhost:9092"); On Fri, Feb 26, 2021 at 7:56 PM Alexey Trenikhun wrote: > I believe bootstrap.servers is mandatory Kafka property, but it looks like > you didn’t set it > >

Using Prometheus Client Metrics in Flink

Hi folks, I’ve just recently started working with Flink and I was in the process of adding some metrics through my existing pipeline with the hopes of building some Grafana dashboards with them to help with observability. Initially I looked at the built-in Flink metrics that were available, but

Re: Does the Kafka source perform retractions on Key?

Hi Rex, imho log compaction and CDC for historic processes are incompatible on conceptual level. Let's take this example: topic: party membership +(1, Dem, 2000) -(1, Dem, 2009) +(1, Gop, 2009) Where 1 is the id of a real person. Now, let's consider you want to count memberships retroactively ea