Re: Documentation and meaning of configuration 'retention.bytes'

Well spotted I think – I was briefly puzzled with the time retention behaviour, as segments seemed to live longer than advertised – until I realised it was min time, deletion is lazy – can occur at some (distant?) time in the future (and is async I think) – this was particularly noticeable for

Support for other OAuth2 grant types in Kafka

Hello All, I see that Kafka by itself supports client credentials as grant type for OAuth2. I see this mentioned in one of the kafka KIP as well: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186877575 Is there a way to support other grant types as well? I came across the follo

Re: Optimizing Kafka Streams Validation for Array Elements

I will provide you with the sample source code on GitHub today. Dnia 25 lutego 2025 20:47:36 CET, Bruno Cadonna napisał/a: >Hi Pawel, > >What is the "automatic json schema validation feature"? >Streams does not have such a thing built-in. > >Do you have an example of a third-party software that d

Re: Optimizing Kafka Streams Validation for Array Elements

Sorry for that, I should be more precise in that from the beginning. By automatic schema validation I mean SerDe combined with Default Deserialisation Exception Handler along with dlq topic as you can find in official streams documentation

Picking up the thread on emit-on-change and at-least-once

Hi kafka-users, We are implementing a Kafka Streams app that computes various streaming statistics over a corpus of data stored in Kafka topics. While some aggregates update often, others like 'min', 'max', or histogram buckets could have relatively few distinct updates relative to the input data.

Re: Optimizing Kafka Streams Validation for Array Elements

To make it clear I will give you an example: On the source topic we have: offset: 0 key: null value: { "request":{ }, "response":{ }, "data":[ { "deviceId":"23dc78ffa6c1ad3c038b", "event":{ "lvl":2, "someValue":{ "hori

Re: Optimizing Kafka Streams Validation for Array Elements

Hi Pawel, What is the "automatic json schema validation feature"? Streams does not have such a thing built-in. Do you have an example of a third-party software that does that validation? Regarding decomposing your input records into the smaller array elements, that should work with builder.s

Re: Optimizing Kafka Streams Validation for Array Elements

Hi Pawel, I am not completely sure I understand the issue, because I am not a JSON expert. Is my understanding correct that the serde that you use to read from the input topic and from the repartition topic also do the validation of the JSON? Regarding point 2 and 3: I agree that the depen

Re: Optimizing Kafka Streams Validation for Array Elements

Ideal solution has two topics, can we somehow do flatMap and change a key without changing creating the internal repartition topic? This will allow us to skip using the internal repartition topic as source for k table. Dnia 25 lutego 2025 15:41:19 CET, Bruno Cadonna napisał/a: >Hi Pawel, > >I a

Optimizing Kafka Streams Validation for Array Elements

Dear Kafka users, The last few days I spent working with Kafka Streams on some tasks which looked very easy at first glance but finally I struggled with the Streams Builder API and did something which I am not proud of. Please help me, I am open to any suggestions. On the input topic we have a mes