Polling behaviour when a consumer assigned with multiple topics-partition

Hi, I want to know the polling behaviour when a consumer is assigned with multiple topic-partitions. 1. In a single poll will it get messages from multiple topic-partitions. Or in one poll only one topic-partition's messages will come? 2. How does it choose topic-partitions for next polling? Any

Re: Send several Serialized Java objects under one Kafka Topic

Thanks you all for the answer, >Use Avro and a discriminated union / sum type to combine all the types. Can you share a bit more about this approach, plase? BR, Pater On Mon, Feb 1, 2021 at 12:54 AM Christopher Smith wrote: > > Use Avro and a discriminated union / sum type to combine all the typ

Re: Send several Serialized Java objects under one Kafka Topic

Use Avro and a discriminated union / sum type to combine all the types. On Sun, Jan 31, 2021, 6:49 AM Peter Penzov wrote: > Hello All, > I'm working on proof of concept for sending several Java Objects > under one Kafka Topic. More about the requirements: > > https://stackoverflow.com/quest

Re: Send several Serialized Java objects under one Kafka Topic

Hi Peter, The biggest issue with Java serialization has always been compatibility between versions of the classes between producer and consumer, it can be avoided via very careful management, but it's a lot more painful (IMO) than other serialisation formats. If you're looking to avoid JSON, I re

Re: Send several Serialized Java objects under one Kafka Topic

" I want to use a serialized Java object, not JSON because I think it's more optimized for performance." => Bad assumption. https://www.robert-franz.com/2013/12/09/comparing-serialization-performance/ Avro and protobuf are oflently used, Json is easier https://blog.softwaremill.com/the-best-serial

Re: Event Sourcing with Kafka Streams and processing order of a re-entrant pipeline

Hi David, Thank you for the question. If I can confirm, it looks like the "operations" topic is the only input to the topology, and the topology reads the "operations" topic joined with the "account" table and generates a "movements" stream. It reads (and aggregates) the "movements" stream to cre

ReplicaManager fetch fails on leader due to long/integer overflow

Hi, I’m having what seems to be this issue: https://issues.apache.org/jira/browse/KAFKA-7656 It happens when a transactional producer sends an event for the 1st time. ``` ERROR [

Send several Serialized Java objects under one Kafka Topic

Hello All, I'm working on proof of concept for sending several Java Objects under one Kafka Topic. More about the requirements: https://stackoverflow.com/questions/65811681/design-kafka-consumers-and-producers-for-scalability I managed to implement this working concept: https://github.com/rcb

Event Sourcing with Kafka Streams and processing order of a re-entrant pipeline

I'm working on a project where I want to use Kafka Streams for Event Sourcing. General idea is that I have a "commands" topic/KStream, an "events" topic/KStream and a "snapshots" topic/KTable. Snapshots contains the current state of the entities. Commands are validated using the "snapshots" and tr