Re: kafka log.retention.bytes

2021-02-24 Thread Peter Bukowinski
log.retention.bytes is a broker-level config that sets the maximum size of a topic partition on a broker, so it will apply to all topics… unless a topic has the retention.bytes property configured — this is a topic-level config and only applies to a single topic — in which case that takes prece

Create a new consumer for a consumer group from within the assign method of the rebalancing interface (AbstractPartitionAssignor)

2021-02-24 Thread Mazen Ezzeddine
I am running a Kafka cluster on Kubernetes. I am implementing a custom PartitionAssignor to personalize the way topic partitions are assigned to existing consumers in the consumer group. To this end, I am overriding the method Map assign( Cluster metadata, Map subscriptions) If inside the assi

kafka log.retention.bytes

2021-02-24 Thread Calvin Chen
Hi all I have question about Kafka topic log retention on bytes, does the log.retention.bytes apply to each topic or it apply to all topics in broker? If it apply to each topic, then, when topic numbers keep growing in broker, how can we make sure total disk size for all topic logs will not exc

Re: NullPointerException after upgrading from 2.5.1 to 2.6.1 in my stream app

2021-02-24 Thread Nitay Kufert
I guess it's possible but very unlikely because it works perfectly with all the previous versions and the current one? (2.5.1) Why did a change in the version introduce NULLS there? On Tue, Feb 23, 2021 at 9:16 PM Guozhang Wang wrote: > Is it possible that the flattened values contain `null` and