Re: Need broker side property which can restrict a fetch.size

Hi Shivam, You can configure quotas per user/client. [1] https://kafka.apache.org/documentation/#design_quotas Regards, Srinath On Jan 9, 2018 5:19 PM, "Shivam Sharma" <28shivamsha...@gmail.com> wrote: > Hi, > > We are having Kafka cluster(0.11.0.0) of 3 Machines. When we increase > *max.fetch

Re: Kafka Streams | Impact on rocksdb stores by Rebalancing

Ok, Matthius. Thanks for correcting. On Wed, Jan 10, 2018 at 3:18 AM, Matthias J. Sax wrote: > Sameer, > > the KIP you are pointing to is not related to Kafka Streams' > task/partition assignment. Kafka Streams uses it's own implementation of > a partitioning assigner (not the default one the co

Re: Kafka per topic retention.bytes and global log.retention.bytes not working

Upgrade? On Wed, Jan 10, 2018, 00:26 Thunder Stumpges wrote: > How would I know if we are seeing that issue? We are running 0.11.0.0 so we > would not have this fix. > > On Tue, Jan 9, 2018 at 11:07 AM Wim Van Leuven < > wim.vanleu...@highestpoint.biz> wrote: > > > What minor version of Kafka ar

Re: Best practice for publishing byte messages to Kafka

Byte Array is essentially "serialized" already isn't it? I mean the message itself is sent as a byte array, so the default byte array serializer is as "efficient" as it gets, as it's just sending your byte array through as the message... there's no serialization happening. -Thunder On Tue, Jan 9,

Re: Best practice for publishing byte messages to Kafka

Thanks, Matt. Have you done any benchmarking to see how using different Serializers may impact throughput/latency? Regards, Ali On Wed, Jan 10, 2018 at 7:55 AM, Matt Farmer wrote: > We use the default byte array serializer provided with Kafka and it works > great for us. > > > On Jan 9, 2018, a

Triggers for offline partition

Hi We have several small kafka clusters (0.10.2) of 4 nodes running on cloud hosted VMs (each node runs on separate VM). We are having an issue of some random partitions going offline intermittently every few days at a fixed time (8 am) in a particular datacenter (identical clusters in other d

Re: Kafka per topic retention.bytes and global log.retention.bytes not working

How would I know if we are seeing that issue? We are running 0.11.0.0 so we would not have this fix. On Tue, Jan 9, 2018 at 11:07 AM Wim Van Leuven < wim.vanleu...@highestpoint.biz> wrote: > What minor version of Kafka are you running? Might you be impacted by > https://issues.apache.org/jira/bro

Broker won't exit...

I only discovered the kafka-server-stop.sh script a couple days ago. I can't seem to make it do its thing (the corresponding zookeeper stop script seems to work just fine). All consumers have been stopped. Lsof still shows the Kafka broker process listening on its port. The last connection left the

Upgrading Kafka from Version 0.10.2 to 1.0.0

Hello All, Is it easy to upgrade from Kafka version 0.10.2 to 1.0.0 or do I need to upgrade to version 0.11.0 first? Anything to expect? Thanks

Re: Kafka Streams | Impact on rocksdb stores by Rebalancing

Sameer, the KIP you are pointing to is not related to Kafka Streams' task/partition assignment. Kafka Streams uses it's own implementation of a partitioning assigner (not the default one the consumer uses). -Matthias On 1/9/18 4:22 AM, Sameer Kumar wrote: > Got It. Thanks. Others can also take a

Re: Best practice for publishing byte messages to Kafka

We use the default byte array serializer provided with Kafka and it works great for us. > On Jan 9, 2018, at 8:12 AM, Ali Nazemian wrote: > > Hi All, > > I was wondering whether there is any best practice/recommendation for > publishing byte messages to Kafka. Is there any specific Serializer

Re: Kafka per topic retention.bytes and global log.retention.bytes not working

What minor version of Kafka are you running? Might you be impacted by https://issues.apache.org/jira/browse/KAFKA-6030? -w On Tue, 9 Jan 2018 at 19:02 Thunder Stumpges wrote: > Hello, I posted this on StackOverflow > < > https://stackoverflow.com/questions/47948399/kafka-per-topic-retention-byte

Kafka per topic retention.bytes and global log.retention.bytes not working

Hello, I posted this on StackOverflow also but haven't gotten any response. thanks in advance, Thunder __ We are running a 6 node cluster of kafka 0.11.0. We have set a

UNKNOWN_TOPIC_OR_PARTITION for kafka producer

Hi All, I have SSL from client to broker,SSL again for broker to borker communication and kerberos from broker to ZK The topics exists when i do describe from ZK Cli. Authorization log clearly says, it can allow principal to describe and write to the topic. When producing the message using kafka

Best practice for publishing byte messages to Kafka

Hi All, I was wondering whether there is any best practice/recommendation for publishing byte messages to Kafka. Is there any specific Serializer that is recommended for this matter? Cheers, Ali

Re: Kafka Streams | Impact on rocksdb stores by Rebalancing

Got It. Thanks. Others can also take a look at https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy -Sameer. On Tue, Jan 9, 2018 at 5:33 PM, Damian Guy wrote: > Hi, > > yes partition assignment is aware of the standby replicas. It will try and > assign

Re: Kafka Streams | Impact on rocksdb stores by Rebalancing

Hi, yes partition assignment is aware of the standby replicas. It will try and assign tasks to the nodes that have the state for the task, but also will try and keep the assignment balanced. So the assignment will be more like your second assignment. If you are interested you can have a look at: h

Need broker side property which can restrict a fetch.size

Hi, We are having Kafka cluster(0.11.0.0) of 3 Machines. When we increase *max.fetch.size.per.partition* to around 50 MB for topic having 48 partition, our Kafka Broker is getting OOM. As our cluster has many consumers, so restricting fetch size on consumer side is very difficult which is continu

Re: Kafka Streams | Impact on rocksdb stores by Rebalancing

Hi Damian, Thanks for your reply. I have some further ques. Would the partition assignment be aware of the standby replicas. What would be the preference for task distribution: load balancing or stand by replicas. For e.g N1 assigned partitions: 1,2 standby partitions: 5,6 N2 assigned partitio

Re: Less poll interval on StoreChangelogReader

State Store restoration is done on the same thread as processing. It is actually interleaved with processing, so we keep the poll time small so that if there is no data immediately available we can continue to process data from other running tasks. On Tue, 9 Jan 2018 at 08:03 Sameer Kumar wrote:

Re: Kafka Streams | Impact on rocksdb stores by Rebalancing

On Tue, 9 Jan 2018 at 07:42 Sameer Kumar wrote: > Hi, > > I would like to understand how does rebalance affect state stores > migration. If I have a cluster of 3 nodes, and 1 goes down, the partitions > for node3 gets assigned to node1 and node2, does the rocksdb on node1/node2 > also starts upda

Less poll interval on StoreChangelogReader

In StoreChangelogReader.restore, we have a very short poll interval of 10 ms. Any specfic reasons for the same. -Sameer.