Re: Transactional markers are not deleted from log segments when policy is compact

2019-06-02 Thread Pranavi Chandramohan
Hi Jonathan, After looking at the changes in PR, I would like to ensure the fix would work for our scenario too. We create a new TransactionId for every producer instance every time it restarts and this would result in creating new ProducerIds. Old producers and their transactions will get expired

Scaling Kafka Cluster in Cloud

2019-06-02 Thread Varun Kumar
Hi, I have a Kafka cluster setup in cloud, I tried a different approach to scale it. I gracefully shutdown a broker with 2 (or more) disks and attach 2 brokers with 1 (or equally distributed) disks each. (Had to use partition reassignment tool change partitions after new broker is attached). I

How to prevent data loss in "read-process-write" application?

2019-06-02 Thread 1095193...@qq.com
Hi I have a application consume from Kafka, process and send to Kafka. In order to prevent data loss, I need to commit consumer offset after committing a batch of messages to Kafka successfully. I investigate Transaction fearture that provided atomic writes to multiple partitions could solv

Tool to automate kafka cluster configuration management

2019-06-02 Thread Navdeep Sekhon
Hi guys, I just created a tool to automate configuration management for kafka cluster. I thought I’d share it here since it might be useful for others: https://github.com/navdeepsekhon/kafkaer Your feedback is welcome. Thanks, Navdeep