Re: Kafka segment.index.bytes

Thanks, Ismael Updating to Java 8 version 202 solved the problem! On Sun, Feb 24, 2019 at 10:09 AM Soheil Pourbafrani wrote: > Thanks, I use CentOS 7 and Java 8 (192) > > > I'll try updating Java to 202 > > On Sat, Feb 23, 2019 at 11:05 PM Ismael Juma wrote: >

Re: Kafka segment.index.bytes

aking all the space. It was fixed in Java 8 update 202. > > Ismael > > On Sat, Feb 23, 2019 at 2:19 AM Soheil Pourbafrani > wrote: > > > Hi, > > > > I have a 2-node Kafka cluster. When I create a new topic it preallocates > > about 21MB for each partition:

Kafka segment.index.bytes

Hi, I have a 2-node Kafka cluster. When I create a new topic it preallocates about 21MB for each partition: 10M /data/1-0/.index 0 /data/1-0/.log 10M /data/1-0/.timeindex 4.0K /data/1-0/leader-epoch-checkpoint And because I have many top

Re: Each empty partition has 21MB by default

sing the property: segment.index.bytes > > You can find more information by searching that property in the > documentation: https://kafka.apache.org/documentation/ > > On Tue, Feb 19, 2019 at 6:39 AM Soheil Pourbafrani > wrote: > > > Recently I noticed that when I create a new em

Each empty partition has 21MB by default

Recently I noticed that when I create a new empty topic, each partition will have 21MB data! I even change the Kafka version from 0.11.3 to 2.1.0 but it shows the same behavior. For example, if I create a new topic using the command : kafka-topics.sh --zookeeper zoo1:2181 --create --topic test --r

Create empty topic result in massive data in the log-dir directory

Hi, I have a single node Kafka broker and 3-node Zookeeper cluster. When I create a new topic with 24 partitions using the command: kafka-topics.sh --zookeeper zoo3:2181 --create --topic test_topic --replication-factor 1 --partitions 24 It results in 481M data in the log-dir!? Actually, every p

Re: Kafka topic data automatically restore after deleting topics

Thanks, > Anji. > > On Sun, 3 Feb 2019, 17:27 Soheil Pourbafrani > > Hi, > > > > I use Kafka 0.11.3 > > > > I delete topics from Kafka using the command *kafka-topic.sh *and after a > > while, I observer Kafka just mark topics for deletion while the >

Kafka topic data automatically restore after deleting topics

Hi, I use Kafka 0.11.3 I delete topics from Kafka using the command *kafka-topic.sh *and after a while, I observer Kafka just mark topics for deletion while the property *delete.topic.enable *was true! So I stop Kafka cluster and delete the data log directory of Kafka using the command *rm -r /da

Re: communication between brokers doesn't establish

EDIT: At the end of the controller.log logs the following line should be added: ERROR [Controller 100]: Error while electing or becoming controller on broker 100 (kafka.controller.KafkaController) java.util.NoSuchElementException: key not found: 0 On Sat, Jan 5, 2019 at 4:21 PM Soheil

communication between brokers doesn't establish

Hi, using Kafka 11.0.3 I set up a 2 node Kafka cluster on CentOS 7 with broker id 100 and 101. There is no firewall enabled. Both nodes can ping each other and establish the SSH connection successfully. When I start the cluster both broker successfully register their id into the zookeeper cluster (

Re: Kafka sometime truncate new messages

at 13:02, Soheil Pourbafrani > wrote: > > > In addition, I should say this behavior is for when topics are created > > automatically. > > > > On Mon, Jul 30, 2018 at 2:01 PM, Soheil Pourbafrani < > soheil.i...@gmail.com > > > > > wrote: > > >

Re: Kafka sometime truncate new messages

In addition, I should say this behavior is for when topics are created automatically. On Mon, Jul 30, 2018 at 2:01 PM, Soheil Pourbafrani wrote: > I have a 3 node Kafka cluster. Upgrading Kafka version from 0.11.0.2 to > 0.11.0.3, I notice that Kafka sometimes truncate new message's

Kafka sometime truncate new messages

I have a 3 node Kafka cluster. Upgrading Kafka version from 0.11.0.2 to 0.11.0.3, I notice that Kafka sometimes truncate new message's topic. Here is the log: [2018-07-30 09:19:39,612] WARN [ReplicaFetcherThread-0-101]: Based on follower's leader epoch, leader replied with an unknown offset in 779

Re: Kafka metrics attribute meaning

here : > http://metrics.dropwizard.io/2.2.0/getting-started/ > > > On Sat, Feb 24, 2018 at 8:31 PM, Soheil Pourbafrani > > wrote: > > > Hi, > > > > What is the exact meaning of the Kafka metrics attributes, for example, > the > > attribute > > kaf

Re: Kafka Zookeeper Connection

Thanks, Manna, Can you say which property (in Kafka or Zookeeper) should I increase? On Sat, Feb 24, 2018 at 7:49 PM, M. Manna wrote: > Have you tried increasing the timeouts for zookeeper nodes and Kafka > brokers to see if they make a difference? > > > > On Sat, 24 Feb 201

Kafka metrics attribute meaning

Hi, What is the exact meaning of the Kafka metrics attributes, for example, the attribute kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec has the attributes MeanRate and OneMinuteRate. I am confused about the attribute OneMinuteRate! It is for the last minutes or for the whole broker runt

Re: Kafka Zookeeper Connection

Hi and Thanks, Excuse me, The Kafka version is 0.11 and Zookeeper version is 3.4.10 I've check Zookeeper logs and sessions are expiring and renewing continuously there. I use the same Zookeeper cluster for Hadoop HA and it works well. I Answer the same question with more details in the following

Kafka Zookeeper Connection

In Kafka 0.11 server.log, it continuously logs [2018-02-24 10:38:11,178] WARN Client session timed out, have not heard from server in 4001ms for sessionid 0x361c65434f1000c (org.apache.zookeeper.ClientCnxn) [2018-02-24 10:38:11,178] INFO Client session timed out, have not heard from server in 4001