Re: Kafka Streams: Is it possible to set heap size for the JVMs running Kafka Streams?

Thanks very much Eno for the promptly help. Very appreciated! Tianji On Thu, Apr 6, 2017 at 7:23 PM, Eno Thereska wrote: > Hi Tianji, > > Yes, with some caveats. First, to set the heap size of the JVM, you can > treat it just like any other Java program, e.g., see > http://stackoverflow.com/que

Are consumer coordinators not replicated in case of failure?

Hey all, Question. I have three brokers. I also have 3 consumers on their own thread consuming 3 partitions of the same topic "scaled-states". Here are the configs when I run: kafka-topics.sh --describe --topic 'scaled-cities' --zookeeper zoo2:2181 Topic:scaled-cities PartitionCount:3 Replicat

recommendation on kafka go client : sarama or confluent-kafka-go

Hi all, We need to write a service in Go to consume from kafka. There are two popular kafka clients: sarama and confluent-kafka-go . Sarama client is written purely in Go. This allow it to show the full Go st

Re: Kafka connector

If you want both N2 and N3 to get all the same messages (rather than each getting an exclusive partitioned subset of the data) then you need to configure N2 and N3 to be in unique Kafka consumer groups which I believe is driven off the "name" of the N2 and N3 connectors. Make sure N2 and N3 have di

Are consumer coordinators not replicated in case of failure?

Hey all, Question. I have three brokers. I also have 3 consumers on their own thread consuming 3 partitions of the same topic "scaled-states". Here are the configs when I run kafka-topics.sh --describe --topic 'scaled-cities' --zookeeper zoo2:2181 Topic:scaled-cities PartitionCount:3 Replicatio

Kafka connector

Hello Sir/Ma'am, I was trying to write a simple case of using kafka connector. My setup involves using three nodes N1,N2 and N3. N1 is the source and N2, N3 are the sink nodes in my case. I am writing data to a text file(say input.txt) on Node N1 and using the standalone kafka connector I wish to s

Re: Kafka Streams: Is it possible to set heap size for the JVMs running Kafka Streams?

Hi Tianji, Yes, with some caveats. First, to set the heap size of the JVM, you can treat it just like any other Java program, e.g., see http://stackoverflow.com/questions/27681511/how-do-i-set-the-java-options-for-kafka

Kafka Streams: Is it possible to set heap size for the JVMs running Kafka Streams?

If so, how to? Thanks Tianji

Re: Topic deletion

What version of kafka you are on? There are issues with delete topic until I guess kafka 0.10.1.0 You may be hitting this issue… https://issues.apache.org/jira/browse/KAFKA-2231 Thanks Zakee > On Apr 6, 2017, at 11:20 AM, Adrian McCague wr

RE: Topic deletion

Hi Sachin, I am told with confidence that the setting is present in server.properties (I do not have access to the server). I have used delete before without issue, and attempted to use it when the setting is false and the behaviour is different. I'm certain that running the command multiple t

Replica selection at the time of topic creation

Hello, I tried to understand the algorithm used in choosing a leader of a topic / partition at the time of creation - rack unaware mode. I went thru TopicCommand.scala and AdminUtils.scala, especially assignReplicasToBrokersRackUnaware() function (pasted below for convenience) and figured that

Re: Topic deletion

Do you have delete.topic.enable=true uncommented or present in server.properties On Thu, Apr 6, 2017 at 11:03 PM, Adrian McCague wrote: > Hi all, > > I am trying to understand topic deletion in kafka, there appears to be > very little documentation or answers on how this works. Typically they ju

NotLeaderForPartitionException in 0.8.x

Hi All, I understand that we get a NotLeaderForPartitionException when the client tries to publish a message to a server which is not the Leader for a given partition. I also understand that once leadership does change the client will learn that and start publishing messages to the right server

Configuring consumer client to throw an exception when Kafka fails

Hi all, We're trying to make our Kafka consumers fail fast, rather than run forever. We're using Kubernetes to run our containers, and we are taking a fail fast approach, and letting K8s manage restarts. What I want to happen 1) Start my consumer. This connects to Kafka 2) Kill my Kafka cluste

Topic deletion

Hi all, I am trying to understand topic deletion in kafka, there appears to be very little documentation or answers on how this works. Typically they just say to turn on the feature on the broker (in my case it is). I executed: Kafka-topics.bat -delete -zookeeper keeperhere -topic mytopic Runn

issues with leader election

HI, I ran into an issue with kafka where the leader was set to -1 : Topic:AP-100PartitionCount:5ReplicationFactor:3 Configs:retention.ms=1579929985 Topic: AP-100 Partition: 0Leader: 26 Replicas: 26,24,25 Isr: 24,26,25 Topic: AP-100 Partition: 1

Re: best practices to handle large messages

Hi, you might be interested by this presentation https://www.slideshare.net/JiangjieQin/handle-large-messages-in-apache-kafka-58692297 On Wed, Apr 5, 2017 at 1:27 AM, Mohammad Kargar wrote: > What are best practices to handle large messages (2.5 MB) in Kafka? > > Thanks, > Mohammad > -- The in