Re: Topics being automatically deleted?

Thanks, yeah, that must be it. So topics don't get deleted if all messages in them expire, right? Good to know :) On Thu, Sep 15, 2016 at 11:29 AM, Manikumar Reddy wrote: > looks like you have not changed the default data log directory. By default > kafka is configured to store the data logs to

Re: hi

It sounds like a network issue. Where are the 3 servers located / hosted? On Thu, Sep 15, 2016 at 11:51 AM, kant kodali wrote: > Hi, > I have the following setup. > Single Kafka broker and Zookeeper on Machine 1single Kafka producer on > Machine 2 > Single Kafka Consumer on Machine 3 > When a pr

hi

Hi, I have the following setup. Single Kafka broker and Zookeeper on Machine 1single Kafka producer on Machine 2 Single Kafka Consumer on Machine 3 When a producer client sends a message to the Kafka broker by pointing at the Zookeeper Server the consumer doesn't seem to get the message right away

Re: Topics being automatically deleted?

looks like you have not changed the default data log directory. By default kafka is configured to store the data logs to /tmp/ folder. /tmp gets cleared on system reboots. change log.dirs config property to some other directory. On Thu, Sep 15, 2016 at 11:46 AM, Ali Akhtar wrote: > I've noticed

Topics being automatically deleted?

I've noticed that, on my own machine, if I start a kafka broker, then create a topic, then I stop that server and restart it, the topic I created is still kept. However, on restarts, it looks like the topic is deleted. Its also possible that the default retention policy of 24 hours causes the mes

Re: How to create a topic using the Java API / Client?

I'm guessing its not possible to delete topics? On Thu, Sep 15, 2016 at 5:43 AM, Ali Akhtar wrote: > Thank you Martin > > On 15 Sep 2016 3:05 am, "Mathieu Fenniak" > wrote: > >> Hey Ali, >> >> If you have auto create turned on, which it sounds like you do, and you're >> happy with using the br

Kafka Source Connector - JSON with Escape double quotes

Hi, I am using Kafka connect to feed data into kafka, please find sample code snippet from Source Task. My question is , why kafka connect dematerialize JSON message by escaping double quotes? Is there way to get the same without escaping? byte[] message = null; Charset charset = Charset.forName

Re: what's the relationship between Zookeeper and Kafka ?

In addition to what Michael noted, this question has been asked a few times before too and here's one such previous discussion https://www.quora.com/What-is-the-actual-role-of-ZooKeeper-in-Kafka -Jaikiran On Wednesday 14 September 2016 03:50 AM, Michael Noll wrote: Eric, the latest versions

Re: kafkaproducer send blocks until broker is available

This is a known issue and is being tracked in this JIRA https://issues.apache.org/jira/browse/KAFKA-3539 -Jaikiran On Saturday 10 September 2016 12:20 AM, Peter Sinoros Szabo wrote: Hi, I'd like to use the Java Kafka producer in a non-blocking async mode. My assuptions were that until the new

Re: Too many open files

What does the output of: lsof -p show on that specific node? -Jaikiran On Monday 12 September 2016 10:03 PM, Michael Sparr wrote: 5-node Kafka cluster, bare metal, Ubuntu 14.04.x LTS with 64GB RAM, 8-core, 960GB SSD boxes and a single node in cluster is filling logs with the following: [20

Re: Too many open files

What does the output of: lsof -p show? -Jaikiran On Monday 12 September 2016 10:03 PM, Michael Sparr wrote: 5-node Kafka cluster, bare metal, Ubuntu 14.04.x LTS with 64GB RAM, 8-core, 960GB SSD boxes and a single node in cluster is filling logs with the following: [2016-09-12 09:34:49,522]

Re: Anyone running Kafka on Kubernetes in production?

Also interested in hearing responses on this, so please respond to the list and not just Ali. On Wed, Sep 14, 2016 at 5:55 PM, Ali Akhtar wrote: > If so, can you please share if you're using a publicly available > deployment, or if you created you own, how you did it? (I.e which services > / rep

Re: Exception while deserializing in kafka streams

Guozhang, I am using 0.10.0.0. Could the below log be the cause? 16/09/14 17:24:35 WARN ConsumerConfig: The configuration schema.registry.url = http://192.168.50.6: 8081 was supplied but isn't a known config. 16/09/14 17:24:35 INFO AppInfoParser: Kafka version : 0.10.0.

Anyone running Kafka on Kubernetes in production?

If so, can you please share if you're using a publicly available deployment, or if you created you own, how you did it? (I.e which services / replication controllers you have) Also, how has the performance been for you? I've read a report which said the performance suffered running kafka as a dock

Re: How to create a topic using the Java API / Client?

Thank you Martin On 15 Sep 2016 3:05 am, "Mathieu Fenniak" wrote: > Hey Ali, > > If you have auto create turned on, which it sounds like you do, and you're > happy with using the broker's configured partition count and replication > factor, then you can call "partitionsFor(String topic)" on you

Consumer stops after reaching an offset of 1644

Hi All, I am trying to do a simple benchmark test  for Kafka using single broker, producer and consumer however my consumer doesn't seem to receive all the messages produced by the producer so not sure what is going on any help? Here is the full description of the problem. http://stackoverflow.com/

Re: Exception while deserializing in kafka streams

Hello Walter, Which version of Kafka were you using? I ask this because there was a bug causing the serde passed through config to NOT being configured when constructed: https://issues.apache.org/jira/browse/KAFKA-3639 Which is fixed in the 0.10.0.0 release, which means you will only hit it if

Kafka consumer group problem

Hi, I created a few consumers that belong to the same group_id, but I noticed that each consumer get all messages instead of only some of the messages. As for the topic, I did create the topic with a few partitions. Anyone else had the same problem? Is there any configuration parameter I need

Re: How to create a topic using the Java API / Client?

Hey Ali, If you have auto create turned on, which it sounds like you do, and you're happy with using the broker's configured partition count and replication factor, then you can call "partitionsFor(String topic)" on your producer. This will create the topic without sending a message to it. I'm no

Re: How to create a topic using the Java API / Client?

You can make following setting to set partition count, etc auto.create.topics.enable = true(enable auto creation of topic on the server) num.partitions = 1 (number of log partitions per topic) On 9/14/16, 2:41 PM, "Ali Akhtar" wrote: It looks like if I

Re: How to create a topic using the Java API / Client?

It looks like if I just send a message to a non-existent topic, it is created. But this has the downside that the first message of the topic is null or otherwise invalid. Also the partition count can't be specified. Is there a way to create a topic without needing to post a null message? Do I nee

Re: MockClientSupplier

Hello Andy, You can just call KafkaStreams.toString(). I'm copying its java doc string here: "Produces a string representation contain useful information about Kafka Streams * Such as thread IDs, task IDs and a representation of the topology. This is useful * in debugging scenarios." Guozhang

Re: Unexpected KStream-KStream join behavior with asymmetric time window

Hello Elias, Thanks for reporting! I will follow-up with you on the created JIRA ticket. Guozhang On Mon, Sep 12, 2016 at 4:01 PM, Elias Levy wrote: > https://issues.apache.org/jira/browse/KAFKA-4153 > https://github.com/apache/kafka/pull/1846 > > On Mon, Sep 12, 2016 at 7:00 AM, Elias Levy >

Exception while deserializing in kafka streams

Hello, I get the below exception when deserilaizing avro records using KafkaAvroDeserializer. 16/09/14 17:24:39 INFO StreamThread: Stream thread shutdown complete [StreamThread-1] Exception in thread "StreamThread-1" org.apache.kafka.common.errors.SerializationException: Error deserializing Avro

Re: KAFKA-3933: Kafka OOM During Log Recovery Due to Leaked Native Memory

Here are the links: https://github.com/apache/kafka/commit/8a417c89d2f0b7861b2dec26f02e4e302b64b604 (trunk) https://github.com/apache/kafka/commit/c47c3b0b583a849fdf3ed0a06835427a2801950a (0.10.0) Ismael On Wed, Sep 14, 2016 at 4:38 PM, feifei hsu wrote: > Hi, Ismael >So many thanks for th

Re: KAFKA-3933: Kafka OOM During Log Recovery Due to Leaked Native Memory

Hi, Ismael So many thanks for the quick reply I checked the trunk tree at github, I did not see the merge. did I make some mistake? sorry for that. for example, the pull 1598. file related to the LogSegment.scala. the one of the PR add try catch to close the leaking resource. but I did no

Re: ZooKeeper client > 3.4.6

We upgraded to 3.4.8 for this exact reason and have not had any issues. On 09/14/2016 09:45 AM, Bill de hÓra wrote: Hi, I was wondering if anyone has experience (good or bad!) running Kafka with a ZooKeeper client > 3.4.6. The reason to do this is to avoid having to bounce brokers when ZooKeep

ZooKeeper client > 3.4.6

Hi, I was wondering if anyone has experience (good or bad!) running Kafka with a ZooKeeper client > 3.4.6. The reason to do this is to avoid having to bounce brokers when ZooKeeper node IPs change; it seems the 3.4.6 client caches IP addresses indefinitely but this may have be fixed in later

RE: handling generics(solved) but new problem unearthed

the end goal was to power thru the kafka parsing/compilation to then see if all kafka java classes would actually compile I found this bug where kafka is attempting implement joptsimple kafka.tools.StreamsResetter.java fails the build https://www.codatlas.com/github.com/apache/kafka/trunk/core

Re: KAFKA-3933: Kafka OOM During Log Recovery Due to Leaked Native Memory

Hi, We did merge the PR to trunk and 0.10.0. Ismael On Wed, Sep 14, 2016 at 9:21 AM, feifei hsu wrote: > Hi Tom and Ismael. >I am following the kafka-3933. the memory leak. but I did not see the > pr #1598 #1614 #1660 are merged into the trunk. > Do you know what the current status? > S

KAFKA-3933: Kafka OOM During Log Recovery Due to Leaked Native Memory

Hi Tom and Ismael. I am following the kafka-3933. the memory leak. but I did not see the pr #1598 #1614 #1660 are merged into the trunk. Do you know what the current status? So many thanks. We are also thinking backport it to 0.9.0.1 --easy