How to Ingest data into kafka

Hi kafkars, I want to write a java code to ingest csv files into kafka,any help. Thanks, Kishore.

Re: How to raise a question in forum

thanks for response. But my questions are still not answered a. Is ConsumerLag is the metrics that is exposed on kafka broker end or on Consumer side b. Can ConsumerLag value be in negative? Regards, Nitin Kumar Sharma. On Tue, Dec 9, 2014 at 5:23 PM, Otis Gospodnetic wrote: > Nitin, > > If y

Re: Possibly leaking socket on ReplicaFetcherThread

Sorry for me not replying in the thread. ignore last email. Hi, Jun We experienced a network device problem. and cause all brokers crashed. After investigation, we found server log throw similar exceptions. this: java.nio.channels.UnresolvedAddressException at sun.nio.ch.Net.checkAddre

Re: Possibly leaking socket on ReplicaFetcherThread

Hi, Jun We experienced a network device problem. and cause all brokers crashed. After investigation, we found server log throw similar exceptions. this: java.nio.channels.UnresolvedAddressException at sun.nio.ch.Net.checkAddress(Net.java:29) at sun.nio.ch.SocketChannelImpl.connec

Re: [DISCUSSION] adding the serializer api back to the new java producer

Hi All, This is very likely when you have large site such as Linked-in and you have thousand of servers producing data. You will mixed bag of producer and serialization or deserialization because of incremental code deployment. So, it is best to keep the API as generic as possible and each org /

Re: [DISCUSSION] adding the serializer api back to the new java producer

> In practice the cases that actually mix serialization types in a single stream are pretty rare I think just because the consumer then has the problem of guessing how to deserialize, so most of these will end up with at least some marker or schema id or whatever that tells you how to read the data

Re: How to raise a question in forum

Nitin, If you don't want to DIY there are already tools and solutions for monitoring Kafka, including consumer lag. Ours is mentioned here: http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/ Others have been mentioned on the list a few times before: http://sea

Re: How to raise a question in forum

Hi Neha, Guozhang Wang My project requirement is to monitor Consumer Lag Mbean.. 1) According to me this MBean is exposed only at the Consumer side and not on Broker server. This is because in my initial PoC, i had used "kafka-consumer-perf-test.sh" and MBean exposed by this consumer had "Consume

Re: How to raise a question in forum

Nitin, You have found the place :) Just send an email to users@ mailing list is fine. Guozhang On Tue, Dec 9, 2014 at 1:26 PM, nitin sharma wrote: > HI, > I need some help related to Kafka Consumer Lag monitoring.. Kindly let me > know how can i post my query to the forum. > > Regards, > Nitin

Re: How to raise a question in forum

You can post it here :-) On Tue, Dec 9, 2014 at 1:26 PM, nitin sharma wrote: > HI, > I need some help related to Kafka Consumer Lag monitoring.. Kindly let me > know how can i post my query to the forum. > > Regards, > Nitin Kumar Sharma. > -- Thanks, Neha

Re: Topic config reset on alter

Filed it here: https://issues.apache.org/jira/browse/KAFKA-1816 Thanks! --  Andrew Jorgensen @ajorgensen On December 9, 2014 at 4:10:38 PM, Neha Narkhede (n...@confluent.io) wrote: Thanks for reporting the bug. Would you mind filing a JIRA

How to raise a question in forum

HI, I need some help related to Kafka Consumer Lag monitoring.. Kindly let me know how can i post my query to the forum. Regards, Nitin Kumar Sharma.

Re: Increased CPU usage with 0.8.2-beta

The following should be sufficient java -agentlib:hprof=cpu=samples,depth=100,interval=20,lineno=y,thread=y,file=kafka.hprof You would need to start the Kafka server with the settings above for sometime until you observe the problem. On Tue, Dec 9, 2014 at 3:47 AM, Mathias Söderberg < mathias.s

Re: Topic config reset on alter

Thanks for reporting the bug. Would you mind filing a JIRA ? On Tue, Dec 9, 2014 at 12:46 PM, Andrew Jorgensen < ajorgen...@twitter.com.invalid> wrote: > I am using kafka 8.1.1 > > If y

Topic config reset on alter

I am using kafka 8.1.1 If you run an alter on a topic to increase the number of partitions the existing configs are removed from the topic. You can confirm this by doing: bin/kafka-topics.sh —zookeeper localhost —alter —topic topic —config retention.ms=360 bin/kafka-topics.sh —zookeeper lo

Re: Reading only the latest message

> > Is this the best way to get the offset and is it safe to decrement the > offset returned as we do in the sample code below. Yes, this should work. Let us know if you see any problems. On Tue, Dec 9, 2014 at 7:44 AM, Orelowitz, David wrote: > I am reposting a question that I posted last w

leaderless topicparts after single node failure: how to repair?

Hi. We've suffered a single node HW failure (broker_id 4) with at least 2 replicas of each topic partition, but some topic parts are now leaderless (all were across 4,5): Topic: topic.with.two.replicas Partition: 0Leader: -1 Replicas: 4,5 Isr: on broker 5, we see warnings like this in t

Re: Kafka Issue [Corrupted broker]

It looks like broker 5 is in a bad state. You are likely going to have to shut it down. From there you have a few options and depending on your environment setup will dictate if you do shut it down and/or what you do after that. Spinning up another server with broker.id == 5 and let replication he

Reading only the latest message

I am reposting a question that I posted last week. On startup or recovery we would like to read the latest message in each partition. The getOffsetsBefore() below seems to return the offset of the next message that will be published to that partition. The code below works correctly as required.

Re: In what condition does getoffsetbefore (latest) not return latest offset?

Thanks Guozhang， I checked the server log, and I am 90% sure that no leader movement happened at that moment. Could it be that the data was in the memory but yet not flushed to the disk? I read the doc in the wiki https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication, and said The

Kafka Issue [Corrupted broker]

Hi, One of the broker seems to have got corrupted in my cluster of 7 brokers. All the topic partitions where this broker was leader are having NoLeader or UnderReplicated partition exceptions. All these partittions have no leader and even no replica in the isr(in-sync replica) set. Corrupt b

Re: Rebalance not happening even after increasing max retries causing conflict in ZK

Neha, The same issue reoccured with just 2 consumer processes. The exception was related to conflict in writing the ephemeral node. Below was the exception. Topic name is "lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin" with 30 partitions. The 2 processes were running

Re: Increased CPU usage with 0.8.2-beta

Hi Neha, Yeah sure. I'm not familiar with hprof, so any particular options I should include or just run with defaults? Best regards, Mathias On Mon Dec 08 2014 at 7:41:32 PM Neha Narkhede wrote: > Thanks for reporting the issue. Would you mind running hprof and sending > the output? > > On Mon