Re: Commit offset with SimpleConsumer in 0.8.2

Thanks for the pointer!! Added myself as a watcher. 2015-02-24 3:12 GMT+01:00 Joel Koshy : > Can you add yourself as a watcher on KAFKA-1729? I will update that > when I fix the example on the wiki. > > On Sun, Feb 22, 2015 at 10:16:44PM +0100, Jochen Mader wrote: > > I have a hard time figuring

Re: NetworkProcessorAvgIdlePercent

Hi Jun, With ~100G of data being pushed per hour across 35 topics (replication-factor 3), the NetworkProcessorAvgIdlePercent is mostly showing below 0.5 sometimes when the producers send on a high rate. Thanks -Zakee On Sun, Feb 22, 2015 at 10:29 PM, Jun Rao wrote: > What kind of load do you h

Re: Broker w/ high memory due to index file sizes

Thanks, Ewen. I will try these. -Zakee On Sun, Feb 22, 2015 at 11:51 PM, Ewen Cheslack-Postava wrote: > If you haven't seen it yet, you probably want to look at > http://kafka.apache.org/documentation.html#java > > -Ewen > > On Thu, Feb 19, 2015 at 10:53 AM, Zakee wrote: > > > Well are there a

Re: Broker w/ high memory due to index file sizes

Hi, I'd start small and grow if needed in general. There is one bit of info missing here. Did you try with a smaller heap and had issues? If so, which issues? A few references from our internal Wiki: - XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:

After Leadership Election and "kafka.log" JMXBean Registration Process

Hi Kafka Team or User Community , After leadership election or switch between follower/leader of partition for given topic, does following metrics JMX bean gets registered (on leader) and de-registered (on follower). LogEndOffset Size LogStartOffset eg: "kafka.log":type="Log",name="TOPIC-17-*Si

Re: Why The Division Between Scala And Java

We will eventually only have Java clients. For your specific question: javaapi.SimpleConsumer and consumer.SimpleConsumer - there are some arguments that contain scala-specific constructs. E.g., scala maps which cannot be created in Java. This is why we expose a javaapi variant which takes Java co

Re: Simple Consumer and offsets

> Assuming one can store the offset in an external location (redis/db > etc), along with the rest of the state that a program requires, > wouldn't it be possible to manage things such that, you use the High > Level API with auto commit turned off and do your custom offset > management followed by t

Re: Commit offset with SimpleConsumer in 0.8.2

Can you add yourself as a watcher on KAFKA-1729? I will update that when I fix the example on the wiki. On Sun, Feb 22, 2015 at 10:16:44PM +0100, Jochen Mader wrote: > I have a hard time figuring out how to do a commit using API 0.8.2 on JDK 8. > > I tried using the examples from 0.8.1.1. > > Fi

Re: ability to delete inactive queues?

We do support delete topic. However, this is a client(admin) operation that is done via zookeeper. It would be useful to do this automatically on the broker-side. Can you file a jira for this? It is not very straightforward to implement this since you would want to check across all partitions whic

Re: Kafka High Level Consumer

Did you enable auto offset commit? Thanks, Jun On Tue, Feb 17, 2015 at 4:22 PM, Pranay Agarwal wrote: > Hi, > > I am trying to read kafka consumer using high level kafka Consumer API. I > had to restart the consumers for some reason but I kept the same group id. > It seems the consumers have s

Re: Broker Shutdown / Leader handoff issue

Hmm, if you shut down the leader, the follower should get a socket exception immediately. Thanks, Jun On Wed, Feb 18, 2015 at 7:01 AM, Philippe Laflamme wrote: > After further investigation, I've figured out that the issue is caused by > the follower not processing messages from the controller

Re: Kakfa question about starting kafka with same broker id

This is likely caused by ZK client session expiration. Which version of ZK are you using? Thanks, Jun On Wed, Feb 18, 2015 at 3:16 PM, Deepak Dhakal wrote: > Hi, > > My name is Deepak and I work for salesforce. We are using kafka 8.11 and > have a question about starting kafka with same broke

Re: Broker ID disappears in Zookeeper

This is fixed in KAFKA-1971 and will be included in the 0.8.2.1 release. Thanks, Jun On Thu, Feb 19, 2015 at 10:22 AM, Sybrandy, Casey < casey.sybra...@six3systems.com> wrote: > Hello, > > We're having the following issue with Kafka and/or Zookeeper: > If a broker (id=1) is running, and you sta

Re: KafkaProducer.send contract

Could you explain a bit more what you want to achieve through batching? Better throughput or atomicity? Thanks, Jun On Thu, Feb 19, 2015 at 4:09 PM, JAmes Atwill wrote: > Hey Jun, > > That's what I've got right now, semaphore before send() and release in the > callback. Am I correct in underst

Re: Simple Consumer and offsets

We are developing a new java consumer that addresses the offset reset issue better. Thanks, Jun On Thu, Feb 19, 2015 at 11:53 PM, Arunkumar Srambikkal (asrambik) < asram...@cisco.com> wrote: > If I may use the same thread to discuss the exact same issue > > Assuming one can store the offse

Re: KafkaProducer from kafka.clients hangs when some partitions are not available

The logic in that code is to cycle through all partitions and return as soon as we see a partition with the leader. I do see an issue that if there are multiple threads sending messages to the same producer concurrently, we may not cycle through all partitions and therefore we could return an unava

Re: Anyone interested in speaking at Bay Area Kafka meetup @ LinkedIn on March 24?

Nevermind! I’m at idiot and missed the first message. I’ll coordinate with Steven and Allen offline. Thanks all! On 2/23/15, 3:26 PM, "Ed Yakabosky" wrote: >I don’t happen to know who is from Netflix but happy to poke that person >if they speak up :). > >I think we can broadcast remotely but

Re: Anyone interested in speaking at Bay Area Kafka meetup @ LinkedIn on March 24?

I don’t happen to know who is from Netflix but happy to poke that person if they speak up :). I think we can broadcast remotely but will confirm that in a few days. Ed On 2/23/15, 3:24 PM, "Ruslan Khafizov" wrote: >+1 For recording sessions. >On 24 Feb 2015 07:22, "Jiangjie Qin" wrote: > >> +

Re: Anyone interested in speaking at Bay Area Kafka meetup @ LinkedIn on March 24?

+1 For recording sessions. On 24 Feb 2015 07:22, "Jiangjie Qin" wrote: > +1, I¹m very interested. > > On 2/23/15, 3:05 PM, "Jay Kreps" wrote: > > >+1 > > > >I think something like "Kafka on AWS at Netflix" would be hugely > >interesting to a lot of people. > > > >-Jay > > > >On Mon, Feb 23, 2015

Re: Latest offset is frozen

Hmm, when the tail offset is frozen, does it freeze forever? Also, do you get the same frozen offset if you run the GetOffsetShell command? Thanks, Jun On Sun, Feb 22, 2015 at 6:45 PM, Stuart Reynolds wrote: > I'm finding that if I continuously produce values to a topic (say, > once every 2 se

Re: Anyone interested in speaking at Bay Area Kafka meetup @ LinkedIn on March 24?

+1, I¹m very interested. On 2/23/15, 3:05 PM, "Jay Kreps" wrote: >+1 > >I think something like "Kafka on AWS at Netflix" would be hugely >interesting to a lot of people. > >-Jay > >On Mon, Feb 23, 2015 at 3:02 PM, Allen Wang >wrote: > >> We (Steven Wu and Allen Wang) can talk about Kafka use ca

Re: Anyone interested in speaking at Bay Area Kafka meetup @ LinkedIn on March 24?

Same here. It would be great specially if the Netflix/AWS use case is presented. Kind Regards, Daniel Moreno On Feb 23, 2015, at 18:06, Xiaoyu Wang mailto:xw...@rocketfuel.com>> wrote: It is possible to attend the meetup remotely? I am in east coast, really want to be able to attend this. On

Re: earliestOrLatestOffset, getOffsetsBefore

1) ConsumerId is for tracking which client makes the request. It's mostly for debugging right now. 2) The latter is more general. You can ask for an offset before an arbitrary timestamp. Thanks, Jun On Sun, Feb 22, 2015 at 6:35 PM, Stuart Reynolds wrote: > In > > > > http://apache.osuosl.org/

Re: Anyone interested in speaking at Bay Area Kafka meetup @ LinkedIn on March 24?

+1 I think something like "Kafka on AWS at Netflix" would be hugely interesting to a lot of people. -Jay On Mon, Feb 23, 2015 at 3:02 PM, Allen Wang wrote: > We (Steven Wu and Allen Wang) can talk about Kafka use cases and operations > in Netflix. Specifically, we can talk about how we scale a

Re: Anyone interested in speaking at Bay Area Kafka meetup @ LinkedIn on March 24?

It is possible to attend the meetup remotely? I am in east coast, really want to be able to attend this. On Mon, Feb 23, 2015 at 3:02 PM, Allen Wang wrote: > We (Steven Wu and Allen Wang) can talk about Kafka use cases and operations > in Netflix. Specifically, we can talk about how we scale and

Re: Anyone interested in speaking at Bay Area Kafka meetup @ LinkedIn on March 24?

We (Steven Wu and Allen Wang) can talk about Kafka use cases and operations in Netflix. Specifically, we can talk about how we scale and operate Kafka clusters in AWS and how we migrate our data pipeline to Kafka. Thanks, Allen On Mon, Feb 23, 2015 at 12:15 PM, Ed Yakabosky < eyakabo...@linkedin

Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 1

Hi, Joe, Left a comment on KAFKA-1724. Not sure if it's a true blocker. However, we can probably include it if it can be fixed in the next couple of days. Thanks, Jun On Sun, Feb 22, 2015 at 2:12 PM, Joe Stein wrote: > Jun, > > Can we also add https://issues.apache.org/jira/browse/KAFKA-1724

Re: Anyone using log4j Appender for Kafka?

^^ I would really appreciate this as well. It's unclear how to get log4j working with Kafka when you have no prior experience with log4j. On Mon, Feb 23, 2015 at 4:39 AM, Scott Chapman wrote: > Thanks. But we're using log4j. I tried setting the type to async but it > generated a warning of no su

0.8.2 OffsetCommitRequest

Hi, Does anyone have a full Java example of making an OffsetCommitRequest in 0.8.2? My use case is for Consumer restart, where we want to specify the offset as a timestamp. We are dealing with multiple partitions. I'm trying to follow the OffsetCommitRequest example here in Java code. However, on

commit offset using SimpleConsumer

I am trying to manually commit offsets using the SimpleConsumer (Kafka 0.8.2 on JDK 8). Using the following code: offsets.put(new TopicAndPartition(cd.getTopic(), cd.getPartition()) , new OffsetAndMetadata(messageAndOffset.offset(), "", -1L)); OffsetCommitRequest ocRequest = new OffsetCom

Re: Consuming a snapshot from log compacted topic

Hey guys, I created a patch based on your feedback. Let me know what you think. https://issues.apache.org/jira/browse/KAFKA-1977 On 20 February 2015 at 01:43, Joel Koshy wrote: > The log end offset (of a partition) changes when messages are appended > to the partition. (It is not correlated w

Anyone interested in speaking at Bay Area Kafka meetup @ LinkedIn on March 24?

Hi Kafka Open Source - LinkedIn will host another Bay Area Kafka meetup in Mountain View on March 24. We are planning to present on Offset Management but are looking for additional speakers. If you’re interested in presenting a use case, operational plan, or your experience with a particular

Apache Samza Meetup Announced (March 4 @6PM hosted at LinkedIn¹s campus in Mountain View CA)

Hi all - I would like to announce the first Bay Area Apache Samza Meetup hosted at LinkedIn in Mountain View, CA on March 4, 2015 @6PM. We plan to host the event every 2-months to encourage knowledge sharing & collaboration in Samz

Re: How to find the number of message in the topic

See SimpleConsumer. getOffsetsBefore and the getLastOffset example here: https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example whichTime=-1 or -2 (depending whether you want the latest or earliest offset. However, I've been banging my head against the wall with this

Re: Auto topic creation occurring when auto.create.topics.enable=false

Does the ruby library write to ZK directly to create topics? That will bypass the checking on the broker side. Thanks, Jun On Mon, Feb 23, 2015 at 3:06 AM, David Corley wrote: > Hey all, > I'm trying to run some basic error-handling validation with some client > code, and I'm attempting to han

Re: Anyone using log4j Appender for Kafka?

Thanks. But we're using log4j. I tried setting the type to async but it generated a warning of no such field. Is there any real documentation on the log4j appender? On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker < sschlans...@opentable.com> wrote: > We just configure our logback.xml to have t

Messages aren't consumed immediately

Hi I'm sending a message using this python code: client = KafkaClient("localhost:9092") producer = SimpleProducer(client) producer.send_messages("mytopic", bytes("some msg", 'utf-8')) Then i am consuming it using kafka-net in the following way: var options = new KafkaOptions(new Uri

Auto topic creation occurring when auto.create.topics.enable=false

Hey all, I'm trying to run some basic error-handling validation with some client code, and I'm attempting to handle an UnknownTopicOrPartitionException. To set this scenario up, I wanted to attempt fetch messages from a topic I know doesn't exist. To that end, I've got a 3-broker cluster with: * au

Re: Ship Kafka in on prem product

Hi, I have shared this question with the devops team and i will keep this thread updated. Regards. 2015-02-16 16:14 GMT+01:00 Ivan Balashov : > David, > > Thanks for sharing this. Any plans to include 0.8.2 to the list of > available packages? > Or, any chance you could share your packaging sc