Re: Interested in contributing to Kafka?

I have been using kafka for quite some time now and would really be interested to contribute to this awesome code base. Regards, Pushkar On Thu, Jul 17, 2014 at 7:17 AM, Joe Stein wrote: > ./gradlew scaladoc > > Builds the scala doc, perhaps we can start to publish this again with the > next r

Re: Has anybody successfully integrated Kafka jar for Android app.

Thanks Philip, I will certainly take a look. I had not considered the fact that Kafka doesn't have any security/authentication mechanism yet, so using some kind of secure gateway certainly makes sense. -Subodh On Wed, Jul 16, 2014 at 10:03 PM, Philip O'Toole < philip_o_to...@yahoo.com.invalid>

Re: Has anybody successfully integrated Kafka jar for Android app.

Agreeing with Philip! Our use of Kafka is with a relatively small number of high volume producers and consumers. I'm pretty sure Kafka wasn't designed for large numbers of either Producers or Consumers, as would be necessary for any significant number of mobile clients. Also, while an Android dev

Re: Has anybody successfully integrated Kafka jar for Android app.

You should find code here that will help you get a HTTP app server together, that writes to Kafka on the back-end. https://cwiki.apache.org/confluence/display/KAFKA/Clients https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem   On Wednesday, July 16, 2014 9:36 PM, Philip O'Toole wrote

Re: Has anybody successfully integrated Kafka jar for Android app.

FWIW, I happen to know Subodh -- we worked together many years back. We discussed this a little off-the-list, but perhaps my thoughts might be of wider interest. Kafka, in my experience, works best when Producers have a persistent TCP connection to the Broker(s) (and possibly Zookeeper). I gues

Re: Has anybody successfully integrated Kafka jar for Android app.

Hello Steven, Thanks for your suggestions, I have certainly seen other stack/presentation where people do forward their messages to their own receiver which forwards the message to kafka -- basically they write their own producer(s). The question I haven't been able to answer for myself- why -- d

RE: Question about ConsoleConsumer

Hi Joe, Thanks for your answer, seems I know what's its purpose. Thanks a lot. Jerry -Original Message- From: Joe Stein [mailto:joe.st...@stealth.ly] Sent: Thursday, July 17, 2014 10:15 AM To: d...@kafka.apache.org Cc: users@kafka.apache.org Subject: Re: Question about ConsoleConsumer

Re: Question about ConsoleConsumer

auto.offset.reset is only when there is not an already established offset to consume from or an offset that no longer exists (and is invalid) in the case of truncated logs or compaction). You can force this behavior by deleting the znode /*** Joe Stein Fo

Question about ConsoleConsumer

Hi folks, A newbie question when investigating ConsoleConsumer's code, When we use ConsoleConsumer by setting "-from-beginning", we will set property as "auto.offset.reset"="smallest", which means moving the offset to the beginning of the data. But in the code, we still need to delete Zookeeper

Re: Interested in contributing to Kafka?

./gradlew scaladoc Builds the scala doc, perhaps we can start to publish this again with the next release and link it on the website. For more related check out the README /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC

Re: Interested in contributing to Kafka?

Here are the newbie labels in JIRA for open Kafka tickets https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20labels%20%3D%20newbie%20AND%20status%20%3D%20Open /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security

Re: Has anybody successfully integrated Kafka jar for Android app.

Hi Subodh, I would think you'd be better off having an app server of some kind as an intermediary that accepts messages from your android app and posts them to Kafka for you, rather than having your app be a Kafka Producer on its own. Steve On Wed, Jul 16, 2014 at 1:22 PM, Subodh Nijsure < subo

Re: Interested in contributing to Kafka?

I am a newbie as well and would be great if you can add a 'newbie' label to the Jira tickets. -Alan On 2014. 7. 17. 오전 8:26, Jarek Jarcec Cecho wrote: I’m completely new to Kafka as well, but I would really like to contribute. On other projects we’re using label “newbie” to denote JIRA that i

Re: Interested in contributing to Kafka?

Is there a scala API doc for the entire kafka library? On Wed, Jul 16, 2014 at 5:34 PM, hsy...@gmail.com wrote: > Hi Jay, > > I would like to take a look at the code base and maybe start working on > some jiras. > > Best, > Siyuan > > > On Wed, Jul 16, 2014 at 3:09 PM, Jay Kreps wrote: > >> He

Re: Interested in contributing to Kafka?

Hi Jay, I would like to take a look at the code base and maybe start working on some jiras. Best, Siyuan On Wed, Jul 16, 2014 at 3:09 PM, Jay Kreps wrote: > Hey All, > > A number of people have been submitting really nice patches recently. > > If you are interested in contributing and are loo

Re: Interested in contributing to Kafka?

I’m completely new to Kafka as well, but I would really like to contribute. On other projects we’re using label “newbie” to denote JIRA that is particularly suitable for someone completely new to the project. I’m wondering if Kafka uses similar mechanism to annotate “newbie” JIRAs/tasks? Jarcec

Re: Interested in contributing to Kafka?

Hi Kane, Jonathan, We tried to pick some JIRAs that we think are relatively easy with label "newbie" or "newbie++" (a bit more than "newbie"). You may want to search with these labels. Guozhang On Wed, Jul 16, 2014 at 4:38 PM, Jonathan Natkins wrote: > I was actually struggling with this, mys

Re: Interested in contributing to Kafka?

I was actually struggling with this, myself. I'm looking for some relatively easy JIRAs to go after to try to get familiar with the codebase. If you have any little pet JIRAs that you've not had time to get around to, I'd love to try to tackle some of those. Let me know. Thanks! Natty On Wed, Ju

Re: Interested in contributing to Kafka?

Hi Jay, I don't have a lot of experience patching kafka, but I would really like to help starting with some minor tasks. Thanks. On Wed, Jul 16, 2014 at 3:09 PM, Jay Kreps wrote: > Hey All, > > A number of people have been submitting really nice patches recently. > > If you are interested in con

Interested in contributing to Kafka?

Hey All, A number of people have been submitting really nice patches recently. If you are interested in contributing and are looking for something to work on, or if you are contributing and are interested in ramping up to be a committer on the project, please let us know--we are happy to help you

Re: [DISCUSS] Kafka Security Specific Features

Pramod, I presented secure kafka configuration and usage at last meet up. So hope this video recording would help. You can skip to about 59 min to jump to security talk. Thanks, Raja. On Wed, Jul 16, 2014 at 5:57 PM, Pramod Deshmukh wrote: > Hello Joe

Re: [DISCUSS] Kafka Security Specific Features

Hello Joe, Is there a configuration or example to test Kafka security piece? Thanks, Pramod On Wed, Jul 16, 2014 at 5:20 PM, Pramod Deshmukh wrote: > Thanks Joe, > > This branch works. I was able to proceed. I still had to set scala version > to 2.9.2 in kafka-run-class.sh. > > > > On Wed, J

Re: [DISCUSS] Kafka Security Specific Features

Thanks Joe, This branch works. I was able to proceed. I still had to set scala version to 2.9.2 in kafka-run-class.sh. On Wed, Jul 16, 2014 at 3:57 PM, Joe Stein wrote: > That is a very old branch. > > Here is a more up to date one > https://github.com/stealthly/kafka/tree/v0.8.2_KAFKA-1477 (

Has anybody successfully integrated Kafka jar for Android app.

Hello, I am trying to make my Android application to be the producer of messages that get sent to Kafka server. I am trying to compile my android code using following libraries - kafka_2.9.2-0.8.1.1.jar, scala-library-2.9.2.jar However I am running into issue where it produces too many methods.

Re: [DISCUSS] Kafka Security Specific Features

That is a very old branch. Here is a more up to date one https://github.com/stealthly/kafka/tree/v0.8.2_KAFKA-1477 (needs to be updated to latest trunk might have a chance to-do that next week). You should be using gradle now as per the README. /*** Joe S

Re: [DISCUSS] Kafka Security Specific Features

Thanks Joe for this, I cloned this branch and tried to run zookeeper but I get Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain I see scala version is still set to 2.8.0 if [ -z "$SCALA_VERSION" ]; then SCALA_VERSION=2.8.0 fi Then I instal

RE: New Consumer Design

Hi Guozhang, Currently, the brokers do not know which high-level consumers are reading which partitions and it is the rebalance between the consumers and the coordinator which would authorize a consumer to fetch a particular partition, I think. Does this mean that when a rebalance occurs, a

Re: Performance/Stress tools

We are working on this Yarn based tool , but it's still in alpha https://github.com/DemandCube/DemandSpike On Wed, Jul 16, 2014 at 11:30 AM, Dayo Oliyide wrote: > Hi, > > I'm setting up a Kafka Cluster and would like to carry out some > performance/stress tests on different configurations. > >

Re: High Level consumer commits

Hello Guy, One thing you can do is to turn off auto.commit.offset and then only commit offset after try/catch block, and then in the catch block stop and restart the consumer so that the message will be re-consumed. Guozhang On Wed, Jul 16, 2014 at 8:16 AM, Guy Doulberg wrote: > Hi, > > Is th

Re: High Level consumer commits

You can turn off auto commits and issue commitOffsets() periodically. See the api in http://kafka.apache.org/documentation.html#highlevelconsumerapi. Thanks, Jun On Wed, Jul 16, 2014 at 8:16 AM, Guy Doulberg wrote: > Hi, > > Is there a way to commit consuming of events only if I had ran code

Re: message loss for sync producer, acks=2, topic replicas=3

Selecting replicas in ISR based on their fetched messages would be quite complicated since it requires the controller to keep track of this information in sync. An alternative solution to your issue would either be using ack=-1 or reduce replica.lag.time.max.ms so that followers not keeping up clos

Re: message loss for sync producer, acks=2, topic replicas=3

The current replication design is that a message is considered committed if it's written to isr. In this case, since isr has all three replicas and the producer is acked when only 2 replicas have received the message, there is no guarantee that the message is committed. So, it is possible for an ac

Re: Durably storing messages in Kafka

Hi Daniel, Your understanding is mostly correct, a few notes to add: 1. fsync is done in a backend thread asynchronously, so event setting log.flush.interval.messages = 1 will not guarantee that when the producer request ack back, the messages is definitely on the disk now. But since you are usin

High Level consumer commits

Hi, Is there a way to commit consuming of events only if I had ran code successfully? In other words, In this code while (it.hasNext()) { /** Iterator on kafka High lever consumer*/ try { MessageAndMetadata current = it.next(); dealWithEvent(cu

Re: New Consumer Design

Hi Rob, Piggy-back the rebalance partition info in HeartbeatResponse may cause inconsistency of the partition assignments to consumers with consecutive triggering of rebalances, since the coordinator no longer has a synchronization barrier any more for re-compute the distribution with a consistent

Performance/Stress tools

Hi, I'm setting up a Kafka Cluster and would like to carry out some performance/stress tests on different configurations. Other than the performance testing scripts that come with Kafka, are there any other tools that anyone would recommend? Regards, Dayo

Re: message loss for sync producer, acks=2, topic replicas=3

Guozhong, So this is the cause of message loss in my test where acks=2 and replicas=3: At one moment all 3 replicas, leader L, followers F1 and F2 are in ISR. A publisher sends a message m to L. F1 fetches m. Both L and F1 acknowledge m so the send() is successful. Before F2 fetches m, L is kill