Re: 0.9.0.0 remaining jiras

2015-09-13 Thread Jason Rosenberg
Hi Jun, Can you clarify, will there not be a 0.8.3.0 (and instead we move straight to 0.9.0.0)? Also, can you outline the man new features/updates for 0.9.0.0? Thanks, Jason On Sat, Sep 12, 2015 at 12:40 PM, Jun Rao wrote: > The following is a candidate list of jiras that we want to complete

How to model topics and partitions for Kafka when used to store all business events?

2015-09-13 Thread Johan Haleby
Hi, We're considering using Kafka as a way to store all our business events (effectively) forever. The purpose is to be able to spin up new "microservices" that we haven't yet thought of that will be able to leverage on all previous events to build up their projections/state. Another use case migh

Re: What can be reason for fetcher thread for slow response.

2015-09-13 Thread Madhukar Bharti
Hi Erik & Prabhjot We are using Kafka-0.8.2.1 and old producer API with below config: request.required.acks=1 request.timeout.ms=2000 producer.type=sync On Kafka broker we are having: num.network.threads=8 num.io.threads=10 num.replica.fetchers=4 replica.fetch.max.bytes=2097154 replica.fetch.wa

Re: port already in use error when trying to add topic

2015-09-13 Thread allen chan
Changing the port to 9998 did not help. Still the same error occurred On Sat, Sep 12, 2015 at 12:27 AM, Foo Lim wrote: > Try throwing > > JMX_PORT=9998 > > In front of the command. Anything other than 9994 > > Foo > > On Friday, September 11, 2015, allen chan > wrote: > > > Hi all, > > > > Firs

Re: Auto preferred leader elections cause data loss?

2015-09-13 Thread Zhao Weinan
Hi group, Sorry I'm confused with *min.insync.replicas: When a producer sets request.required.acks to -1, min.insync.replicas specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. *When talking about number of replicas, dose it include th

Re: open source projects based on kafka

2015-09-13 Thread Jeff Holoman
You could try implementing your own client. https://cwiki.apache.org/confluence/display/KAFKA/Clients On Sun, Sep 13, 2015 at 7:46 PM, Stevo Slavić wrote: > Have a look at https://github.com/allegro/hermes > > On Mon, Sep 14, 2015, 01:28 David Luu wrote: > > > The toy project idea is good. Ano

Re: open source projects based on kafka

2015-09-13 Thread Stevo Slavić
Have a look at https://github.com/allegro/hermes On Mon, Sep 14, 2015, 01:28 David Luu wrote: > The toy project idea is good. Another option I think could be to look at > the various Kafka client langague bindings and/or utilities (like > kafkacat). And from there, another option is to build a c

Re: open source projects based on kafka

2015-09-13 Thread David Luu
The toy project idea is good. Another option I think could be to look at the various Kafka client langague bindings and/or utilities (like kafkacat). And from there, another option is to build a client language binding for a language that's kind of lacking Kafka support, some have better support th

Re: open source projects based on kafka

2015-09-13 Thread Hisham Mardam-Bey
Hi Li, You can take a look at mypipe. "MySQL binary log consumer with the ability to act on changed rows and publish changes to different systems with emphasis on Apache Kafka." https://github.com/mardambey/mypipe It uses Kafka as well as some related concepts and technologies like Avro, a sche

Re: Broker restart, new producer and snappy in 0.8.2.1

2015-09-13 Thread Joe Stein
Hi, the 0.8.2.2 release (which vote just passed and should be announced soon) has a patch that may be related https://issues.apache.org/jira/browse/KAFKA-2308 not sure. Here are the 0.8.2.2 artifacts https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/ I don't see them yet in Maven central

Re: open source projects based on kafka

2015-09-13 Thread Manasvi Gupta
Here's a toy project - analyzing twitter stream. 1) Create dev. account on twitter 2) Using your dev credentials, connect to twitter stream api to retrieve stream of tweets 3) Store tweets in Kafka (using Kafka producer) 4) Retrieve tweets (using Kafka consumer) 5) For each tweet (or group of twee

Re: open source projects based on kafka

2015-09-13 Thread Li Tao
Hi Roger, Thanks for your recommendation. I just got to know Samza. and checked its code base. It is a little too huge for me. Maybe for now, I need to start a small project/application which utilize kafka as its infrastructure, so that I can use Kafka's API a lot and know Kafka better. It's har