Re: [VOTE] 0.10.2.1 RC0

2017-04-12 Thread Gwen Shapira
gt; +1 >>> >>> On Wed, Apr 12, 2017 at 8:53 AM, Michal Borowiecki < >>> michal.borowie...@openbet.com> wrote: >>> >>>> FWIW, I upgraded without issue and noticed the speedup from >>>> KAFKA-4851/KAFKA-4876. >>>> >&g

[VOTE] 0.10.2.1 RC1

2017-04-12 Thread Gwen Shapira
1 tag: https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=e133f2ca57670e77f8114cc72dbc2f91a48e3a3b * Documentation: http://kafka.apache.org/0102/documentation.html * Protocol: http://kafka.apache.org/0102/protocol.html /** Thanks, Gwen Shapira

Re: [VOTE] 0.10.2.1 RC1

2017-04-14 Thread Gwen Shapira
Verified my own signatures, ran quickstart and created few Connectors. +1 (binding) On Wed, Apr 12, 2017 at 5:25 PM, Gwen Shapira wrote: > Hello Kafka users, developers, client-developers, friends, romans, > citizens, etc, > > This is the second candidate for release of Apache Ka

Re: [VOTE] 0.10.2.1 RC1

2017-04-17 Thread Gwen Shapira
. I'm inclined to roll another RC for 0.10.2.1 that includes this patch. If there are no objections, I will roll out another RC and re-initiate the vote. Thank you everyone for your patience. Less bugs is good for all of us :) On Wed, Apr 12, 2017 at 5:25 PM, Gwen Shapira wrote: > Hel

[VOTE] 0.10.2.1 RC2

2017-04-18 Thread Gwen Shapira
Hello Kafka users, developers and client-developers, This is the third candidate for release of Apache Kafka 0.10.2.1. It is a bug fix release, so we have lots of bug fixes, some super important. Release notes for the 0.10.2.1 release: http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc2/RELEASE_

[VOTE] 0.10.2.1 RC3

2017-04-21 Thread Gwen Shapira
t the door already :P Thanks, Gwen -- *Gwen Shapira* Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter <https://twitter.com/ConfluentInc> | blog <http://www.confluent.io/blog>

[ANNOUNCE] New committer: Rajini Sivaram

2017-04-24 Thread Gwen Shapira
The PMC for Apache Kafka has invited Rajini Sivaram as a committer and we are pleased to announce that she has accepted! Rajini contributed 83 patches, 8 KIPs (all security and quota improvements) and a significant number of reviews. She is also on the conference committee for Kafka Summit, where

Re: Kafka source connector for avro files

2017-04-25 Thread Gwen Shapira
es into kafka connect > format and back in a source connector? > > Thank you, > Stanislav. > -- *Gwen Shapira* Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter <https://twitter.com/ConfluentInc> | blog <http://www.confluent.io/blog>

Re: [VOTE] 0.10.2.1 RC3

2017-04-26 Thread Gwen Shapira
t; > >> * Maven artifacts to be voted upon:https://repository. > apache.org/content/groups/staging/ > >> > >> > >> > >> On 22/04/17 19:23, Shimi Kiviti wrote: > >> > >> Is there a maven repo with these jars so I can test it aga

Re: [VOTE] 0.10.2.1 RC3

2017-04-26 Thread Gwen Shapira
12:12 PM, Shimi Kiviti wrote: > +1 > > I compiled our (Rollout.io) kafka-stream project, run unit-tests and > end-to-end tests (against streams 0.10.2.1 and broker 0.10.1.1) > Everything works as expected > > On Wed, Apr 26, 2017 at 10:05 PM, Gwen Shapira wrote: > > &

Re: [VOTE] 0.10.2.1 RC3

2017-04-26 Thread Gwen Shapira
, 2017 at 12:16 PM, Gwen Shapira wrote: > > Vote summary: > +1: 6 (3 binding) - Eno, Ian, Guozhang, Jun, Gwen and Shimi > 0: 0 > -1: 0 > > W00t! 72 hours passed and we have 3 binding +1! > > Thank you for playing "bugfix release". See you all at the next round :)

[ANNOUCE] Apache Kafka 0.10.2.1 Released

2017-04-27 Thread Gwen Shapira
lack-Postava, Guozhang Wang, Gwen Shapira, Ismael Juma, Jason Gustafson, Konstantine Karantasis, Marco Ebert, Matthias J. Sax, Michael G. Noll, Onur Karaman, Rajini Sivaram, Ryan P, simplesteph, Vahid Hashemian We welcome your help and feedback. For more information on how to report problems, and to

Re: [VOTE] KIP-156 Add option "dry run" to Streams application reset tool

2017-05-10 Thread Gwen Shapira
+1. Also not sure that adding a parameter to a CLI requires a KIP. It seems excessive. On Tue, May 9, 2017 at 7:57 PM Jay Kreps wrote: > +1 > On Tue, May 9, 2017 at 3:41 PM BigData dev > wrote: > > > Hi, Everyone, > > > > Since this is a relatively simple change, I would like to start the > vo

KIP-162: Enable topic deletion by default

2017-05-26 Thread Gwen Shapira
Hi Kafka developers, users and friends, I've added a KIP to improve our out-of-the-box usability a bit: KIP-162: Enable topic deletion by default: https://cwiki.apache.org/confluence/display/KAFKA/KIP-162+-+Enable+topic+deletion+by+default Pretty simple :) Discussion and feedback are welcome. Gw

Re: KIP-162: Enable topic deletion by default

2017-05-27 Thread Gwen Shapira
topic (similar to how removing ACLs > works). > > Thanks. > --Vahid > > > > From: Gwen Shapira > To: "d...@kafka.apache.org" , Users > > Date: 05/26/2017 07:04 AM > Subject:KIP-162: Enable topic deletion by default > > > >

[VOTE] KIP-162: Enable topic deletion by default

2017-06-05 Thread Gwen Shapira
Hi, The discussion has been quite positive, so I posted a JIRA, a PR and updated the KIP with the latest decisions. Lets officially vote on the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-162+-+Enable+topic+deletion+by+default JIRA is here: https://issues.apache.org/jira/browse/KA

Re: Kafka on yarn

2014-07-23 Thread Gwen Shapira
Hi, Can we discuss for a moment the use-case of Kafka-on-YARN? I (as Cloudera field engineer) typically advise my customers to install Kafka on their own nodes, to allow Kafka uninterrupted access to disks. Hadoop processes tend to be a bit IO heavy. Also, I can't see any benefit from co-locating

Re: request.required.acks=-1 under high data volume

2014-07-30 Thread Gwen Shapira
Super useful slide-deck. Thank you! On Wed, Jul 30, 2014 at 9:44 AM, Jun Rao wrote: > Just want to send out the link that describes how replication works in > Kafka, in case people are not aware of it. > > http://www.slideshare.net/junrao/kafka-replication-apachecon2013 > > Thanks, > > Jun > > >

Re: Issue with kafka-server-start.sh

2014-08-06 Thread Gwen Shapira
These are info messages, not necessarily errors. Do they occur in a never ending loop? Do you also see errors? Do you ever get a successful broker registration message in the log? (Registered broker 1 at path) Gwen On Wed, Aug 6, 2014 at 1:53 PM, Shikha Srivastava wrote: > Hi, > > I am work

Re: Issue with kafka-server-start.sh

2014-08-06 Thread Gwen Shapira
For completeness, you are probably running into: https://issues.apache.org/jira/browse/KAFKA-1451 On Wed, Aug 6, 2014 at 2:26 PM, Gwen Shapira wrote: > These are info messages, not necessarily errors. > > Do they occur in a never ending loop? Do you also see errors? > Do yo

Re: delete topic ?

2014-08-06 Thread Gwen Shapira
Hi Timothy, While we are on the subject, few questions/comments (based on the trunk implementation of delete topic command): * After deleting a topic, I still see it when listing topics. Is the expected behavior? Should it disappear after some time? * When does the actual deletion gets triggered?

Re: delete topic ?

2014-08-07 Thread Gwen Shapira
he >> brokers that has partitions for the topic to delete all the >> partitions, then finally delete the topic from zk. >> >> Therefore once the deletion finishes it will also deleted from zk. The >> topic command can definitely however join the topic list with the

Re: Improving the Kafka client ecosystem

2014-08-19 Thread Gwen Shapira
Does it make sense to merge the Camus mailing list? (i.e. ask the Camus community to merge?) Its a fairly large and popular client. On Tue, Aug 19, 2014 at 1:27 PM, Joe Stein wrote: > I also opened issues on 3 of the clients on github that I frequently > use/involved in often enough would be

Re: Please fix broken link and update blurb

2014-08-22 Thread Gwen Shapira
Done :) On Fri, Aug 22, 2014 at 11:22 AM, Igal Levy wrote: > Could someone with edit rights in the wiki please fix the broken link for > Metamarkets that's on this page: > > https://cwiki.apache.org/confluence/display/KAFKA/Powered+By > > The right URL is: http://metamarkets.com > > Also, if you

Re: EBCDIC support

2014-08-25 Thread Gwen Shapira
Hi Sonali, Kafka doesn't really care about EBCDIC or any other format - for Kafka bits are just bits. So they are all supported. Kafka does not "read" data from a socket though. Well, it does, but the data has to be sent by a Kafka producer. Most likely you'll need to implement a producer that w

Re: EBCDIC support

2014-08-25 Thread Gwen Shapira
; Thanks Gwen! makes sense. So I'll have to weigh the pros and cons of doing an > EBCDIC to ASCII conversion before sending to Kafka Vs. using an ebcdic > library after in the consumer > > Thanks! > S > > -Original Message- > From: Gwen Shapira [mailto:gshap...

Re: More partitions than consumers

2014-08-26 Thread Gwen Shapira
I hope this helps: https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example "if you have more partitions than you have threads, some threads will receive data from multiple partitions" On Tue, Aug 26, 2014 at 10:00 AM, Vetle Leinonen-Roeim wrote: > Hi, > > As far as I can see, t

Re: Consumer sensitive expiration of topic

2014-08-28 Thread Gwen Shapira
Maybe I misunderstand the proposal, but it sounds like an "irresponsible" consumer can accidentally delete data that others did not consume yet? On Thu, Aug 28, 2014 at 10:06 AM, Prunier, Dominique wrote: > Jay, > > I understand perfectly. I think you have all the reasons in the world to keep >

Re: kafka unit test in Java, TestUtils choosePort sends NoSuchMethodError

2014-08-29 Thread Gwen Shapira
So it looks like you are depending on a Kafka artifact which was build with Scala 2.9.2, and importing Scala 2.10.1 Scala is not binary compatible between versions, so you need either both 2.10 or both 2.9. Gwen On Fri, Aug 29, 2014 at 11:41 AM, Parin Jogani wrote: > so I always had this in my

Re: High Level Consumer and Commit

2014-09-02 Thread Gwen Shapira
I believe a simpler solution would be to create multiple ConsumerConnector, each with 1 thread (single ConsumerStream) and use commitOffset API to commit all partitions managed by each ConsumerConnector after the thread finished processing the messages. Does that solve the problem, Bhavesh? Gwen

Re: High Level Consumer and Commit

2014-09-03 Thread Gwen Shapira
said in my first reply. :-) > > - > http://www.philipotoole.com > > > On Tuesday, September 2, 2014 10:37 PM, Gwen Shapira > wrote: > > > I believe a simpler solution would be to create multiple > ConsumerConnector, each with 1 thread (single ConsumerStream) and use >

Re: High Level Consumer and Commit

2014-09-03 Thread Gwen Shapira
gt; topicDirs.consumerOffsetDir()+"/"+metaData.getPartitionNumber(),nextOffset+""); > checkPointedOffset.put(key,nextOffset); > } > } > > -Original Message- > From: Gwen Shapira [mailto:gshap...@cloudera.com] > Sent: Tuesday, September 02, 2

Re: High Level Consumer and Commit

2014-09-03 Thread Gwen Shapira
dden by thread that finish > first Indexing (Thread 2 will finish first and Thread 1 will commit offset > then ZK will have older offset). I guess it does not matter in our case > since indexing same document will over write. > > Thanks, > > Bhavesh > > > > On We

Re: Leader became -1 and no ISR for all topics/patitions

2014-09-04 Thread Gwen Shapira
Was there any data loss? On Sep 4, 2014 3:52 PM, "TianYi Zhu" wrote: > No log in the recent 7 days. > > kafka-preferred-replica-election.sh won't work in this stage, as the IRS is > null. I restarted the servers one by one then it's back to normal. > > > On Fri, Sep 5, 2014 at 1:11 AM, Jun Rao w

Re: Use case

2014-09-05 Thread Gwen Shapira
Especially since the same code will need to process all these messages. Topics are typically used to separate messages between apps, modules, etc. If all your apps consume all the topics, there's something wrong in the design. Gwen On Fri, Sep 5, 2014 at 9:31 AM, Philip O'Toole wrote: > Yes, IMH

<    1   2   3   4   5   6